Why Search Engine "Spiders" Are Important

LEE ODDEN: What is a search engine spider and why is it important? Well, the way search engines find out about websites is they send out computer programs following new links to content. If the search engine "spider" isn't able to follow the links within your website, that's sort of like bringing half of your team to a baseball game. In other words, if you have 10,000 pages on your website and only 6,000 of them have been crawled by a search engine, that's putting you at a pretty big disadvantage. So it's important to make sure that your url's are crawlable. Well, how do you do that? As a general rule, simple is best. If you have a website, something like www.domainname.com/product.htm, that's pretty easy for a search engine to understand and crawl. If you have something a little more complex and sophisticated, like www.com/productid=456&categoryid=4789, and so on and so forth, that can get a little trickier. It's not impossible for a search engine to crawl some of the more complex urls, but its something that can definitely make it more difficult. So make sure that your urls are crawlable, and your rankings will improve.