The Difference Between Spidering and Indexing
Shari Thurow’s column over at ClickZ today has an interesting tidbit in it than many small business owners and new SEM practitioners should read and consider. It’s an explanation of the difference between a site being spidered and a site being indexed. Most site owners assume that the two words are one and the same…meaning you get spidered and you are then in the index. As Shari points out, that’s not always the case.
From the article:
The biggest misconception at this year’s conference was the difference between the spidering and indexing processes. The index is a subset of the spider. Search engines access Web content through the spidering process. Then, they filter out duplicates and other bad content and create the index.
A Web site can be spidered and not included in a search engine index.
Go back and read that last line again. Then think about it. It’s not uncommon for me to talk to a site owner or to get an email from someone that’s confused as to why they aren’t showing up in Yahoo! or Google. Usually, these site owners have seen proof in their log files that the spiders from these engines have been to their sites. Sometimes the spiders have been there dozens of times and have visited every single one of their pages, yet the site still fails to show up in the index.
This is where it’s important to remember that there are plenty of stumbling blocks that can keep a site from being indexed and ranked. That may mean a simple technical errors like a robots.txt file that’s set to exclude the indexing of an entire site (rather than of certain areas) or an improperly routed 301 redirect. It could also mean that there are serious errors in your code, or that the content of your site is somehow being dubbed as duplicate to another site.
In other words, just because you aren’t showing up in Google doesn’t mean that Googlebot hates you. Sometimes it means that there are serious issues that need to be addressed.
Source: searchenginenews.com
To Comment about this article go here
