Robots.txt

Sometimes you would like Search Engines to not list your pages on their search result pages. Or maybe you want certain pages not to be indexed, because they will lower your PR (linking to bad neighbours) or your keyword density. In that case you will want to use a Robots.txt file to tell search engines what pages they can index.

Robots.txt: is a file in the root of a site that will tell spiders where the can go and where that can not go. Google has developed the idea and many other search engine followed afterwards.
By default Google spiders all linked content from the main page. Only quite recently Google seems to not spider all content and gives sites with higher PageRank more space in their database.

Tutorial

But anyway, here’s some examples and tips on how to use robots.txt:

Always place the robots.txt file in the root of the website!
Make sure it's named robots.txt and not robot.txt!

The following "robots.txt" file specifies that no robots should visit any pages in the directory "/content/article/" or "/temp/", or /badgoogle.html:

The following "robots.txt" file specifies that no robots should visit any pages in the directory "/content/article/", only "goodspider" can visit them.

Please note that a spider reads the robots.txt file from top to bottom and stops whenever it finds a User-agent refering to it. Therefor when disallowing certain spiders, always put them first and the * at the end.

The following "robots.txt" file specifies that no robots should spider any content:

Off course the User-agent stands for the name of the spider visiting your website, where ‘*’ means all spiders .

User Agents

A list with popular user-agents can be found here: http://www.jafsoft.com/searchengines/webbots.html. Please note that some spiders will change their name or IP-address sometimes.