Robots.txt

Robots.txt is a file in the root of a site that will tell spiders where the can go and where that can not go. Google has developed the idea and many other search engine followed afterwards.
By default Google spiders all linked content from the main page.

Go to the Robots.txt tutorial to read how to make robots.txt files.