How to Set Up a robots.txt to Control Search Engine Spiders (thesitewizard.com)
Tutorial on setting up a robots.txt to exclude search engine robots/spiders as part of the Robots Exclusion Standard. ... The first block of text disallows all spiders from the images directory and the cgi-bin directory. The second block of code disallows...