How To Use Robots.Txt File? - Free Web Resources – Web Resources Depot
JrrMaster May 13th, 2012 at 5:39 am This is some example using robots.txt file User-agent: * Disallow: / Allow: /private Request-rate: 1/10 Crawl-delay: 10 #means after one request crawl successfully executed, the bot should wait to crawl after 10 seconds...