Why Pages Disallowed in robots.txt Still Appear in Google
Is Google indexing the pages you explicitly disallowed in robots.txt? Craig discusses why it happens and what you can do to prevent it. ... It’s the difference between crawling and indexing. Google won’t crawl a page disallowed in robots.txt, but it might...