Hi All,
I have a directory restricted by robots.txt, however, in Google webmaster tools, it shows all 18 files in the folder as "Not Found: URLS restricted by robots.txt". I did not take a rankings hit for this, however, could I? Do I have ways around doing it this way?
Can you post the format of your code here? I bet there's something wrong with your code.. If you wanna bots crawl those pages, put do-follow codes like for instance: <meta name="robots" content="index,follow,all" />
A robots.txt file provides restrictions to search engine robots (known as "bots") that crawl the web. These bots are automated, and before they access pages of a site, they check to see if a robots.txt file exists that prevents them from accessing certain pages.
There are some ways for preventing the bots from indexing certain pages, changing the robots file, adding the no follow and no index tags to the meta tags.