Robots.txt & Restricted & Google...

[rj71]

Hi All,
I have a directory restricted by robots.txt, however, in Google webmaster tools, it shows all 18 files in the folder as "Not Found: URLS restricted by robots.txt". I did not take a rankings hit for this, however, could I? Do I have ways around doing it this way?

Regards,
Rob

[web67]

check the format of ur robots.txt

[ikshayar]

There is whole explanation for robots.txt usage in google help. You may have to play a little with it but its very good.

[dee_el07]

Can you post the format of your code here? I bet there's something wrong with your code.. If you wanna bots crawl those pages, put do-follow codes like for instance: <meta name="robots" content="index,follow,all" />

[Ka-canor]

A robots.txt file provides restrictions to search engine robots (known as "bots") that crawl the web. These bots are automated, and before they access pages of a site, they check to see if a robots.txt file exists that prevents them from accessing certain pages.

[bluesky]

There are some ways for preventing the bots from indexing certain pages, changing the robots file, adding the no follow and no index tags to the meta tags.

[Sonal]

You have to check your robot.txt format. If it is correct then follow the meta robot txt format on home page. And wait for google crawled

[delhiseo]

Check your robots.txt file. Also use noindex,nofollow to the pages you want to restrict.

[skysoldier]

http://googlewebmastercentral.blogspot. ... -tags.html

This might help!

[Digitoko]

Do check your robot.txt. And Google webmaster tool may help you.