Page 1 of 1
Robots.txt & Restricted & Google...
Posted: Mon Mar 17, 2008 9:34 am
by rj71
Hi All,
I have a directory restricted by robots.txt, however, in Google webmaster tools, it shows all 18 files in the folder as "Not Found: URLS restricted by robots.txt". I did not take a rankings hit for this, however, could I? Do I have ways around doing it this way?
Regards,
Rob
Posted: Fri Apr 11, 2008 1:14 pm
by web67
check the format of ur robots.txt
Posted: Sun May 04, 2008 4:51 am
by ikshayar
There is whole explanation for robots.txt usage in google help. You may have to play a little with it but its very good.
Posted: Mon May 05, 2008 1:05 am
by dee_el07
Can you post the format of your code here? I bet there's something wrong with your code.. If you wanna bots crawl those pages, put do-follow codes like for instance: <meta name="robots" content="index,follow,all" />
Posted: Mon May 05, 2008 10:35 pm
by Ka-canor
A robots.txt file provides restrictions to search engine robots (known as "bots") that crawl the web. These bots are automated, and before they access pages of a site, they check to see if a robots.txt file exists that prevents them from accessing certain pages.
Posted: Wed May 07, 2008 7:59 am
by bluesky
There are some ways for preventing the bots from indexing certain pages, changing the robots file, adding the no follow and no index tags to the meta tags.
Robot.txt Format
Posted: Wed May 07, 2008 9:35 pm
by Sonal
You have to check your robot.txt format. If it is correct then follow the meta robot txt format on home page. And wait for google crawled
Posted: Mon May 12, 2008 1:09 am
by delhiseo
Check your robots.txt file. Also use noindex,nofollow to the pages you want to restrict.
Posted: Mon May 12, 2008 2:25 pm
by skysoldier
Posted: Wed May 14, 2008 1:57 am
by Digitoko
Do check your robot.txt. And Google webmaster tool may help you.