Search Engine Forums, SEO Forum

What is a robots.txt?

robots.txt is a standard used by websites to communicate with web crawlers and other web robots. The standard specifies how to inform the web robot about which areas of the website should not be processed or scanned.

The robots exclusion normal, additionally referred to as the robots exclusion protocol or just robots.txt, may be a normal employed by websites to speak with net crawlers and different web robots. the quality specifies a way to inform the net mechanism concerning that areas of the web site mustn't be processed or scanned.

robot.txt or robots exclusion protocol is used by websites to communicate with the web crawlers to inform that which pages should not be scanned or crawled. For this the robot file should be uploaded in the website root directory and linked with the html code of the website.

The robots exclusion standard, also known as the robots exclusion protocol or simply robots.txt, is a standard used by websites to communicate with web crawlers and other web robots. The standard specifies how to inform the web robot about which areas of the website should not be processed or scanned.

It is a file used to block certain areas or pages in a website from search engine.

It is great when search engines frequently visit your site and index your content but often there are cases when indexing parts of your online content is not what you want. For instance, if you have two versions of a page (one for viewing in the browser and one for printing), you'd rather have the printing version excluded from crawling, otherwise you risk being imposed a duplicate content penalty.