Search Engine Forums, SEO Forum

hi friends,

What is a robots.txt file?

Robots.txt helps to crawler or indicate webmaster tool which files are important to index on search engine. It also helpful to make privacy of web pages from other users.

Robots.txt is a text file we give the permission to the search engine allow which pages are taken and disallow which pages are not taken by the search engine. this purpose use robots.txt and it is created by webmaster

With your robots file you can block search engines to crawl your website.