Search Engine Forums, SEO Forum

How do you create a Robots.Txt file? and how can it be used to pass on instructions to search engine robots? Tell me as soon as.

it is just text file, which give the search engine what should do with your website and what should not. If you wanna avoid some page to make crawl so you can make from using robot.txt

It gives you the permission to search over the search engines...further details along with guidelines are available on internet too.

text file located in the root directory to restrict SE spiders access to certain files/folders of a web site.

Here's a basic "robots.txt":

User-agent: *
Disallow: /

you may not want Google to crawl the /images directory of your site, as it's both meaningless to you and a waste of your site's bandwidth. "Robots.txt" lets you tell Google just that.

We can restrict the personal information from the search engine by using robots.txt & we can hide the file or folder by using this file.

The use of robot.txt can help you shape bot-traffic. Spiders will crawl all your files if you don’t instruct them. Depending on your technical skills, there are two ways to generate a sitemap. You can either download or install a sitemap generator, or you can use an online sitemap generation tool.

It is easy to create one, try searching up on Google terms like "robots.txt example..." and you will get examples and tutorials that will guide you how to make robots for your website / blog.

Instruct google search engine either to crawl or not to crawl