What do you mean by Robots.txt? Why it is used?

This forum is for discussion of Google only. Post anything Google here, including news and SEO related questions.
phil12
Posts: 7
Joined: Sat Nov 24, 2012 11:43 am
Contact:

What do you mean by Robots.txt? Why it is used?

Postby phil12 » Sat Feb 09, 2013 2:35 am

Robot.txt is used to in order to instruct the search engine crawler about the following and not following of the links. The link with Nofollow tags are not visited or consider by the spiders.


Family Dentist Toronto

elenabrown
Posts: 27
Joined: Sun Feb 10, 2013 10:06 pm
Location: USA
Contact:

Re: What do you mean by Robots.txt? Why it is used?

Postby elenabrown » Mon Feb 11, 2013 2:58 am

Robots.txt is stand for Robots Exclusion Protocol. :)

1. The robots.txt is a simple text file in your web site that inform search engine bots how to crawl and index website or web pages.
2. By default search engine bots crawl everything possible unless they are forbidden from doing so. They always scan the robots.txt file before crawling the web site.
3. Declaring a robots.txt means that visitors (bots) are not allowed to index sensitive data but it doesn’t mean that they can’t. The legal/good bots follow what is instructed to them but the Malware robots don’t care about it, so don’t try to use it as a security for your web site.


SEO World Class
Posts: 21
Joined: Tue Jan 24, 2012 11:32 pm
Location: Boat Quay
Contact:

Re: What do you mean by Robots.txt? Why it is used?

Postby SEO World Class » Tue Mar 05, 2013 9:46 am

Hi Friends

It is simple text file that is created using text editors. It provides essential information to search engine. Best practice robots.txt file improve search engine rankings for the site. So this is one of the important file to have nice ranking on search engine and SEO.

When search crawls our website, it will look for this particular file on our site. This special file tells the search engine spider, to priority the web pages to be indexed. In the sense, which page should be index and which page should be discarded. It requests the search engine crawler not to index these files as they are not request to be viewed by users.

Thanks and Regards
seoworldclass.com

hugorane12
Posts: 5
Joined: Thu Mar 07, 2013 12:20 am
Location: USA
Contact:

Re: What do you mean by Robots.txt? Why it is used?

Postby hugorane12 » Thu Mar 07, 2013 12:38 am

Robots.txt file is very important for any website. It restricts search engine which part of website crawler can index and crawl. Its very helpful to keep your privacy secret.

Shelly47
Posts: 20
Joined: Wed Mar 08, 2017 5:23 am
Contact:

Re: What do you mean by Robots.txt? Why it is used?

Postby Shelly47 » Wed Mar 08, 2017 5:36 am

Dental is as important as your physical health. Never neglect any kind of oral or dental discomfort and consult your dentist if there is any problem.

RashidS
Posts: 37
Joined: Wed Feb 22, 2017 1:21 am
Contact:

Re: What do you mean by Robots.txt? Why it is used?

Postby RashidS » Thu Mar 09, 2017 4:30 am

Robots.txt is a file which tells crawlers to crawl website or not.

mujahid
Posts: 4
Joined: Tue Oct 23, 2018 1:54 pm
Contact:

Re: What do you mean by Robots.txt? Why it is used?

Postby mujahid » Tue Oct 23, 2018 2:37 pm

Robots.txt is a standard text file is used for websites or web applications to communicate with web crawlers (bots).
if you want to learn more about Robots.txt visit here https://www.twinztech.com/robots-txt-file/

Arpita
Posts: 236
Joined: Mon Jan 14, 2019 11:10 pm
Contact:

Re: What do you mean by Robots.txt? Why it is used?

Postby Arpita » Tue Jan 15, 2019 2:28 am

The robots exclusion standard, also known as the robots exclusion protocol or simply robots.txt, is a standard used by websites to communicate with web crawlers and other web robots. The standard specifies how to inform the web robot about which areas of the website should not be processed or scanned.

mraj27030
Posts: 12
Joined: Thu Sep 06, 2018 3:18 am
Contact:

Re: What do you mean by Robots.txt? Why it is used?

Postby mraj27030 » Tue Jan 15, 2019 2:55 am

Robot.txt is file which commands the bots whether or not to crawl any particular page or website.

riyajindal
Posts: 401
Joined: Fri Jan 04, 2019 12:08 am
Contact:

Re: What do you mean by Robots.txt? Why it is used?

Postby riyajindal » Tue Jan 15, 2019 5:27 am

The robots.txt file, also known as the robots exclusion protocol or standard, is a text file that tells web robots (most often search engines) which pages on your site to crawl. It also tells web robots which pages not to crawl. ... The slash after “Disallow” tells the robot to not visit any pages on the site.

abanote
Posts: 5
Joined: Tue Jan 15, 2019 6:48 am
Contact:

Re: What do you mean by Robots.txt? Why it is used?

Postby abanote » Tue Jan 15, 2019 10:36 am

Robots.txt is used to allow or prohibit bots to visit or not the files in question.
You can also specify which robot can crawl your site or not

webappi
Posts: 12
Joined: Tue Jan 22, 2019 4:13 am
Location: Dubai
Contact:

Re: What do you mean by Robots.txt? Why it is used?

Postby webappi » Thu Jan 31, 2019 4:58 am

Web site owners use the /robots.txt file to give instructions about their site to web robots; this is called The Robots Exclusion Protocol.

User-agent: *
Disallow: /
The "User-agent: *" means this section applies to all robots. The "Disallow: /" tells the robot that it should not visit any pages on the site.

There are two important considerations when using /robots.txt:

robots can ignore your /robots.txt. Especially malware robots that scan the web for security vulnerabilities, and email address harvesters used by spammers will pay no attention.
the /robots.txt file is a publicly available file. Anyone can see what sections of your server you don't want robots to use.

ptiacademy
Posts: 3
Joined: Sat Jan 12, 2019 4:22 am
Location: 418-421, Jaipur Textile Market, Near Jagatpura Flyover, Jagatpura, Jaipur, 302017
Contact:

Re: What do you mean by Robots.txt? Why it is used?

Postby ptiacademy » Thu Jan 31, 2019 11:07 pm

Robot.txt means, you have a web page and you want to block any inner page of that URL you use Robot.txt. If you want that only you can see any particular page on your website then you use Robot.txt.
It is used in all website because it is mandatory. It's an algorithm of Google that if you have a website then you must use Robot.txt on any page, that is for single or multiple pages.

aezioninc
Posts: 2
Joined: Wed Nov 07, 2018 10:39 pm
Location: Dallas,Texas, USA
Contact:

Re: What do you mean by Robots.txt? Why it is used?

Postby aezioninc » Thu Jan 31, 2019 11:21 pm

Robot.txt is used to show allow or disallow links, it is helpful to protect any unwanted links from the users.
visit: https://www.aezion.com/robots.txt


Return to “Google”

Who is online

Users browsing this forum: No registered users and 2 guests