What is Robot.txt file

Discuss other engines here. AllTheWeb, AltaVista, Excite, Lycos, etc...
xenelsoft
Posts: 15
Joined: Thu Apr 04, 2019 12:25 am
Location: Noida
Contact:

Re: What is Robot.txt file

Postby xenelsoft » Tue Aug 13, 2019 5:02 am

The robots.txt file is known as the robots exclusion protocol or standard that is a text file that tells search engines which pages on your site to crawl. It also tells web robots which pages not to crawl.
Xenelsoft is a leading Digital Media company in India established in 2013. we are creative digital marketing company offers SEO, SMO, ORM, Website designing and Website development services.

sophiawils59
Posts: 133
Joined: Mon Sep 02, 2019 4:29 am
Contact:

Re: What is Robot.txt file

Postby sophiawils59 » Tue Sep 24, 2019 1:38 am

Robots.txt is a simple text file that sites in the root directory of your site. It tells “robots” (such as search engine spiders) which pages to crawl on your site, which pages to ignore. While not essential, the Robots.txt file gives you a lot of control over how Google and other search engines see your site.

atocloud
Posts: 52
Joined: Fri Jun 21, 2019 11:56 pm
Location: Faridabad
Contact:

Re: What is Robot.txt file

Postby atocloud » Wed Oct 23, 2019 10:23 pm

A robots.txt file tells search engine crawlers which pages or files the crawler can or can't request from your site. This is used mainly to avoid overloading your site with requests; it is not a mechanism for keeping a web page out of Google.
Contact Atocloud: Salesforce Development Company
Contact No : 9953170767
Business Email : contact@Atocloud.com

sophiawils59
Posts: 133
Joined: Mon Sep 02, 2019 4:29 am
Contact:

Re: What is Robot.txt file

Postby sophiawils59 » Thu Oct 24, 2019 2:35 am

Enough answres are posted @op should close the thread.

e4apple
Posts: 8
Joined: Mon Oct 28, 2019 7:30 am
Contact:

Re: What is Robot.txt file

Postby e4apple » Mon Oct 28, 2019 11:40 pm

Robots.txt file works as a site moderator. Using it, you can control what you want google to crawl on your website. It's a very useful code I'd suggest you make full use of it.
For Niche-Driven Content Writing Services: https://pennovate.co/

ritesh3592
Posts: 186
Joined: Wed Jul 31, 2019 3:42 am
Contact:

Re: What is Robot.txt file

Postby ritesh3592 » Sun Nov 03, 2019 9:26 pm

Robots.txt is a file associated with your website used to ask different web crawlers to crawl or not crawl portions of your website. The robots.txt file is primarily used to specify which parts of your website should be crawled by spiders or web crawlers.The content within robot.txt files are publicly available.


Return to “All Other Engines”

Who is online

Users browsing this forum: No registered users and 1 guest