What is Robot.txt file

Discuss other engines here. AllTheWeb, AltaVista, Excite, Lycos, etc...
photoeditor
Posts: 430
Joined: Wed Mar 22, 2017 5:58 am
Contact:

Re: What is Robot.txt file

Postby photoeditor » Mon Apr 09, 2018 11:13 pm

Robots.txt file helps to index any website in different search engines.

dombowkett
Posts: 86
Joined: Tue Feb 13, 2018 3:30 am
Location: CA
Contact:

Re: What is Robot.txt file

Postby dombowkett » Sat Aug 25, 2018 10:00 am

Robots.txt file is set of indexing instructions for search engine. http://raptorhomes.com

Josh44
Posts: 20
Joined: Thu Aug 30, 2018 11:26 pm
Contact:

Re: What is Robot.txt file

Postby Josh44 » Mon Sep 03, 2018 12:36 am

Robot.txt file helps search engine and other bots to crawl your website. You can instruct bots what to crawl and how to behave while indexing your website by including certain rules into the robots text file.








-----------------------------------------------------------------------------
https://www.netguru.co/services/nodejs
Hire The Best Node.js Developers

garmincomexpress
Posts: 5
Joined: Wed Oct 10, 2018 5:32 am
Location: Canada
Contact:

Re: What is Robot.txt file

Postby garmincomexpress » Mon Oct 15, 2018 10:11 pm

Robots.txt is a text (not html) file you put on your site to tell search robots which pages you would like them not to visit. Robots.txt is by no means mandatory for search engines but generally search engines obey what they are asked not to do.
Garmin.com/express

Rob45
Posts: 23
Joined: Mon Sep 03, 2018 5:16 am
Contact:

Re: What is Robot.txt file

Postby Rob45 » Tue Oct 30, 2018 4:42 am

It is an important text file which can block search engines to crawl your site pages.

davidweb09
Posts: 99
Joined: Sun Nov 23, 2014 9:42 pm
Location: CA
Contact:

Re: What is Robot.txt file

Postby davidweb09 » Sat Feb 02, 2019 4:53 am

Robots.txt file send instructions to search engines for website crawling.

TejasPatil1991
Posts: 19
Joined: Sun Feb 03, 2019 11:37 pm
Contact:

Re: What is Robot.txt file

Postby TejasPatil1991 » Sun Feb 10, 2019 1:38 am

robot.txt file that gives instructions to the crawler as which page to crawl and which page not to crawl.
The basic format of a robots.txt file is:
User-agent: *
Disallow: /

heena
Posts: 4
Joined: Thu Feb 07, 2019 10:55 pm
Location: germany
Contact:

Re: What is Robot.txt file

Postby heena » Mon Feb 11, 2019 11:24 pm

A robot.txt file is a file at the root of your site that indicates those parts of your site you don’t want accessed by search engine crawlers.
office.com/setup

SteveS
Posts: 7
Joined: Mon Feb 11, 2019 11:24 pm
Contact:

Re: What is Robot.txt file

Postby SteveS » Mon Feb 11, 2019 11:50 pm

If you have a website then you probably should know about Robot.txt file. Google craws every pages of your website and this file tells Google that what pages should be crawl and what pages should not crawls as we want.
User -agent : *
Disallow: /


Return to “All Other Engines”

Who is online

Users browsing this forum: No registered users and 1 guest