What is robots.txt

For discussion of general Search Engine Optimization, not specific to any engine.
priyakumari95
Posts: 28
Joined: Thu Sep 07, 2017 2:16 am
Contact:

What is robots.txt

Postby priyakumari95 » Fri Jan 19, 2018 3:46 am

Hello friends,









What is robots.txt......?

ranjitshah
Posts: 2
Joined: Fri Jan 19, 2018 10:28 pm
Contact:

Re: What is robots.txt

Postby ranjitshah » Fri Jan 19, 2018 10:50 pm

Robots.txt is a text file, that is present in the root folder of the server. It contain commands about how the website should be crawled. The commands are understood by search engine crawlers and accordingly search engine bots crawls or blocks specific pages, directories or subdomains.
Digital Marketer with 8 years of experience. I have a successful track record of improving ranking, traffic & revenue by using different channels like SEO, SEM, SMM, Affiliate Marketing and Email Marketing. https://ranjitshah.com/seo-company-mumbai

AmandaCherry
Posts: 90
Joined: Sat Dec 02, 2017 7:36 am
Contact:

Re: What is robots.txt

Postby AmandaCherry » Mon Jan 22, 2018 2:13 am

The robots exclusion standard, also known as the robots exclusion protocol or simply robots.txt, is a standard used by websites to communicate with web crawlers and other web robots. The standard specifies how to inform the web robot about which areas of the website should not be processed or scanned.

vishwa01
Posts: 15
Joined: Sat Jan 20, 2018 12:39 am
Contact:

Re: What is robots.txt

Postby vishwa01 » Mon Jan 22, 2018 2:26 am

robots.txt is the text file for control the search engine crawls which pages want to crawl and which pages don't want to crawl.

AmandaCherry
Posts: 90
Joined: Sat Dec 02, 2017 7:36 am
Contact:

Re: What is robots.txt

Postby AmandaCherry » Sun Jan 28, 2018 6:48 am

The robots exclusion standard, also known as the robots exclusion protocol or simply robots.txt, is a standard used by websites to communicate with web crawlers and other web robots. The standard specifies how to inform the web robot about which areas of the website should not be processed or scanned.

rachana
Posts: 22
Joined: Tue Dec 12, 2017 4:34 am
Contact:

Re: What is robots.txt

Postby rachana » Wed Feb 14, 2018 10:43 pm

Robots.txt a text file created by webmasters that instruct web robots how to crawl pages in their website. This file also indicates whether the pages of their website should be crawled or not. The crawl instructions are “allowing” or “disallowing” which are specified by user agents.

HokuApps
Posts: 94
Joined: Sun Sep 10, 2017 11:38 pm
Contact:

Re: What is robots.txt

Postby HokuApps » Wed Feb 14, 2018 11:24 pm

The robots exclusion standard, also known as the robots exclusion protocol or simply robots.txt, is a standard used by websites to communicate with web crawlers and other web robots. The standard specifies how to inform the web robot about which areas of the website should not be processed or scanned.

praveenitech1
Posts: 137
Joined: Mon Feb 19, 2018 6:06 am
Contact:

Re: What is robots.txt

Postby praveenitech1 » Wed Feb 21, 2018 1:27 am

Robots.txt is a text file, that is present in the root folder of the server. It contain commands about how the website should be crawled. The commands are understood by search engine crawlers and accordingly search engine bots crawls or blocks specific pages, directories or subdomains.


Return to “SEO Forum”

Who is online

Users browsing this forum: No registered users and 13 guests