What is robots.txt

For discussion of general Search Engine Optimization, not specific to any engine.
Post Reply
priyakumari95
Posts: 36
Joined: Thu Sep 07, 2017 2:16 am
Contact:
Contact priyakumari95

What is robots.txt

Post by priyakumari95 »

Hello friends,









What is robots.txt......?
Top
ranjitshah
Posts: 2
Joined: Fri Jan 19, 2018 10:28 pm
Contact:
Contact ranjitshah

Re: What is robots.txt

Post by ranjitshah »

Robots.txt is a text file, that is present in the root folder of the server. It contain commands about how the website should be crawled. The commands are understood by search engine crawlers and accordingly search engine bots crawls or blocks specific pages, directories or subdomains.
Digital Marketer with 8 years of experience. I have a successful track record of improving ranking, traffic & revenue by using different channels like SEO, SEM, SMM, Affiliate Marketing and Email Marketing. https://ranjitshah.com/seo-company-mumbai
Top
AmandaCherry
Posts: 90
Joined: Sat Dec 02, 2017 7:36 am
Contact:
Contact AmandaCherry

Re: What is robots.txt

Post by AmandaCherry »

The robots exclusion standard, also known as the robots exclusion protocol or simply robots.txt, is a standard used by websites to communicate with web crawlers and other web robots. The standard specifies how to inform the web robot about which areas of the website should not be processed or scanned.
Top
vishwa01
Posts: 15
Joined: Sat Jan 20, 2018 12:39 am
Contact:
Contact vishwa01

Re: What is robots.txt

Post by vishwa01 »

robots.txt is the text file for control the search engine crawls which pages want to crawl and which pages don't want to crawl.
Top
AmandaCherry
Posts: 90
Joined: Sat Dec 02, 2017 7:36 am
Contact:
Contact AmandaCherry

Re: What is robots.txt

Post by AmandaCherry »

The robots exclusion standard, also known as the robots exclusion protocol or simply robots.txt, is a standard used by websites to communicate with web crawlers and other web robots. The standard specifies how to inform the web robot about which areas of the website should not be processed or scanned.
Top
rachana
Posts: 26
Joined: Tue Dec 12, 2017 4:34 am
Contact:
Contact rachana

Re: What is robots.txt

Post by rachana »

Robots.txt a text file created by webmasters that instruct web robots how to crawl pages in their website. This file also indicates whether the pages of their website should be crawled or not. The crawl instructions are “allowing” or “disallowing” which are specified by user agents.
Top
HokuApps
Posts: 92
Joined: Sun Sep 10, 2017 11:38 pm
Contact:
Contact HokuApps

Re: What is robots.txt

Post by HokuApps »

The robots exclusion standard, also known as the robots exclusion protocol or simply robots.txt, is a standard used by websites to communicate with web crawlers and other web robots. The standard specifies how to inform the web robot about which areas of the website should not be processed or scanned.
Top
praveenitech1
Posts: 137
Joined: Mon Feb 19, 2018 6:06 am
Contact:
Contact praveenitech1

Re: What is robots.txt

Post by praveenitech1 »

Robots.txt is a text file, that is present in the root folder of the server. It contain commands about how the website should be crawled. The commands are understood by search engine crawlers and accordingly search engine bots crawls or blocks specific pages, directories or subdomains.
Top
Post Reply

Return to “SEO Forum”

Who is online

Users browsing this forum: No registered users and 34 guests