robots.txt

For discussion of general Search Engine Optimization, not specific to any engine.
Post Reply
Ravishshah
Posts: 22
Joined: Wed May 08, 2019 4:07 am
Location: Connecticut USA
Contact:

robots.txt

Post by Ravishshah »

What is robots.txt?

gautamseo
Posts: 7
Joined: Tue May 21, 2019 3:01 am
Contact:

Re: robots.txt

Post by gautamseo »

It is a text file which tells Google which pages to crawl and which not to crawl.
BEST SEO Expert in India, SEO EXPERT IN KOLKATA, SEO EXPERT IN MUMBAI, SEO EXPERT IN CHANDIGARH, SEO FREELANCER IN HYDERABAD

riya11
Posts: 15
Joined: Thu Jun 20, 2019 11:26 pm
Contact:

Re: robots.txt

Post by riya11 »

Robot.txt is a file also know as google bot is used to tells Google which pages to crawl and which not to crawl.

rais26
Posts: 76
Joined: Wed Jul 03, 2019 3:10 am
Contact:

Re: robots.txt

Post by rais26 »

A robots.txt file tells search engine crawlers which pages or files the crawler can or can't request from your site.

Rasull
Posts: 28
Joined: Mon Jul 29, 2019 12:44 am
Location: Dubai, UAE
Contact:

Re: robots.txt

Post by Rasull »

Robots.txt is a text file which tells Google which pages to crawl and which not to crawl.

Jvimala
Posts: 11
Joined: Mon Jul 29, 2019 5:03 am
Contact:

Re: robots.txt

Post by Jvimala »

Robots.txt is a text file webmasters create to instruct robots (typically search engine robots) how to crawl & index pages on their website.

regards,
Ansularjs Training Institute in Chennai

gagarihuhe
Posts: 1
Joined: Mon Jul 29, 2019 12:25 pm
Contact:

Re: robots.txt

Post by gagarihuhe »

Jvimala wrote:Robots.txt is a text file webmasters create to instruct robots (typically search engine robots) how to crawl & index pages on their website dermatologo pavia.

regards,
Ansularjs Training Institute in Chennai
very interesting, thanck you :D

vijaykl32
Posts: 25
Joined: Thu Jul 25, 2019 1:50 am
Contact:

Re: robots.txt

Post by vijaykl32 »

robots.txt file mention the google'c crawler that which website to crawl or not

Marquestech
Posts: 3
Joined: Tue Jul 30, 2019 4:01 am
Contact:

Re: robots.txt

Post by Marquestech »

We tell Google crawler 'to crawl and not to crawl' since this is will help your valuable data to be hidden like admin data.
So we tell google using robots.txt file on the website.
Very simple and easy right.

Post Reply

Who is online

Users browsing this forum: Google Adsense [Bot] and 20 guests