ROBOTS.TXT & SITEMAP

rj71 · Post by **rj71** » Fri Jun 15, 2007 8:46 am

Is it true that it is important to add sitemap to robots.txt now?

i.e...

User-agent: *
Disallow: /cgi-bin/
Disallow: /images/
Sitemap: http://www.XYZ.com/sitemap.xml.gz

I have a site in the top 3 in Google and it does not have the sitemap variable in the robots.txt.....by adding it, will it "mess me up"?
Thanks !!!

thusharaherath · Post by **thusharaherath** » Fri Jun 15, 2007 10:00 am

Yes, it is important to add sitemap to robots.txt file. Site map helps the Search Engine crawler to go through your site very easily. Don't worry, it doesn't mess up your site by adding site map to your robots.txt file.

rj71 · Post by **rj71** » Fri Jun 15, 2007 11:31 am

Thanks!
And is this syntax OK?

User-agent: *
Disallow: /cgi-bin/
Disallow: /images/
Sitemap: http://www.XYZ.com/sitemap.xml.gz

Thiagwou · Post by **Thiagwou** » Wed Jan 18, 2012 12:27 am

A robots.txt file is a text file in a simple format which gives information to web robots (such as search engine spiders) about which parts of your website they are and aren't allowed to visit.

anneparker786 · Post by **anneparker786** » Sat Feb 04, 2012 2:21 am

Yes, it’s very important to add site maps and robot.txt in your code, it will save your time and in case of searching , customer will only look those pages which you want them to look not the wasted pages or important pages. Adding sites maps explore your pages into one segment and when crawlers look for searching result , it’s easy to find related keywords.

SEO expert

atcomaartltd · Post by **atcomaartltd** » Mon Mar 05, 2012 2:29 am

Site map is important for the crawling of your website easily. When spiders visit your website, they first go through your site map so your site will be crawled in one visit of spiders. Because site map have all the main links of your website pages. But it will not effect on the result that you get now.

VladTepes · Post by **VladTepes** » Thu Mar 08, 2012 12:13 am

Robots.txt issues command to bots to index or not index a page or sections of a site, I would not include it for site is a good reference for bots to see and index all pages of the site.

Flash Design · Post by **Flash Design** » Thu Mar 08, 2012 11:43 pm

Every search engine has a “robot” (a software program) that does the job of visiting a website. Their purpose is to gather a copy of the site and keep them in their database. So, if your site is not there in their database it never shows up in the search results.

Muirenn · Post by **Muirenn** » Mon Mar 26, 2012 2:05 am

Through Robots.txt File search engine spiders or robots knows which web page of files in your website should be indexed and which pages should be ignored. Sitemaps is basically an additional page on your website that incorporates a list of links to every single page on your site and can also detail the hierarchy of page on your site.

NileSmeeth · Post by **NileSmeeth** » Sat Apr 07, 2012 1:09 am

To begin we need to know what a web robot is, and is not. Thus, a Web robot is sometimes called spiders or web crawlers.Search engine crawlers are a lot like simple insects. They have a purpose in life to index website content and index they will - everything, unless instructed otherwise.

jerrydu · Post by **jerrydu** » Mon Apr 09, 2012 8:33 am

I think it is not important as in robots.text you tell search engine which website pages should not be crawled or index... This is my understand of robots.text file...

addlinksweb · Post by **addlinksweb** » Tue Apr 10, 2012 2:14 pm

yes, the link to your sitemap must be included in the end of robots.txt because Google read robot.txt before crawling your website and robots.txt tells it which URLs are blocked and also tell the location of xml sitemap

hirephpexpert · Post by **hirephpexpert** » Thu Apr 12, 2012 12:19 am

Both are on page factor's. These both are important for website.

Robots.txt is a file that is used to leave out content from the crawling method of search engine spiders / bots.

Search Engine Forums, SEO Forum