Page 1 of 1
Completely dropped out of Google
Posted: Thu Feb 10, 2005 2:30 am
I was able to successfully search for and reach exact phrases at my site until this week (albeit home page only). Now the google cache seems to have suddenly and magically disappeared completely and no searches reach my cache page. Any ideas? Where did the cache go?
Completely dropped out of Google
Posted: Fri Feb 11, 2005 7:28 am
To dyn4mik3 (see last posting): Thanks for pointing out the "error re-direct". I have now adjusted the index file to accept robots (I didn't use one of the cut/paste routines to create a separate robots.txt file. Sound cool?
Also, I am confused: 1) where can I find the redirect instruction you mention? 2) why would any web design software create such a file? (I didn't do anything outside the application to create an erro redirect? 3) I have created a very similar website alongside this one and it doesn;t suffer from this problem
Posted: Fri Feb 11, 2005 7:43 am
I had thought one potential cause for this change in robot performance could have been that when I had completely revamped my site (just before this happened), I had inadvertantly copied the wrong text into the "description" and "keywords" areas - meaning the content would have been totally irrelevant. I have now corrected this and am waiting. Any comments on whether this might have caused google to stop and remove my cache pages per my original posting?
Still interested to get answers to my questions in the previous posting above.
Posted: Fri Feb 11, 2005 8:55 am
dyn4mik3: Thanks I'll do it now. But I am still puzzled - I didn't ever have one in the first place (and I was crawled regularly) - and my other site doesn't have one (and I am crawled regularly) - in neither case did I ever have this problem. It's not that I dont believe you - I simply dont understand the "data" based on my previous actions and results: Comments?
Posted: Fri Feb 11, 2005 12:27 pm
dyn4mik3, I have never created such a (404) page - I am not sure how to. Give me some more clues about this scenario - this still doesn't add up. Although I have now put in the robots file on the current site as suggested, I have also gone back to my old site files off line and can find no evidence of robot file ever being there or indeed anything different between the two in the header stuff. The strongest clue I still have is that my metas on the revised site were completely wrong and from a completely irrelevant piece of text. NOthing else significant has changed.
Posted: Fri Feb 11, 2005 1:04 pm
I will leave the robots.txt file addition and the correction of meta stuff to see what happens if google comes back - I'll share the outcome here
Thanks for your help thus far
Posted: Sat Feb 12, 2005 8:04 pm
are you guys saying to create a robot.txt file just like you would an index file or other page? and do you have to make any reference to it in your source code?
Also, should I add a robot.txt file in my sub directory(s)?
Posted: Thu Feb 17, 2005 9:33 am
My site has returned to Google and now it goes further than just my home page unlike before. I did two things
1) added a simple robots file (with stuff to repel bad robots)
2) I corrected my erroneous meta fields (completely irrelevant garbage before)
So, I don't know either what really brought google back or what caused it to now go past my home page.
I must say, I am still highly puzzled by our previous entries in this thread where emphasis has been placed on the re-direct piece. I have never placed a redirect in my site - my other (very similar) site has similar structure without robots.txt or anything to do with redirect (I woudln't even know how to add it), and it has never suffered from either getting google past the home page or dropping out as this one did.
All I know is that when I rebuilt the site using cleaned up text and layout, it disappeared for about three weeks and then came back. (btw, it still doesn't get google down to the third level - but I am thankful it is now back to at least this stage).
Posted: Sun Mar 06, 2005 7:52 pm
A robots.txt file (notice the plural) is key for search engines to index your site. There are several good tutorials for them and make sure you have one on any site.
Re-directs are setup in the .htaccess file. That is typically where you set up a 404 redirect. Get a good book on .htaccess, because this file is an unbelievable tool. You can do a million different things with the .htaccess file. For example, you can create a page with php on it and name the file something.htm, then you can use the .htaccess to make sure the php parser is invoked when someone calls that page (usually you would need to end the file with .php). You can re-direct pages, you can even hide variables from the search engines with the .htaccess. For example, most search engines don't really like the following type of URL:
http://www.mysite.com?somevariable=abc& ... riable=def
This is because search engines don't like following variables. But, using the .htaccess file, you can mask variables so the above link becomes:
http://www.mysite.com/somevariable/abc/ ... riable/def
Much more search engine friendly, meaning they are more likely to index deeper into your site.
All of this is done with the .htaccess file. It's a cool tool.
Posted: Thu Mar 10, 2005 2:24 pm
Things have changed yet again.
Although G came back and indexed the whole site starting earlier in Feb, as of Feb 23 it has gone back to listing only the home page. This is an on-again, off-again thing for this site and I can't understand it - versus my other site which is always indexed at every page I chuck at it all the time.