Search Engine High School

Abandoned by Google! Googlebot, Wherefore Art Thou Googlebot?

As a search engine optimization specialist I often optimize existing web pages for small business clients, upload them to the site and see pages re-indexed by Google within a week. This only happens with existing business sites that have been online for a few years. Google seems to be updating their index as often as every other week at this point and older established sites that are already indexed seem to be re- crawled on that twice a month schedule on a fairly routine basis.

Two clients that hired me for recent work saw their rankings shoot to the top for a newly targeted search phrase in a weekend when I did optimization on a Thursday and they were ranked instantly by Saturday. Now keep in mind that this doesn't happen for everyone, only those that have been online for some period and already have significant content that simply needs tweaking and proper title and metatag information added. They usually have relatively good existing PageRank and do well for other RELEVANT search phrases already. I offer that warning only to avoid instilling false hopes in anyone hoping to achieve the same instant ranking boost overnight.

Those clients that do succeed in this way are often thrilled with the results accomplished in such short order. I'd love to be able to offer that type of ranking boosts to everyone, but some are more equal than others when it comes to easy, inexpensive SEO tune-ups that rev up your rankings overnight. Your mileage may vary.

WHY DO NEW SITES SUFFER?

What is going on with newer sites that don't get crawled for months? I've got a client, a newer attorney directory that offers tons of great information in the form of articles on specific areas of law, links to incredibly valuable and relevant legal sites and over 600,000 attorneys listed by practice area and state. Yet the site has not been re-crawled by Google for over 3 months! Now this would not be such a big issue for many sites, but this site is relatively new and we've optimized all the titles, tags & page text, created a complete site map and placed links to all these resources on the front page.

I know that the site is not being crawled because Google's cached copy of the front page shows it before we did the work four months ago, without the new links and without title tags. We've submitted the site by hand, (manually) once a month for three months via the Google Add URL page. http://www.google.com/addurl.html When the hand submission failed to get it re-indexed for four months, we submitted the sitemap page, which has not been crawled at all. Google shows only ONE page on this site, when in fact it has thousands of pages, a sitemap and dozens static pages!

Part of the problem is that this site must be dynamic, since a database of over 632,000 attorneys must be accessed, retrieved and served for any of those law firms searched for to be returned to the site visitor. Google warns owners of dynamic sites that Googlebot may not crawl dynamically generated pages with "?"" question marks in the URL. This is to avoid crashing the server with too many concurrent page requests from Google's spider. http://www.google.com/webmasters/2.html#A1

The solution to this dynamic URL problem has been discussed widely in search engine forums and solutions have been bandied about including software provided by SEO's, URL re-write techniques for dynamic pages on APACHE servers http://www.alistapart.com/articles/urls/ and PHP pages http://www.stargeek.com/php-seo.php to generate search engine friendly URL's. Others recommend simply adding static HTML sitemap pages as alternatives for the search engine spiders. In this instance the client's developer simply said "I can't do that (PHP solution) on this server". So we resorted to putting up the static HTML sitemap pages with hard-coded

URLS to the main 54 pages of the site at http://lawfirm411.com/Law-Firm-411-sitemap.html This should get at least those fifty pages crawled by Googlebot, but Googles' spider appears not to be crawling this site at all. How do we know this? See for yourself by using the following query in the search box at Google: allinurl:www.lawfirm411.com where the result page shows ONE page in the results. If you try that query on your own site (replace your own domain name for lawfirm411.com), you'll see the results lists ALL your pages.

The site home page was crawled by Google four months ago, when they took their "Cached Snapshot" of the page. You can see this by visiting the Google cached page here: http://66.102.7.104/search?sourceid=navclient&;ie=UTF-8&q=cache:www.lawfirm411.com where the date of this snapshot is "Apr 20, 2004 07:42:19 GMT" and they haven't been back since. The page in that snapshot has none of the newly added links, an outdated title tag, and old content.

This problem is not unique to this site. One client we worked with two years ago had a dynamically generated, framed site! Those two site structures have always given search engines trouble. Their site was not crawled at all and only the front page showed up. Our solution was to create a second domain (owned by the client), which had static HTML pages that precisely mirrored the content of the client's framed, dynamically generated site. Guess what happened after Googlebot crawled the static site? Google indexed the framed site in full and then banned the static site from the index! Not an approach we advocate, but the one that worked for this client.

We're still searching for ways to get Googlebot back to LawFirm411.com before creating that new static site, but decided to share this odd experience with the SEO community before going to any extremes. Google provides over 70% of most search engine referred traffic to ALL of our clients and we realized we can't site idly by and see a major client languish because Googlebot didn't like what it found at the client site on the first visit four months ago. This issue dogs newer sites in other places as well. The Open Directory Project has also become notoriously slow in adding new sites to the directory and in this case, has not picked up this site even after 6 regular monthly submissions. The web playing field may have begun tilting toward older, established sites and away from new ones.

------------------------------------------------------------

Mike Banks Valentine is the SEO for http://www.lawfirm411.com

Contact him at http://www.seoptimism.com/SEO_Contact.htm

Improve Your Small Business Online at our Ecommerce Tutorial

http://website101.com/Free-Tutorials/index.html

------------------------------------------------------------

MORE ARTICLES:

Advantages Disadvantages Of internet marketing: The Good The Bad And The Ugly
The Advantages Disadvantages Of internet marketing are many.A few Advantages Disadvantages Of internet marketing are:* It's on the internet so it's low cost* Very fast* And you can reach a global audienceHowever it:* Can leave the businessman feeling isolated* Hard to tell if people are lying because you can't see their face* And you can be overloaded with information.

Software Companies: Generate New Revenue Streams and Decrease Costs with Custom e-Learning Content
It's no secret that Software Companies operate in a very competitive space where rivalry is increasingly fierce and where profit margins can be razor thin. New, smaller Software Companies are sprouting up each month and the leading Software Companies continually make strong advancements forward leveraging massive cash flow reserves.

Search Engine Marketing 101 For Corporate Sites
When most people want to find something on the web, they use a search engine. Millions of searches are conducted every day on search engines such as: google.

Where is King Content?
One bit of advice you'll read over and over about business websites is that "content is king". Whether this is true or not is debatable, but most marketing and web advisors agree that content is very important.

SEO 101 – The Basics of How to Get to the Top of the Search Engines
Depending on your site's subject matter, you might even be able to start earning a significant income purely from the popularity of your site. For example, if you have a site on jokes you might be able to get a substantial amount of traffic, but you're unlikely to earn a huge income.

Abandoned by Google! Googlebot, Wherefore Art Thou Googlebot?
As a search engine optimization specialist I often optimize existing web pages for small business clients, upload them to the site and see pages re-indexed by Google within a week. This only happens with existing business sites that have been online for a few years.

An SEO Checklist
search engine optimization is on every webmaster's mind these days. Achieving a favorable ranking for the right keywords can mean a steady stream of targeted traffic to your site, and all for free - that's hard to beat.

Why Some Experts Are Completely Wrong About linking
Why Some Experts Are Completely Wrong About linking And How You Can Get Your Website Marketing Back on Track! (Free Software Included)If you have read some information about reciprocal linking and other linking tactics in order to better promote your website, you have probably found some seriously differing opinions as to what the purpose of linking is.There are a wide range of articles and books about linking that come from polar opposite views as to why and how you should conduct a linking promotion campaign.

When internet marketing Gurus Sell Their Souls
If you're just getting started in internet marketing, you may be awed by the amount of "Internet Gurus" out there, each one making millions of dollars (or so they claim) and for some reason seem more than willing, for a fee, to teach you their biggest, most intimate secrets. Well, I believe in one thing - "If it sounds too good to be true, it most probably is!" While I have no doubt about the specialized skills and techniques used by these gurus, it can get a little too frustrating simply getting tangled and lost in all their hype, and find almost no value at the end of the day.

linking for Traffic not Positioning!
With more and more experts and search engine enthusiastsclaiming the right way and the wrong way to handle linkswapping, link exchanging or reciprocal linking! You can tell something is important when there is more thanone name for it! GRIN! There are also two schools of thought on the reasons linkswapping. The first reason for link swapping has always been to carryfavour with Search engine rankings.

Are Keywords Destroying the Flow of Your SEO Copy?
With all the shuffling that's been seen in the search engine world within the last year, the issue of obvious optimizing has become a hot button. The current line of thinking is that most engines (especially Google) are on the lookout for sites that purposely make an effort to optimize their pages in order to get high rankings.

Getting A Solid internet marketing Foundation
As we look at building our online businesses, many of us acknowledge, early-on, that we don't know it all. Those who don't, are almost certain to fail because they often attempt to apply techniques that simply don't work online - to their online businesses.

Writing, Publishing, Self Publishing, Website Promotion, Blogging, internet marketing
Are You Ready For The Publishing Revolution?Writing has traditionally been considered a solitary craft. You wrote you article or book in seclusion.

Content Management Systems Eyeball SEOs
Content Management Systems and search engine optimization (SEO) used to be mutually exclusive terms. But the SEO community has been driving the developers of Content Management Systems to integrate more SEO-friendly methodology within their systems.