How to Help Dynamic Websites Get Indexed

October 6th, 2010

SEO Cartoon

Rate this cartoon!

1 Star2 Stars3 Stars4 Stars5 Stars

(13 votes, average 5.00 out of 5)

 
Loading ... Loading ...

Today’s article

The more people go online without “Web” training, the more tools Web professionals offer them so they can create content or sell things without knowing much about webpage coding.

Nearly all of the newer generation websites are built using one of the data-base driven site building techniques that organize content such as products, news, articles, etc.

Optimizing such sites is a formidable challenge from the SEO standpoint.

Dynamic vs. static

What’s a dynamic page and how is it different from static? Unlike a static .HTML/.HTM page, a dynamic page does not exist as such until the information for the page is requested and pulled from a data base with the help of a server-side program, and as a result such a page is built on the fly. Search engines have difficulties with crawling dynamic sites basically because their requests may result in an infinite number of URLS, especially if URLs contain session IDs.

Search engines don’t want to store multiple copies of the same page in their indexes (if the same page can be built with the help of different request syntax elements that results in different URLs or also, if parameters are used to produce unique URLs, but do not change the page content).

You – as a webmaster – should help a search engine to solve these problems.

What approaches can be applied to dynamic site optimization?

First, webmasters should focus their efforts and follow webmaster guidelines for Google and Bing. Google is the top referrer among search engines (as of August 2010, more than 71% of US searches according to HitWise and 83% worldwide according to HitLens), and Bing is its main rival since Yahoo! has been losing ground on the SE battlefield (as of August 2010, Bing+Yahoo! have more than 24% of US searches according to HitWise and 13% worldwide according to HitLens). Moreover, Google’s Chairman has named Bing as Google’s primary competitor.

Second, one should consider rewriting URLs for all search engines to crawl more effectively.

As Apache servers and Microsoft servers are most often used (56% and 17% share respectively of active servers across all domains), we’ll mention how to optimize URLs of sites hosted on Apache and IIS servers.

What Google recommends

Basically, Google recommends three things:

  1. Use of the rel=”canonical” tag in the head section of the dynamic pages to specify a preferred URL of a page that Google will store in its index:
    <link rel="canonical" href="http://www.example.com/dresses/greendress.html">
  2. Creating and submitting a sitemap that lists your canonical (preferred) URLs for the dynamic pages.
  3. Specifying URL parameters for Google to ignore in your Google Webmaster Tools account. In this way, you can make Google omit all kinds of IDs, “sessionid”, “source” and a multitude of other parameters making a URL unique but not really creating a unique page.

Note that Google gives no guarantee that it will take into account all the information that you gave to help it better index your site.

What Bing recommends

  1. Bing does support the rel=”canonical” tag (together with Google and Yahoo!), so you may follow the above Google advice for better indexing of your dynamic content by Bing. Read this article for more information.
  2. A sitemap will also help you have Bing index all of your important site pages.
  3. Although MSNBot (Bing’s crawler) claims to be able to read and follow URLs using more than 30 variables, Bing recommends minimizing usage of parameters. Unlike Google Webmaster Tools, there is no possibility for webmasters to tell Bing directly which URL parameters it should ignore.

Server-side rewrite engines

These programs are used as a part of server software to replace pieces of URL syntax in a query string (question marks and parameters) with static-looking slashes and directory names so that the URLs could look as plain HTML. The replacement rules based on usage of regular expressions or more simple replacement techniques are entered into the configuration file (for instance, .htaccess for an Apache server).

For example, with the URL rewriting rules in place, the below URL

http://www.some.site.com/agent/answer/256

corresponds to the following GET request:

http://www.some.site.com/index.php?person=agent&command=answer&id=256

If your site is hosted on an Apache server…

You may use the favorite technique of many SEOs – use of the “mod_rewrite” module. More information and step-by-step instructions can be obtained on Apache Module mod_rewrite page.

If your site is hosted on a Microsoft IIS server…

A similar regular expression-based technique can be used to rewrite dynamic URLs on IIS servers. Refer to Rewriting URLs for detailed description.

If you have a WordPress-powered blog…

Use the All in One SEO Pack,  a WordPress plugin that optimizes your blog in many ways, including support for advanced canonical URLs.

Bookmark and Share

Web CEO Metrics

Here we are sharing the generalized numbers from our HitLens Web Analytics service. It covers 300,000+ websites from all over the world.

This September the world searched Google more than they did in September 2009: Google referred 83% of all visitors. Yahoo referred 7% of visitors to sites against 10% in September 2009. Bing is stable: it referred 6% of all visitors in September 2010 as well as in the same period of 2009.

se-share-september-2010

Global Search Engines (%)

You can see how visitors are being referred to websites.

In September 2010 visitors used search engines in 47% of all searches (down 2 percent compared to September 2009). Bookmarking and linking are more popular – 2 and 1 percent up correspondingly. The amount of people who used paid advertising has decreased – 6%.

site-ref-september-2010

Visitor Referrers (%)

3 Responses to “How to Help Dynamic Websites Get Indexed”


[...] This post was mentioned on Twitter by SEOMixTour, Deborah Van Hoose and Lana Zaitseff, WebCEO. WebCEO said: http://goo.gl/igBP How to Help Dynamic Websites Get Indexed – SEO MixTour by Web CEO #seo [...]

Randy Kemp says:

Excellent website statistics. I also like the cartoon. Does the same person that writes the article draw the cartoon?
Randy

    Lana Zaitseff says:

    Randy, thank you! We work as a team: the editorial team creates the cartoons, our designer draws them, our editors write the articles – all at one place :)

E-mail(required - will not be published)




Comment