No sooner had the first website been created, then I am sure the owner asked – is anyone looking?

Everyone wants to know how many people are visiting their website, how long they are staying, where did they come from and so on. This is the realm of website analytics.

Log analysis

All web servers keep logs of activity and the first analytics programs processed the logs to give reports.

The first thing we are interested in is how many visitors there were in a given time period.  This is not a simple question to answer because there are four different numbers.

• how many server requests (aka hits)
• how many web pages have been looked at (page views)
• how many different times someone has come onto your web site and looked at pages before going away (visitor sessions)
• how many different people have come onto the website (unique visitors)

So a statement like ‘we had a million hits last month’ doesn’t mean a million people visited the web site, it might equate to:

• 50,000 page views
• in 20,000 visitors sessions
• by 10,000 unique visitors.

In fact ‘hits’ is the least useful measure because it depends on how many server requests there are per page, which depends on the page design and construction.  A modern web page will often require 10-20 server requests to load it up.

What else can we find out?

We can also find out

• what web page the user came to the site from
• and, if that web page was a search engine, what keywords were searched for 
• what type of computer and operating system 
• and what web browser
• what internet address and from that (sometimes)
• what company or what location

So there is a lot of information here, which can be sliced and diced in a number of ways.

Is log analysis the only way?

The problem with log analysis is that it involves processing very large log files and this needs a lot of computer power.  It often takes time, and to get decent numbers, money. But this is the internet, this is the 21st century, we want answers NOW.

So as the technology developed it became possible to produce results in real time. This approach depends on inserting a small amount of code into every page on your web site (tags). These tags send a message to a central server somewhere which is recording every page view. By logging into the server you can get results in real time – or close to it.

Log analysis vs Tags

The two aproaches, log analysis and tag-based analysis measure different things and estimate things in different ways, so will never give identical results, in fact they will often be quite different.  For example:

• tag-based analysis will never give a figure for hits. No big deal because it is a meaningless number anyway.
• tag-based analysis is a bit short on server performance measurement, for example it can’t work out how much bandwidth the site has been using.
• tag-based analysis will not record visits by non-browser visitors, particularly search engine spiders. So you won’t know when Google is indexing you.
• log analysis will not record visits when web pages are cached locally, by the browser or by the visitor’s network

However trends should be the same and with enough work you can make the results reasonably consistent.

If you want to a really top-notch job you need to do both and then spend a lot of time correcting and rationaising the data. You really have to want to do this because it is not easy.

Google wades in

There have always been a number of free analytics programs. The most well-known is Webalizer. The two most well-known of the commercial systems in the late 90’s were Webtrends and Urchin. Both offered log anlysis and tag based systems.

Then in 2005 Google bought Urchin. They did (or didn’t do) two things of note:

• they continued to develop the tag-based service and offered it for free (it was previously $495 dollars a month)
• work on the log analysis product seemed to come to a halt. Finally, the first new release since 2004 was announced in April 2008.

Don’t let the ‘free’ price-tag fool you, it is a significant product and gives pretty good results. It was worth $495 a month, and for most users it is all they will ever need.

When your main competitor offers their product for free there is only one way to go. Upmarket. This is what Webtrends have done. A product that was once $495 dollars to puchase is now thousands of pounds plus a hefty annual maintenance fee. This cut out their previous mass market but opened up a new market – the corporate users who don’t feel they have bought a proper piece of technology unless they have handed over a significant wedge.

Major players

In a recent survey 66% of responders used Google Analytics. Over 50% used Webtrends or Omniture. The rest were around 5% each or less.

There were over 50 vendors at Internet World this year listed under the Analytics/Reporting heading – which did not include Webtrends or Omniture. 
 

There is a very interesting dicsussion of the strengths and weaknesses of several of the products here.

Tips