Analyzing Website Traffic

Website traffic reports are generated from the log files that are kept by a web server. These logs contain records of every request for pages, graphics and files on the server. The records include the IP address of the requesting computer (not the actual identity of the visitor), the time and date of the request, what kind of browser requested the file, what kind of computer was being used, and whether or not the request was successfully fulfilled. The raw log file is a running list of technical information and it's not very helpful to look at. That's where software like Urchin comes in. Urchin reads these log files and then presents the information in ways that are much more comprehensible and helpful to site owners.

We have set up Urchin to compile and generate our traffic reports on a nightly basis. One of Urchin's nice features is that it allows you to set and view your traffic information by whatever date range you set. Urchin's interface is very easy to use. Simply click on the "enter range" link below the calendar display and a window pops up where you can visually select the start and end date for the range you are interested in analyzing. The report adjusts its numbers on the fly when you click "Apply Date Range."

There are six groups of reports available in the Urchin system: Traffic, Pages & Files, Navigation, Referrals, Domains & Users, and Browsers & Robots. The "Traffic" group is the default group when you first open up Urchin. It contains four different ways of reporting traffic. It reports site activity by session, pageviews, hits, bytes, and then gives a summary view of all four. The default screen upon log-in is perhaps the most useful bit of information; it describes the number of site sessions or unique visits.

One of the terms frequently thrown around regarding website traffic is "hits." What most people mean by "hits" is the number of visits the site gets. But technically, the word "hits" actually refers to the total number of files that are requested from the server. Therefore the number of hits to a site is always going to be significantly higher than the actual number of visits to the site. This is because a typical visit to a website will include "hits" on a number of pages. Not only is each page counted as a hit, but all the graphics on every page requested also register as uniquely requested files, a.k.a. "hits." Given the number of graphics on a typical webpage (including the navigation bar, highlight states of graphic menus and so forth), the difference between hits and visits is substantial. It would not be uncommon for a traffic report to show ten or even twenty times as many hits as actual visits.

When analyzing your traffic report you can, for the most part, ignore hits and bytes (bytes being the total amount of data sent out from your site). Instead, it's much more helpful to review the trends in sessions, pageviews, and length of stay. Urchin provides both totals and averages for these traffic stats.

Tracking your overall website sessions is the best and most accurate way to determine your site's performance. A session is a unique visit by a singe individual. One session is recorded for each unique site visit whether the visitor looks at one page or every page on the site.

Because every site is different, comparing your site's performance to another's is not very useful. Some sites are designed to attract daily readers. Some are simply used to support other marketing materials. Therefore, comparing overall traffic volume is not a useful effort. Rather than worrying about how many sessions your site receives compared to other sites, it's better to understand how your site adjustments and marketing efforts impact your site's overall traffic patterns. Therefore, tracking session data from month to month is the most fundamental aspect of analyzing your website traffic reports. Hopefully, you will see growth in your overall session traffic over time.

Another gauge of performance is the average number of pageviews and average length of a session. This information is useful because it shows the "stickiness" of your site. Having started a session, do your users stick around and look at lots of information, or do they leave after hitting the home page? The average number of pageviews tells you how your site is performing. The average length of stay is related to the pageviews, but indicates how intensely people are looking at your site. For example, you might only have two or three average page views, but a long average session time. This would imply that while people are only looking at a few pages, they are really reading those few pages carefully. Conversely, you might have high page views but low length of time, indicating that people are clicking through your site a lot but aren't really reading it very thoroughly.

Again, comparing these aspects of your site to other sites is problematic. Some kinds of sites are always going to out-perform others simply by their nature. But for the sake of generic benchmarking I went through a sampling of the sites on our server and ran some averages. Here is a breakdown of the sites we've developed.

The average number of pageviews (based on roughly six months of data) was 4.6 pages viewed per session. The highest number of pageviews among the sites we reviewed was 13.5 per session and the lowest was 1.7.

The average duration of site sessions was 4:53 minutes with the longest average being 12:16 minutes and the lowest length of session being only 2 minutes.

The second group of reports in Urchin is the Pages & Files group. This shows you the most requested pages on your site. While Newfangled believes strongly in the advantages of database driven websites, there is one small downside to such sites, that is, log files record page URLs (website address), not the actual page titles. This means that when you view the "pages stats" you will see pages listed by their database driven URLs such as "/contentmgr/showdetails.php/id/3353." This is not very helpful information since you have no idea what "/id/3353" refers to. Fortunately, Urchin turns all these references into links. By clicking on the title in the report, the page in question will open up in a new browser window so that you can see what it is.

Also, in the "Pages & Files" report, the top most view page of a site is usually listed as "/". This "/" simply refers to the "root" of your site a.k.a your home page. Additionally, you will see a page listed called "/robots.txt." This is a system file that gives some information to search engines when they index your site. You can ignore this item.

The third group of Urchin reports is "Navigation." It contains reports of the top entrance pages, exit pages, click paths, click to and from, length of pageview, depth of session, and length of session.

Now don't freak out when the most common exit page of your site is your sites homepage. The preponderance of sessions will leave after only viewing the homepage. Another potentially disturbing stat is that the most common average session length to your site will be less than 10 seconds. These numbers are always disturbing at first because what it seems to indicate is that most visitors hit your home page and then leave your site right away. It's not. This quirk is due to the high number of automated systems that hit your site and leave a record in your logs of quick, one-page visits. Because these hits are not from human beings, they do not go any further than your homepage. These automated systems include helpful things, like search engine spiders that index your site, as well as not so helpful things like automated email scrapers looking to harvest email addresses off you site for spammers. You will see a similar distortion in your reports in the depth of session, and length of session reports as well, for the same reasons.

As you peruse these reports, simply disregard the first sets of numbers that indicate such activity. However, if you go to the top exit page report and you find that your homepage ("/") is the exit page more than 20% percent of the time, you might want to look into it further. Less than 20% of such activity would be considered normal.

I find the referral reports very interesting. They tell me whenever a visit is generated from a link on another website - including links from search engines. They include which search engines are sending the most traffic, and what keywords people are using to find you. It's obviously very helpful to know where your site traffic is coming from and how people are finding you through search engines. I'm often surprised by what I find in these reports. The information I glean can also help me fine-tune my terminology or discover areas of my site to augment with offers, or additional information.

I have never found this set of reports very useful. It simply tells you what network a visitor was using when they came to your site. Because there are so many networks, and since knowing what network a visitor used doesn't really tell me anything about the actual visitor, I don't pay much attention to this information. The countries report is interesting. It's neat to know that people all over the world are reading my site, but it's still not terribly useful to me.

This set of reports is particularly helpful to us at Newfangled because it determines which browsers and platforms to support. As older browsers drop in their use we can safely stop developing for them. For more information on the impact of browsers and platforms on web development check out our two part newsletter from July and August '02 called "Browser Battles." It's good to check this information out, from time to time, just to have a sense of which browsers are currently in use. This is helpful if a client receives an email complaining about browser issues from someone using an archaic browser or platform. Sooner or later someone out there is going to see something weird on their screen due to a site's incompatibility with older browsers. Being able to provide exact numbers of how infrequently such a browser or platform is used can help to resolve any feelings that a site is "broken." Newfangled generally will support a browser if it is used by more than 3% of users. If a particular browser drops below that threshold, we stop testing and supporting it to maximize our client's development budgets.

Urchin provides very helpful site-wide global statistics. In addition to providing Urchin reports for our clients we also have an optional advanced NewfangledCMS applications for tracking specific user sessions, particularly when they result from online marketing campaigns. We've mentioned these tools in past newsletters but we'll go into more detail on what they do and how they differ from Urchin reports next month.