What in the World is Your Website Doing?

This month, for the first time, we've produced our Web Smart Newsletter as an online video. The embedded video below is the full length version (19:40). We've also published the text with shorter corresponding inserts from the full video.

This month's newsletter (and video) is longer than usual since it also serves as a training resource for our clients in interpreting their Urchin website traffic reports and introduces Google Analytics.

Since this is our first video newsletter we'd love to hear your comments, questions, and critique. Please either add comments below or email us through our contact form. Thanks and enjoy! next >

Computers keep track of everything. Every website request is stored in a log file. And not just page requests--every image, every pdf, and every document request is logged. Each entries contain the date and time of the request, the IP address, the site that the click came from, and whether or not the request was successfully fulfilled. That's a lot of data!

These logs provide a wealth of knowledge, but they're not very useful in their raw form. They need to be digested into reports that are easier to understand. Let's see what we can learn from Newfangled's traffic report.

We use Urchin for our traffic reports. Urchin is one of many software packages that interpret raw log files. I'm also going to review Google Analytics but there are a lot of other systems out there too. Check out this blog post for a list of others.

So, let's see how many hits I'm getting--WOW! I'm getting upwards of 30,000 hits a day! Too bad hits is such a useless form of measure. See, a "hit" is any request for any file from my website. When someone goes to Newfangled.com they request not only the home page, but also all the files needed to make the page. Every graphic is a hit. So one page request creates dozens of hits. Let's look at a a better way to evaluate my traffic.

Sessions are a much more accurate means of measuring. One session is counted whenever an individual visits the site. A session could be one page or it include visits to every page. Tracking sessions is a much more helpful way to view my activity than "hits."

Another helpful perspective is individual page views. Because sessions can include just one page or dozens, my session numbers alone won't give me a full picture. By comparing the ratio of sessions to total page views I begin to get a good baseline for overall site activity. Urchin's summary report averages these numbers.

But are these numbers any good? How do I know if my site's doing well?

There really isn't a direct answer to this question. I can use sites like Alexa.com to compare my traffic to competitors. But what's more important than how my site competes, is how it compares to itself over time. This is where the real insight is--which can help me improve my site.

By entering an extended date range I'll display the last years worth of data. Ok, my traffic was growing, but here I notice a significant drop off in November. That's because we re-designed our site and changed the way we display our URLs. We knew this would hurt us for a while in the search engines, but we hoped it would ultimately improve things--and it looks like it's happening since our traffic is now up quite a bit.

And my page views are increasing as well, that means when people get to the site they're sticking around awhile. By the way, these spikes came from a hacker who was trying to mess with our site. When I saw this report it clued me in to the hacker and our system administrator was able to turn away these malicious hits.

Overall this looks good. But there's still a lot more to discover.

So far we've looked at overall stats. But not all of my pages perform equally. What I'd like to know is which pages are viewed most--Let's take a look at the Pages & Files report.

Here are the most accessed pages in my site. Usually, the home page (represented by a "/" ) tops the list. But because of that hacking attempt, the Content Management page (which was the target of the hack) is at the top. I don't want these hack attempts to slant my percentages so I'll use Urchin's filtering tool to remove this page from the list.

When reviewing a website's traffic reports there are usually a few "pages" that can be ignored. For example, the robots.txt is a system file that tells search engine spiders what to do, but it's invisible to visitors so I just ignore it. And in our case the search page gets lots of traffic. That's because when we redesigned our site, and changed all our web addresses, we knew we'd get lot's of 404 errors--that's the page not found error--so we set our 404 page to go to our search page so that visitors could easily look for the content that had been moved. Now I notice that because my date range is still set to the past year I'm seeing our old pages in this list. I'll change the report's date range to a point after we launched the new site to remove these old pages.

Ok. Aside from the home page, my portfolio and pricing pages are among my top performers. After that my newsletter content is getting a lot of activity. Let's take a look the one I wrote about RSS feeds. Clicking this chart icon shows how this page performed over time. Looks like I had a big boost around March. That's because I began running an AdWords campaign to promote my newsletters--looks like its generating some good activity.

Knowing which pages are most popular is very helpful. But another facet of site traffic is not just which pages are viewed most, but which pages were viewed first. See not every session begins at the home page. We can see this under Navigation, by picking the Entrance page report. Now look at this, my home page is the entry page only about 10% of the time. This is an important observation, and it's typical of most sites. That's because search engines index all of my pages, and because sub-pages usually contain the richest, most focused content, it's only natural that these sub-pages would be the first pages viewed from search engine results. This means that the honor of being an Entrance page is shared quite broadly across all the pages of my site. In fact, it seems that most of my top entry pages are actually parts of my monthly newsletters. Since there are a lot of these pages and each one gets a few visits each month--this adds up to a lot of traffic!

That's why, by regularly creating compelling content--which adds more content rich sub-pages--you can experience significant growth in search traffic--but that's a different video.

There are a few pages in this list to be ignored. The robots.txt is one, and these XML pages are another. Because of the way RSS works, RSS XML pages, if you have them, will always show up high on this report. But I'm not going to go into analyzing RSS traffic in this newsletter.

The top exit page report lists the last pages viewed before a session ends. I admit, looking at the exit page report can make me a little sad. And it can be a bit perplexing when I see that many of my most popular pages are also in my top exit page list. If they're so popular why do people go away after seeing them? I have to remember that many visits to my website will be "bounce" visits. Bounce visits are sessions where a visitor hits just one page and then leaves the site right away. This happens a lot with search engine traffic. People do tend to click a lot of links from search results before they find what their looking for. So it's not at all unusual to see my best performing search engine pages are also high in the exit page list, since their activity get there share of bounces.

I can get detail on my exact bounce pages in the bounce rate report.

I don't find the Click Path report very helpful. It reports on the most common paths, or page visit sequences. But because each visit to my website is so unique there really aren't many common paths. There is a variant to this report available in Google Analytics that I do find helpful--we'll be heading there soon.

Another perspective on web traffic is the depth and length of the sessions. The opposite of a bounce session is a "sticky" session, meaning that the visitor stuck around taking the time to look at a few pages. There will always be a high percentage of these 1 page "bounce" sessions. But what I like to look for is a nice distribution of the rest. In this case I see quite a few sessions that look at 2, 3, 4, 5, and 6 pages and then it flattens out until I get to 20 and more where I see a boost. That means I have descent retention in many overall sessions, and a considerable number of in-depth readers. I should see a similar structure in my length of time report since the number of pages viewed and the time it takes to read then should correspond.

I'm skipping the Bytes and File Type reports. Bytes keeps track of the total bandwidth used and Files identifies downloads other than web pages (like images, PDFs, and videos).

One point of interest in comparing Urchin with Google Analytics is that Google doesn't measure bandwidth or files. That's because Urchin uses our raw server logs to generate its reports. Google Analytics doesn't use our log files. Instead it uses a special bit of code we embedded on every page. When each page loads it triggers a call to Google Analytics which uses these hits to generate it's reports. As a result they don't have data on bandwidth, and because non HTML files like images, PDFs, and videos don't have this bit of code embedded in them they don't register with Google.

But Google Analytics does have other advantages. In fact, since the rest of the reports in Urchin--Referrals, Domains & Users, and Browsers & Robots are handled just as well or better in Google Analytics, let's use their system to review the rest of our website activity.

The basic reports in Google Analytics are very similar to Urchin. For example, Google has it's own version of the reports we've already looked at in Urchin.

The visitor overview screen shows visits (sessions in Urchin), and page views. We've mentioned that Google doesn't track "hits"--no matter--the data on hits report is useless as we've seen. I like how Google displays my stats in both graphs and numeric reports. It's very clean and easy to read and easy to use.

Just like in Urchin we have visits, page views, time on site, and bounce rates. Similarly, under the Content tab we'll find the most viewed list which here they call "Top Content." One thing I really like about Google's system is that they have the Top Content by page title. Typically, traffic reports use a web page's file names in their reports. This can be a problem for sites whose file names are obscured by database driven IDs and codes that don't easily differentiate one page from another.

For our site there's not much difference because we've implemented a feature makes our file names and page titles very similar. But if you did have undecsriptive URLs you could use this title view to list the pages using each page's browser title instead of the URL. And if you're doing the right thing with your browser titles--describing each page's subject for maximum search engine optimization this list view will be much more helpful than the URL display.

But let's get to the good stuff. Analytics displays the same info as Urchin in their list of top content, but it lets you drill further down to learn all sorts of things about any particular page's performance. From the Content Drilldown list I can click trough on any page--let's take a look at the page about my book Client vs. Developer Wars. In addition to this page's overview stats I can use this pull down to analyze it from several different perspectives. I can examine this page's performance based on Navigation, and its Entrance Paths, Entrance Sources, and Entrance Keywords.

The Navigation summary shows the most common pages that preceded this page being viewed in a session, and then which pages where most commonly viewed after this page. In this case a good number of people came to this page from our development process page, and then proceed to read the book. The Entrance path view shows where people went, when they landed on this page first, from a search engine or some other website linking to my book page.

Speaking of which, I wonder what sites out there do link to my book--let's check out the Entrance source report.

Thinkvitamin.com is a web development site where I posted a link to my book in the comments of a blog post relating to the web development process. Looks like that five minute activity was well spent - well over 300 visits from the link in that comment alone--nice. I also answer a lot of questions in LinkedIn where I reference my book.

For search engine optimization the Entrance Keywords report is extremely helpful. It shows the keyword phrases used in search engines which led to this specific page. I'm going to switch to my pricing page. These are all the phrases visitors typed into Google whose results linked them here. This is very useful to know when I'm reviewing my title tags for search engine optimization.

There's one more way to slice this report up. For any page I can segment the report based on just about any other factor available in Google Analytics. I can break down the visits, specifically to this page, by the source, keywords used, region or any of these other options.

You'll find the report for overall referring traffic sources, under the Referrals section in Urchin, but in Google Analytics they break their traffic source reports into a few categories. Direct traffic results when no link was followed--the visitor just typed our web address directly into their browser, referring sites come via links on other sites, and also Search Engines traffic. Their combined report shows visits from all these sources.

Our referring sites list shows many of the blogs I post comments on as well as our clients' sites--since we usually add a "built by Newfangled Web Factory" link on them. The Search engine report shows me a break down by each engine. One thing I've observed since we redesigned our site is a much higher percentage of Yahoo traffic. This is due to the change in URL structure I mentioned. Yahoo likes our new shorter URLs. I can click through on each search engine to see which keyword phrases are performing best in each engine.

The keyword report shows me which phrases are performing best in search engines. But there's more. If I click through on a particular phrase like "fun websites" and segment that phrase by source I can see exactly which search engines this phrase is performing well on.

You can really slice and dice site data to an amazing degree in Google Analytics.

Another highlight is that AdWords campaigns can be viewed right from with Google Analytics.

And an additional highlight is goal setting. I've set a goal for Web Smart newsletter signups. I just had to add a snippet of code to our sign-up response page. When that page is viewed Google registers a completed goal. I can see my goal conversion rates for any kind of data, in this case for my RSS newsletter AdWord campaign. We don't sell anything directly on the Newfangled site so I don't have a ton of experience with goals and conversion reports--but you can get a lot of detail on e-commerce activity, AdWords campaigns, and any other kind of search marketing effort.

Earlier, I said that I wasn't interested in Click path reports because most session paths are unique. But this reverse click path, based on sessions that fulfilled the goal I set is helpful. Each time a goal is met I can see where the goal-achieving session came from. And in this case I see that my goals are met from many different sessions and pages. It seems that people come from all sorts of paths when they ultimately do sign up for my newsletter. This is good. It means my site is working broadly. But it also means that my paid advertising isn't producing significantly more sign ups than my organic traffic, so that might have an impact on what I decide to spend on my AdWord campaigns.

There's one more view I want to show that's really cool, it's called the Site Overlay view. When this view is clicked it launches my actual website in a new window and overlays it with Google Analytics data. These little bars show me from page to page where people clicked. my home page shows that the preponderance of visitors go to my pricing page. As I click through I can see where the traffic patterns tend. This is just a real fun perspective for viewing site activity.

Well we're getting close to the end, the last set of reports I'll demo are the Browser capabilities and Network properties reports. Day to day this data is not real import to track. But it is important when considering any design changes or when implementing any new technologies that depend certain browser versions or plug ins. Looking at exactly which browsers are being used helps to gauge how much effort to spend making our site compatible with older browsers.

Clearly Internet Explorer, Firefox, and Safari are the majority. But this isn't really enough. I need to click through to find out which versions of these browsers are being used. For example, I see that for Internet Explorer versions 6 and 7 make up almost all its usage. So if we did decide to implement a new feature that I know won't work in the older 5.x versions of Internet Explorer, based on this report, I won't bother with the 5.x browsers. I can get similar data for platform, screens sizes, colors settings and more. Oh, and these days checking Flash versions is important since new Flash features, like this video player depend on the later versions of Flash.

And last but not least I like to keep an eye on my Network Properties info. These days we pretty much assume high bandwidth internet connections, but there are still a few stragglers that don't have affordable high bandwidth access. In particular I look at my bounce rates based on connection speed. If I see a much higher bounce rate with dial up speed I might want to re-consider my design decisions, in this case it's pretty in-line with the rest so I won't concern myself about it.

Well there you have it. We've covered the most important aspects of website traffic reports. There are plenty of other things you can learn from these reports by playing around with them. You can dig pretty deep into what's actually going on with your site. These reports and the things you'll learn from them inform the changes, updates, and tweaks you'll want to make to maximize your site's performance. It will also help to craft your content development strategy--which is key to using a website effectively.

You don't have to spend lot's of time in these reports. I suppose if you're doing lots of e-commerce or running a lot of AdWord campaigns you might want to look at these reports pretty often. But for non-e-commerce sites opening them up every couple weeks, or even once a month will give you real insight into how your online marketing and search engine optimization efforts are paying off.

And in this world of every expanding Advertising 2.0 opportunities it's good to know which efforts are most worth pursuing. Speaking of which I have some RSS feeds and blog posts to catch up on--see ya!