Wikis and Swikis and Blogs, Oh My!

The term Web 2.0 is being thrown around a lot these days. It sometimes stands for just about anything deemed new online. New web tools, new technologies, or new design trends are often given the moniker "Web 2.0." No doubt the term is suffering from over use. In general Web 2.0 refers to websites and web based tools that are trying to address, in one way or another, the rising need to manage our rapidly expanding access to vast sources of information, consumer products, relationships, software tools, and more. We are confronted with many choices in almost every sector of our lives. So many that we need help navigating them. Google is, of course, currently the best known tool for searching for web based information - but they are quickly applying their mission "to make all the world's information accessible" in other areas like books and video too.

Google is the mathematician's approach to organizing the world's information. But there are other approaches in development. These approaches might be described as the sociologist's approach to information processing. Such concepts as web based collaboration, social bookmarking, community tagging, and other popularity ranking mechanisms are different ways we can evaluate an ever expanding world of choices. Web based tools arising to serve this need are grouped under the term Web 2.0.

Next month, in part two of this newsletter I'll examine at a few specific sites and tools, such as wikis, swikis and blogs and the sites that organize, search and rank them. But first some background as to why these tools are rising so rapidly.

Historical economic rules are shifting. As the internet makes the world's information accessible, certain well-established marketing and distribution dynamics are radically changing. The constraints of product to shelf space, television shows to lineup schedules, and newspapers to circulation are being broken. The internet is having an effect on consumer products, channels, and news sources and it's breaking all the rules. Chris Anderson, senior editor of Wired magazine has a new book coming out in July called "The Long Tail." We've written about the Long Tail's effect as it relates to search engine optimization, but that's only one aspect of the Long Tail's broader economic impact. The Long Tail effect happens when the exponentially growing availability of stuff, like news, consumer products, entertainment and even software becomes readily available to well, just about everyone.

Let's start by examining the Long Tail as it relates to the world of consumer products. Retailers like Border's, Blockbuster and Tower Records have to carefully decide what products to include in their inventories. Managing shelf space constraints is a tricky job. Retailers have to figure out which, among all the vast options available, will sell. Despite careful research and analysis, most retailers will confirm that ultimately only 20% of their inventory will account for 80% of their sales. The most popular books, biggest hits, and blockbuster movies deliver the bulk of sales. This has been a consistent retail reality for brick and mortar establishments. But now, the internet has made shelf space an obsolete limitation. Online retailers can now "stock" far more than what they could have otherwise kept in inventory or put out on display racks. But if only the most popular items make up 80% of sales, why should a traditional retailer worry about an online competitor with a vastly larger inventory? In the end aren't huge inventories just distributing the left over 20% of sales among a lot more items? So what's the big deal?

The big deal is that when a much larger pool of options becomes available something remarkable happens to the 80%-20% rule. It shifts closer to 50%-50%. Chris Anderson's article "The Long Tail" and his blog www.thelongtail.com goes into detail about this effect. I'll try to summarize.

The shorthand reason why 80% of the less popular inventory of traditional brick and mortars only adds up to 20% of sales is that they've had to radically shorten the tail of available items due to shelf space constraints. They have only so much room, so while there may be hundreds or thousands possible books, movies, and CDs the actual in-store choices are limited to just a few thousand.

For example, let's compare a real world Tower Records with an imaginary one. A real Tower Records has only so many racks to store and display their CDs. If we were to view a chart representing their monthly sales we would see a tall head at the beginning of the chart representing the high sales of the most popular items. After the head a quick sweeping descent would drop down close to the base line but extend to the right as the remaining 80% of less popular products contribute their meager sales to the total. The combined sales of the under performing inventory ultimately only add up to 20% of total sales. The sales chart starts with a tall narrow column followed by a longer tail.

But interestingly, the humble sales percentage of the larger block of under performing inventory is actually a consequence of shortening the tail due to inventory constraints. The tail ends abruptly at the exact point where inventory ends. The sales chart stops when there are no more products to report on. Unstocked items obviously have no sales at all.

Now imagine walking into the biggest Tower Records you've ever seen. This Tower Records has just about every record ever made. Whether you're looking for the latest releases or an old folk ballad, you'll find it there. (For this illustration to work this immense Tower Records would also have to be just around the corner from everyone on the planet.) Some items at this mega Tower Records sell like hot cakes, but the older or more obscure titles don't sell nearly as much, maybe just once or twice a month. While these less popular items aren't selling fast, they do all sell - albeit just a little. If such a store could exist, the sales receipts would show that the most popular items no longer add up to 80% of sales, surprisingly they now make up only 50% of overall sales. The remaining 50% is made up from a little of everything else. The fact that the long tail of products is not limited by space constraints means the low selling tail goes on further to the right than it otherwise would. Much further.

But interestingly as the tail extends to the right with very low sales levels, the items at the end of the tail still sell, the tail doesn't drop to zero sales until the very last segment of the sales chart. The long tail, because it is not chopped off at the physical inventory limit, continues to add sales. Until, at final count the combination of piecemeal sales totals 50% of overall sales. To get why this shift is so dramatic you need to see just how long the long tail gets.

A real world example of this phenomenon is Netflix. A typical Blockbuster has about 3,000 movies in stock. (Numbers are from Chris Anderson's article.) Netflix, without a brick and mortar inventory limitation, has over 40,000. Applying the 80-20 rule, only about 600 titles make up 80% of Blockbuster's sales. If we examine the sales data of Blockbuster's 3000th title we observe that it only sells maybe once or twice per month. But what of the 3001st title? For Blockbuster there is no number 3001, nor are there numbers 3002 through 40,000. The tail is shortened and so only adds up to 20%. But if you keep extending the tail from 3001 to 40,000 and all the tiles sell if only a couple times - they'd add up to many more overall sales at the end of the tail. In aggregate, the end of the tail contributes 50% to overall sales rather than just 20%. Netflix inventory is over 13 times larger than a typical Blockbuster store and yet all of their titles sell - if only a few times per month. Christopher Null in his article1 on Netflix in Bussness 2.0 stated "On any given day, in fact, 98 percent of the 15,000 titles in Netflix's inventory are in circulation with customers." (His article was from July 2003 - Netfilx inventory is now over 40,000.)

Other examples where the shift from 80-20 is changing to 50-50 can be observed in Amazon.com sales percentages compared to a typical Border's or Rhapsody's or iTune's compared to a Strawberries or Tower Records. But enough explanation - Chris Anderson does a much better job at it anyway.

When an economic shift like this takes place there will always be opportunity in the marketplace. That's why venture capitalists and current big time internet players like Google, Yahoo, and Microsoft are so interested investing in and buying up so many Web 2.0 internet startups. But this shift is not just about selling consumer products like books and CDs. Information and content itself is a hot commodity. Especially since Google figured out an effective way to monetize content through AdWords and AdSense. In fact, for any industry where distribution can be expanded using the internet and the logistics of delivery can also be facilitated, the Long Tail can flip the current market upside down. It's already a reality in music, movies and books, and it starting to be felt in the news and publishing industry with the rise of the blogosphere, and now in software development too.

Information and the expansion of choices among consumer products are proliferating. There remains one big problem. To grasp this problem, a problem which is leading to the explosion of all the Web 2.0 stuff, let's return to our illustration.

If you walk into a Tower Records you will have a buying experience that has been carefully planned out. Inventory has been carefully researched. The point of purchase displays have been planned and designed. New releases are grouped on one wall, and aisle end caps feature the most popular artists in their genre. Sections are labeled by category and organized alphabetically. But what about our imaginary record store? How would it be laid out? How big would it be? How would you find what you were looking for? What if each genre has several rooms of music not just a couple rows? What if there were aisles and aisles of sub-genres and categories you'd never heard of?

If such a store existed you would probably walk out before long, simply overwhelmed with choices and unable to find what you were looking for. There would be just too much information to handle. Well this is the current dilemma of the online marketplace. Google does a good job helping you find stuff you're looking for. But what about the things you aren't actually looking for, but you would prefer if you knew about? How do we take a world of choices and narrow them down to a manageable few? And how do we do this without retail product managers condensing our options, or editors deciding which articles we can read, or Siskel & Ebert helping us sort through all the videos?

Given the vastness of what is available there really can't be human filters. Yahoo started off being a human reviewed directory of websites, but before too long the exponential growth in the number of websites made that approach impossible - and Google came along with a better - automated algorithm and took the lead in search.

And while Google still has a dominant position there are other ways to identify, discover, rank and filter all the stuff out there. That's what all these new Web 2.0 social collaboration, tagging, and sharing systems are all about. And that's where we will pick up next month.