Newfangled works with independent agencies to create lead development web platforms for their clients.

Newfangled works with independent agencies to create lead development web platforms for their clients.

Quality Assurance



Quality Assurance (QA), according to the online Project Management glossary, is defined as "a planned and systematic pattern of all actions necessary to provide adequate confidence that the item or product conforms to established technical requirements." In other words, QA is a standardized method that ensures that everything works as it was intended to work and looks as it was intended to look. QA for the web should include an initial site-specific test plan, a round of browser testing, and a generous integration phase during which the client can evaluate functionality while entering content.

qa.jpg

Quality Assurance Test Plans

Every website built around a content management system (CMS) will have a significant amount of common functionality that will require testing. This kind of general evaluation might include anything from testing the CMS login to exporting website form data, but is not primarily concerned with how things appear visually. Of course, if something is radically out of place visually, it should be noted. However long the testing list, this first step should be to identify any flaws in the standard operation of the site and CMS and can probably be performed once the whitescreen is complete and before design application has occurred.

The second step is to evaluate functionality specific to the website. Again, this stage is less concerned with how things appear visually than with how things operate. For example, an e-commerce website's store should be thoroughly tested with every relevant combination of products, accessories, and discount codes to make sure that even the most minor variable isn't overlooked. A site with a large content database that relies heavily upon an advanced search tool should be tested by running a large number of various search queries. A site with complex form options should have many test form submissions sent covering all options and combination of options, and so on. While the test plan should be drawn up by the team--naturally, those most familiar with how the site is intended to function--the person performing the test plan should be someone familiar with the technology, process, and purpose of QA, but new to the particular project being tested. Even if you don't have a dedicated QA role, you should endeavor to have fresh eyes on the site for this stage of QA.

folder.jpgFor a more in-depth look at what the QA role should look like, read Nolan Caudill's blog post on The Most Important Person on the Team.

 

Browser Testing

crossbrowsertesting.jpg Realistically, there is going to be some overlap between the test plan and browser testing. Common site functions, like form submissions, for example, can have unpredictable issues in different browsers. But once the site functionality has been thoroughly vetted, the site needs to be tested, page by page, in every browser you officially support.

Browser testing can be done in a variety of ways, including running multiple physical machines, running virtual machines locally, running a centralized virtual machine server, or using a third-party testing service. We've been experimenting with using CrossBrowserTesting.com, which can test browser performance and some interactive functionality for any URL you submit. It can even generate screenshots of any URL in every current browser simultaneously (I ran this test on my blog page, which you can see in the image above). Plans range from as low as $19.95 per month to as high as $199.95 per month, but the value of being able to quickly check against the particularities of multiple browsers is well worth the cost.

By the way, despite being nine years old and out of compliance with today’s web standards, Internet Explorer 6 is still being used by a considerable portion of the population. While tools like CrossBrowserTesting.com make it easy to check out how a website looks in IE 6, they don't change the fact that it often requires a lot of extra effort to make a current site look and function correctly in it. At this point, it doesn't make much sense to do more than ensure that websites degrade well to IE 6; guaranteeing perfect performance in it is a losing battle. Even our site isn't perfect in IE 6!

browser_comparison.jpg

arrow.jpg These screenshots of my blog page (click image to enlarge) were automatically generated by CrossBrowserTesting.com. At first glance, they look pretty close, but there are some slight spacing differences in the page as viewed by Internet Explorer 6.0.

 

Integration is QA

Once the test plan is complete, the site is probably ready for content entry (we'll cover this a bit more in next month's article). Remember, this often happens before the design has been applied, so it's important that the client is prepared to see and use a work in progress. Though it's not officially considered a QA method, I believe that content entry, or integration, is one of the most effective and important QA efforts for any project.

steve_quote.jpg

Typically, integration is the point in a project when a client is able to fully experience the reality of their site for the first time. While they have worked closely with the team on prototyping and design, the process of using the CMS to create and enter content is when all the "dots" are connected and made real, and often the first point at which expectations are clarified. With that in mind, here's some straight talk:

No matter how thorough the prototype is, sometimes there are concepts or needs that cannot be communicated until you are immersed in an actual working and producing environment. This is similar to the "blank-slate-shopper" phenomenon: Have you ever seen a review of a book and thought that you'd like to purchase it, only to find that the next time you are actually in a bookstore you have no idea what you want or where to start? This is because we tend towards reactive rather than proactive thinking. We hear about a book and react to it with, "Yes, I'd like to read that," yet when we get to the store and are surrounded by thousands of books, we react to them all by drawing a blank. In web development, things are reversed a bit. Prototyping can be like the store, offering lots of attractive options unrefined by the future reality of how a site will be used. "Yes, I'd like my site to do that!" But integration will always catch the flaws in a site, be they many or few, because users will quickly react when they can't do what they expected to be able to. "Sure, the slideshow is nice, but I need to change the address in the footer!"

Finally, QA does not ensure that a project will be 100% bug free. While some bugs are due to sloppiness or haste and can be quickly identified by QA, others are the result of unforeseen functionality conflicts that may not become evident until a site has been used for a while--despite the best intentions and foresight of the team. As with any development project, bugs like these should be expected and encountered with patience. (Need I remind you of how buggy some expensive operating systems are when they launch?) While we hope that the various steps of QA will mitigate the frequency of any bugs occurring, we are definitely not surprised when they show up.

 

Coming Up in Part 2...

It's not done yet! Next month, we'll look at the last two steps of a web project, content entry and going live. Once a site is live, it begins its real life, so we'll also cover content strategy and nurturing for the future of a website.





Comments

Justin | February 2, 2010 11:04 AM
Chris, Great post! Nice to see our process so well articulated. Perhaps this will help dispel the myth that we have an extra button on our keyboards that simply says, "Make Website."
Chris Butler | February 2, 2010 12:59 PM
Justin, Shhh! Don't let that secret get out. Otherwise, what would I write about every month? ;-)
Joanne Cirillo | February 2, 2010 2:08 PM
I watched your webinar on Personas and found it very instructive. Is there an application for this process if you're already way down the road on your web site development?
Jeri A Hastava | February 2, 2010 3:02 PM
Great post, and I would love to see the same post with the addition of content strategy - gathering, analyzing, and planning for content today and into the future - at the very beginning during the planning phase. Planning, designing and developing in the absence or "real" content can result in content that simply doesn't "fit" the site, or fails to convey the message - not to mention frustrated customers. :-)
Chris Butler | February 2, 2010 3:09 PM
Joanne: Definitely. In fact, I was speaking with one of our project managers just this morning about planning for persona development for a client that has been using their site actively for almost two years now. Specifically, this client does a lot of television advertising but has not yet focused their website on a particular segment. As a result, they don't really have any content strategy at all and are assuming that the same people that respond to a television ad are their web audience too. Our hunch is that assumptions is incorrect. The value of identifying personas is there, really no matter when you do it. It's always going to provide some insight that wasn't available beforehand.

Jeri: That's a good point, which is why the persona stuff is so critical. If you don't identify who you are speaking to with a site, you won't have any significant or accurate model for content. I'll be covering content strategy in the second part, not because it's secondary, but just because I wanted to deal with the fundamental/mechanical stuff first.
Robert | February 2, 2010 5:27 PM
Nice post, and when did you guys do the redesign? Looks great! Just curious though on the choice for navigation, use of images and a table? Seems odd, surely that could have been achieved with plain text and css. Just curious if there was a good reason for it.
Jason | February 2, 2010 7:11 PM
I see the value of having the project managers create and implement test plans. Being the most familiar with the spec makes them the most qualified to assess whether the project measures up. But what about the code itself? Who does a QA review of the code for security issues, leanness, web standards, etc.? I'm not a big shop like you all, so the biggest problem I've had has been with working with freelancers in the past and not having a way to ensure code quality, so even if a site looks great when it's first launched, it degrades fast. I've learned a ton as a result about how little I really know about code ;-) So now I depend upon other programmers to check the work in addition my own review.
Chris Butler | February 3, 2010 9:27 AM
Robert: We went live with our redesign on January 12. You can read all about the process in a blog post I published that morning called How We Redesigned Newfangled.com. As for your question about why the navigation is programmed the way it is, I'm going to let the developer who built it, Dave Mello, answer. From my point of view, the new site is significantly faster than the old one, which makes me happy!

Jason: You raise a good point. I neglected to mention developer code reviews in the QA part of the article. I can only fit so much in, but it is definitely a critical piece of the QA process. Right now, our developers have a code review prior to the site going live with our Lead Software Engineer, who evaluates their work from the perspectives you listed out. We were actually discussing this just the other day, and are going to move to a new model of having developer code reviews every two weeks- on a schedule independent of specific projects- to make it more of a developer department culture thing, rather than just a step in the project process.
Dave Mello | February 4, 2010 9:30 AM
Robert: Thanks for the feedback! In rebuilding the new site, we drastically reduced our use of images *and* tables. The result is a must faster load and rendering time. In the case of the main navigation though, I chose to stick with a simple table/image structure for a couple of reasons.

Since the menu system is dynamic, I wanted to utilize a little bit more control of how the columns appeared if the ordering or placement changed at all, particularly in older browsers. As for using graphics, since the menu actually serves as one of the main visual elements, I wanted it to be as consistent and attractive as possible, particularly in cases where no browser/os induced anti-aliasing was taking place. You are correct in that the same structure could have been built using only CSS, but I made the call to take the minor trade-off in performance to ensure consistent design. We are also using ALT tags in the image itself to avoid any SEO issues.
Chuck | February 4, 2010 5:43 PM
I read something good in another article about website mood boards from WebDesignerDepot - "Words fail miserably when trying to translate design concepts... Visuals communicate things words cannot." It's true. Why try to pitch a visual concept in a written form to get your client to buy into the design you haven't created yet? So mood boards are great at getting to the point faster. But I like how you emphasize that the strategy happens with mood boards and the production happens with the layout iterations after. Most desigers try to do both in layouts and end up redoing their work over and over again. In the WebDesignerDepot article they talk about when clients don't want to pay for mood boards. How do you handle that? Do you breakout the costs and let them choose?
Chris Butler | February 5, 2010 2:40 PM
Chuck, That's a good article you linked to- thanks. Our process always includes mood boards. We don't break out the price and allow clients to choose a path. The only time we wouldn't do mood boards is if we decided to not include them in the process for some reason- perhaps to keep the scope narrow and work within a lower budget. But in that case, we would have very specific contractual language to keep the scope of the design phase as limited as necessary.
Jessie Nunez | March 27, 2010 2:28 AM
Wow! I skimmed over many of your articles and plan to read them in depth. Great writing and content. By the time I'm done, I'll have a UI degree (of sorts).

I primarily focus on visual design for smaller businesses. These businesses generally don't have large budgets. However, I believe that best practices should be the rule and I ask lots of questions to help my clients think beyond "We need a cool web site with such and such bells and whistles".

I would appreciate any advice on how to implement and maintain good UI principles when the budget is under 10K (way under 10K).

Great work and much continued success to your firm.
Jann Mirchandani | December 20, 2010 4:39 PM
Chris,

Great article. I love seeing the process from a large shop; gives some perspective to what we're doing here; ways to improve and validating what we're getting right. Thanks!
Joel Milne | January 28, 2012 7:03 PM
Interesting idea, I like the concept of a more specific way of profiling users instead of just borrowing the traditional marketing plan approach to forming target profiles.
Adekunle | March 22, 2013 12:59 PM
Wonderful article.! Looks like mood boards breaks down the process into small bites for any design project. Thanks!

↑ top