Early testing can free up resources for additional rounds of testing later, and still stay within the project's budget.

John B Smelcer, MBA, PhD 

"Vote early – and vote often" – William Hale Thompson, former Mayor of Chicagoevery vote counts button

In Chicago during elections, they say "Vote Early and Vote Often." We will hear this often during next Tuesday's contest, and the winners will all claim the election a victory for the people.

User experience designers also want a victory for the people, and it also requires involving more people more often in the feedback process. More feedback means a more productive, usable, and successful information system (or website or app).

What is the most effective, least difficult way to get feedback from users? Usability testing. Steve Krug showed that anyone can conduct a usability test, and he wrote a handy little book to prove it, Rocket Surgery Made Easy. With a pair of conference rooms, an overhead projector for the audience, and screen-sharing software for the user and the audience, one-on-one usability testing with just three participants can provide useful, actionable feedback. And the usability test can be done on any system - the old system, the new system, and rough sketches of the proposed system.

Let's look at this concept in more detail. Test early means testing the new system before all the details are worked out. The benefits are clear, because early feedback tells the project manager to drop features that won't be used or aren't needed. The challenge is testing a system that is still rough, often just a sketch. Yes, the sketch needs to be visible, perhaps with only the navigation and a few screens. The beauty of this lies in the ease and simplicity of change. Changing a sketch is easy and cheap; changing a finished system is hard and expensive. Most managers would choose cheap over expensive any day.

Early testing with a limited budget can actually save money.

Here's an example. We were brought in to redesign the information architecture for the extranet of a large financial planning and investment firm. The project was also large because there was so much content that needed to be organized for efficient use by their financiers. As part of our planning for usability testing, we quickly identified that nearly 20% of their content was outdated and unused. By reducing its scope, we saved over 10% of the development budget. This more than paid for our consulting fee. So early testing can free up resources for additional rounds of testing later, and still stay within the project's budget.

Someone might ask, "What should we do if we have a limited budget? We can only do one test. Shouldn't we test the final system, which has everything that needs testing?" If you find yourself in this situation, then there is not much you can do. You are probably nearing the end of development, and the system is scheduled to go live in a few weeks. If testing uncovers substantial problems, then your limited budget and tight schedule prevent you from making major changes. At best you can make the system look good with exciting graphics, and plan for the next round of iterations with some early and frequent usability testing.

Recently we were approached by a client in the state government. They had spent most of a year redesigning and rebuilding their website. The launch was scheduled for next month, and nothing was going to stop it. They wanted a usability test of the site, which was designed to be used by five different user groups, and they had a very limited budget. They could afford just one usability test, and it was going to be at the end of the development process. They were clearly in the "exciting graphics" situation. As they put it, they wanted to make sure that "the website doesn't suck." Ultimately we agreed that we couldn't help them. And they looked elsewhere for someone to perform the impossible.

More Feedback = Greater Success

Research conducted by Erran Carmel and Shirley Becker (1995) showed that the more often designers and developers get feedback from the users, the more successful their system is. This is true whether measured by user adoption, completion on schedule, or completion within budget.

Bill DeLone and Eph McLean found the powerful importance of use and user satisfaction on information system success. These two factors directly impact the individual, and getting feedback on that impact is essential to information system success.

 

DeLone and McLean IS success model

DeLone and McLean I.S. success model

Testing often is also consistent with agile software development. Every sprint gets a usability test. Sometimes the testing is done on the product of the sprint. Ideally the user experience team would work ahead of the the software developers to test (and refine) the design before it is coding. Either way, the repetitive, iterative nature of testing often works well with agile development.

When the project has resources for only one round of usability testing, there is another approach that will produce a better, more usable system. Instead of one big test, usually at the end of a project, several smaller tests throughout the project will produce a better user experience. Instead of a big test with 25 participants, conduct four or more tests with three participants. This identifies problems early and fixes them before they grow to be insurmountable. A little advanced planning for early usability testing will make the project manager look like a hero.

References


Carmel, E. and Becker, S. (1995). A process model for packaged software development, IEEE Transactions on Engineering Management, 41(5), pp. 50-61.

DeLone, W. and McLean, E. (1992). Information systems success: The quest for the independent variable, Information Systems Research, 3(1), pp. 60-95.

Krug, S. (2010). Rocket Surgery Made Easy: The Do-It-Yourself Guide to Finding and Fixing Usability Problems, New Riders, Berkeley, CA.

Tweets

"Deal with the world the way it is, not the way you wish it was." John Chambers
"We need some ability which allows us to envision the objective from afar, and this ability is intuition." Henri Poincare
"In order to have judgment, you have to have knowledge and experience." - Nancy Pelosi (D, California)
"If you're building a product or delivering a service that solves a real problem, you can do well and grow fast." Marina Hatsopoulos
"You already know the answers. You just don't want to hear them from yourself." - Sunny Vanderbeck
Go to top