News from Apr 01, 2008

Recently, I had a discussion with Shane Nuessler from The Australian National University where he mentioned it would be nice if I could put a presentation together for the next conference on design analysis, usability testing, and any other methods for understanding how to find and document problems in Sakai's UX using user-centered methods.

I loved the idea but just not sure how my schedule will look in the coming months. So in the meanwhile, I thought I'd take a few minutes and whip together a quick blog post to summarize some of the different methods I've used.

Some of these methods can be used for testing existing products, but I like to think more in terms of testing new designs. Either way, I hope you find them useful.

----------------

1. Surveys

The trick with surveys is knowing how and what to ask. Without a direct dialog to affirm and discuss the real issues, it's easy to miss the heart of the problem.

But that's not to say they aren't useful. Just try to ask very specific questions like: Do you have dial-up or broadband where you use your system?

Asking general questions like "Do you think the portal is easy to use?" can lead to fairly mixed results.

2. Focus Groups

These are good for general discussions about a product, but they do carry the common pitfalls of group-thought, personality conflicts, shyness, etc.

It also depends on the methodology used. If a group of random people were asked to perform certain tasks in an application, then later came together to discuss their opinions, that could turn out to be a rewarding discussion.

But if the group is lead by a facilitator who walks them through a design and then asks for their opinions, the feedback is fairly tainted since the users weren't given a chance to figure out the UI on their own.

3. One-On-One Interview

When I do these I like to have a few goals in the back of my mind to help keep the discussion moving along – but not much more in terms of preparation. Bringing in a questionnaire usually makes the process too structured to really hear what the user is saying. Instead, I suggest trying to be as apathetic and philosophical with the interviewee as possible. If you can go out and grab a beer with him/her, even better. You really want to try to get them to let their hair down and share issues that go beyond the specifics of the tool, but more into the nature of why they use the tool.

But despite your best efforts, some users just don't know what the problem is. Sometimes they don't realize there are be better ways of doing something than how they're currently doing it. So you might have a lot of users tell you they like something, when in fact the UI or functionality is problematic.

4. Contextual Inquiries (observing people in their own environment)

This is a really nice way to get a glimpse into what a user is doing without relying on them to remember or explain some issue they've had with the software in the past.

If you're only at the design stage with your product, this might not help give you any insight into how your product is being used, but it's still nice to watch a user use other products in their environemtns – particualrly if the user fits your target "persona".

You'd be surprised how shadowing and quietly observing a user can clue you into all sorts of things that spur innovative improvements to software.

5 Heuristic analysis

There's a ton of examples on this all over the web, so I won't re-purpose them here. But I am working on the UX Scorecard which will have a good format for performing these types of reviews.

Basically, the idea is to get a panel of experts together (by experts, I mean designers who have an eye and vocabulary for design heuristics) and have them review the product.

This tends to be a good way to get some very general insight into the some of the common problem areas, but it rarely leads to any deep interaction analysis beyond just screen-level issues. I still think this is a great way to get your feet wet with a fairly nice method for reviewing and documenting a products UI.

6. Usability Testing

Last but not least, good 'ol usability testing. IMO, there is no better way to understand if a design is going to fly or not. There's a lot of different standards in testing – some are even ISO approved. But rather than going the fancy route, I tend to use quick and dirty tests just to get feedback early on while still in a design phase of a project.

When preparing a test, the thing you put in front of users can range from hand paper sketches to a semi-functional html prototype. Whatever is most economic for your organization. I prefer hand sketches myself since you can easily change them in mid-test and they don't take much to prepare.

To get going, I suggest you come up with a set of screens that revolve around a few tasks that you want to test. Don't try to test too much in one sitting as your prototype will get large and unwieldy. Just a few simple screens will do. Once you have your tasks outlined in your mind and have prepared the screens you'll be putting in front of the user, you're pretty much ready to go. Just schedule a few users and have them come in on a day that works well for everyone.

Oh, despite the central limit theorem (having at least 30 observations), Jakob Nielsen seems to think that 5 users is enough:

http://www.useit.com/alertbox/20000319.html

From my perspective, the statistical accuracy is not as important as simply seeing how users react to a design. From that perspective, 1 user is better than none.

Anyway, if you can get your screens into an electronic format (i.e. sketch them out in Visio, Photoshop, etc.. or scan in hand sketches), you can use a video recording tool like Camtasia to do all the documenting for you. That can be a real joy once you have everything setup properly.

A few things you may want to do to make the testing workout a little better for you:

a) Try to let the user know it's not them but rather the software that's being tested. You may need to reiterate this point a few times... and even still, you'll probably have some user's say things like "Oh, sorry! That was dumb of me!". Just let them know that they found a design flaw and that's exactly what you're looking for.

You may also want to stay away from calling it "user testing" as you're not testing them, but the software – so "usability testing" is possibly a less intimidating way to refer to it.

b) Invoke the verbal protocol (but don't also test for how long it takes for them to complete a task). Basically, keep encouraging the user to think out loud. They'll stop periodically, as it's not natural for them to talk about what they're thinking at every step, but give them a light prod and they should perk back up

c) Try to avoid hinting in anyway! Keep the questions focused on the task or goal. If the user asks how to do something, feel free to play the amateur psychologist and ask "how do you think you should do it?" If they're completely stumped, you've found a design flaw.

Sometimes I setup a system where if the user asks more than once, I ask them if they want to call tech support? If they say "yes", that's when I answer their question. You might have an interim step where you guide them to online help documentation – which makes for a nice way to test two birds with one stone: the product and the help docs.

With usability testing, you could try to focus on the following areas:

• Effectiveness – have they completed the task? This is categoric (yes or no)
• Efficiency – I'm tempted to suggest you avoid this one entirely. But basically, you could look for how long it took them to complete a task, or whether they chose the "preferred" path, etc. This one's a bit contentious though and should certainly be used with caution – especially if the verbal protocol is also used.
• User Satisfaction – at the end of a task and/or the entire test, it's a good time to get some free-form feedback about likes and dislikes.

Anyway, I hope this helps for now. As for how to capture and analyze all this data, that will have to be something I cover in a future post.

Regards!