286 User Experience Re-Mastered: Your Guide to Getting the Right Design THE PILOT TEST Before any actual evaluation sessions are conducted, you should run a pilot test as a way of evaluating your evaluation session and to help ensure that it will work It is a process of debugging or testing the evaluation material, the planned time schedule, the suitability of the task descriptions, and the running of the session Participants for Your Pilot Test You can choose a participant for your pilot test in the same way as for your actual evaluation However, in the pilot test, it is less important that the participant is completely representative of your target user group and it is more important that you feel confident about practicing with him or her Your aim in the pilot test is to make sure that all the details of the evaluation are in place Design and Assemble the Test Environment Try to your pilot test in the same place as your evaluation or in a place that is as similar as possible Assemble all the items you need: ■ ■ ■ ■ ■ Computer equipment and prototype, or your paper prototype Keep a note of the version you use Your evaluation script and other materials Any other props or artifacts you need, such as paper and pens for the participants The incentives, if you are offering any If you are using video or other recording equipment, then make sure that you practice assembling it all for the pilot test As you put it together, make a list of each item There is nothing more aggravating than forgetting some vital part of your equipment Run the Pilot Test Run the pilot participant through the evaluation procedure and all the supporting materials The session should be conducted in the same way as the actual evaluation session Ideally, the evaluator(s) who will conduct the actual evaluation session should participate in the pilot test They should observe, take notes, and facilitate the pilot test, just as they would in the actual session For example, they should consider the following questions: ■ ■ ■ ■ ■ Is the prototype functioning as required for the session? Is the introductory material clear enough to the evaluator(s) and the participants? Are the observation and data collection procedures working? Are the evaluator(s) aware of their roles and responsibilities for the evaluation session? Can the task descriptions be accomplished within the planned session time? Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark Final Preparations for the Evaluation CHAPTER While observing the pilot participant, make a note of where the evaluation materials and procedures may need to be improved before conducting the actual usability evaluation sessions It is often helpful to analyze and interpret the data that you get from the pilot test This often points out that an important facet of the evaluation has been overlooked and that some essential data, which you need to validate certain usability requirements, has not been collected If you are short of time, then you might consider skipping the pilot test If you omit the pilot test, then you will find that you forget to design some details of the tasks or examples, discover that some item of equipment is missing, realize that your interview plan omits a topic of great importance to the participants, or find that your prototype does not work as you had intended Doing a pilot test is much simpler than trying to get all these details correct for your first participant Often, the pilot test itself reveals many problems in the user interface (UI) You may want to start redesigning immediately, but it is probably best to restrain yourself to the bare minimum that will let the evaluation happen If the changes are extensive, then it is probably best to plan another pilot test SUMMARY In this chapter, we discussed the final preparations for evaluation: ■ ■ ■ ■ Assigning roles to team members (or adjusting the plan to allow extra time if you are a lone evaluator) Creating an evaluation script Deciding whether you need forms for consent and for nondisclosure Running a pilot test Once you have completed your pilot test, all that remains is to make any amendments to your materials, recruit the participants, and run the evaluation Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark 287 This page intentionally left blank Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark CHAPTER 10 Usability Tests 289 Michael Kuniavsky EDITOR’S COMMENTS Think-aloud usability testing, where participants verbalize their reactions to a product as they work on a series of tasks, is a popular technique in the repertoire of usability practitioners because it is regarded as relatively easy to learn, straightforward to use, capable of generating useful data, convincing, and (relatively) inexpensive You can use think-aloud usability testing to follow with: ■ ■ ■ ■ ■ ■ ■ ■ Obtain first impressions of a product Uncover features or components of the product that cause confusion Reveal initial learning problems Reveal clues about the user’s mental model of a system Reveal general likes and dislikes Determine if the language is understood Explore navigation and workflow efficiency Uncover how users recover from errors This method is applicable from requirements analysis through product release You can use the think-aloud testing method to get feedback on concept sketches, storyboards, wireframes, paper prototypes, existing products, working prototypes, and competitive products The optimal time to use this method in new product development is generally in the exploratory stages of design when you are focused on high-level issues like overall navigation, major feature design, and high-level organization This chapter provides a detailed guide for planning and conducting a usability test The author of this chapter, Michael Kuniavsky, is a very wise practitioner who provides a wealth of tips, tricks, and templates for a successful usability test Copyright © 2010 Elsevier, Inc All rights Reserved Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark 290 User Experience Re-Mastered: Your Guide to Getting the Right Design USABILITY TESTS A one-on-one usability test can quickly reveal an immense amount of information about how people use a prototype, whether functional, mock-up, or just paper Usability testing is probably the fastest and easiest way to tease out showstopping usability problems before a product launches Usability tests are structured interviews focused on specific features in an interface prototype The heart of the interview is a series of tasks that are performed by the interface’s evaluator (typically, a person who matches the product’s ideal audience) Tapes and notes from the interview are later analyzed for the evaluator’s successes, misunderstandings, mistakes, and opinions After a number of these tests have been performed, the observations are compared, and the most common issues are collected into a list of functionality, navigation, and presentation problems Using usability tests, the development team can immediately see whether people understand their designs as they are supposed to understand them Unfortunately, the technique has acquired the aura of a final check before the project is complete, and usability tests are often scheduled at the end of the development cycle – after the feature set has been locked, the target markets have been determined, and the product is ready for shipping Although testing can certainly provide insight into the next revision of the product, the full power of the technique remains untapped They can be better used much earlier, providing feedback throughout the development cycle, both to check the usability of specific features and to investigate new ideas and evaluate hunches WHEN TO TEST Because usability testing is best at seeing how people perform specific tasks, it should be used to examine the functionality of individual features and the way they’re presented to the intended user It is better used to highlight potential misunderstanding or errors inherent in the way features are implemented rather than to evaluate the entire user experience During the early to middle parts of a development cycle, usability testing can play a key role in guiding the direction of functionality as features are defined and developed Once the functionality of a feature is locked in and its interaction with other features has been determined, however, it’s often too late to make any fundamental changes Testing at that point is more an investment in the next version than in the current one Moreover, usability testing is almost never a one-time event in a development cycle for a product and should not be seen as such Every round of testing can focus on a small set of features (usually no more than five), so a series of tests is used to evaluate a whole interface or fine-tune a specific set of features Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark Usability Tests CHAPTER 10 The first thing the development team needs to is decide on the target audience and the feature set to examine This means that a good time to start usability testing is when the development cycle is somewhat underway, but not so late that testing prevents the implementation of extensive changes if it points to their necessity Occasionally, usability testing reveals problems that require a lot of work to correct, so the team should be prepared to rethink and reimplement (and, ideally, retest) features if need be In the Web world, this generally takes a couple of weeks, which is why iterative usability testing is often done in two-week intervals WARNING Completely open-ended testing, or “fishing,” is rarely valuable When you go fishing during a round of user research – often prompted by someone saying, “Let’s test the whole thing” – the results are neither particularly clear nor insightful Know why you’re testing before you begin A solid usability testing program will include iterative usability testing of every major feature, with tests scheduled throughout the development process, reinforcing, and deepening knowledge about people’s behavior and ensuring that designs become more effective as they develop Example of an Iterative Testing Process: Webmonkey 2.0 Global Navigation Webmonkey is a cutting-edge Web development magazine that uses the technologies and techniques it covers During a redesign cycle, they decided that they wanted to create something entirely new for the main interface Because much of the 1.0 interface had been extensively tested and was being carried through to the new design, they wanted to concentrate their testing and development efforts on the new features The most ambitious and problematic of the new elements being considered was a DHTML global navigational panel that gave access to the whole site (see Figs 10.1 and 10.2) but didn’t permanently use screen real estate Instead, it would slide on and off the screen when the user needed it Webmonkey’s previous navigation scheme worked well, but analysis by the team determined that it was not used often enough to justify the amount of space it was taking up They didn’t want to add emphasis to it (it was, after all, secondary to the site’s content), so they decided to minimize its use of screen real estate, instead of attempting to increase its use Their initial design was a traditional vertical Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark 291 292 User Experience Re-Mastered: Your Guide to Getting the Right Design FIGURE 10.1 The Webmonkey 2.0 Navigation Panel design (open) FIGURE 10.2 The Webmonkey 2.0 Navigation Panel design (closed) Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark Usability Tests CHAPTER 10 navigation bar, identical to that found in the left margin of the 1.0 site, but in its own panel The panel was hidden most of the time but would reveal its contents when an arrow at the top of a striped bar on the left side was clicked The target audience of Web developers would hopefully notice the striped bar and arrow and click on it out of curiosity Webmonkey developed on an iterative development cycle, so Web developers and sophisticated users were invited to a series of tests, with each test phase being followed by a design phase to incorporate the findings of the test Although the purpose of the test was to examine the participants’ entire user experience, the developers paid special attention to the sliding panel In the first round of testing, none of the six evaluators opened the panel When asked whether they had seen the bar and the arrow, most said they had, but they took the striped bar to be a graphical element and the arrow to be decoration Two weeks later, the visual design had not changed much, but the designers changed the panel from being closed by default to being open when the page first loaded During testing, the evaluators naturally noticed the panel and understood what it was for, but they consistently had trouble closing it and seeing the content that it obscured Some tried dragging it like a window; others tried to click inside it Most had seen the arrow, but they didn’t know how it related to the panel and so they never tried clicking it Further questioning revealed that they didn’t realize that the panel was a piece of the window that slid open and closed Thus, there were two interrelated problems: people didn’t know how the panel functioned and they didn’t know that the arrow was a functional element A third design attempted to solve the problem by providing an example of the panel’s function as the first experience on the page: a short pause after the page loaded, the panel opened and closed by itself The designers hoped that showing the panel in action would make the panel’s function clearer It did, and in the next round of testing, the evaluators described both its content and its function correctly However, none were able to open the panel again The new design still did not solve the problem with the arrow, and most people tried to click and drag in the striped bar to get at the panel Having observed this behavior, and (after some debate) realizing that they could not technically implement a dragging mechanism for the panel, the designers made the entire colored bar clickable so that whenever someone clicked anywhere in it the panel slid out (or back, if it was already open) In the end, people still didn’t know what the arrow was for, but when they clicked in the striped panel to slide it open, it did, which was sufficient to make the feature usable, and none of the people observed using it had any trouble opening or closing the panel thereafter Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark 293 294 User Experience Re-Mastered: Your Guide to Getting the Right Design HOW TO DO IT Preparation A full-on usability test (say six to 10 users) can easily take three to four weeks from conception to presentation of the results (see Table 10.1) You should start preparing for a usability testing cycle at least three weeks before you expect to need the results SETTING A SCHEDULE Before the process can begin, you need to know whom to recruit and which features you want them to evaluate Both of these things should be decided several weeks before the testing begins Table 10.1 A Typical Usability Testing Schedule Timing Activity t  weeks Determine test audience; start recruiting immediately t  weeks Determine feature set to be tested t  week Write first version of script; construct test tasks; discuss with development team; check on recruiting t  days Write second version of guide; review tasks; discuss with development team; recruiting should be completed t  days Complete guide; schedule practice test; set up and check all equipment t  day Do a practice test in the morning; adjust guide and tasks as appropriate T Test (usually 1–2 days, depending on scheduling) t  day Discuss with observers; collect copies of all notes t  days Relax; take a day off and something else; you will often be pressured to get a report out immediately, but this period of reflection is important for considering how small problems might be indicative of larger themes t  days Watch all tapes; take notes t  week Combine notes; write analysis t  week Present to development team; discuss and note directions for further research Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark Usability Tests CHAPTER 10 RECRUITING Recruiting is the most crucial piece to start on early It needs to be timed right and to be precise, especially if it’s outsourced You need to find the right people and match their schedules to yours That takes time and effort The more time you can devote to the recruiting process, the better (although more than two weeks in advance is generally too early since people often don’t know their schedules that far in advance).You also need to choose your screening criteria carefully The initial impulse is to recruit people who fall into the product’s ideal target audience, but that’s almost always too broad You need to home in on the representatives of the target audience who are going to give you the most useful feedback Say you’re about to put up a site that sells upscale forks online Your ideal audience consists of people who want to buy forks In recruiting for a usability test, that’s a pretty broad range of people Narrowing your focus helps preserve clarity since different groups can exhibit different behaviors based on the same fundamental usability problems Age, experience, and motivation can create seemingly different user experiences that are caused by the same underlying problem Choosing the “most representative” group can reduce the amount of research you have to in the long run and focus your results The best people to invite are those who are going to need the service you are providing in the near future or who have used a competing service in the recent past These people will have the highest level of interest and knowledge in the subject matter, so they can concentrate on how well the interface works rather than on the minutia of the information People who have no interest in the content can still point out interaction flaws, but they are not nearly as good at pointing out problems with the information architecture or any kind of content-specific features since they have little motivation to concentrate and make it work Say your research of the fork market shows that there are two strong subgroups within that broad range: people who are replacing their old silverware and people who are buying wedding presents The first group, according to your research, is mostly men in their 40s, whereas the second group is split evenly between men and women, mostly in their mid-20s and 30s You decide that the people who are buying sets of forks to replace those they already own represent the heart of your user community They are likely to know about the subject matter and may have done some research already They’re motivated to use the service, which makes them more likely to use it as they would in a regular situation So you decide to recruit men in their 40s who want to buy replacement forks in the near future or who have recently bought some In addition, you want to filter out online newbies, and you want to get people Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark 295 ... User Experience Re-Mastered: Your Guide to Getting the Right Design CHOOSING FEATURES The second step is to determine which features to test These, in turn, determine the tasks you create and the. .. closing the panel thereafter Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark 293 294 User Experience Re-Mastered: Your Guide to Getting the Right Design HOW TO DO IT... situation they may have found themselves in that your product could address Then, when the times comes for a task, ask them to try to use the product as if they were trying to resolve the situation they