How many participants?

‘Over the years, there has been plenty of debate over how many participants are enough for a study. It turns out we were looking in the wrong direction. When you focus on the hours of exposure, the number of participants disappears as an important discussion. We found 2 hours of direct exposure with one participant could be as valuable (if not more valuable) than eight participants at 15-minutes each. The two hours with that one participant, seeing the detailed subtleties and nuances of their interactions with the design, can drive a tremendous amount of actionable value to the team, when done well.’

JARED M. SPOOL: Fast Path to a Great UX – Increased Exposure Hours

Biased answers in usability testing

Self-reported data is typically three steps removed from the truth:

• In answering questions (particularly in a focus group), people bend the truth to be closer to what they think you want to hear or what’s socially acceptable.
• In telling you what they do, people are really telling you what they remember doing. Human memory is very fallible, especially regarding the small details that are crucial for interface design. Users cannot remember some details at all, such as interface elements that they didn’t see.
• In reporting what they do remember, people rationalize their behavior. Countless times I have heard statements like “I would have seen the button if it had been bigger.” Maybe. All we know is that the user didn’t see the button.

Jakob Nielsen (2001): First Rule of Usability? Don’t Listen to Users

10 Killer Ways to Boost Your Site’s Usability

1. Make the most important links large and easy to read.
2. Create multiple access points to important content like subscription options.
3. Eliminate unnecessary obstacles like captcha systems or mandatory logins.
4. Move your advertisements so that they don’t get in the way of the content.
5. Place a link to the home page on every page in case a user gets lost.
6. Provide a clear and complete about page and contact page.
7. Don’t clutter your sidebar with unnecessary links or widgets.
8. Provide targeted content and links in the post footers.
9. Use a large, comfortable font and provide space between each paragraph.
10. Conduct simple, old-fashioned usability testing.

by Nicholas Z. Cardot

Usability testing: essentials

1. Usability testing is one of the most important aspects of professional website production.
2. These task-based tests should ideally be conducted at regular intervals throughout the development process.
3. Heuristic evaluations by usability experts provide valuable feedback regarding the generic usability of the site.
4. You need to test actual members of the target audience – not just your colleagues in the office.
5. Information architects are not the best people to carry out usability testing on their own sites. This should be left to impartial experts.
6. Simple paper-based tests can show the information architect if labels are accurate and have the right scent.
7. Structural prototypes can be used to identify navigational problems such as excessive pogo-sticking.
8. During beta-tests of a nearly completed site, users can be asked to make comparisons and judgments in addition to factfinding.
9. Usability tests provide hard facts. They don’t depend on your own “best guess.”

From: Reiss, E.L. (2000), Practical Information Architecture, Harlow: Pearson Education, p.170

Doing user testing

1. Determine the goals and explore the questions
2. Choose the paradigm and techniques
3. Identify the practical issues: Design typical tasksIdentify the practical issues: Select typical users
4. Identify the practical issues: Prepare the testing conditions
5. Identify the practical issues: Plan how top run the tests
6. Deal with ethical issues
7. Evaluate, analyze, and present the data

From: Preece, J., Rogers, Y., Sharp, H. (2002), Interaction Design: Beyond Human-Computer Interaction, New York: Wiley

Interviewing: Asking users and experts

1. There are three styles of interviews: structured, semi-structured and unstructured.
2. Interview questions can be open or closed. Closed questions require the interviewee to select from a limited range of options. Open questions accept a free-range response.
3. Many interviews are semi-structured. The evaluator has a predetermined agenda but will probe and follow interesting, relevant directions suggested by the interviewee. A few structured questions may also be included, for example to collect demographic information.
4. Structured and semi-structured interviews are designed to be replicated. Focus groups are a form of group interview.
5. Questionnaires are a comparatively low-cost, quick way of reaching large numbers of people.
6. Various rating scales exist including selection boxes, Likert, and semantic scales.
7. Inspections can be used for evaluating requirements, mockups, functional prototypes, or systems.
8. Five experts typically find around 75% of the usability problems.
9. Compared to user testing, heuristic evaluation is less expensive and more flexible.
10. User testing and heuristic evaluation often reveal different usability problems.
11. Other types of inspections include pluralistic and cognitive walkthroughs.
12. Walkthroughs are very focused and so are suitable for evaluating small parts of systems.

From: Preece, J., Rogers, Y., Sharp, H. (2002), Interaction Design: Beyond Human-Computer Interaction, New York: Wiley, p. 424

Observation tecniques (PDF)

Grid about advantages and constraints of different observation techniques

Observation techniques (PDF)

From: Preece, J., Rogers, Y., Sharp, H. (2002), Interaction Design: Beyond Human-Computer Interaction, New York: Wiley

Observing users

1. Observation in usability testing tends to be objective, from the
   outside. The observer watches and analyzes what happens.
2. In contrast, in participant observation the evaluator works with
   users to understand their activities, beliefs and feelings within
   the context in which the technology is used.
3. Ethnography uses a set of techniques that include participant
   observation and interviews. Ethnographers immerse themselves in
   the culture that they study.
4. The way that observational data is collected and analyzed depends
   on the paradigm in which it is used: quick and dirty, user
   testing, or field studies.
5. Combinations of video, audio and paper records, data logging, and
   diaries can be used to collect observation data
6. In participant observation, collections of comments, incidents,
   and artifacts are made during the observation period. Evaluators
   are advised to discuss and summarize their findings as soon after
   the observation session as possible.
7. Analyzing video and data logs can be difficult because of the
   sheer volume of data. It is important to have clearly specified
   questions to guide the process and also access to appropriate
   tools.
8. Evaluators often flag events in real time and return to examine
   them in more detail Identifying key. events is an effective
   approach. Fine-grained analyses can be very time-consuming.

From: Preece, J., Rogers, Y., Sharp, H. (2002), Interaction Design: Beyond Human-Computer Interaction, New York: Wiley, p.386

User testing – 5 reasons!

Five reasons to invest in user testing (according to B Tognazzini):
1. Problems are fixed before the product is shipped, not after.
2. The team can concentrate on real problems, not imaginary ones.
3. Engineers code instead of debating.
4. Time to market is sharply reduced.
5. Finally, upon first release, your sales department has a rock-solid design without having to pepper their pitches with how it will all actually work in release 1.1 or 2.0

From: Preece, J., Rogers, Y., Sharp, H. (2002), Interaction Design: Beyond Human-Computer Interaction, New York: Wiley

Sample tasks for usbility testing

1. Go to www.usps.com and find out how much it costs to send a
   postcard to China.
2. Go to www.sandiego.gov and find the name of the city council
   member for one area
3. You are planning a family reunion at Sugarloaf Ridge,
   California. Go to www.parks.ca.gov and make a reservation for a
   campsite that can hold 35 people.
4. You are looking for a snack that you can eat during your
   workout. Go to www.nestle.com and see what products they have for
   you.
5. Go to www.pixar.com and see if you can find out how they came
   up with the idea for the movie Monsters, Inc.
6. You want to visit the J. Paul Getty Museum this week. Go to
   www.getty.edu and find out the cost to enter and the time it
   opens.
7. You want to put $1,000 in the bank and keep it there for a long
   time. Go to www.bankone.com and find the Lccounts with the best
   interest rates.
8. You read an article about how fuel cell technology may world.
   Go to www.cun1mins and find the top two pros and cons of fuel
   cell technology.

From: Nielsen, J. and Loranger, H. (2006), Prioritizing Web Usability, Berkeley, CA.: New Riders