Talking with Participants During a Usability Test

3 safe and productive approaches for interrupting or answering users during usability tests and other research studies:

  • Echo – With the echo technique, the facilitator repeats the last phrase or word the user said, while using a slight interrogatory tone.
  • Boomerang – With the boomerang technique, the facilitator formulates a generic, nonthreatening question that she can use to push the user’s question or comment back to him.
  • Colombo – With the Columbo technique, be smart but don’t act that way. (…) One way to do this is to ask just part of a question, and trail off, rather than asking a thorough question.

Kara Pernice: Talking with Participants During a Usability Test

Behavioural economics

Behavioural economics teaches us that human behaviour is often automatic and unthinking, and is shaped far more heavily by momentary contextual factors than we realise. These contexts can be both internal and external. They include our automatic habitual responses, the heuristics or rules of thumb that we follow every day, and the constraints of the physical environment.


In this way, the ‘unconscious’ factors that often trigger behaviour are not a pool of deep desires and feelings such as that conceived of by Freud, but an active, adaptive management system, focused on navigating life and responding to the physical environment as efficiently and effectively as possible. One frequently observed outcome is that people do not usually seek out ideal solutions; instead they ‘satisfice’, adopting choices that are ‘good enough’ but avoid them expending angst and energy seeking something better.


‘Fridge fit’ is a well-known example of a satisficing heuristic that leads shoppers to make unexpected choices in categories like salad dressing. They choose their dressing not based on how well it fits with their beliefs about food and nutrition, but on how well
it fits into their fridge


This approach aligns with the human brain’s inherent striving for meaning, interpreting, categorising and generally making sense of our actions for future reference.

TNS: The value of context

Big Data, qualitative research and the need for behavioural insight

To begin with, Big Data delivers thin data. (…) Big Data focuses solely on correlation, paying no attention to causality. What good is thin “information” when there is no insight into what your consumers actually think and feel?
Big Data Gets the Algorithms Right but the People Wrong

“Qualitative research is uniquely positioned to uncover the true drivers of consumer behaviour, but can only do so if it starts to look beyond our articulated wants and needs.”
Research excellence – The value of context, or what qualitative research can learn from behavioural economics (PDF)

ROI on User Experience

That said, ROI is always calculated in terms of increase or decrease of a key variable.

These increases and decreases usually fall into one of six categories:

  • Increased sales – either incremental revenue from add-ons or through an easier sales process, which is typically called “conversion rate” though that often gets mixed up with “website conversions”. The metrics for this are typically found in the sales and finance department. Coupled with the estimated costs of usability testing and UX development, you’ll be able to work out over what timeframe it will take for the UX work to pay for itself.
    Increased productivity – Most often found when there are large pools of employees completing certain repetitive tasks. Metrics are found in operations, HR, and finance (for overhead items like office space, equipment, etc.) For example, if you optimize the UX on a series of screens so that what was once a 5 minute task is now a 2.5 minute task, then you’ve increased a person’s productivity by 100%. That’s huge. HUGE. If the company has 100 phone agents who have an average salary of $40,000 + benefits (~$8,000) (+ an unknown amount for overhead), you could either release or retask those agents on other activities with a savings of $2,4000,000/year. (half of 100 agents x $48,000)
  • Increased customer satisfaction and loyalty – This is extremely difficult to measure, and should be used as a primary cost justification only in extreme cases. That said, one method of measuring customer loyalty is in terms of customer retention. (Important especially in organizations that know when their customers are most likely to defect to a competitor or a more mature product offering.) The calculation takes into account the cost of new customer acquisition and amortizes the cost over the average customer retention period. Increasing average customer retention is a cost savings for the company, and can cost justify UX & usability work related to customer loyalty. The metrics of success will come from sales and marketing.
  • Decreased training and support costs – If customers generate calls or support tickets or other uncompensated overhead costs to the business, then those costs should be measured. The easiest way to do this is how airlines handle the it. When a passenger, for example, calls an agent to check flight status, the airline knows how long that call will last, and based on the average cost/minute for phone agents (including salary, benefits, and overhead like the phone line, office space, electricity, etc), the airline can assign a $ value to the cost of the call. Because airlines handle large volumes of calls every day, if there is an increase of 2500 check flight status calls/day at a cost of $2.00/call (theoretical cost!), then we could expect that would increase uncompensated support costs by $5,000/day. If the airline wanted to do a $50,000 eye tracking usability study to get that number down to 1225 calls/day (assuming a clean 50% improvement), then that usability study would pay for itself in 20 days.
    Be wary in cases where training and support are part of the company’s core business model – otherwise it doesn’t make sense to decrease these costs, (i.e., if the company’s revenues are earned from consulting)
  • Decreased development time and costs
    Measured in terms of development resources and time to market. For example, if moving to a system with design patterns and object oriented CSS will decrease the need for an additional UX hire and 5 developer hires (a reasonable ratio), then that’s a cost savings of all of those non-hires salaries + benefits + overhead (computers, software, office space, desk, etc). Similarly, if a strong UX framework decreases the development time from 6-months to 3-months for a project that will increase revenues by $100k per month, then bringing the project to market 3 months early adds $300k to this year’s revenues.
  • Decreased maintenance costs – Again, this is a function of the number of developers needed to complete project work. Bad UX is often a reflection of poor design and complexity in the underlying code. A hallmark of good UX is the elimination of unnecessary or marginal features. This reduces the complexity of the code base, making it more robust and less buggy. As a wise man once said, “there are no bugs in the code you don’t write.”


Baseline information for Discovery phase

Prior to any interviews, gather information from the following documents:

  • Mission
  • Vision
  • Values statement
  • Value proposition
  • Organisation chart
  • Strategic planning documents
  • Business plans
  • Marketing plans
  • Annual reports
  • Existing Marketing research
  • Cultural assessments
  • Employee surveys
  • CEO speeches
  • Press releases
  • News clips
  • History
  • Domains
  • Intranet access

Alina Wheeler: Designing Brand identity, Hoboken (2013)