News

Stay in the loop with company updates and insights from our talented team.

Better Data in Five Steps: UX Testing

eye tracking / research design / usability / user experience / blog

Last week, Jen wrote about resolutions for being a UX Champion. This week, I want to highlight five simple ways that UX research teams can improve the UX testing environment. Improving the testing environment in the following ways will ensure the validity of data and provide participants with a more pleasant experience.

  1. Don’t interrupt. The UX data collection process often relies on concurrent think aloud protocol. In this method, researchers ask participants to think aloud and narrate everything they are doing (e.g., reading, searching for links, interpreting graphics). This method is valued by UX researchers because it allows us to hear things like initial impressions of design features and experiences using site search and navigation. Users’ comments are very powerful when inserted into final reports, but they can obstruct the collection of other valuable data. During silent moments in think aloud protocols, moderators may prompt users to continue talking or to explain why they are having difficulty with a task. Interrupting users may interfere with complex cognitive processing (1) and thinking aloud may invalidate other metrics like task completion time and eye tracking (2). Interruptions can also decrease the user experience and could result in poor perceptions of the tested product (3). In 2014, UX researchers should not interrupt participants while they work on tasks.

  2. Let participants be themselves. Novice users of a product are frequently recruited for UX testing because their experience with the product is unbiased by previous exposure. Researchers then design tasks for these novices that evaluate a wide variety of product functions. To get participants in the right mindset for the tasks, instructions frequently start with phrases like, “Imagine you are...” or “Let’s say that you are interested in...” Phrasing questions in this way provides motivation for participants to complete the task. It is often necessary to phrase questions in this way, but one drawback of asking participants to “imagine” is that it introduces multitasking that can increase mental workload and negatively impact information processing (4). A better method is to recruit participants from specific, targeted demographics. These participants will be able to complete tasks as they normally would. In 2014, UX Researchers should recruit participants that match typical user demographics.

  3. Create specific tasks.Error rates and successful task completion are common UX metrics. These metrics can become difficult to calculate when the wording of a task is vague or multiple correct options are available. On sites with redundant pages and sources of data, it becomes difficult to assess successful task completion when a person finds only part of the answer and then says they are finished. When tasks are worded vaguely, participants begin to say things like, “Is this the right place?” or “Is this the information I am looking for?” In 2014, UX Researchers should create questions that are direct and have a clearly defined end location so participants can confidently say, “I have found the right answer".

  4. Measure physiological responses.Measuring implicit emotional responses via eye tracking and electrodermal activity (EDA) allows researchers to analyze data that would otherwise be impossible. For example, if a researcher asked a participant to explain their emotional responses to an advertisement, the participant may 1) not remember how they feel about the ad or 2) not feel comfortable expressing their emotions publicly. Similarly, participants will have difficulty recalling where they looked on a screen first, or how many times they fixated the top navigation during a particular task. EDA allows researchers to evaluate small changes in emotional states, and eye tracking allows researchers to collect data about attention. Collecting physiological data helps researchers understand the user experience in an unbiased fashion. For 2014, UX Researchers should consider implicit measures to enhance typical usability methods.

  5. If you see something, say something. While DC Metro riders may be tired of hearing the PSA, “If you see something, say something,” UX researchers would benefit from practicing this mantra as well. During a usability testing session, if we notice a small inconsistency, broken link, or missing element, we write it down and bring it to the attention of the team. It has been my experience that when our team of researchers contribute their ideas and observations to a working document or whiteboard, a more complete story can be told about the user’s experience with the product. In 2014, UX Researchers should create a complete picture of the user experience through excellent note taking.

By working on these five steps, your research team can improve testing methods and ensure that you are collecting viable information from participants. These methods have been successful for the FMG UX Team, and we will continue to use and improve them. For us, it is important to produce UX findings and recommendations that are grounded in research, thereby increasing the user experience of future surveys, websites, and mobile applications.


  1. Cooke, L. (2010). Assessing concurrent think-aloud protocol as a usability test method: A technical communication approach. Professional Communication, IEEE Transactions on, 53(3), 202-215.
  2. Olmsted-Hawala, E. L. & Romano Bergstrom, J. C. (2012). Think-aloud protocols. Does age make a difference? Proceedings from the Society for Technical Communication Summit, May 2012, Chicago, IL.
  3. Bailey, B. P., Konstan, J. A., & Carlis, J. V. (2001, July). The effects of interruptions on task performance, annoyance, and anxiety in the user interface. In Proceedings of INTERACT (Vol. 1, pp. 593-601).
  4. Ryu, K., & Myung, R. (2005). Evaluation of mental workload with a combined measure based on physiological indices during a dual task of tracking and mental arithmetic. International Journal of Industrial Ergonomics, 35(11), 991-1009.
comments powered by Disqus