Stay in the loop with company updates and insights from our talented team.

Threats to User Experience Research Validity

featured / ux / usability

Many parallels can be drawn between experimental psychology research and summative user experience research. In experimental research, we take a systematic and scientific approach to understanding the effect of the manipulation of one or more variables, while holding other variables constant. User experience (UX) research is usually more qualitative in nature, but at FMG, we feel that UX deserves the same systematic and scientifically rigorous treatment as experimental research. In most laboratory experiments, researchers only have a small number of variables that change and these studies are tightly controlled. Independent variables are those factors manipulated by the experimenter or demographic in nature, while dependent variables are where the effects of the independent variable are measured. In UX testing, we have many independent variables (for example, the location of the navigation, the words used for labels, the images and graphics). Each of these design elements could serve as potential independent variables in a UX test. However, because there are often many independent variables changed between designs, it is not often clear which specific design feature(s) caused the observed differences in the dependent variable (e.g., time to complete task, satisfaction with the product, success in completing the task).

Although we have additional challenges in determining causality, a strong summative UX research study should still be grounded in the same scientific research principles as experimental research. Just as with experimental research, UX research is subject to threats of internal and external validity. These threats can jeopardize our UX test conclusions and are therefore important to consider when designing and conducting UX research studies.

Internal Validity

Internal validity in user experience research reflects the extent to which a conclusion based on a study is warranted. When we conduct UX testing, we want our conclusions about the design to reflect users’ true experience. However, if there is a threat to internal validity present, we have a difficult time confidently stating that the design caused the change in experience. Let’s first take a closer look at each of the internal threats to validity that apply to UX and how best to address and control for them.

Threats to Internal Validity

External Validity

External validity in UX testing reflects the extent to which a conclusion can be generalized to other situations and other user groups. For example, when we conduct an in-lab UX test, we want to generate conclusions that will generalize to the situations in which actual users will use the product in the real world. Two types of validity can impact external validity: population validity and ecological validity. Population validity pertains to the representativeness of our participants to the actual users. The better the participants represent the actual users, the more confident we can be that our sample (participants) generalizes to the population (users). Ecological validity pertains to the extent to which the setting is representative of the actual setting that users will use the product. A threat to external validity makes it difficult to state that our conclusions will hold true when users use the product in their natural environment. Let’s take a look at some threats to external validity and how best to address and control for them.

Threats to External Validity

At FMG, we pride ourselves on taking a systematic and scientific approach to testing methods and have taken steps to ensure internal and external validity in all studies. If you’ve encountered threats to validity while conducting your own UX studies, we hope some of the solutions above help you navigate the challenges and improve your studies.

comments powered by Disqus