Stay in the loop with company updates and insights from our talented team.

Your Web-Based Survey Should be “Medium Rare”

behavioral research / user experience / blog

On Wednesday evening, I had dinner at Medium Rare , a fun, friendly, neighborhood restaurant in the Cleveland Park neighborhood of Washington DC, where “customers can just relax, and have a simple yet exceptional meal served to them.” This was my third time there, and each time I go, I am amazed by how busy it is. Honestly, it’s good but what’s so fantastic is the experience – there is a pre-fixed menu of artisan rustic bread, mixed green salad, and culotte steak and fries – the only thing you have to choose is the temperature of your steak. Even their wine and beer lists and the dessert menu are short. Everything is prepared well and consistent – something many people, including me, value in restaurants. The atmosphere is warm and friendly, and the staff is attentive. I have seen many restaurants come and go from Cleveland Park, and Medium Rare stays...and is always packed, just like on this random Wednesday night. It got me thinking about why this place is so popular. Is it the consistent experience you get every time you go? Do the apple pie and layered chocolate cake and the Dubble Bubble gum they give you with your check bring back childhood memories? Is it the feeling of knowing you don’t have to choose which steak you are going to have?

Medium Rare DC

I turned to Yelp to gather some qualitative data. I read many at Medium Rare dinner reviews , and satisfaction seemed to come down to one experience: the fact that people don’t have to make a choice. Diners had posted things like, “For dinner, the only choice you had to (or could) make was how you wanted your steak cooked -- that's it!” and “It’s a pre-set menu so you don't have to make any decisions except how you want your steak done.” And “They have a set menu, and it's kind of nice to not have to make a decision once in a while.” So that was it! People enjoy not making decisions! Brilliant! I found this very interesting, especially considering how Americans LOVE choices. Take, for example, the “more than 200 item” menu at Cheesecake Factory, another restaurant that is always packed (and gives me great anxiety).

It got me thinking about the workshop I had just taught earlier in the day at the Census Bureau with Amy Anderson Riemer about the usability of web-based surveys. During the class, Amy had discussed “mode paralysis” and how survey respondents do not make a decision when given one. In trying to increase survey response rates, researchers often give people the choice of how to respond, usually by mail or web. The notion is that respondents have a preferred method of responding; thus, if we provide a choice between different modes, then people are more likely to choose the preferred method, and response rates will be high. Yet, contrary to what we might expect, researchers (1) have found that giving respondents the option to respond to surveys via mail or web actually decreases response rates. Millar, O’Neill and Dillman (2009) (2) actually suggest that withholding the paper option altogether could drive people to the web. This could have the profound effect of increasing response rates and decreasing costs. (Note that providing different response modes after the first contact appears to lead to increased responses from those who do not respond the first time around) (3).

One very interesting finding from Millar et al. (2009) was that respondents actually expressed that they preferred the survey mode they were “pushed” to (i.e., that they must use). So when we don’t give people an option, they are forced to respond in the way we ask them to, and they are more likely to do it than if we give options, and they report actually liking it!

This was exactly what I was seeing at Medium Rare. People do not have a choice in what they are going to eat, and then they rave about the experience and keep coming back for more. Who would have thought that this simplicity would lead to such success? (Well the owners of Medium Rare, that’s who. I wonder if they have read the survey methodology literature and at The Paradox of Choice…). So just like Medium Rare, we should consider the effects of not providing choices. Eliminating choices may greatly reduce the anxiety and busyness in our lives, freeing up time to do other things we enjoy, like socializing with family and friends at dinner. Next time you are designing a survey, consider not providing a choice, and instead, make your survey “Medium Rare.”

  1. Israel, G. D. (2009). Obtaining responses by mail or web: Response rates and data consequences. Proceedings from the American Association of Public Opinion Research (AAPOR) conference.
  3. Millar, M. & Dillman, D. A. (2011). Improving response to Web and mixed-mode surveys. Public Opinion Quarterly, 75, 249-269.
comments powered by Disqus