The More You Ask, the Less You Get: When Additional Questions Hurt External Validity

Preference elicitation tasks are widely used to predict behavior in marketing, finance, and public policy. We suggest that every time a respondent answers an additional elicitation question, two things happen: (1) We obtain information about some parameter(s) of interest, such as their discount rate or the partworth for a product attribute, and (2) the respondent increasingly adapts to the task—i.e., using a task-specific decision process specialized for this task but that may not apply to other tasks. Importantly, adaptation comes at the cost of potential mismatch between the task-specific decision process and real-world processes that generate the target behaviors, so that asking more questions can reduce external validity. Using mouse-tracking to trace decision processes, we show evidence that respondents increasingly rely on task-specific decision processes as more questions are asked, which reduces external validity of the task for predicting other related tasks and behavior. Two large studies found that external validity for predicting real-world intertemporal choices and credit scores—peaked after only a few elicitation questions. When measuring preferences, less can be more.