Order Effects in Eliciting Preferences
Summary
Economic decisions are governed by 3 trade-offs: risk vs. return; today vs. tomorrow & self vs. others. Good accounts of such preferences help governments & organizations better address their citizens’/customers’ needs.
Traditionally, such measures were obtained in one of two ways: either by inferring preferences by asking people to make choices involving monetary outcomes (Quantitative) or by asking them to directly self-report their preference through some scale (Qualitative).
Ideally, conclusions about underlying preferences would not depend on the method (Quantitative or Qualitative) they were elicited with. Alarmingly, there is now considerable evidence that this is not so.
Therefore, it is becoming increasingly popular to include both elicitation methods in surveys in order to harness the best of the two worlds by combining them into a hybrid score.
In this paper we ask: ‘Are there order effects between Qualitative and Quantitative elicitation methods and how do they affect conclusions about preferences?’ We find order effects affecting the level of the measures (Magnitude) & their cross-method correlation (Consistency).
We run 3, pre-registered studies with 3,000 participants where we elicit pref/ces for risk, time & altruism in 2 conditions: ‘Quantitative First’ (people saw the quantitative items before the qualitative ones for each pref/ce dimension) & ‘Qualitative First’ (reverse order). In Study 1 we use hypothetical monetary incentives for Quantitative items while Qualitative items are expressed in general context.
We find significant Magnitude order effects: seeing the qualitative item first increases patience and altruism in both qualitative and quantitative measures relative to the alternative of seeing the quantitative item first (scales are always standardized between 0 & 1).
We also see that eliciting quantitative measures first increases significantly the cross-method correlation for risk and time preferences (Pearson’s-r correlation coefficient reported).
But why?! We suspect that Magnitude effects are affected by the ‘talk is cheap’ criticism & Consistency effects by context misalignment. We test this by introducing incentive compatibility for quantitative items (Study 2) & frame qualitative ones in financial terms (Study 3).
Indeed, using an incentive compatible payment scheme inoculates quantitative measures from magnitude order effects for risk and time-discounting preferences:
Moreover, introducing financial context in qualitative items (e.g. ‘how do you evaluate your attitude towards risk regarding financial investments?’), renders the Consistency effect no longer statistically significant for time & mitigates that for risk.
We interpret the Magnitude effects – & their mitigation when quantitative items are incentivized – through self-image concerns. People like to see themselves as risk tolerant, patient & altruistic but are less prone to ‘live up’ to this image when actions are costly.
Interestingly, the quantitative measure for altruism is an exception to this. People increase their actual donations by 24% when asked to report how altruistic they consider themselves before asked to donate. This result is redolent of the findings related to moral nudges.
The mitigation of the Consistency order effects in Study 3 corroborates our intuition that (part of) the reason behind the apparent incongrunce between preference elicitations is due to context misalignment between quantitative & qualitative items. The commonly used abstract framing of qualitative items evokes different contexts to different people (e.g. health, sports, financial decision making). Quantitative items-on the other hand- involve monetary trade-offs, imposing a context of financial decision making. To the extent that preferences are context dependent and someone’s willingness to take risks in health-related decisions differs from that in financial ones, eliciting qualitative measures first will lead to lower correlation scores.
Although context-adjusted items for risk are not uncommon (e.g. German Socio-Economic panel), it’s harder to find equivalently adjusted qualitative items for time and altruism. Our results suggest that these are important & our paper includes such adjusted formulations.
In short, the order in which qualitative and quantitative questions are answered affect systematically elicited preferences. We explore the implications of this result for our understanding of the nature of preferences and discuss how these order effects can be mitigated or leveraged in nudging interventions designed to boost charitable contributions. You can read the latest version of this working paper here.
