Zachary del Rosario (Olin College)
Abstract
Background. It is widely accepted that statistical thinking involves thinking about variability (e.g., Wild and Pfannkuch, 1999, ISR). However, not all disciplines recognize the importance of variability, which can lead to serious issues. Previous work on practicing engineers found that some neglect variability (del Rosario 2024, JSDSE)---they make choices as though variability has no consequence. This is in contrast with targeting variability---making choices that address the consequences of variability. It remains an open question whether targeting is common or rare among the general population.
Targeting is a decision-making behavior wherein a person formulates their decision in response to statistical variability. For instance, using the average of previous bids in an auction neglects the possibility of larger bids, while using the maximum of previous bids targets variability. To target effectively, a person needs both comprehension of the task (e.g., how an auction works) and of variability (e.g., that people bid different amounts). Therefore, presentation of data (e.g., as different visuals) can impact comprehension of variability, and therefore targeting.
We answer two questions:
RQ 1: How often do U.S. adults target the consequences of variability?
RQ 2: How does the presentation of data impact targeting?
Methods. This quantitative study developed, validated, and deployed a novel instrument to measure targeting. We report results from a n=306 sample representative of U.S. adults, randomized into one of three treatments (sinaplot, bar chart, or text only).
To measure targeting, we elicit a response to a task (numeric answer), and the perception of variability (perceived minimum and maximum value). For instance, in the Auction task we provide data on other bids, ask participants to report the lowest and highest values, and request their own bid. To judge a response as targeted, a participant must both observe that variability exists (via their perceived minimum and maximum value) and make an appropriate decision under variability (via their numeric answer). The instrument consists of 10 similar tasks.
We followed the Measurement of Abstract Graph Interpretation (MAGI) principles (Kerns and Wilmer, 2021 Journal of Vision)---a set of design criteria to promote quality of a survey instrument. For instance, following MAGI principle Ground-truth Linkage, we designed all tasks to have a correct response. This enables objective assessment of participants' behavior to determine if they targeted variability.
Findings. Approximately 99% of participants targeted at least once in the sinaplot and text treatments, demonstrating that targeting is a common behavior among U.S. adults. However, ~9% of participants in the bar treatment failed to target even once. This indicates that showing averages as a bar chart leads to poorer decisions.
Participants in the sinaplot treatment also gave significantly more accurate answers than participants in other treatments, indicating that U.S. adults can make use of the information in more complex graphs. Additional analyses support the validity and reliability of the instrument in measuring targeting.
Implications For Teaching and For Research. For statistics education researchers, our survey constitutes a novel instrument for studying decision making in data science; particularly,
for studying how people make decisions under variability. We hope to spark discussion about future work in other domains (e.g., in health or business contexts) and with other populations (e.g., with K-12 students).
For teaching statistics, our results call attention to an aspect of reasoning under variability that is under-emphasized: targeting the consequences of variability. Our survey is appropriate for administration in college classrooms, and may be used to spark discussion with students.