Abstract

A viewer's existing beliefs can prevent accurate reasoning with data visualizations. In particular, confirmation bias can cause people to overweigh information that confirms their beliefs, and dismiss information that disconfirms them. We tested whether confirmation bias exists when people reason with visualized data and whether certain visualization designs can elicit less biased reasoning strategies. We asked crowdworkers to solve reasoning problems that had the potential to evoke both poor reasoning strategies and confirmation bias. We created two scenarios, one in which we primed people with a belief before asking them to make a decision, and another in which people held pre-existing beliefs. The data was presented as either a table, a bar table, or a bar chart. To correctly solve the problem, participants should use a complex reasoning strategy to compare two ratios, each between two pairs of values. But participants could also be tempted to use simpler, superficial heuristics, shortcuts, or biased strategies to reason about the problem. Presenting the data in a table format helped participants reason with the correct ratio strategy while showing the data as a bar table or a bar chart led participants towards incorrect heuristics. Confirmation bias was not significantly present when beliefs were primed, but it was present when beliefs were pre-existing. Additionally, the table presentation format was more likely to afford the ratio reasoning strategy, and the use of ratio strategy was more likely to lead to the correct answer. These findings suggest that data presentation formats can affect affordances for reasoning.

Instructions:

> What does the variables mean?

experiment: the two experiments in the paper.

p: their ID from the respective study (not meaningful)

visualization: did they see a table, a bar table or a bar chart?

effectiveness: the counterbalancing on whether their saw the version of the dataset that indicated the skin cream/legislation was effective or ineffective

gun belief - for experiment 2, how much did they agree with the statement "greater gun restriction laws are necessary to reduce violence" on a scale from 1 (strongly disagree) and 7 (strongly agree).

congruency: whether the effectiveness was aligned with the skin cream prime/their identified attitude towards guns (congruent) or not (incongruent)

congruency_weight: doesn't mean anything for exp1 and exp1b. For exp2, this depends on the strength of their pro and anti-gun sentiment. We explicitly asked them "how much do you agree with this statement: Greater gun restriction laws are necessary to reduce violence" (on a scale from 1-strongly disagree to 7-strongly agree) to compute a belief-strength score related to gun control. People at the two ends of the spectrum are considered strong-believers, middle is considered neutral. Then, whether this belief is algined with what the dataset is saying (effective or uneffective), helps us decide whether this is congruent or incongruent. For example, a strong anti-gun believer who saw the dataset showing that the legistlation on gun control is effective would be in "congruent_strong"

prime: were they primed to think the company is good or bad in exp1 and exp1b? Doesn't mean anything for exp2.

accuracy: their answer to whether the skin cream or the legislation was effective, is it correct?

answer: what did they actually answer? (note that the answer is a 2-choice question)

confidence: how confident were they in their answer?

confidencenum: transforming their verbal confidence rating to a numeric one (higher number, more confident)

strategy_actual: their veral response on the strategy question when they explained the reasoning behind their judgement

strategy_coding_coder1: categorization of their strategy, coded by the first coder, blind to the conditions

strategy_coding_coder2: categorization of their strategy, coded by the second coder, blind to the conditions

strategy_coder_agree: did coder 1 and 2 agree on the codes?

reasoning_strategy: which reasoning strategy did they ultimately use?

rt_prime: total time spent on reading the prime in exp1 (we excluded people who spent less than 10 seconds on prime)

age, sex, race, education, income: as indicated

political_ideology: 1 means extremely liberal

isratio: based on the reasoning strategy, we categorized whether it's the correct ratio strategy or not

Dataset Files

ConfirmationBias_ExperimentalData.csv (420.55 kB)
Statistical Analysis ConfirmationBias_Analysis.Rmd (18.46 kB)

Datasets

Standard Dataset

Reasoning Affordances with Tables and Bar Charts Dataset

Abstract

Dataset Files

QUESTIONS?