A new study published in PLOS ONE provides evidence that a person’s political ideology and their capacity for analytical thinking shape how they gather information. The findings suggest that liberals and individuals with higher cognitive reflection skills are more likely to seek out comprehensive statistical data, whereas conservatives and those who rely more on intuition tend to focus on singular data points or expert opinions.
Public policy debates are often characterized by intense disagreement, even when facts are available. While much psychological research has examined how people process information they have already received, less attention has been paid to the earlier step of information seeking.
The authors of the new study aimed to understand the standards of evidence people apply when they want to learn about the world. They sought to determine why some individuals demand rigorous statistical comparisons while others are satisfied with anecdotes or expert testimonials.
“There is a solid research agenda that evaluates how people treat evidence presented to then. However, there is much less research on how people look for evidence in the first place,” said study author Florian Justwan, an associate professor of political science at the University of Idaho.
“This is particularly important in the social world where one finds a lot of contested cause-and-effect claims. Examples of such claims are: is a particular law responsible for an undesirable / desirable social outcome? Does a particular vaccine decrease the risk of contracting a disease?”
For their study, the researchers recruited a diverse sample of 583 adults from the United States using an online platform. The study was conducted in August 2023.
The participants were presented with a fictionalized but realistic scenario regarding a policy initiative known as cash bail reform. They were informed that out of the 300 most populous American cities, 100 had implemented this reform while 200 had not. The participants were then tasked with evaluating whether the policy was effective at reducing crime.
The core of the experiment was an “evidence bank” containing ten different pieces of information. Participants could choose to view as many or as few items as they wished before making their final assessment. This setup allowed the researchers to track exactly what kind of proof each respondent felt was necessary to form an opinion.
The evidence options fell into two broad categories: statistical data and expert testimony. The statistical options allowed participants to see the number of cities with or without the reform that experienced an increase or decrease in crime.
The expert testimony options provided evaluations from political sources, specifically the Democratic Party, the Republican Party, the National Rifle Association (NRA), and the Center for American Progress.
The researchers classified the statistical information seeking behavior into two types. The first type was labeled “categorical.” This involved looking at only one specific outcome, such as the number of reformed cities where crime went up. Relying on categorical evidence is often considered less reliable because it ignores the context provided by a control group.
The second type was labeled “fully associative.” This involved gathering all four necessary pieces of data: crime increases and decreases in both the reformed and non-reformed cities. This method allows for a calculation of probabilities and provides a complete picture of the policy’s impact.
In addition to tracking evidence selection, the researchers measured the participants’ political ideology and their level of “cognitive reflection.” Cognitive reflection refers to a person’s ability to override an immediate, intuitive response to engage in deeper, analytical thinking. This was measured using a seven-question test containing brain teasers that have an obvious but incorrect answer and a correct answer that requires slightly more thought.
The results showed a link between political ideology and the type of evidence gathered. Self-identified conservatives were more likely to rely on categorical standards of evidence. For example, the probability that a respondent would rely on a single type of data point rose from roughly 4 percent for very liberal participants to over 37 percent for very conservative participants.
Conservatives were also less likely to seek out the fully associative data required to make a mathematically sound comparison. Liberals, by contrast, demonstrated a strong preference for collecting comprehensive statistical information.
Cognitive reflection also played a substantial role in these behaviors. Participants who scored higher on the cognitive reflection test were much less likely to rely on categorical evidence.
Instead, they tended to gather all available statistical data to compare the outcomes properly. These analytical thinkers were also less likely to request expert testimony. They preferred to look at the raw numbers themselves rather than deferring to the judgment of political organizations.
The researchers also found nuance in how people used expert sources. When individuals with high cognitive reflection scores did choose to consult experts, they were more likely to look for a mix of opinions. They sought out evaluations from both their own political “in-group” and the opposing “out-group.”
For instance, a cognitively reflective Democrat might check what both the Center for American Progress and the NRA said about the policy. In contrast, participants with lower cognitive reflection scores who sought expert advice tended to look exclusively at sources that aligned with their own political preferences.
“There are different ‘types of evidence seekers’ in the world,” Justwan told PsyPost. “For instance, some people look for what we call ‘categorical standards of evidence.’ In practice, this means that they only look for one type of data when they examine a particular cause-and-effect relationship. This might take the form of looking up the number of people who received a vaccine for a particular disease and who still got sick.”
“Other people, by contrast, look for what we call ‘fully associative’ forms of evidence. In other words, they seek information on all possible combinations of treatment and outcome. Furthermore, people also vary systematically according to (1) their likelihood on looking for expert testimony and (2) their likelihood of consulting ideologically congruent and incongruent sources.”
“Importantly, our study shows that two major individual-level variables help to predict what type of ‘evidence seeker’ a given person is: whether or not they are ‘cognitively reflected’ and whether or not they are liberal/conservative,” the researcher explained. “Indeed, people’s political beliefs influence how they look for information (often without them realizing it).”
But the study, like all research, has some caveats. The study used a fixed list of evidence options, which does not perfectly mimic the open-ended nature of searching the internet. Real-world searches often involve typing queries into search engines, a process that might yield different results than selecting from a menu.
Additionally, the topic of cash bail reform is politically polarized. It is possible that people might gather evidence differently for less controversial topics. The researchers also noted that they did not evaluate whether the participants interpreted the evidence correctly, but simply observed what evidence they chose to access.
“Our work does not address how people process information,” Justwan noted. “In other words, we do not investigate what people do with information once they have collected it. We also do not examine whether people interpret information ‘correctly.’ Instead, the focus of this research only focuses on what type of information people rely on when they assess whether an external stimulus has an effect on a given outcome or not.”
Future research could explore how these patterns hold up in different contexts. The rise of artificial intelligence and chat-based search tools might alter how people gather information.
The researchers plan to investigate how people weigh the credibility of sources against the actual data those sources provide. Understanding these dynamics is essential for grappling with how citizens become informed in an increasingly complex and divided media landscape.
“Long term, we seek to understand general patterns between how people think about a source of information (e.g. trust, reliability, expertise) and the first-order information from that source,” Justwan said. “We anticipate this relationship will be further complicated by the rapid growth of AI systems in our information landscapes.”
The study, “The effects of ideology and cognitive reflection on evidence gathering behavior in the political domain,” was authored by Florian Justwan and Bert Baumgaertner.
