How accurate are psychological scientists in predicting societal change? A series of four studies published in American Psychologist suggest that psychologists are no better at such predictions compared to laypeople.
The onset of the COVID-19 pandemic brought with it media appearances of psychologists discussing their predictions regarding what changes we ought to expect in various domains of life. However, these predictions were often outside their area of expertise. Across a series of four studies, Igor Grossmann, PhD (@psywisdom) and colleagues looked into the accuracy of psychologists and laypeople in predicting future societal change and compared these predictions to what unfolded in the real world.
“My interest in this topic stemmed from the lack of insight about how scientists viewed and anticipated major societal risks like pandemics in the past. In this and related projects (such as WorldafterCovid and the Forecasting Collaborative) I wanted to explore how scientists think such major societal shifts may unfold, assess the accuracy of these forecasts, and identify areas where predictions have been more or less successful,” said Grossmann, a professor of psychology at the University of Waterloo.
“By examining whether and how serious scientists made predictions about such uncertain events like the pandemic, and investigating how public intellectuals and scientists engaged with the media at the onset of the pandemic, my team also sought to uncover the domains in which predictions were made, thereby aiming to enhance the understanding of scientific accuracy and how it can be improved.”
Study 1 examined psychological scientists’ discussion regarding the pandemic in the news media, utilizing The Coronavirus Corpus which included over 1.8 million texts of news content containing an interview with an academic psychologist regarding the pandemic. A total of 169 unique articles were retained, which included 719 unique judgments from 213 different scientists (e.g., impact of pandemic on child development).
Study 2 was conducted in two parts. In part one, 401 scientists from 39 countries made forecasts in April 2020 regarding societal change due to the pandemic. This included cultural change in the United States across 11 domains, such as generalized trust, expected birth rates, delay of gratification, among others (verbatim questions can be found here).
Participants made their forecasts for 6 months, 1 year and 2 years into the future, with response options ranging from a 50% or greater decrease to a 50% or greater increase. Of the 11 domains that were tested, accuracy of predictions was reliably assessed for seven (including “polarization, traditionalism, individualism, trust, climate change, life satisfaction, and depression”). Participants also indicated one psychological or social issue in the United States that was not mentioned in the study, but that they believed would evolve in the coming months/year.
Part two of Study 2 was conducted after the initial peak of COVID-19. This included another group of 316 psychological scientists from 26 countries. At the same time, a sample of 394 participants who were nationally representative of the United States were recruited via Prolific. The procedure was approximately the same as that of part one.
However, alongside the 11 domains of Study 2, participants also predicted changes for four additional domains (e.g., charitable giving). In addition to predicting change, participants also provided confidence ratings of their predictions on a scale of 1 (not at all) to 5 (extremely). Reliable benchmarks of accuracy were attainable for 10 of the domains (including “loneliness, charitable giving, violent crimes, polarization, traditionalism, individualism, trust, climate change, life satisfaction, and depression”).
Study 3 was also conducted in two parts. A total of 411 laypeople and 270 scientists were prompted to provide retrospective judgements of change and confidence ratings for the same domains as in Study 2. Specifically, they provided an estimate of the amount of change they perceived on a given issue compared to six months prior. Participants also indicated the types of information they considered when providing their judgments.
In Study 4, 203 participants were prompted to consider scientists, practitioners and laypeople, and rate how accurate they would be when predicting societal change throughout the COVID-19 pandemic across the various domains of interest (e.g., life satisfaction, loneliness). They also indicated who they would prefer to hear recommendations from regarding the societal issues tackled in this project (e.g., scientist with expertise in epidemiology, practitioner with expertise in social work, average american, for a total of 10 groups to consider).
“The average person should understand that psychological scientists’ predictions regarding societal changes during the COVID-19 pandemic were found to be no more accurate than those of laypeople,” Grossmann told PsyPost.
“Despite their formal training and expertise, these scientists often based their judgments on intuition and heuristics rather than empirical evidence. This work also showed that neither specific expertise nor experience significantly improved the accuracy of these ‘off-the-cuff’ predictions. This underscores the complexity of forecasting societal responses to unprecedented events like the pandemic. At the same time, and in contrast to laypeople, scientists were more uncertain about their predictions, thus showing a sign of ‘meta-accuracy’ – they were potentially more aware of their limitations.”
An important question that emerges from this work is how to improve scientists’ predictive accuracy regarding societal effects of major events such as the pandemic.
“In this project, the lack of accuracy in predicting societal changes is not confined to one domain or level of expertise, raising questions about the underlying reasons for these inaccuracies. More investigation is needed to determine how psychological scientists can improve their forecasting abilities, perhaps by implementing models that focus on prediction-oriented designs rather than solely relying on post-hoc explanations,” Grossmann explained.
“We must also explore how to use psychological expertise in ways that take uncertainty into account and how biases like negativity bias might be corrected in both expert and policy considerations. Finally, this project does not examine what happens when scientists make predictions as a group or by relying on formal modeling of past data – a typical way many scientists operate; this question was addressed in a parallel Forecasting Collaborative initiative we ran in parallel, results from which appeared in Nature Human Behaviour this year (and were similarly disappointing).”
The researcher added, “I would like to emphasize that while the study found shortcomings in the predictive capabilities of psychological scientists, it doesn’t diminish the importance of psychological/social science in informing public understanding and policy – after all, in this study people expect scientists to be at the decision table and they were more aware of uncertainty associated with making societal predictions.”
“The findings also highlight the need for improved methods of prediction and communication of uncertainty in times of crisis. Future work should explore ways to enhance accuracy, including training strategies at both institutional and individual levels.”
The research, “On the Accuracy, Media Representation, and Public Perception of Psychological Scientists’ Judgments of Societal Change”, was authored by Cendri A. Hutcherson, Konstantyn Sharpinskyi, Michael E. W. Varnum, Amanda Rotella, Alexandra S. Wormley, Louis Tay, and Igor Grossmann.
