Subscribe
The latest psychology and neuroscience discoveries.
My Account
  • Mental Health
  • Social Psychology
  • Cognitive Science
  • Psychopharmacology
  • Neuroscience
  • About
No Result
View All Result
PsyPost
PsyPost
No Result
View All Result
Home Exclusive Artificial Intelligence

Groundbreaking AI model uncovers hidden patterns of political bias in online news

by Eric W. Dolan
May 23, 2025
in Artificial Intelligence, Political Psychology
[Adobe Stock]

[Adobe Stock]

Share on TwitterShare on Facebook
Follow PsyPost on Google News

A new study published in PLOS One introduces a large-scale method for detecting political bias in online news sources using artificial intelligence. By analyzing hundreds of thousands of news articles, the researchers developed a model that predicts political leaning and explains why an outlet is categorized in a particular way.

Bias in media reporting is widely acknowledged, but studying and measuring that bias at scale has been difficult. Traditional approaches often rely on human annotations, which are limited in scope and can themselves be biased. At the same time, most studies focus on narrow expressions of bias—such as loaded wording in headlines—while overlooking broader behavioral patterns like which topics are covered, how frequently, or how much space they are given. The new study addresses these limitations by building a data-driven system that examines a range of bias indicators across a vast number of sources.

“This project actually started as part of my master’s thesis, and I was determined to apply my technical skills toward analyzing a societally important topic,” explained study author Ronja Thelen-Rönnback, a PhD student at Tilburg University and member of the Tilburg Algorithm Observatory.

“People are becoming more and more skeptical or disenchanted with news, and a large part of this is (according to surveys) due to perceived political bias. There is a lot of excellent academic work about news bias, but much of it relies on human experts analyzing and labeling bias in articles. This is thorough but, of course, slow—which has inspired some data-driven approaches that can detect bias much faster.”

“However, the data-driven approaches usually do not provide the same level of detail and understanding about news political bias as expert labels, and they also tend to focus on very simple forms of bias. For example, they detect sensational headlines or biased wording in an article. Crucially, experts have long known that there are many more, sometimes subtle, ways that news outlets can be biased. For example, an outlet might entirely avoid reporting on a specific topic, or only report on it very briefly. These kinds of bias are only really visible when one examines an outlet’s overall behavior, not just the text in its articles.”

To conduct a large-scale investigation into political bias in news media, the researchers developed a machine learning-based system capable of classifying the political orientation of web-based news outlets. They grounded their analysis in data from the Global Database of Events, Language, and Tone (GDELT), one of the most comprehensive open platforms for monitoring global news coverage. The study focused on English-language articles from 2022, allowing the researchers to examine how thousands of news web domains behaved across a range of topics and how this behavior related to political bias.

The first step involved transforming raw article-level data into a structured, outlet-level dataset. GDELT tags news articles with thematic labels—ranging from crime and immigration to climate change and economic topics—and provides metadata such as tone, word count, and whether the article included visual content like images or videos. These data points served as proxies for different types of media bias, including tone bias (how emotionally charged or neutral a story is), selection bias (which topics are covered or ignored), and size bias (how much space is given to different topics).

To build a representative sample and reduce noise, Thelen-Rönnback and her colleagues filtered out themes that were either too obscure or appeared too infrequently, ending up with over 500 themes and nearly 7,000 features per web domain.

In addition to the GDELT data, the researchers supplemented their dataset with information from Media Bias Fact Check, an independent organization that evaluates media outlets based on factors like political leaning, factual accuracy, web traffic, and press freedom in the outlet’s country of origin. These outlet-level features—such as whether a domain was a newspaper or TV station, or how credible it was rated—were incorporated into one version of the experiment to assess whether they could improve classification accuracy.

The researchers then created two sets of ground truth political bias labels. One came from Media Bias Fact Check, where human experts assigned each outlet a political classification on a five-point scale: left, left-center, least biased, right-center, and right. The other came from a study by Robertson and colleagues, which inferred political leaning based on Twitter user behavior. In that method, websites frequently shared by registered Democrats were assumed to lean left, and those shared by Republicans were assumed to lean right. These continuous scores were grouped into the same five political categories for comparison.

Using these ground truth labels, the researchers trained and tested multiple machine learning models, including a feed-forward neural network, support vector machines (SVM), AdaBoost, and XGBoost classifiers. They also included two large language models (GPT-4o-mini and LLaMA 3.1) as baseline comparisons, asking them to classify political bias without any fine-tuning or additional training.

Across all experiments, model performance was evaluated based on classification accuracy and the area under the receiver operating characteristic curve (AUC), which measures how well a model can distinguish between classes.

The neural network consistently outperformed other models. When trained on the full set of features—including both traditional tone-based and alternative bias indicators like article counts and image presence—it achieved an accuracy of 76% and an AUC score of 81% when using the Media Bias Fact Check labels. This marked a significant improvement over the majority baseline model, which simply predicted the most common class and achieved only 45% accuracy. The language model baselines, surprisingly, performed no better than the majority baseline, typically defaulting to the “least biased” label for most outlets.

“We used large language models (GPT-4o-mini and LLaMA 3.1) to see how they would perform compared to smaller, traditional machine learning models,” Thelen-Rönnback told PsyPost. “They did not do well at all, though it is worth noting our implementation of them was very straightforward. Nevertheless, given that there is currently a lot of hype surrounding large language models, we show that they do not always perform the best, and that smaller models can be more than sufficient for many tasks.”

The researchers also examined whether different types of features influenced model performance. When models were trained only on traditional bias features such as tone and sentiment, performance was lower. When trained only on alternative features like topic coverage and media presence, performance improved. But the best results came from using all features together, suggesting that a multifaceted approach to detecting bias—one that includes what topics are covered, how much space is given to them, and whether visuals are used—yields a more accurate picture of a news outlet’s political slant.

“Our work uses an existing database that tracks news worldwide (GDELT) to automatically label political bias of news outlets with machine learning,” Thelen-Rönnback explained. “We account for multiple forms of bias, which is, as of now, somewhat rare in the field. We show that (unsurprisingly) this makes it much easier to detect political bias compared to just looking at a single, narrow expression of bias.”

To make the results interpretable, the researchers used a model-agnostic explainability technique called SHAP (Shapley Additive Explanations). SHAP assigns importance values to each feature used in the model, showing which variables had the most influence on a specific prediction. These explanations revealed that features related to article counts on politically charged themes—such as gun ownership, environmental regulation, and election fraud—were often among the most informative. In some cases, more surprising themes like natural disasters or sanitation also played a role, though the reasons for their relevance were less clear.

In one illustrative example, the model accurately classified Breitbart as a right-wing outlet due to its high frequency of negatively toned articles about crime-related themes such as cartels, kidnapping, and robbery. Similarly, The Guardian was correctly identified as left-leaning due to a strong emphasis on inequality and social movements. These insights provided a window into not just what label the model assigned, but why it reached that conclusion—addressing a major critique of previous machine learning approaches that treated models as “black boxes.”

“We use an explainability tool to provide ‘reasonings’ for each classification—so our models do not just say ‘Breitbart is right biased,’ but they actually show that Breitbart discusses a lot of themes related to crime, and that this is what pushed the model to classify it as right-wing,” Thelen-Rönnback told PsyPost.

To test the reliability of the two labeling systems, the researchers compared how often Media Bias Fact Check and the Twitter-based method agreed. They found that only 46% of web domains shared the same label across both systems, indicating a significant level of disagreement. However, this isn’t entirely surprising given the subjective nature of bias detection. Even two human-annotated sources—Media Bias Fact Check and another platform called AllSides—agreed on only 57% of domains. This suggests that while human labels remain the gold standard, automatically derived labels are not dramatically worse and could be useful when manual evaluations are unavailable.

“We try to use machines to characterize how news sources are biased, rather than humans,” Thelen-Rönnback said. “Machine learning allows us to determine if a news outlet is showing bias at a much larger scale, much faster than what humans can—and potentially in ways that humans might not consider. For example, we see that some unexpected topics like climate disasters are informative to the models, but these are quite often not considered by human experts.”

“News outlets can be biased in many different ways, and some of them are currently not being considered sufficiently. We hope that changing this might help the field advance. We also try to provide some transparency into why a specific outlet is deemed to be biased, which is going to become increasingly important to ensure that the public can understand and trust the news they consume. Hopefully, this can, down the line, lead to a better-informed society and a healthier political environment.”

But there are some caveats to consider. “Right now, getting the output requires some technical expertise with programming and AI models,” Thelen-Rönnback noted. “We also relied on the left-right political split, but this might not be the most appropriate one worldwide. Nevertheless, our approach does easily allow for a different set of political labels, if need be.”

Looking ahead, the researchers hope to expand their system to classify other dimensions of media content. “Ideally, we would be able to expand this from political bias to other relevant aspects like disinformation or unreliability,” Thelen-Rönnback explained. “We are currently also looking at not just online news, but the internet more broadly. Specifically, we’re looking at how search engines react to polarizing topics, which is very exciting since search engines are nowadays the most trusted source of news.”

The study, “Automatic large-scale political bias detection of news outlets,” was authored by Ronja Rönnback, Chris Emmery, and Henry Brighton.

TweetSendScanShareSendPin1ShareShareShareShareShare

RELATED

Political ambivalence has a surprising relationship with support for violence
Authoritarianism

New study sheds light on the psychological roots of collective violence

June 21, 2025

A new study from Lebanon finds that people with authoritarian beliefs tend to oppose violence against political leaders, while those high in social dominance orientation are more likely to support violence against rival group members.

Read moreDetails
Epistemic mistrust and dogmatism predict preference for authoritarian-looking leaders
Authoritarianism

Epistemic mistrust and dogmatism predict preference for authoritarian-looking leaders

June 20, 2025

A new study suggests that the way people learn to trust others early in life can shape their political ideology and preference for strong, dominant leaders—though not directly, but through dogmatic thinking and broader political attitudes.

Read moreDetails
Generative AI chatbots like ChatGPT can act as an “emotional sanctuary” for mental health
Artificial Intelligence

Do AI tools undermine our sense of creativity? New study says yes

June 19, 2025

A new study published in The Journal of Creative Behavior offers insight into how people think about their own creativity when working with artificial intelligence.

Read moreDetails
Individual traits, not environment, predict gun violence among gun-carrying youth
Political Psychology

Republican women and Democratic men often break with party lines on gun policy

June 19, 2025

New research shows that Americans’ views on gun policy are shaped by the intersection of gender and partisanship, with Republican women and Democratic men often expressing positions that differ from those typically associated with their party.

Read moreDetails
Troubling study shows “politics can trump truth” to a surprising degree, regardless of education or analytical ability
Donald Trump

Racial insecurity helped shield Trump from Republican backlash after Capitol riot, study suggests

June 18, 2025

Despite widespread condemnation of the January 6th attack, many white Republicans remained loyal to Trump—especially those who perceived anti-white discrimination. A new study shows how racial status threat can protect political leaders from the consequences of norm violations.

Read moreDetails
Dark personality traits and specific humor styles are linked to online trolling, study finds
Artificial Intelligence

Memes can serve as strong indicators of coming mass violence

June 15, 2025

A new study finds that surges in visual propaganda—like memes and doctored images—often precede political violence. By combining AI with expert analysis, researchers tracked manipulated content leading up to Russia’s invasion of Ukraine, revealing early warning signs of instability.

Read moreDetails
Teen depression tied to balance of adaptive and maladaptive emotional strategies, study finds
Artificial Intelligence

Sleep problems top list of predictors for teen mental illness, AI-powered study finds

June 15, 2025

A new study using data from over 11,000 adolescents found that sleep disturbances were the most powerful predictor of future mental health problems—more so than trauma or family history. AI models based on questionnaires outperformed those using brain scans.

Read moreDetails
Poor sleep may shrink brain regions vulnerable to Alzheimer’s disease, study suggests
Political Psychology

Christian nationalists tend to imagine God as benevolent, angry over sins, and engaged

June 14, 2025

Christians who believe God is benevolent, engaged, and angered by sin are more likely to support religious nationalism, according to a new study. This worldview was also associated with conspiracy mentality and xenophobic attitudes.

Read moreDetails

SUBSCRIBE

Go Ad-Free! Click here to subscribe to PsyPost and support independent science journalism!

STAY CONNECTED

LATEST

Exposure to heavy metals is associated with higher likelihood of ADHD diagnosis

Eye-tracking study shows people fixate longer on female aggressors than male ones

Romantic breakups follow a two-stage decline that begins years before the split, study finds

Believing “news will find me” is linked to sharing fake news, study finds

A common parasite not only invades the brain — it can also decapitate human sperm

Almost all unmarried pregant women say that the fetus resembles the father, study finds

New neuroscience research reveals brain antioxidant deficit in depression

Scientists uncover kidney-to-brain route for Parkinson’s-related protein spread

         
       
  • Contact us
  • Privacy policy
  • Terms and Conditions
[Do not sell my information]

Welcome Back!

Login to your account below

Forgotten Password?

Retrieve your password

Please enter your username or email address to reset your password.

Log In

Add New Playlist

Subscribe
  • My Account
  • Cognitive Science Research
  • Mental Health Research
  • Social Psychology Research
  • Drug Research
  • Relationship Research
  • About PsyPost
  • Contact
  • Privacy Policy