Subscribe
The latest psychology and neuroscience discoveries.
My Account
  • Mental Health
  • Social Psychology
  • Cognitive Science
  • Psychopharmacology
  • Neuroscience
  • About
No Result
View All Result
PsyPost
PsyPost
No Result
View All Result
Home Exclusive Artificial Intelligence

Groundbreaking AI model uncovers hidden patterns of political bias in online news

by Eric W. Dolan
May 23, 2025
in Artificial Intelligence, Political Psychology
[Adobe Stock]

[Adobe Stock]

Share on TwitterShare on Facebook
Follow PsyPost on Google News

A new study published in PLOS One introduces a large-scale method for detecting political bias in online news sources using artificial intelligence. By analyzing hundreds of thousands of news articles, the researchers developed a model that predicts political leaning and explains why an outlet is categorized in a particular way.

Bias in media reporting is widely acknowledged, but studying and measuring that bias at scale has been difficult. Traditional approaches often rely on human annotations, which are limited in scope and can themselves be biased. At the same time, most studies focus on narrow expressions of bias—such as loaded wording in headlines—while overlooking broader behavioral patterns like which topics are covered, how frequently, or how much space they are given. The new study addresses these limitations by building a data-driven system that examines a range of bias indicators across a vast number of sources.

“This project actually started as part of my master’s thesis, and I was determined to apply my technical skills toward analyzing a societally important topic,” explained study author Ronja Thelen-Rönnback, a PhD student at Tilburg University and member of the Tilburg Algorithm Observatory.

“People are becoming more and more skeptical or disenchanted with news, and a large part of this is (according to surveys) due to perceived political bias. There is a lot of excellent academic work about news bias, but much of it relies on human experts analyzing and labeling bias in articles. This is thorough but, of course, slow—which has inspired some data-driven approaches that can detect bias much faster.”

“However, the data-driven approaches usually do not provide the same level of detail and understanding about news political bias as expert labels, and they also tend to focus on very simple forms of bias. For example, they detect sensational headlines or biased wording in an article. Crucially, experts have long known that there are many more, sometimes subtle, ways that news outlets can be biased. For example, an outlet might entirely avoid reporting on a specific topic, or only report on it very briefly. These kinds of bias are only really visible when one examines an outlet’s overall behavior, not just the text in its articles.”

To conduct a large-scale investigation into political bias in news media, the researchers developed a machine learning-based system capable of classifying the political orientation of web-based news outlets. They grounded their analysis in data from the Global Database of Events, Language, and Tone (GDELT), one of the most comprehensive open platforms for monitoring global news coverage. The study focused on English-language articles from 2022, allowing the researchers to examine how thousands of news web domains behaved across a range of topics and how this behavior related to political bias.

The first step involved transforming raw article-level data into a structured, outlet-level dataset. GDELT tags news articles with thematic labels—ranging from crime and immigration to climate change and economic topics—and provides metadata such as tone, word count, and whether the article included visual content like images or videos. These data points served as proxies for different types of media bias, including tone bias (how emotionally charged or neutral a story is), selection bias (which topics are covered or ignored), and size bias (how much space is given to different topics).

To build a representative sample and reduce noise, Thelen-Rönnback and her colleagues filtered out themes that were either too obscure or appeared too infrequently, ending up with over 500 themes and nearly 7,000 features per web domain.

In addition to the GDELT data, the researchers supplemented their dataset with information from Media Bias Fact Check, an independent organization that evaluates media outlets based on factors like political leaning, factual accuracy, web traffic, and press freedom in the outlet’s country of origin. These outlet-level features—such as whether a domain was a newspaper or TV station, or how credible it was rated—were incorporated into one version of the experiment to assess whether they could improve classification accuracy.

The researchers then created two sets of ground truth political bias labels. One came from Media Bias Fact Check, where human experts assigned each outlet a political classification on a five-point scale: left, left-center, least biased, right-center, and right. The other came from a study by Robertson and colleagues, which inferred political leaning based on Twitter user behavior. In that method, websites frequently shared by registered Democrats were assumed to lean left, and those shared by Republicans were assumed to lean right. These continuous scores were grouped into the same five political categories for comparison.

Using these ground truth labels, the researchers trained and tested multiple machine learning models, including a feed-forward neural network, support vector machines (SVM), AdaBoost, and XGBoost classifiers. They also included two large language models (GPT-4o-mini and LLaMA 3.1) as baseline comparisons, asking them to classify political bias without any fine-tuning or additional training.

Across all experiments, model performance was evaluated based on classification accuracy and the area under the receiver operating characteristic curve (AUC), which measures how well a model can distinguish between classes.

The neural network consistently outperformed other models. When trained on the full set of features—including both traditional tone-based and alternative bias indicators like article counts and image presence—it achieved an accuracy of 76% and an AUC score of 81% when using the Media Bias Fact Check labels. This marked a significant improvement over the majority baseline model, which simply predicted the most common class and achieved only 45% accuracy. The language model baselines, surprisingly, performed no better than the majority baseline, typically defaulting to the “least biased” label for most outlets.

“We used large language models (GPT-4o-mini and LLaMA 3.1) to see how they would perform compared to smaller, traditional machine learning models,” Thelen-Rönnback told PsyPost. “They did not do well at all, though it is worth noting our implementation of them was very straightforward. Nevertheless, given that there is currently a lot of hype surrounding large language models, we show that they do not always perform the best, and that smaller models can be more than sufficient for many tasks.”

The researchers also examined whether different types of features influenced model performance. When models were trained only on traditional bias features such as tone and sentiment, performance was lower. When trained only on alternative features like topic coverage and media presence, performance improved. But the best results came from using all features together, suggesting that a multifaceted approach to detecting bias—one that includes what topics are covered, how much space is given to them, and whether visuals are used—yields a more accurate picture of a news outlet’s political slant.

“Our work uses an existing database that tracks news worldwide (GDELT) to automatically label political bias of news outlets with machine learning,” Thelen-Rönnback explained. “We account for multiple forms of bias, which is, as of now, somewhat rare in the field. We show that (unsurprisingly) this makes it much easier to detect political bias compared to just looking at a single, narrow expression of bias.”

To make the results interpretable, the researchers used a model-agnostic explainability technique called SHAP (Shapley Additive Explanations). SHAP assigns importance values to each feature used in the model, showing which variables had the most influence on a specific prediction. These explanations revealed that features related to article counts on politically charged themes—such as gun ownership, environmental regulation, and election fraud—were often among the most informative. In some cases, more surprising themes like natural disasters or sanitation also played a role, though the reasons for their relevance were less clear.

In one illustrative example, the model accurately classified Breitbart as a right-wing outlet due to its high frequency of negatively toned articles about crime-related themes such as cartels, kidnapping, and robbery. Similarly, The Guardian was correctly identified as left-leaning due to a strong emphasis on inequality and social movements. These insights provided a window into not just what label the model assigned, but why it reached that conclusion—addressing a major critique of previous machine learning approaches that treated models as “black boxes.”

“We use an explainability tool to provide ‘reasonings’ for each classification—so our models do not just say ‘Breitbart is right biased,’ but they actually show that Breitbart discusses a lot of themes related to crime, and that this is what pushed the model to classify it as right-wing,” Thelen-Rönnback told PsyPost.

To test the reliability of the two labeling systems, the researchers compared how often Media Bias Fact Check and the Twitter-based method agreed. They found that only 46% of web domains shared the same label across both systems, indicating a significant level of disagreement. However, this isn’t entirely surprising given the subjective nature of bias detection. Even two human-annotated sources—Media Bias Fact Check and another platform called AllSides—agreed on only 57% of domains. This suggests that while human labels remain the gold standard, automatically derived labels are not dramatically worse and could be useful when manual evaluations are unavailable.

“We try to use machines to characterize how news sources are biased, rather than humans,” Thelen-Rönnback said. “Machine learning allows us to determine if a news outlet is showing bias at a much larger scale, much faster than what humans can—and potentially in ways that humans might not consider. For example, we see that some unexpected topics like climate disasters are informative to the models, but these are quite often not considered by human experts.”

“News outlets can be biased in many different ways, and some of them are currently not being considered sufficiently. We hope that changing this might help the field advance. We also try to provide some transparency into why a specific outlet is deemed to be biased, which is going to become increasingly important to ensure that the public can understand and trust the news they consume. Hopefully, this can, down the line, lead to a better-informed society and a healthier political environment.”

But there are some caveats to consider. “Right now, getting the output requires some technical expertise with programming and AI models,” Thelen-Rönnback noted. “We also relied on the left-right political split, but this might not be the most appropriate one worldwide. Nevertheless, our approach does easily allow for a different set of political labels, if need be.”

Looking ahead, the researchers hope to expand their system to classify other dimensions of media content. “Ideally, we would be able to expand this from political bias to other relevant aspects like disinformation or unreliability,” Thelen-Rönnback explained. “We are currently also looking at not just online news, but the internet more broadly. Specifically, we’re looking at how search engines react to polarizing topics, which is very exciting since search engines are nowadays the most trusted source of news.”

The study, “Automatic large-scale political bias detection of news outlets,” was authored by Ronja Rönnback, Chris Emmery, and Henry Brighton.

TweetSendScanShareSendPin1ShareShareShareShareShare

RELATED

Dark personality traits and specific humor styles are linked to online trolling, study finds
Artificial Intelligence

Memes can serve as strong indicators of coming mass violence

June 15, 2025

A new study finds that surges in visual propaganda—like memes and doctored images—often precede political violence. By combining AI with expert analysis, researchers tracked manipulated content leading up to Russia’s invasion of Ukraine, revealing early warning signs of instability.

Read moreDetails
Teen depression tied to balance of adaptive and maladaptive emotional strategies, study finds
Artificial Intelligence

Sleep problems top list of predictors for teen mental illness, AI-powered study finds

June 15, 2025

A new study using data from over 11,000 adolescents found that sleep disturbances were the most powerful predictor of future mental health problems—more so than trauma or family history. AI models based on questionnaires outperformed those using brain scans.

Read moreDetails
Poor sleep may shrink brain regions vulnerable to Alzheimer’s disease, study suggests
Political Psychology

Christian nationalists tend to imagine God as benevolent, angry over sins, and engaged

June 14, 2025

Christians who believe God is benevolent, engaged, and angered by sin are more likely to support religious nationalism, according to a new study. This worldview was also associated with conspiracy mentality and xenophobic attitudes.

Read moreDetails
Fear predicts authoritarian attitudes across cultures, with conservatives most affected
Authoritarianism

Fear predicts authoritarian attitudes across cultures, with conservatives most affected

June 13, 2025

A sweeping study of over 84,000 people across 59 countries found that individuals who feel threatened by crime, poverty, or instability are more likely to support authoritarian governance—especially in Western nations and among politically right-leaning individuals.

Read moreDetails
New research links certain types of narcissism to anti-immigrant attitudes
Artificial Intelligence

Fears about AI push workers to embrace creativity over coding, new research suggests

June 13, 2025

A new study shows that when workers feel threatened by artificial intelligence, they tend to highlight creativity—rather than technical or social skills—in job applications and education choices. The research suggests people see creativity as a uniquely human skill machines can’t replace.

Read moreDetails
New research links certain types of narcissism to anti-immigrant attitudes
Narcissism

New research links certain types of narcissism to anti-immigrant attitudes

June 13, 2025

New research published in Behavioral Sciences shows that certain narcissistic traits are linked to anti-immigrant attitudes through competitive worldviews and ideological beliefs, highlighting the role of personality in shaping how people view immigrants and social hierarchies.

Read moreDetails
Smash or pass? AI could soon predict your date’s interest via physiological cues
Artificial Intelligence

A neuroscientist explains why it’s impossible for AI to “understand” language

June 12, 2025

Can artificial intelligence truly “understand” language the way humans do? A neuroscientist challenges this popular belief, arguing that machines may generate convincing text—but they lack the emotional, contextual, and biological grounding that gives real meaning to human communication.

Read moreDetails
Uncertainty about immigration intensifies nationalist sentiments
Political Psychology

From well-being to white replacement: What psychology research says about immigration

June 11, 2025

Immigration sparks fierce debates—but what does the research say? These seven studies uncover surprising findings about mental health, crime, social media, nationalism, and political violence.

Read moreDetails

SUBSCRIBE

Go Ad-Free! Click here to subscribe to PsyPost and support independent science journalism!

STAY CONNECTED

LATEST

Frequent pornography use linked to altered brain connectivity and impaired cognitive performance

Childhood trauma linked to changes in brain structure and connectivity, study finds

COVID-19 coverage linked to rise in anti-Asian sentiment, especially among Trump supporters

Some dark personality traits may help buffer against depression, new psychology research suggests

Dementia risk begins in childhood, not old age, scientists warn

Millennials are abandoning organized religion. A new study provides insight into why

Sleep regularity might be protective of adolescents’ mental health, study suggests

Different parts of the same neuron learn in different ways, study finds

         
       
  • Contact us
  • Privacy policy
  • Terms and Conditions
[Do not sell my information]

Welcome Back!

Login to your account below

Forgotten Password?

Retrieve your password

Please enter your username or email address to reset your password.

Log In

Add New Playlist

Subscribe
  • My Account
  • Cognitive Science Research
  • Mental Health Research
  • Social Psychology Research
  • Drug Research
  • Relationship Research
  • About PsyPost
  • Contact
  • Privacy Policy