Groundbreaking AI model uncovers hidden patterns of political bias in online news

A new study published in PLOS One introduces a large-scale method for detecting political bias in online news sources using artificial intelligence. By analyzing hundreds of thousands of news articles, the researchers developed a model that predicts political leaning and explains why an outlet is categorized in a particular way.

Bias in media reporting is widely acknowledged, but studying and measuring that bias at scale has been difficult. Traditional approaches often rely on human annotations, which are limited in scope and can themselves be biased. At the same time, most studies focus on narrow expressions of bias—such as loaded wording in headlines—while overlooking broader behavioral patterns like which topics are covered, how frequently, or how much space they are given. The new study addresses these limitations by building a data-driven system that examines a range of bias indicators across a vast number of sources.

“This project actually started as part of my master’s thesis, and I was determined to apply my technical skills toward analyzing a societally important topic,” explained study author Ronja Thelen-Rönnback, a PhD student at Tilburg University and member of the Tilburg Algorithm Observatory.

“People are becoming more and more skeptical or disenchanted with news, and a large part of this is (according to surveys) due to perceived political bias. There is a lot of excellent academic work about news bias, but much of it relies on human experts analyzing and labeling bias in articles. This is thorough but, of course, slow—which has inspired some data-driven approaches that can detect bias much faster.”

“However, the data-driven approaches usually do not provide the same level of detail and understanding about news political bias as expert labels, and they also tend to focus on very simple forms of bias. For example, they detect sensational headlines or biased wording in an article. Crucially, experts have long known that there are many more, sometimes subtle, ways that news outlets can be biased. For example, an outlet might entirely avoid reporting on a specific topic, or only report on it very briefly. These kinds of bias are only really visible when one examines an outlet’s overall behavior, not just the text in its articles.”

To conduct a large-scale investigation into political bias in news media, the researchers developed a machine learning-based system capable of classifying the political orientation of web-based news outlets. They grounded their analysis in data from the Global Database of Events, Language, and Tone (GDELT), one of the most comprehensive open platforms for monitoring global news coverage. The study focused on English-language articles from 2022, allowing the researchers to examine how thousands of news web domains behaved across a range of topics and how this behavior related to political bias.

The first step involved transforming raw article-level data into a structured, outlet-level dataset. GDELT tags news articles with thematic labels—ranging from crime and immigration to climate change and economic topics—and provides metadata such as tone, word count, and whether the article included visual content like images or videos. These data points served as proxies for different types of media bias, including tone bias (how emotionally charged or neutral a story is), selection bias (which topics are covered or ignored), and size bias (how much space is given to different topics).

To build a representative sample and reduce noise, Thelen-Rönnback and her colleagues filtered out themes that were either too obscure or appeared too infrequently, ending up with over 500 themes and nearly 7,000 features per web domain.

In addition to the GDELT data, the researchers supplemented their dataset with information from Media Bias Fact Check, an independent organization that evaluates media outlets based on factors like political leaning, factual accuracy, web traffic, and press freedom in the outlet’s country of origin. These outlet-level features—such as whether a domain was a newspaper or TV station, or how credible it was rated—were incorporated into one version of the experiment to assess whether they could improve classification accuracy.

The researchers then created two sets of ground truth political bias labels. One came from Media Bias Fact Check, where human experts assigned each outlet a political classification on a five-point scale: left, left-center, least biased, right-center, and right. The other came from a study by Robertson and colleagues, which inferred political leaning based on Twitter user behavior. In that method, websites frequently shared by registered Democrats were assumed to lean left, and those shared by Republicans were assumed to lean right. These continuous scores were grouped into the same five political categories for comparison.

Using these ground truth labels, the researchers trained and tested multiple machine learning models, including a feed-forward neural network, support vector machines (SVM), AdaBoost, and XGBoost classifiers. They also included two large language models (GPT-4o-mini and LLaMA 3.1) as baseline comparisons, asking them to classify political bias without any fine-tuning or additional training.

Across all experiments, model performance was evaluated based on classification accuracy and the area under the receiver operating characteristic curve (AUC), which measures how well a model can distinguish between classes.

The neural network consistently outperformed other models. When trained on the full set of features—including both traditional tone-based and alternative bias indicators like article counts and image presence—it achieved an accuracy of 76% and an AUC score of 81% when using the Media Bias Fact Check labels. This marked a significant improvement over the majority baseline model, which simply predicted the most common class and achieved only 45% accuracy. The language model baselines, surprisingly, performed no better than the majority baseline, typically defaulting to the “least biased” label for most outlets.

“We used large language models (GPT-4o-mini and LLaMA 3.1) to see how they would perform compared to smaller, traditional machine learning models,” Thelen-Rönnback told PsyPost. “They did not do well at all, though it is worth noting our implementation of them was very straightforward. Nevertheless, given that there is currently a lot of hype surrounding large language models, we show that they do not always perform the best, and that smaller models can be more than sufficient for many tasks.”

The researchers also examined whether different types of features influenced model performance. When models were trained only on traditional bias features such as tone and sentiment, performance was lower. When trained only on alternative features like topic coverage and media presence, performance improved. But the best results came from using all features together, suggesting that a multifaceted approach to detecting bias—one that includes what topics are covered, how much space is given to them, and whether visuals are used—yields a more accurate picture of a news outlet’s political slant.

“Our work uses an existing database that tracks news worldwide (GDELT) to automatically label political bias of news outlets with machine learning,” Thelen-Rönnback explained. “We account for multiple forms of bias, which is, as of now, somewhat rare in the field. We show that (unsurprisingly) this makes it much easier to detect political bias compared to just looking at a single, narrow expression of bias.”

To make the results interpretable, the researchers used a model-agnostic explainability technique called SHAP (Shapley Additive Explanations). SHAP assigns importance values to each feature used in the model, showing which variables had the most influence on a specific prediction. These explanations revealed that features related to article counts on politically charged themes—such as gun ownership, environmental regulation, and election fraud—were often among the most informative. In some cases, more surprising themes like natural disasters or sanitation also played a role, though the reasons for their relevance were less clear.

In one illustrative example, the model accurately classified Breitbart as a right-wing outlet due to its high frequency of negatively toned articles about crime-related themes such as cartels, kidnapping, and robbery. Similarly, The Guardian was correctly identified as left-leaning due to a strong emphasis on inequality and social movements. These insights provided a window into not just what label the model assigned, but why it reached that conclusion—addressing a major critique of previous machine learning approaches that treated models as “black boxes.”

“We use an explainability tool to provide ‘reasonings’ for each classification—so our models do not just say ‘Breitbart is right biased,’ but they actually show that Breitbart discusses a lot of themes related to crime, and that this is what pushed the model to classify it as right-wing,” Thelen-Rönnback told PsyPost.

To test the reliability of the two labeling systems, the researchers compared how often Media Bias Fact Check and the Twitter-based method agreed. They found that only 46% of web domains shared the same label across both systems, indicating a significant level of disagreement. However, this isn’t entirely surprising given the subjective nature of bias detection. Even two human-annotated sources—Media Bias Fact Check and another platform called AllSides—agreed on only 57% of domains. This suggests that while human labels remain the gold standard, automatically derived labels are not dramatically worse and could be useful when manual evaluations are unavailable.

“We try to use machines to characterize how news sources are biased, rather than humans,” Thelen-Rönnback said. “Machine learning allows us to determine if a news outlet is showing bias at a much larger scale, much faster than what humans can—and potentially in ways that humans might not consider. For example, we see that some unexpected topics like climate disasters are informative to the models, but these are quite often not considered by human experts.”

“News outlets can be biased in many different ways, and some of them are currently not being considered sufficiently. We hope that changing this might help the field advance. We also try to provide some transparency into why a specific outlet is deemed to be biased, which is going to become increasingly important to ensure that the public can understand and trust the news they consume. Hopefully, this can, down the line, lead to a better-informed society and a healthier political environment.”

But there are some caveats to consider. “Right now, getting the output requires some technical expertise with programming and AI models,” Thelen-Rönnback noted. “We also relied on the left-right political split, but this might not be the most appropriate one worldwide. Nevertheless, our approach does easily allow for a different set of political labels, if need be.”

Looking ahead, the researchers hope to expand their system to classify other dimensions of media content. “Ideally, we would be able to expand this from political bias to other relevant aspects like disinformation or unreliability,” Thelen-Rönnback explained. “We are currently also looking at not just online news, but the internet more broadly. Specifically, we’re looking at how search engines react to polarizing topics, which is very exciting since search engines are nowadays the most trusted source of news.”

The study, “Automatic large-scale political bias detection of news outlets,” was authored by Ronja Rönnback, Chris Emmery, and Henry Brighton.