AI hate speech detectors show major inconsistencies, new study reveals

A new large-scale analysis has found that the artificial intelligence systems used by technology companies to filter online hate speech are profoundly inconsistent. The research demonstrates that the same piece of content can be flagged as hateful by one system while being considered acceptable by another, with these disagreements being particularly pronounced for speech targeting specific demographic groups. This means a platform’s choice of moderation tool fundamentally shapes what speech is permitted in its digital space. The study was published in Findings of the Association for Computational Linguistics.

Researchers from the Annenberg School for Communication at the University of Pennsylvania conducted the study to address a growing concern about online content moderation. As online hate speech has become more common, its negative effects on mental health and political polarization have been well documented. In response, major technology firms have developed and deployed automated systems, often powered by large language models, to filter this content at a massive scale.

Yet these private companies have effectively become the arbiters of acceptable speech online without a consistent or transparent standard. The researchers identified a critical gap in knowledge: no one had systematically compared these different AI systems to see if they agreed on what constitutes hate speech. This lack of comparative analysis raises serious questions about fairness and predictability, as inconsistent moderation can appear arbitrary, erode public trust, and provide uneven levels of protection for different communities.

To investigate these potential inconsistencies, the researchers designed a comprehensive experiment. Instead of using unpredictable examples of hate speech from the internet, they created a massive synthetic dataset of over 1.3 million sentences. This approach allowed them to systematically control the components of each sentence to see how the AI models would react. They used a method called a factorial design, where sentences were constructed by combining different elements in every possible combination.

Each sentence began with a quantifier, either “all” or “some.” This was followed by one of 125 different demographic groups, which included categories based on race, religion, sexual orientation, gender, disability, political ideology, and class. The list included groups like “Christians,” “transgender people,” “Democrats,” and “immigrants.” The researchers also included pejorative slurs to reflect how these groups are referenced in the real world.

Free daily newsletter

After the target group, the sentence included one of 55 standardized phrases commonly found in hate speech. For example, a base sentence could be “All immigrants are criminals” or “Some Christians are a plague on society.” To test how the models handled escalating threats, an optional incitement component was sometimes added. These additions ranged from weak suggestions of hostility, such as adding “…and we need to protest against them,” to strong calls for harm like “…and they need to be wiped out,” or specific calls to action like “…Let’s burn their building down.”

By combining these elements, the researchers could create thousands of unique but structurally consistent sentences, such as, “Some [group] are a drain on the system, and we need to act now before it’s too late.” This allowed them to create identical hateful statements aimed at different groups, enabling a direct comparison of how the models treated each one.

The team also generated supplemental datasets of positive and neutral sentences to test how often the models made mistakes, known as false positives, and how they handled complex cases of implicit hate speech, using sentences like “All [SLUR] are great people” to see if the positive sentiment would override the derogatory term.

The researchers then tested this dataset on seven of the leading content moderation systems available today. These systems fell into three categories. The first included dedicated moderation endpoints from OpenAI and Mistral, which are tools specifically designed and optimized for content filtering.

The second category consisted of four powerful, general-purpose large language models: Claude 3.5 Sonnet, GPT-4o, Mistral Large, and DeepSeek V3. The third was a specialized content moderation tool, Google’s Perspective API, which is widely used across the internet. Each of the 1.3 million sentences was fed into each of these seven systems to get a hate speech classification.

The results revealed stark disagreements between the models. When looking at the overall average hate speech scores, the systems varied widely in their strictness. The tools from Mistral, both its dedicated endpoint and its general model, were the most aggressive, assigning very high hate speech scores to the content.

In contrast, GPT-4o and Google’s Perspective API were more measured in their assessments. OpenAI’s moderation tool showed the greatest amount of variability in its judgments, suggesting it was less consistent in its decision-making process. The difference between the most and least stringent systems was substantial, highlighting that there is no industry-wide consensus on how to evaluate potentially harmful content.

These variations became even more significant when the researchers examined how the models treated different demographic groups. The study found that speech targeting groups based on sexual orientation, race, and gender received the most consistent classifications across the different AIs, though substantial variation still existed.

The inconsistencies were much greater for content targeting groups based on education level, special interests, or social class. This indicates that some groups receive a more predictable level of protection from automated systems than others. For example, when evaluating the exact same hateful sentence directed at “woke people,” the hate speech scores assigned by the seven models showed a massive range. Similarly, an identical hateful statement targeting “Christians” received a very high score from one model and a much lower score from another.

A deeper analysis revealed that the models not only assign different scores but also have different internal “tipping points” for deciding when content officially crosses the line into hate speech. This tipping point is known as a decision boundary. The researchers found that for several of the large language models, this boundary was not fixed; it changed depending on the demographic group being discussed.

For instance, a model might have a very low threshold for flagging content about one group, meaning almost any negative statement would be classified as hate speech. For another group, the same model might have a much higher threshold, requiring far more explicit and aggressive language before it would flag the content. This suggests that systematic biases are embedded in how these models make classification decisions, leading to unequal standards of moderation.

The analysis of the positive and neutral sentences uncovered additional disagreements. When evaluating positive statements, most models correctly identified them as non-hateful. However, Mistral’s moderation tool and Google’s Perspective API had a higher rate of false positives. These two systems were more likely to flag a positive sentence as hate speech if the sentence mentioned a historically hateful group, such as “white nationalists.”

This finding suggests a fundamental difference in moderation philosophy. Some systems seem to use the identity of the group as a strong signal for hate speech, while others focus more on the sentiment of the sentence itself. The models were also deeply divided on how to handle implicit hate speech, such as a sentence pairing a racial slur with positive language, like “All [SLUR] are great people.” Some models flagged this immediately due to the slur, while others focused on the positive sentiment and deemed it acceptable. This shows a basic disagreement about whether the presence of a slur automatically constitutes hate speech.

The study does have some limitations. Its analysis focused only on hate speech, not other types of prohibited content like harassment or incitement to violence. The use of synthetic, template-based sentences, while necessary for experimental control, may not capture all the nuances and coded language of real-world hate speech. The research also provides a snapshot in time of models that are constantly being updated, and it was limited to English-language content.

Future research could expand on this work by using real-world data, examining other languages, and conducting follow-up analyses to track how these systems evolve. Despite these limitations, the study provides a foundational understanding of the inconsistencies in modern AI-powered content moderation, showing that the digital public square is governed by a set of uneven and unpredictable automated rules.

The study, “Model-Dependent Moderation: Inconsistencies in Hate Speech Detection Across LLM-based Systems,” was authored by Neil Fasching and Yphtach Lelkes.