1111

Clara Shih, CEO of Salesforce AI, says, “In my career, I’ve never seen a technology get adopted this fast.”

That said, it’s getting harder and harder to tell which is human- and which is AI-generated content. And companies that develop Gen AI tools are aware of challenges and do their best to mitigate the risks. 

Take Synthesia, one of the leading AI video-generation platforms. Their technology creates digital versions of real humans and uses them to generate full videos from plain text in a matter of seconds.

The company enforces stringent safeguards around avatar creation and the use of these videos. They proactively stress test their systems under rigorous, independent read-team evaluations. And they’re highly invested in preventing deepfakes and damaging content from slipping through.

With this context, let’s take a closer look at the main challenges of content filtering in the gen AI era.

Harmful AI-Generated Content

AI is trained on human-made content. By definition, its whole purpose is to be as realistic as possible. Plus, when someone uses AI content to harm, they’ll intentionally design it to bypass filters. 

All these factors, combined with the volume and speed at which AI content is created, make it increasingly more complex to detect such instances. Hence, the growing number of resources designed to teach us how to tell what’s real and what's not.

Over- and Under-Enforcement

Over-enforcement occurs when a system has too strict thresholds or misinterprets sarcasm and harmless mentions as harmful. Wrongfully-sanctioned members become frustrated with their experience and may choose to step back (low engagement) or even step away from the community (high churn).

Under-enforcement occurs when a system frequently fails to detect high-quality harmful AI-generated content, coded words, and subtle threats, and under-blocks violations.

Members are exposed to this content, the platform suffers reputational damage and, in more serious cases, even legal risks.

None of these is desirable, and systems have to carefully balance their scores and thresholds to avoid model errors.

Missing Context

Imagine being banned from a gaming community for writing in the chat, “I’m gonna destroy you tonight.” That’s friendly banter that AI may not interpret as such, depending on the context it has. Often, factors like irony, sarcasm, cultural references, or in-jokes can make it harder to evaluate content in its broader context.  Without deeper contextual queues, AI moderation systems can easily misinterpret intent and either miss violations or flag non-existent ones.

Language Gaps

The less familiar a language is, the less training data that the model will have, and the less reliable its output will be.  When content includes terms from languages like Maori, Welsh, Icelandic, or Basque, AI can wrongly flag as dangerous even the most harmless idioms or common expressions of those languages. All because the model doesn’t understand the nuance, since it hasn’t been trained on data that included it.

Benefits of AI Content Moderation

Paradoxically, AI is both part of the problem and part of the solution to moderation challenges. The benefits of automating processes with artificial intelligence are vast and powerful.

For starters, AI gives speed and scalability to any moderation system. Since the system operates on clear rules, it’s more likely to return more accurate results (as opposed to human moderators, who might interpret the same rule differently depending on who’s reviewing).

Also, with AI doing the heavy lifting and handling most moderation, humans are under less pressure. They have more bandwidth and mental clarity to interpret the fewer but more challenging situations that the AI sends in for revision.

Once a system is implemented, costs become more predictive, and the process finds its groove. With faster, more reliable moderation, users have an improved experience, which will directly impact community engagement. 

DAU and retention naturally increase because members feel safe sharing and consuming content within the community.

UGC is an ecosystem that expands at lightning speed. In this ecosystem, AI moderation serves as the foundation that enables communities to scale in the safest possible ways.

Best Practices for AI Moderation

Success depends on whether you use a hybrid moderation system, how thoroughly you log decisions, and how often you audit and refine your system.

The following best practices will help you get there.

Measuring and Tuning AI Moderation Quality

Set-it-and-forget-it doesn’t apply to moderating with AI tools. You can’t let a system flag content without looking at how often it’s right (precision) and how much toxic content it correctly flags (recall). Or whether it scores false positives/negatives (over- or under-enforces).

Error analysis and threshold tuning are critical steps for improving the process.

Policy Design, Appeals, Compliance, and Governance

The moderation policy, along with the violation categories, is the bedrock. You must clearly define:

• What counts as a violation
• What severity levels will the system use to categorise violations
• What are the threshold rules (when to allow, escalate, or block)
• How should the system proceed finding a threshold

Any content decision has to be handled within a transparent appeal process. Moderating content involves operating with user data, which carries many legal responsibilities. The platform owner has to deal not just with building trust and transparency, but follow the regulations:

• GDPR privacy laws (data privacy and user consent)
• Online platform regulations, such as the EU Digital Services Act (platform accountability and risk mitigation)
• Safety requirements under the UK Online Safety Act (proactive detection and removal of illegal content)

The Future of AI Moderation

With Gen AI pervading online, moderation must focus on developing increasingly more complex and performative models that are:

• Multimodal: can evaluate various types of content simultaneously.
• Context-aware: can understand the whole picture, not simply isolated elements, by examining an intent history, considering cultural nuances, and situational clues.
• Agent-based: uses AI agents that can detect violations and act in real time.
• Cross-contextual: notice the safety signals and share them across digital platforms, preventing repeated violations.

Build Safer In-App Communities with Watchers

If you plan to build communities in-app, you need to protect your users. We offer community chats which are not only engaging but safe and trustful. We build 4-layer AI moderation system and constantly develop and improve it. Learn more and get to know how to build safe space for users, book a call with our team. 

FAQs About AI Moderation

What is AI moderation?

It’s an automated process of reviewing and managing UGC by leveraging ML, NLP, and computer vision. AI moderation detects in real time content that’s harmful or violates policies and can block it directly or dispatch it for human moderation.

How accurate is AI moderation?

Moderating with AI is highly accurate, although results vary by use case and false positives/negatives still occur (especially with nuanced content). Modern systems keep humans in the loop for ambiguous contexts or high-stakes decisions.

What are the problems with AI moderation?

• Challenges in comprehending nuances that leads to false positive or false negative decisions)
• Limited adaptability to slang, memes, and other conversations linguistics elements
• Weaker performance with rarer languages and dialects
• Difficulties in detecting evasion tactics, such as intentional misspellings
• Limited ability to indentify with deepfakes
• Reasons can be unclear

Is AI moderation better than human moderation?

Rather than “better”, AI screening is faster and more scalable. While AI can cover a significantly higher volume of content in real time, it can’t do it all, and it still relies on human review for the less clear contexts. In practice, the best results come from combining AI with human moderation.

References

• Social media content moderation and removal - statistics & facts | Statista
• Data Never Sleeps 12.0 | Domo
• Inside Facebook’s African Sweatshop | TIME
• Five machine learning types to know | IBM
• What is natural language processing (NLP)? | Tech Target
• What is computer vision? | Azure Microsoft
• Perspective API | Perspectiveapi.com
• Open AI Platform - Moderation | OpenAI
• Use of Natural Language Processing in Social Media Text Analysis | Research Gate
• Natural Language Processing for Messenger Platform | Facebook
• Build Natural Language Experiences | Wit.ai
• Vision AI | Google Cloud
• Amazon Rekognition | Amazon
• Our Approach To Responsible AI Innovation | Inside YouTube
• Synthesia’s Content Moderation Systems Withstand Rigorous NIST, Humane Intelligence Red Team Test | Synthesia Blog
• Fact check: How to spot AI-generated newscast | DCNews