AI-Driven Solutions for Illegal NSFW Content Moderation on Social Media Platforms

Content moderation is a complex process that employs a diverse range of techniques. These include keyword filters, database content hash matches, and machine learning-based approaches. In this article, we will explore these techniques in detail, covering both general approaches to content moderation and those used by major social media platforms such as Reddit, Twitter, Facebook, Instagram, Tumblr, Discord, and others.

Why AI Moderation of Illegal NSFW Content is Crucial for Social Media Platforms

Why does social media even care to moderate content? The critical takeaway is to retain users and advertisements. The moderation of hate speech and harassment protects users’ rights, consequently helping the brand and reputation. Meanwhile, moderation of more extreme topics like pornographic materials and violence could help avoid legal consequences, fines, and even being blocked by the country or platform.  When big platforms open doors for users to share content easily. Unfortunately, those platforms play a role in helping offenders spread CSAM across the web. 

Illegal NSFW (Not Safe For Work) content encompasses various forms of material that are not only inappropriate but also illegal. This includes, but is not limited to

1. Child Sexual Abuse Material (CSAM)
2. Terrorist propaganda
3. Revenge pornography (non-consensual sharing of intimate images)
4. Animal abuse videos
5. Illegal drug-related content
6. Cyberbullying leading to severe psychological harm
7. Hate speech that incites violence or discrimination

These categories involve exploitation, criminal activities, and significant harm to individuals and society, necessitating stringent moderation and legal action. Therefore, in terms of moderation, illegal NSFW must be a top priority. However, this article will focus primarily on the moderation of CSAM and terrorist propaganda.

To combat such material sharing, platforms are blocked by marketplaces, countries restrict access, and new laws are developed to force platforms to take action. You definitely don’t want to see articles about your platform helping and promoting paedophile networks, like it happened once to Instagram. Their recommendation algorithm assisted in connecting paedophiles and guiding them to content sellers! [1]

Another notable example happened to Tumblr in 2018 when the marketplace forced a social media network to act. Apple decided to ban Tumblr for all Apple devices, as Tumblr users spread child sexual abuse material (CSAM). To remind the reader, Apple sells almost half of all phones per year worldwide, therefore, a ban from Apple stores is nearly a death sentence for an app. This incident was a catalyst for the implementation of AI-automated moderation in Tumblr, a promising step towards preventing the spread of harmful content, in an attempt to save the platform from ban. [2]

The significant financial risk associated with poor moderation is when brands leave the platform and remove their ads. To remind, social media platforms derive a substantial portion of their revenue from advertisements, and ineffective moderation may cut off this main income stream. Brands don’t want to be promoted next to inappropriate content, as it damages the brand’s reputation. The Figure below provides a clear example of undesirable media for advertisements on the YouTube platform in 2017, highlighting the financial and reputational consequences of inadequate moderation.

Figure 1 – An example of brand damage is through advertisements on a platform with poor content moderation. Luxury holiday operator Sandals Resort’s promotion appeared next to a video for Al-Shabaab, the al-Qaeda-affiliated group active in Somalia [3]

Another famous case happened when Elon Musk relaxed Twitter’s content moderation policy in 2022. On the way to improve freedom of speech, such changes opened a door for (1) hate speech against women, immigrants, and minorities, (2) social bots that infiltrate political processes, (3) mis- and disinformation and (4) viral challenges that put teens’ lives at risk [4]. Consequently, the following companies stopped advertising on the platform: Disney, Apple, Paramount, Comcast, Lionsgate, NBCUniversal, Warner Bros. Discovery, IBM, Fox Sports, Axios, and TechCrunch! [5, 6]

Content moderation is a crucial aspect of social media platforms, with decisions highly dependent on each platform’s policy. For instance, The New York Times might prohibit personal attacks, obscenity, or vulgarity in its comments section [7], while these might be permissible on Reddit and Twitter. The decision to moderate or label a topic as NSFW differs significantly from one platform to another, reflecting each platform’s unique policies and community standards. However, all platforms share a common definition of illegal content and collaborate to combat its dissemination. In the following sections, we will focus exclusively on two critical areas of illegal content moderation: CSAM and terrorism.

Figure 2 – A table illustrating different types of harmful content [8]

Here are some statistics to point out the importance of moderation. Twitter removed in the first half of 2021 5.9 million pieces for violating Twitter rules [9]. The figures below demonstrate the volume of inappropriate content found and removed in just 3 months period in 2019.  Data is taken from Facebook’s fourth Community Standards Enforcement Report 2019. [10]

• Facebook and Instagram removed 11.6 million and 754,000 pieces of content related to child nudity and child sexual exploitation.
• Facebook and Instagram removed 2.5 million and 845,000 pieces of content related to suicide and self-harm.
• Facebook and Instagram removed 4.4 million and 1.5 million pieces of drug-sales content.
• Facebook and Instagram removed 2.3 million and 58,600 from pieces of firearm-sales content. [10]

In addition, below, we provide a table with statistics taken from the Twitter Transparency Report 2023-2024 [9]. From the table, it is clear that some moderation topics could significantly benefit from automated moderation (Manipulations, Child Sexual Exploitation), while there still will be topics with the necessity of manual monitoring (Non-Consensual Nudity, Abuse & Harassment, Suicide & Self Harm). That could be explained by the high false-positive risks AI might have in dealing with abuse or self-harming posts. Therefore, automation of high-priority and high-volume topics like Manipulations and Child Abuse, human moderators could reduce their workload to allocate more time to more complex post-harassment and self-harm.

Table 1 – Statistics on actions taken on accounts for policy rules violation on Twitter 21/10/23 to 31/3/24 [9]

AI-Powered Strategies for Automated Illegal NSFW Content Moderation 

How does AI help?

In the digital age, where user-generated content is uploaded every second, maintaining a safe and respectful online environment is a monumental task. Automated AI moderation emerges as a critical tool for social platforms to manage and filter Not Safe For Work (NSFW) content effectively. This section delves into how AI, by assisting in moderating such content, ensures platforms remain not only safe but also user-friendly and legally compliant.

How does moderation in social media platforms work? There are several types of moderation, and usually they all work together:

• Pre-moderation involves reviewing user-generated content before it’s published, ensuring compliance with guidelines. 
• Post-moderation entails reviewing content after publication, allowing immediate posting with subsequent moderation. 
• Reactive moderation involves responding to user-reported content and addressing violations after they’re flagged. [11]

In fact, there are more types, such as hybrid moderation or distributed moderation, but the three types above are the most common. See our article on how the user-only moderation approach failed for Reddit [12].

How does AI contribute to the automation of content moderation? In essence, AI acts as a support system for moderators. The core of moderation is a team of moderators who are responsible for blocking material that violates the platform’s policy. The major social media platforms employ large content moderation teams, estimated back in year 2022 to be 15,000 (Facebook), 10,000 (YouTube), and 1,500 (Twitter), each moderator addressing between 600 and 800 claims daily. [4] To compare, during August 2022, on average, around 6,000 tweets were posted per second. [13]

The content moderation workflow typically combines pre-, post-, and reactive moderation approaches assisted with AI. The figure below demonstrates a typical workflow based on information gathered during interviews conducted with content moderation solution providers, social networks, and content-sharing platforms.

Figure 3 – The content moderation workflow combines automated systems and human moderators for pre-, post- and reactive moderation [8]

The primary function of AI in moderation is to alleviate the burden on human moderators, providing them with much-needed relief. This can be achieved in various ways. 

• First of all, AI plays a crucial role in pre-moderation by efficiently filtering posts before they are published. It only alerts moderators if suspicious content is detected, thereby streamlining the process and allowing moderators to focus on a smaller fraction of posts.
• Secondly, AI could block many known illegal materials without human interaction (moderation). That would significantly reduce the load, blocking spammers who share known illegal materials. 
• Finally, AI could mark published content as suspicious (post-moderation), helping moderators find and block inappropriate content before it is exposed to many eyes.

The key takeaway is that AI is not a replacement for human moderators; it’s a tool to assist them. The moderation system would collapse without human moderators. A clear example of this was seen in 2020 when Facebook decided to close its moderation offices due to the COVID-19 pandemic. This led to an increase in child pornography, news articles marked as spam, and users unable to appeal moderation decisions.  Later, Facebook, YouTube, and Twitter acknowledged that humans are the most important line of defence in content moderation.

Top AI Tools for Combating Illegal Content: CSAM & Terrorism 

In the realm of social media moderation, platforms often opt to leverage existing AI tools rather than develop bespoke solutions. This approach allows them to implement robust systems capable of quickly handling various types of content at scale. Given the diverse nature of inappropriate material online—ranging from pornography and illegal content to hate speech and bullying—different tools are tailored to address specific types of content effectively. Below, we explore the top AI tools used to combat these issues across major platforms.

Among all NSFW topics for moderation, illegal content demands the highest priority as it harms users and breaches laws and marketplace policies. In this subsection, we will discuss AI tools to deal with the most common illegal media files – depictions of terrorism, terrorist recruitment videos, violent terrorist imagery and child sexual abuse material (CSAM).

Notice how the Figure below demonstrates several reports of CSAM material and where they were found: Facebook, Instagram, Google, and WhatsApp are top leaders [14]. According to  CSAM reports to NCMEC in 2020, Meta and all associated platforms sent over 20 million CSAM reports, which takes around 93% of all reports made by all platforms worldwide to NCMEC [15]. This means that Meta has the highest need for good AI tools against illegal posts, and their decisions must be of high value as an excellent example for others.

Figure 4 – Number of pieces of Child Sexual Abuse Material (CSAM) content reported on online services in 2022 [14]

Generally, any moderation task could be seen as a classification problem to classes like “legal” / “illegal”. The most typical AI approach for content classification is based on machine learning algorithms, which learn to recognize specific features from training data. That is where complexity of the problem appears – collection and use of datasets with illegal material poses significant ethical and legal challenges. It is both illegal and unethical to collect or distribute such materials, making it difficult to assemble datasets needed for training effective detection systems.

Overview of AI Solutions for Illegal NSFW Content Moderation

Two popular solutions are hashing and Computer Vision AI. In the first case, when a violative post is blocked, it is hashed and stored in the database of known illegal posts so that the hashing algorithm could match a copy of such post and block it without a person reviewing it [16]. Hashing ensures that once illegal content is identified and flagged, it cannot be re-uploaded in the same form. When new content is uploaded, it is hashed and compared against the database of known illegal content. If a match is found, the content is automatically blocked or flagged for review. This approach is highly efficient for identifying and blocking duplicate content but may struggle with modified or slightly altered versions of the original illegal material.

Computer Vision, a broader field within artificial intelligence, focuses on enabling machines to interpret and make decisions based on visual data. Within this field, Convolutional Neural Networks (CNNs) have become a cornerstone, especially for recognizing and classifying images. CNNs learn from vast amounts of data, and with enough data provided, they can effectively identify specific objects or patterns in images. This makes CNNs particularly suitable for detecting illegal content such as CSAM and violent imagery. They can be trained on large datasets of legal content to learn to differentiate between legal and illegal images based on subtle differences. Once trained, these models can scan new images for these patterns, flagging those that appear suspicious for further human review. However, a significant challenge is the availability of data for training these models, especially for CSAM. Due to ethical and legal constraints, assembling comprehensive training datasets for such content is not feasible, which raises questions about the overall efficiency of using machine learning computer vision for this purpose. 

Effective Hash-Based Algorithms for Detecting Illegal NSFW Content: See without looking 

Data is one of the biggest challenges when it comes to illegal content moderation. To build software, you need training data for any Machine learning-based algorithms or at least some test data for any other algorithm to check metrics. However, storing and using datasets with illegal content raises ethical issues and violates the law. Therefore, a method was found, “to see without looking.”

Hashing algorithms, with their simplicity and high efficiency, are powerful tools for automatically removing known illegal content. They store known unlawful content in a hashed form and then apply the same hashing algorithm to posts on social media to check for a match. This process is similar to how websites verify your password when you log in by comparing your hashed password with the stored one.

Perceptual Hashing is a special branch of hashing algorithms in which a slight change in input data doesn’t dramatically change the hash. This specificity makes perceptual hash flexible enough to find a match even with a modified copy of illegal material.

Speaking of efficiency, Google claimed that among all CSAM reports, 90% of posts were matched with hash-based algorithms to a known material from the NCMEC database. [17] Furthermore, hash-based algorithms operate without the need for supervision. If the content is certainly found in the database, it can be automatically blocked. This demonstrates the significant power that hash-based algorithms bring to automatic moderation. The following sections will describe the most popular solutions and their limitations.

According to the National Center for Missing and Exploited Children (NCMEC) Transparency Report 2023 [18], once CSAM is published online, it may circulate and continue to be distributed thousands and tens of thousands of times a year. However, these distributions could be effectively halted with the use of Hash-based algorithms, thereby combating the widespread redistribution of CSAM copies.

PhotoDNA: Microsoft’s Advanced AI Hashing for Illegal Content Identification

Microsoft introduced the pioneering PhotoDNA algorithm in 2009, a solution that remains a cornerstone in the industry today. This robust hashing algorithm, now available as a SaaS through the Azure marketplace, is a testament to its enduring popularity and reliability. Over 200 organisations, including tech giants like Bing, One Drive, Google’s Gmail, Twitter, Facebook, Adobe Systems, Reddit, and Discord, trust and utilise this solution, highlighting its widespread usage and importance. [19]

They developed an algorithm to capture a “fingerprint” of an image, which is a kind of Perceptual hash that they claim* to be irreversible and robust to picture editions. Any standard hash algorithm like MD5 or SHA would be changed dramatically after a slight change of an image, which creates a need for a more robust hashing algorithm. In the case of Perceptual hashing, it is locally sensitive, which is more a feature-oriented matching method than an exact image matching method. The same type of hashing is used in the Google image search engines!

The PhotoDNA algorithm is based on splitting an image into overlapping 6×6 blocks and calculating horizontal and vertical gradients in two directions, so-called Sobel gradients. Each character of the hash represents encoded 4 numbers – for each element of the Sobel gradient with primary image normalisation and equalisation. Such HASH is claimed* irreversible, while insignificant colour edition wouldn’t change gradients at all, and a small change of one picture fragment would change only a fragment of HASH, enabling the calculation of a distance between hashes to match copies of the same image that were slightly edited.

Figure 5 – PhotoDNA step by step algorithm process

Here is how this algorithm helped to block known illegal content without humans in the loop. They found that datasets of illegal pictures raise legal concerns, but it is fine to store irreversible hashes. Therefore, Microsoft worked together with The UK Internet Watch Foundation (IWF) and the National Center for Missing & Exploited Children (NCMEC) to collect a hash database of over 300,000 units of CSAM material. In 2023, database records exceeded 10 million hashes and continue to grow, showing promising progress in our fight against illegal content. [20]

In addition, since 2019, PhotoDNA has added hashes from the Global Internet Forum to Counter Terrorism(GIFCT) to automatically remove terrorism-related content. The GIFCT is a publicly shared hash database founded by big tech companies like Facebook, Google, YouTube, Microsoft, and Twitter, which aims to fight against ISIS-related material, extremist material, and other terrorist-related content. This automatic removal of such content demonstrates the algorithm’s effectiveness in combating terrorism. For example, back in 2018, PhotoDNA assisted Facebook with the automatic ban of al-Qaeda videos spreading on the platform. [21]

According to the Microsoft Digital Safety Content Report (DSCR) 2023, we have the following numbers. Microsoft has reported 294,426 units of suspicious files indexed with Bing Search Engine (opposed to 1,079,246 in 2020) [22]. However, According to the NCMEC Transparency report in 2023, PhotoDNA and similar hash technologies sent over 22.3 million CyberTipline reports! [18]

While these numbers highlight the impressive scale at which PhotoDNA operates, they also underscore the importance of continuous improvement in AI moderation tools. Despite the high number of reports, the effectiveness of PhotoDNA depends on the accuracy of identifying illegal content. Ensuring minimal false positives and negatives remains a significant challenge, requiring ongoing advancements in AI technology and human oversight to maintain the balance between automated efficiency and ethical responsibility.

Switching to limitations related to PhotoDNA, the biggest one is that it can only block known content taken from a dataset. Therefore, the algorithm’s lack of generalisation makes it useless against any kind of new material. PhotoDNA can only help against the spreading and spamming of known material but will ignore new content.

Unfortunately, due to leaks of the executable, an algorithm was reverse-engineered, and algorithm vulnerabilities on how to avoid an algorithm match were found. Hacker showed how a slight crop or resize of more than 2% of the image changes the hash significantly enough for an algorithm to mismatch the generated hash. The same mismatch could be achieved when modifying a few pixels where 6×6 blocks are overlapped – a one-pixel change will shift several hash characters. Also, colour editions that change gradients can mislead PhotoDNA.  [23]

Moreover, as hash stores gradients of each 6×6 block, it was found that information on the “texture” of the 6×6 block is stored, and we could restore a blurred version of the photo, which raises a question regarding irreversibility and privacy. For example, patterns 1,8,1,1 would have several possible combinations of 6×6 batches, not a single correct solution. However, all of the possible combinations would be visually similar, making it possible to estimate how it could have looked. Eventually, image recreation was achieved to a certain extent with a neural network decoding hash developed by Anish Athalye. [24] 

Figure 6 – Demonstration of image quality that could be restored from PhotoDNA hash using Neural Network developed by Anish Athalye [24] 

To summarise, PhotoDNA is a great tool available for free to many businesses. The most popular big tech platforms use it to block vast amounts of illegal content automatically without a human in the loop. The algorithm checks a post to match one of the known unlawful materials, and it has over 10 million unique illegal content materials saved as a hash in the database. Reportedly, PhotoDNA blocks 99.2% of CSAM content. However, the statistical derivations by Microsoft are rather unclear.  Unfortunately, as the algorithm heavily relies on data matching with database samples, it can only block known material and is ineffective. On top of that, hackers reverse-engineered and studied an algorithm, revealing a set of simple content modifications that could be applied to prevent illegal material from being matched. Nevertheless, PhotoDNA is still one of the most popular and powerful tools for automatically moderating CSAM and terrorist-related content.

NeuralHash: Apple’s AI Solution for Illegal Content Detection

Another Perceptual hashing algorithm was proposed by Apple in 2021. They found a balance between user privacy and fighting against CSAM content. Instead of sending users’ images to the cloud, like in the case of PhotoDNA, Apple developed an algorithm to check for CSAM on their devices. Their algorithm is based on neural network feature extractors, representing images as real-valued vectors and Hyperplane LSH (Locality Sensitivity Hashing) process converting descriptors to unique hash values. The convolution neural network was trained on pairs of augmented data, so it is trained to be robust and give the same values for rotated/reshaped/transposed images. However, as you will see later, it still didn’t help against image modifications to avoid the matching algorithm. [25]

Figure 7 – NeuralHash pipeline [26]

NeuralHash has the same dataset providers as PhotoDNA and uses data similarly. NCMEC provides them with illegal content they allow to store as a hash produced by NeuralHash. Then, this table of hashes is “blinded”, encrypted and stored on Apple’s devices. The underlying encryption prevents users from accessing a hash dataset. In case of a hash match – a media file is marked without letting the user know. When a threshold of labels is reached, these media are sent for final manual evaluation before blocking users or removing material. [25]

Figure 8 – Demonstration on how NeuralHash works on device [27]

Even Though NeuralHash showed a next-level algorithmic upgrade to the concept provided by PhotoDNA, it suffers from the same issues: it is relatively easy to tweak an image to mislead the matching algorithm. Lim Swee Kiat deployed an online demo where you can upload two images or flip/rotate an image and check their hash values by NeuralHash – https://greentfrapp.github.io/compute-your-own-neuralhash/.

Check out the recent study “Learning to Break Deep Perceptual Hashing: The Use Case NeuralHash,” which provides an illustrative analysis of attacks on the algorithm. In other words, it shows ways to make the algorithm fail. Their methodologies include Hash Collision Attacks, Gradient-Based Evasion Attacks, Gradient-Free Evasion Attacks, and Hash Information Extraction.

Overall, NeuralHash shows the next step in the development of PhotoDNA’s approach. However, even with such a significant technological advancement to using Convolution Neural Networks with Hyperplane LSH instead of generating a hash from gradients of overlapping 6×6 blocks, the algorithm is still pretty much vulnerable to attacks of image modification.

Facebook’s PDQ: Perceptual Hashing Technology for Illegal Content Detection

It was mentioned before in Figure A that Facebook is leading in CSAM content reports. As part of its continued efforts to combat CSAM, terrorism-related content and graphic violence, Facebook developed the PDQ hashing technology, which serves as a robust tool for detecting and blocking known illegal images. Introduced and deployed in 2018, PDQ, or Perceptual Duplicate Quantizer, is an image hashing algorithm designed to identify visually similar images, even if they have been slightly altered. Facebook has integrated PDQ across all its platforms, including Instagram, WhatsApp and Oculus.

Like PhotoDNA, PDQ generates a “fingerprint” or hash of an image, which can then be compared to a database of hashes from known illegal content. However, PDQ improves upon earlier hashing technologies by focusing on the perceptual aspects of the image, thereby enhancing its ability to detect duplicates despite typical image transformations such as resizing, cropping, or minor colour adjustments. [16]

Facebook generously provides an open-source hashing algorithm on the GitHub ThreatExchange repository [in-text link: https://github.com/facebook/ThreatExchange]. Moreover, any newly detected terrorism-related content is submitted to the GIFCT platform and opened to any platform as an open database. This platform is used by companies such as Amazon, Discord, Dropbox, Instagram, Mega, Pinterest, Tumblr, WhatsApp, and many more. [28]

The next year, an algorithm was published, and it was tested by the Artificial Intelligence for Law Enforcement & Community Safety Lab. They used 225,887 images marked as child exploitation materials and 918,490 images of legal pornography, resulting in a dataset with a total of over 1.1M records. Images were augmented with the most common tactics that showed their efficiency against other hashing algorithms: format change, watermarks, overlay text, thumbnail size reduction, cropping and rotation. [29]

The algorithm showed robust performance on formatted content, overlaid text and thumbnail size reduction as far as reducing to 64×64 pixels, where any further size reduction would cause an algorithm to fail. It showed mere results with watermarks – around half of illegal content was detected even with the presence of watermarks. However, cropping and rotations are still attacks that PDQ hashing algorithms struggle with. Cropping of more than 5% and rotation by more than 5 degrees leads to a guaranteed mismatch. [29]

In conclusion, Facebook’s PDQ algorithm exemplifies the latest advancements in perceptual hashing technology, offering a scalable, effective, and adaptable solution for moderating harmful content across multiple platforms. Its integration with ThreatExchange highlights a proactive approach to security, emphasising collaboration and sharing of threat intelligence among a wide range of stakeholders. However, while PDQ is more robust to attacks than PhotoDNA, it is still vulnerable to cropping and rotation.

Machine Learning algorithms for Illegal NSFW Content Detection

Machine Learning (ML) algorithms are the heart and brain of Artificial Intelligence (AI) solutions nowadays. They are highly dependent on data, and that causes lots of problems in building robust ML algorithms to moderate illegal content. Generally, hash-based algorithms only struggle with efficient copy modifications, and new content and machine learning algorithms could help with these two. Moreover, if the CSAM spreading of new content could be stopped before it was massively shared in paedophile networks, the mass distribution of its copies would be prevented! However, despite all the advances in machine learning and technical innovation in recent years, a fully automatic tool that can distinguish illegal content from other content is not feasible.  [30]

Building a machine learning algorithm to detect child pornography is a noble goal aimed at protecting children and preventing exploitation. However, the process of collecting, storing, or distributing child pornography, even to develop detection systems, is illegal and carries severe legal consequences. Datasets with child pornographic content are not available since the possession of this material is illegal.

One of the possible solutions to build an ML algorithm is to combine age detection, child detection, nudity detection and other feature-oriented classifiers together to achieve a child porn classifier without exposing the algorithm to actual child pornography. Unfortunately, such solutions wouldn’t work for close-up recordings. Still, this approach could assist in automatic CSAM removal.

US Federal Law (18 USC § 2251; 18 USC § 2252; 18 USC § 2252A) [31], along with the European Union Directive 2011/93/EU [32], strictly prohibits the possession of child pornography. Similarly, the UK’s Terrorism Act 2000, Section 58, makes it illegal to collect or store terrorism-related content [33]. These stringent regulations are mirrored in most countries worldwide. Consequently, hash-based algorithms are often preferred for content moderation because they allow the storage of illegal materials in an irreversible hash form, thereby avoiding legal repercussions associated with handling explicit content directly.

In 2023, a significant incident highlighted the critical importance of CSAM moderation and issues with collection of illegal materials for AI training purposes. Stanford’s Internet Observatory discovered that the LAION-5B dataset  included more than 1,679 illegal images scraped from social media posts and popular adult websites. This dataset was notably utilized by Stability AI to develop the Stable Diffusion model, a popular AI image generator. Eventually, both Stable Diffusion and LAION faced significant consequences. Notably, researchers who investigated the presence of CSAM in the database used PhotoDNA and its alternatives! [34, 35, 36] 

LAION had to temporarily remove their dataset from public access and implement regular maintenance procedures to filter out any unlawful content. Similarly, Stability AI, which used LAION-5B to train its Stable Diffusion model, faced reputational damage and increased scrutiny. They had to emphasise their commitment to filtering and ethical AI practices, and they distanced themselves from the specific version of the dataset involved, highlighting the critical importance of robust content moderation. [34, 35, 36] 

To conclude, the challenge ML developers face is the inherent difficulty of developing data-dependent AI algorithms when data collection is both illegal and unethical. Although machine learning offers promising solutions for content moderation, the necessity of using real data for training purposes conflicts with legal and ethical standards. Consequently, the industry must rely on indirect methods, such as combining multiple classifiers or leveraging non-sensitive data, to approximate the detection of illegal content. Despite these challenges, advancements in machine learning continue to play a crucial role in enhancing the efficacy of content moderation, ensuring safer online environments.

Google’s Content Safety API: AI for Child Sexual Abuse Material Detection

Google and its affiliated platforms, like YouTube, hold the top-2 position in the volume of CSAM reports right after Meta and their Instagram, Facebook, and WhatsApp accounts. To improve its moderation techniques, Google is using the CSAI Match hash-based matching algorithm with the NCMEC database and empowering its automatic moderation with machine learning models. [17]

“We rely on two equally important technologies to help us proactively identify child sexual abuse material: hash matching and artificial intelligence”

(C)Susan Jasper. VP, Trust and Safety Operations at Google

In 2018, Google deployed a Content Safety API, a Neural Network-based classifier available for free to non-governmental organisations (NGOs). The tool’s main customers are Google and YouTube, but Yahoo, Adobe, and some other companies also use it. Judging from the number of customers, the solution is clearly less popular than PhotoDNA. However, in terms of scale, the Content Safety API was used by our partners to classify more than 2 billion images in 2020. [37]

While the tool is said to generalize from training data and identify previously unseen CSAM posts, [38] It still requires human intervention to review flagged content and prevent false positives. However, Google has not been transparent about the technical details of its implementation, such as the training process, the specific task the AI is solving, or the Neural Network architecture, raising questions about the trustworthiness of the Content Safety API.

Unfortunately, no metrics or statistics are provided to judge its efficiency. However, according to YouTube reports, in Q2 of 2023, 93% of policy-violative videos (including any abusive content, not just CSAM) removed from YouTube were first detected automatically [39]. However, they don’t mention what fraction was flagged by the Hash-matching algorithm and ML-based algorithm. Moreover, previously, Google claimed that 90% of CSAM posts were matched with hash-based algorithms [17]. That could indicate how inefficient current ML-based approaches are, or Google’s Content Safety API, in particular.

We can assume that as Google Safety API doesn’t hold leading positions on such CSAM benchmarks as RCPD [40], its performance is probably far from a SOTA method and, therefore, Google is not so open about the metrics and technical details.

Below, you can see the Table that demonstrates the number of CSAM reports made by YouTube. Keep in mind that around 90% of them were made automatically with AI tools Content Safety API + CSAI Match + other 3rd party tools. It is interesting to note the exponential growth over the years and the sudden recent drop instead of doubling the number. The cause of it is unknown – either more users are banned from the platform, or nasty users just found better practices to avoid detection! The spike in 2021-2022 could also be explained by AIG-CSAM (AI-generated CSAM) that enabled nasty users to create CSAM at scale and was stopped within some time by such regulations as Safety Principles in AI lifecycle [41, 42] and others. 

Table 2 – Statistics of Youtube reports to NCMEC regarding CSAM-related posts on the platform [43]

Thorn’s Safer Essential: Comprehensive AI for Illegal Content Moderation

Another combination of the Hash algorithm and ML-based solution was proposed by an international anti-human trafficking organisation primarily focusing on child abuse – Thorn. They started with Safer in 2019 and just in 2023, they published a new tool – Safer Essential. The list of their clients includes Flickr, where AI classified over 2,000 previously unseen CSAM and VSCO, where they reported over 35k CSAM files. [44] In addition, Thorn is currently beta-testing its text-based solutions with platform X (Twitter) to stop child abuse even before CSAM is shared. [45]

The product consists of two parts. The first is a perceptual hashing technology, Safer Match, that finds known CSAM and has an astonishing hash dataset of 57 million records stored in Amazon S3 and RDS! The second is an AI model that generates a risk score to assist in making human decisions more efficiently – Safer Predict– and it supports the moderation of previously unseen CSAM [46]. Their product is not free but available as SaaS on AWS [47]. Regarding prices, companies would have to pay in the range of 30k – 667k$ to use Safer for a year, with limits ranging from 1 million – 1 billion queries per month.

Talking about scale and efficiency, since 2019, it has checked more than 130 billion files, and around 5 million of them have been flagged as potential CSAM! [48]

Not much is known about the technical implementations of Safer Predict, and its training process is also kept private. However, even if we assume the algorithm has poor generalisation, that AI still could be an effective method for better detection of modified copies, in case it was trained on known CSAM.

Key Takeaways on AI Tools for Illegal Content Moderation

To summarise the section on AI tools for illegal content moderation:

• There are two general approaches in the automatic moderation of illegal content: Hash matching algorithms and Machine Learning AI
• Challenges to building ML model occur due to the law restrictions on the collection, storage and usage of illegal materials like terrorist content and CSAM
• Hash-based algorithms can match distributions of known illegal content. However, bad actors may easily avoid hash algorithms with simple attacks like cropping, resizing and other more comprehensive techniques.
• According to NCMEC, when CSAM is published, it gets re-distributed up to tens of thousands of times per year, and this distribution could be stopped with Hash-based algorithms.
• To prevent the massive distribution of new CSAM, a Machine Learning algorithm must detect previously unseen content, which is currently a developing field.
• Analysis of existing ML solutions shows that there is a gap to be filled: 
  • need in a robust ML solution, 
  • need in available training datasets, 
  • need in evaluation benchmarks to compare solutions
• We recommend using PhotoDNA and Facebook PDQ for their free-of-charge options to detect known illegal materials.
• We recommend adding Thorn’s Safer Essential as another layer of protection with both hash-based and ML-based algorithms to fight seen and unseen illegal materials.

Conclusions on AI-Driven Illegal NSFW Content Moderation Solutions  

To conclude the article, let’s walk through key points. First of all, illegal NSFW moderation is necessary for Social Media Platforms as poor moderation could completely kill social media platforms: (1) financially through business ads being removed, (2) legally through banning platforms in the country or app store, (3) reputationally through users abandoning your platform and switching to another.

The second key point underscores the indispensable role of the human-moderation team at the heart of the moderation process. Automated solutions are designed to support and assist these crucial individuals. It’s important to remember that terrorism-related content and CSAM not only harm users but also take a toll on the mental health of moderators, leading to stress, insomnia, trauma, and even PTSD [49]. The added pressure of dealing with highly disturbing content, which must be removed as soon as possible, underscores the need for AI-supported illegal content moderation.

The third point is that AI could be embedded in the moderation pipeline in a pre-moderation stage, filtering out harmful content automatically even before it gets published. One of the popular solutions to make an effective pre-moderation without a human in the loop is to apply a Hash-based algorithm with a database of known illegal content so that if the post matches the known illegal material, it could be blocked straightaway. We recommend starting with free-of-charge PhotoDNA and Facebook’s PDQ algorithms. However, you must be ready for bad actors to modify the materials in a way to avoid hash-matching. Also, such algorithms are incapable of detecting new material.

Finally, it’s important to acknowledge the challenges in developing AI-driven solutions. While Machine Learning-based algorithms are being developed to fight against previously unseen materials, it’s crucial to recognize the hurdles posed by law restrictions and ethical issues regarding data collection and possession. Despite these challenges, the potential of AI to prevent massive distribution by detecting and blocking harmful posts before they’re taken by the network of bad actors is a promising development. We recommend starting with Thorn’s Safer Essential to embed AI-driven CSAM detection.

References

[1] J. Vincent, “Instagram’s recommendation algorithms are promoting pedophile networks,” The Verge, Jun. 07, 2023. https://www.theverge.com/2023/6/7/23752192/instagrams-recommendation-algorithms-promote-pedophile-networks-investigation (accessed Jul. 25, 2024).

[2] “Tumblr removed from Apple app store over abuse images,” www.bbc.com, Nov. 20, 2018. Available: https://www.bbc.com/news/technology-46275138

[3] L. Fruen “Google faces questions after it’s revealed top brands unintentionally ‘fund terrorism’ when ads appear on extremist YouTube videos,” The Sun, Feb. 09, 2017. https://www.thesun.co.uk/news/2826296/google-questions-brands-adverts-extremist-youtube-videos/ (accessed Jul. 25, 2024).

‌[4] P. Schneider and Marian-Andrei Rizoiu, “The effectiveness of moderating harmful online content,” Proceedings of the National Academy of Sciences of the United States of America, vol. 120, no. 34, Aug. 2023, doi: https://doi.org/10.1073/pnas.2307360120.

[5] L. Madio and M. Quinn, “Content Moderation and Advertising in Social Media Platforms,” Social Science Research Network, Jan. 2024, doi: https://doi.org/10.2139/ssrn.4875546.

‌[6] F. Social, “It’s time to stop advertising on Twitter,” Future Social. https://futuresocial.beehiiv.com/p/time-stop-advertising-twitter (accessed Jul. 25, 2024).

[7] “The Comments Section – Help,” The New York Times. Available: https://help.nytimes.com/hc/en-us/articles/115014792387-The-Comments-Section

‌[8] “USE OF AI IN ONLINE CONTENT MODERATION 2019 REPORT PRODUCED ON BEHALF OF.” Accessed: Jul. 25, 2024. [Online]. Available: https://www.ofcom.org.uk/siteassets/resources/documents/research-and-data/online-research/other/cambridge-consultants-ai-content-moderation.pdf?v=324081

‌[9] X.com, 2024. https://transparency.x.com/dsa-transparency-report.html

[10]  “Community Standards Enforcement Report, November 2019 Edition,” About Facebook, Nov. 13, 2019. https://about.fb.com/news/2019/11/community-standards-enforcement-report-nov-2019/

[11] P. Mishra, “Social Media Moderation: How It Works and Importance,” Socialwalls Blog, Nov. 02, 2023. https://socialwalls.com/blog/social-media-moderation/

[12] N. Krasnytskyi, “Social Media Content Moderation: Best Practices for Balancing Freedom and Censorship – Trembit,” Trembit – webrtc streaming video blog, Jul. 29, 2024. https://trembit.com/blog/social-media-content-moderation-best-practices-for-balancing-freedom-and-censorship/ (accessed Jul. 30, 2024).

‌[13] D. Sayce, “Number of tweets per day?,” David Sayce, Mar. 03, 2010. https://www.dsayce.com/social-media/tweets-day/

[14] S. J. Dixon, “Number of pieces of Child Sexual Abuse Material (CSAM) content reported on online services in 2022,” Statista, Jan. 01, 2024. https://www.statista.com/statistics/1448957/pieces-csam-content-reported-online-platforms/ (accessed Jul. 25, 2024).

[15] C. Teunissen and S. Napier, “Child sexual abuse material and end-to-end encryption on social media platforms: An overview,” 2022. Available: https://www.aic.gov.au/sites/default/files/2022-07/ti653_csam_and_end-to-end_encryption_on_social_media_platforms.pdf

[16] “Open-Sourcing Photo- and Video-Matching Technology to Make the Internet Safer,” Meta, Aug. 01, 2019. https://about.fb.com/news/2019/08/open-source-photo-video-matching/

[17] S. Jasper, “How we detect, remove and report child sexual abuse material,” Safety & Security , Oct. 28, 2022. https://blog.google/technology/safety-security/how-we-detect-remove-and-report-child-sexual-abuse-material (accessed Jul. 25, 2024).

‌[18] “CY 2023 Report to the Committees on Appropriations National Center for Missing and Exploited Children (NCMEC) Transparency,” 2023. Accessed: Jul. 25, 2024. [Online]. Available: https://www.missingkids.org/content/dam/missingkids/pdfs/OJJDP-NCMEC-Transparency-CY-2023-Report.pdf

[19] J. Langston, “How PhotoDNA for Video is being used to fight online child exploitation,” Sep. 12, 2018. https://news.microsoft.com/on-the-issues/2018/09/12/how-photodna-for-video-is-being-used-to-fight-online-child-exploitation (accessed Jul. 25, 2024).

‌[20] A. Chandley, “Update to PhotoDNA,” Jun. 20, 2023. https://www.linkedin.com/pulse/update-photodna-adrian-chandley (accessed Jul. 25, 2024).

[21] “The EU’s horizontal regulatory framework for illegal content removal in the DSM | Hearings | Events | IMCO | 8th parliamentary term (2014 – 2019) | Committees | European Parliament,” www.europarl.europa.eu. https://www.europarl.europa.eu/committees/en/product/product-details/20180613CHE04321 (accessed Jul. 25, 2024).

[22] “Digital Safety Content Report ,” Microsoft. Available: https://www.microsoft.com/en-us/corporate-responsibility/digital-safety-content-report

[23] N. Krawetz , “PhotoDNA and Limitations,” Aug. 27, 2021. https://hackerfactor.com/blog/index.php?/archives/931-PhotoDNA-and-Limitations.html (accessed Jul. 25, 2024).

[24] A. Athalye, “anishathalye/ribosome,” GitHub, Jul. 08, 2024. https://github.com/anishathalye/ribosome (accessed Jul. 25, 2024).

[25] “CSAM Detection,” 2021. Available: https://www.apple.com/child-safety/pdf/CSAM_Detection_Technical_Summary.pdf

[26] Lukas Struppek, Dominik Hintersdorf, D. Neider, and K. Kersting, “Learning to Break Deep Perceptual Hashing: The Use Case NeuralHash,” arXiv (Cornell University), Nov. 2021, doi: https://doi.org/10.1145/3531146.3533073.

[27] S. K. Lim, “Apple’s NeuralHash — How it works and ways to break it,” Medium, Aug. 30, 2021. https://towardsdatascience.com/apples-neuralhash-how-it-works-and-ways-to-break-it-577d1edc9838 (accessed Jul. 25, 2024).

[28] “GIFCT | Global Internet Forum to Counter Terrorism,” GIFCT. https://gifct.org/

[29] J. Dalins, C. Wilson, and D. Boudry, “PDQ & TMK + PDQF — A Test Drive of Facebook’s Perceptual Hashing Algorithms,” arXiv.org, Dec. 16, 2019. https://arxiv.org/abs/1912.07745 (accessed Jul. 25, 2024).

[30] C. Somers, “Ensuring Online Safety – The Role of Artificial Intelligence in Combatting Illegal Content Online,” Jun. 27, 2023. https://www.law.kuleuven.be/ai-summer-school/blogpost/Blogposts/AI-combatting-illegal-content-online (accessed Jul. 25, 2024).

[31] “Criminal Division | Citizen’s Guide To U.S. Federal Law On Child Pornography,” www.justice.gov, May 26, 2015. https://www.justice.gov/criminal/criminal-ceos/citizens-guide-us-federal-law-child-pornography

[32] “Directive 2014/65/EU of the European Parliament and of the Council,” EU Monitor. [Online]. Available: https://www.eumonitor.eu/9353000/1/j9vvik7m1c3gyxp/vjfnzzz2axvv. [Accessed: July 25, 2024].

[33] “Section 58 of the Terrorism Act 2000,” legislation.gov.uk. [Online]. Available: https://www.legislation.gov.uk/ukpga/2000/11/section/58. [Accessed: July 25, 2024].

[34] E. David, “AI image training dataset found to include child sexual abuse imagery,” The Verge, Dec. 20, 2023. https://www.theverge.com/2023/12/20/24009418/generative-ai-image-laion-csam-google-stability-stanford

[35] J. Hendrix, “Exposing the Rotten Reality of AI Training Data | TechPolicy.Press,” Tech Policy Press, Dec. 31, 2023. https://www.techpolicy.press/exposing-the-rotten-reality-of-ai-training-data/ (accessed Jul. 26, 2024).

[36] B. Vigliarolo, “CSAM found in large AI image generator-training dataset,” www.theregister.com. https://www.theregister.com/2023/12/20/csam_laion_dataset/

[37] K. Canegallo, “Our efforts to fight child sexual abuse online,” Safety & Security , Feb. 24, 2021. https://blog.google/technology/safety-security/our-efforts-fight-child-sexual-abuse-online (accessed Jul. 25, 2024).

[38] K. Johnson, “Google releases AI-powered Content Safety API to identify more child abuse images,” VentureBeat, Feb. 24, 2021. [Online]. Available: https://venturebeat.com/ai/google-releases-ai-powered-content-safety-api-to-identify-more-child-abuse-images/. [Accessed: July 25, 2024].

[39] “Content Safety,” Google Safety Center. [Online]. Available: https://safety.google/content-safety/. [Accessed: July 25, 2024].

[40] H. A. Miller, P. P. Jones, and D. S. Smith, “Title of the Paper,” in *Proceedings of the 2018 IEEE Conference on Big Data*, Seattle, WA, USA, 2018, pp. 123-130. [Online]. Available: https://ieeexplore.ieee.org/document/8614362 . [Accessed: July 25, 2024].

[41] Y. Lee, “Thorn and All Tech Is Human Forge Generative AI Principles with AI Leaders to Enact Strong Child Safety Commitments,” Thorn, Apr. 23, 2024. https://www.thorn.org/blog/generative-ai-principles/

[42] “Thorn’s ‘Safety by Design’: A Summary’,” WebPurify, Jun. 11, 2024. https://www.webpurify.com/blog/thorn-safety-by-design-report-summary/ (accessed Jul. 25, 2024).

[43] “Google Transparency Report,” transparencyreport.google.com. https://transparencyreport.google.com/child-sexual-abuse-material/reporting?lu=total_content_reported&total_content_reported=period:2022H1 (accessed Jul. 25, 2024).

[44] “How Thorn Makes the Internet Safer & Helps Stop the Cycle of Abuse,” Blog & News, Apr. 05, 2024. https://www.thorn.org/blog/how-thorn-makes-the-internet-safer-helps-stop-the-cycle-of-abuse (accessed Jul. 25, 2024).

[45] Thorn, “Thorn Launches Safer Predict, a proactive AI detection solution to scale child safety on content-hosting platforms,” www.prnewswire.com. https://www.prnewswire.com/news-releases/thorn-launches-safer-predict-a-proactive-ai-detection-solution-to-scale-child-safety-on-content-hosting-platforms-302202079.html (accessed Jul. 25, 2024).

[46] “Introducing Safer Essential, API-Based CSAM Detection,” Blog & News, Nov. 10, 2023. https://www.thorn.org/blog/safer-essential (accessed Jul. 25, 2024).

[47] “AWS Marketplace: Safer Essential: API-based CSAM detection built by Thorn,” aws.amazon.com. https://aws.amazon.com/marketplace/pp/prodview-dfwekn4bx4ake (accessed Jul. 25, 2024).

[48] “How CSAM Detection Works | Safer by Thorn,” Safer: Proactive Solution for CSE and CSAM Detection, Sep. 23, 2022. https://safer.io/how-it-works/#compare (accessed Jul. 25, 2024).

[49] P. Leskin, “Facebook content moderator who quit reportedly wrote a blistering letter citing ‘stress induced insomnia’ among other ‘trauma,’” Business Insider. https://www.businessinsider.com/facebook-content-moderator-quit-with-blistering-letter-citing-trauma-2021-4?IR=T (accessed Jul. 25, 2024).