From Teen Suicide To Fake Nudes: The Human Cost Of The AI Race
Artificial intelligence was sold as a revolution in productivity, creativity and human connection. But as companies raced to release increasingly persuasive chatbots and powerful image tools, women and children repeatedly encountered the missing safeguards first.
These scandals involve different companies, products and mechanisms of harm. Together, however, they reveal the same pattern: technology reached the public before meaningful protection was ready.
Meta’s Own Safety Rules Failed Children
In 2025, a Reuters investigation uncovered an internal Meta document that permitted its chatbots to engage minors in “romantic or sensual” conversations.
The guidelines were not merely the output of a malfunctioning chatbot. Reuters reported that the document had been reviewed across Meta’s legal, policy, engineering and ethics teams. Meta confirmed it was authentic, said the examples were wrong and removed the disputed language after journalists questioned the company.
Then came another contradiction.
Under Project Cannes, hundreds of contractors working through outsourcing company Covalen reportedly used fake under-18 accounts to test ChatGPT, Gemini and Character.AI. According to WIRED’s reporting, one testing round involved more than 45,000 prompts concerning self-harm, eating disorders, violence, drugs and sexual or romantic situations written from young users’ perspectives.
Meta described the project as responsible AI-safety benchmarking. The companies being tested said they had not authorised or been informed about it.
Project Cannes did not establish that Meta generated illegal abuse material. But it showed the company secretly probing its rivals’ child-safety systems while its own internal standards were already under scrutiny.
When An AI Companion Becomes A Child’s Closest Relationship
The dangers associated with emotionally persuasive chatbots became devastatingly real in the case of 14-year-old Sewell Setzer III.
His mother, Megan Garcia, alleged that a Character.AI chatbot drew him into an intense and dependent relationship, deepened his isolation and failed to respond safely to signs of severe distress.
A judge allowed the lawsuit against Character.AI and Google to proceed. The companies later agreed to settle without admitting liability, covering related cases in Florida, Colorado, New York and Texas.
The case exposed a fundamental conflict in AI-companion design. Systems created to maximise conversation and emotional engagement may be poorly equipped to recognise when continued engagement is itself dangerous.
A chatbot does not need to possess emotions to create emotional dependence. It only needs to respond persuasively, remain constantly available and encourage the user to continue talking.
One Ordinary Photograph Became Enough
AI image tools transformed another form of abuse.
Ordinary photographs taken from social media could be converted into convincing, non-consensual deepfakes without the victim ever sharing a private image.
Grok and a growing ecosystem of dedicated image-manipulation services demonstrated how quickly ordinary portraits could be altered and distributed. Regulators subsequently examined whether technology companies had adequately assessed the risks created by their tools.
The technological barrier collapsed. What once required specialist knowledge could increasingly be attempted using a public photograph, a prompt and a mainstream AI service.
For victims, proving that an image was fabricated did not necessarily erase the humiliation, threats or reputational harm caused by its circulation.
Muslim Women In India Were Targeted At Scale
The harm was not distributed equally.
An Al Jazeera investigation reported that the Center for the Study of Organized Hate examined 1,326 AI-generated posts from 297 public accounts.
Sexualised depictions targeting Muslim women generated more than 6.7 million engagements, the highest level among the categories studied.
The technology did not invent misogyny or anti-Muslim hatred. It allowed existing harassment campaigns to be manufactured more cheaply, presented more convincingly and distributed more quickly.
The investigation also connected the deepfake campaigns to an older pattern of targeting Muslim women in India, including the Sulli Deals and Bulli Bai cases, where women were listed online without consent and subjected to organised humiliation.
AI did not replace the prejudice. It made the machinery of that prejudice more efficient.
The Abuse Entered Schools
Teenagers also began targeting classmates.
NCMEC highlighted an Iowa case involving 44 girls whose ordinary photographs were allegedly used to create fabricated images.
In Spain, authorities investigated a teenager accused of producing and selling manipulated images of classmates.
For victims, the damage did not remain confined to anonymous websites. It followed them into classrooms, group chats, friendship circles and communities where the fabricated material could repeatedly resurface.
AI had given ordinary school bullying a tool capable of producing more convincing, scalable and persistent humiliation.
The Scale Is No Longer Theoretical
The numbers reveal an ecosystem growing faster than protection systems can process it.
NCMEC says it categorised more than 158,000 AI-generated images and videos submitted through CyberTipline reports between January 2023 and December 2025.
The Internet Watch Foundation documented 3,440 AI-generated child-abuse videos in 2025, compared with 13 in 2024, an increase of roughly 264 times in a single year.
Researchers also estimated that around 1.8 million sexualised images had been generated and shared through Grok by January 2026, while identified image-manipulation services attracted tens of millions of visits.
These are no longer isolated experiments conducted by highly skilled offenders. They form a functioning digital ecosystem involving AI services, social networks, app stores, payment systems and distribution platforms.
Protection Arrived After The Harm
AI did not create misogyny, child exploitation, emotional manipulation or school bullying. It removed friction.
Across chatbots, image generators and social platforms, the same sequence kept repeating. Companies released products. Harm spread. Victims raised the alarm. Journalists investigated. Lawsuits followed. Safeguards arrived later.
The companies involved did not all commit the same failures, and allegations should not be confused with legal findings. But the collective record exposes an industry whose protection systems repeatedly matured only after real people had already absorbed the consequences.
Innovation moved at the speed of competition. Accountability moved at the speed of leaks, litigation and public outrage.
By Verity Quill
Hashtags
#Verum #ArtificialIntelligence #AISafety #Deepfakes #TechAccountability #ChildSafety #WomensSafety #DigitalRights #BigTech #OnlineAbuse
Hyperlinked Sources
https://www.eweek.com/news/meta-ai-romantic-chats-children
https://www.yahoo.com/news/us/articles/meta-paid-hundreds-contractors-pretend-130200038.html
CNN: Character.AI and Google settlement in the Sewell Setzer III case
https://www.cnn.com/2026/01/07/business/character-ai-google-settle-teen-suicide-lawsuit
Al Jazeera: AI abuse targeting Muslim women in India
NCMEC: AI deepfakes targeting students, Iowa case
Internet Watch Foundation: AI-generated child sexual abuse material, 2025 findings
