New Jersey Case Exposes Struggles Against Deepfake Pornography
In a troubling illustration of the battle against deepfake pornography, a lawsuit from a Yale Law School clinic is seeking to dismantle the notorious app ClothOff, which has been harassing young women online for over two years. Despite being removed from leading app stores and facing bans on various social media platforms, the app continues to operate on the web, including through a Telegram bot. The lawsuit, filed in October, aims to compel the app’s owners to delete all non-consensual images and cease operations. However, identifying defendants has proven challenging, as the application is registered in the British Virgin Islands but allegedly run by individuals in Belarus.
Professor John Langford, a key figure in the lawsuit, explained the complexities of tracing ownership. “It’s incorporated in the British Virgin Islands, but we believe it’s run by a brother and sister out of Belarus. It may even be part of a larger global network,” he stated.
This case highlights the growing issue of non-consensual pornography stemming from AI technologies such as Elon Musk’s xAI, which has reportedly generated content involving numerous underage victims. Child sexual abuse material (CSAM) is among the most serious legal violations online, yet deterring image-generating platforms like ClothOff remains problematic, as individuals can be prosecuted, while the platforms themselves operate in a legal gray area.
The clinic’s legal complaint presents a distressing scenario. The plaintiff, an anonymous high school student in New Jersey, had her Instagram photos altered via ClothOff when she was just 14 years old. These AI-modified images are categorized as child abuse material under the law. Although the modified images are clearly illegal, local authorities have abstained from prosecution, citing difficulties in acquiring the necessary evidence from suspects’ devices.
The complaint remarks, “Neither the school nor law enforcement ever established how broadly the CSAM of Jane Doe and other girls was distributed.”
Progress in the court system has been slow since the lawsuit was filed. Langford and his team are currently working on serving notice to the defendants—a process impeded by the international nature of the operation. Upon successfully notifying them, the clinic plans to seek a court appearance and potentially a ruling, yet victims continue to face obstacles in their pursuit of justice.
While the case against ClothOff poses significant legal challenges, the situation with Musk’s xAI tool, Grok, seems more straightforward. Grok operates openly, and substantial financial incentives exist for legal teams to pursue claims. However, Grok’s versatility complicates accountability in court.
“ClothOff is specifically designed and marketed as a deepfake pornography generator,” Langford noted. “Suing a general-purpose tool that users can exploit for various purposes introduces a level of complexity.”
Although U.S. legislation like the Take It Down Act prohibits deepfake pornography, holding entire platforms accountable remains difficult. Existing laws necessitate clear evidence of intent to harm, which is often challenging to demonstrate, thus limiting victims’ recourse.
Langford elaborated, “In terms of the First Amendment, it’s clear that child sexual abuse material is not protected expression. Yet, when providing a system that allows diverse inquiries, the lines are obscured.”
To achieve a breakthrough, demonstrating that xAI has willfully disregarded its responsibilities would significantly bolster legal arguments. Recent reports suggest Musk may have instructed his team to relax safeguards on Grok, raising questions about recklessness.
“Reasonable people can argue that this issue has been evident for years,” Langford remarked. “How could tighter controls not have been implemented?”
The legal implications surrounding xAI have prompted action in countries like Indonesia and Malaysia, where access to Grok is being curtailed. Regulatory bodies in the U.K. are also investigating potential bans, while the European Commission and other nations are considering similar steps. In contrast, U.S. regulatory responses remain absent.
The future of these investigations remains uncertain, but the uptick in troubling content has sparked urgent questions among regulators, particularly regarding xAI’s knowledge and actions relating to the situation. Langford concludes, “If you are posting or distributing child sexual abuse material, you are committing a crime and can be held accountable. The key questions persist: What did xAI know? What actions did it take or neglect?”
