San Francisco sues maker of AI-generated ‘deepfake’ pornography in groundbreaking lawsuit

“We must be clear that this is not innovation – this is sexual abuse,” Chiu said in a statement obtained by KQED. “This is a large, multifaceted problem that we as a society must solve as quickly as possible. We all need to do our part to take action against bad actors who use AI to exploit and abuse real people, including children.”

Deputy City Attorney Yvonne Meré first brought the issue to Chiu this year after reading in the news about young girls who had fallen victim to these deepfake images.

She was “horrified and terrified when I think of my own 15-year-old daughter and how she would feel if her autonomy was taken away, her image distorted and her privacy completely disregarded,” Meré said during a press conference on Wednesday. “And as a lawyer, I was frustrated. How can this evil practice continue?”

The lawsuit, which Chiu's team says is the first such lawsuit before a government, seeks to ban websites that allow users to create “sexually explicit images without their consent” or “undress” women – and in some cases children.

One of the companies’ websites states: “Imagine wasting time asking her out on dates when you can simply [website] to get nude photos of her,” the prosecutor said. Another asks the user: “Do you have someone to undress?”

According to the city's prosecutor's office, the websites use open-source generative AI models that are available to the public for customization and training on specific content.

“Even if the developers of these open source models later build security safeguards into new versions of the model, earlier versions – and optimized versions trained to generate pornographic content – continue to circulate online,” the complaint states.

Chiu's office claims the companies are violating state and federal laws against deepfake pornography, revenge pornography and child pornography. While several states have proposed or passed laws to criminalize such non-consensually AI-generated images, the lawsuit asks the San Francisco Superior Court to order the sites shut down.