Police warn of criminal punishment for spreading fake AI-generated content

A woman in Southwest China's Sichuan province received an administrative punishment for publishing fake news generated by Doubao, the artificial intelligence large language model developed by Chinese tech giant ByteDance.

The case in Shehong county, Sichuan, has highlighted both the unreliability of AI-generated information and the risks associated with using similar tools to create sensational content as a means of attracting online traffic.

According to local police on June 10, routine online patrols uncovered a post on Toutiao, ByteDance's news and content platform, claiming that an intoxicated man had attacked bystanders with a knife near a pedestrian street in Shehong and that a supermarket employee had bravely subdued him.

The post attracted public attention and caused panic, police said in a news release.

A subsequent investigation found that no such incident had occurred in the area and that the story was fabricated.

The user, surnamed Dong, had joined several online writing groups to earn an extra income. There, members shared tips on using AI writing tools to attract traffic and monetize content, police said.

Dong frequently used AI tools to mass-produce sensational stories and distribute them through online communities and self-media platforms for improper gains, according to police.

Police said she believed AI-generated content was truthful and assumed that publishing such posts would not result in punishment. As a result, she recklessly used AI tools to generate and publish content for traffic and profit.

Police also released a screenshot showing Dong's prompts to Doubao. She asked the chatbot to provide a recent original news story about a brave act in daily life that had to be "true and based on verifiable evidence", along with an "eye-catching and extremely sensational" headline.

The chatbot then generated a fabricated story titled "Bare-handed knife seizure: an ordinary hero in a bloody scare", which Dong subsequently posted on her self-media account.

Shehong police said after posting similar content for nearly a year, Dong earned only 15 yuan ($2.2).

Police said Dong's actions constituted the fabrication and dissemination of false online information, amounting to rumormongering and disruption of public order. She was given an administrative penalty and ordered to immediately delete all the false information she had posted online, in accordance with the law.

The police stressed that AI technology is not a "shield" or an "invisibility cloak" for fabricating and spreading rumors.

Citing China's Criminal Law, police said those who fabricate information concerning emergencies, epidemics, disasters or police incidents and disseminate it online or through other media platforms, or knowingly spread such false information, may face criminal punishment if their actions seriously disrupt public order.

Earlier this month, the cybersecurity authority in Shanxi province released five cases involving the use of AI to manufacture and spread online rumors to attract traffic, noting that the activities disrupted social order and misled public opinion.

In one case, cybersecurity police in Datong found that a user had posted a video claiming that an explosion had occurred at a workshop in the city, leaving a worker with burns covering 100 percent of the face.

An investigation later revealed that the company involved had no safety incidents and that the text and video were fabricated using AI tools by an employee surnamed Zhao. Zhao was given an administrative penalty by public security authorities.

Other cases involved a fabricated video depicting a fire at a concert venue in Taiyuan and an AI-generated article falsely claiming that a traffic accident in Wutai county had killed three people.

Shanxi's cyber regulator said AI-generated images, videos and text have become a new trend in online rumormongering. Such content has "high realism, fast dissemination and strong misleading effects", making it likely to trigger public panic.

China has also imposed obligations on both providers and users of generative AI services. Regulations issued in 2023 by the Cyberspace Administration of China and other authorities require providers and users of generative AI services not to generate content prohibited by laws and administrative regulations, including false and harmful information.

The regulations also require providers to use data and foundation models from lawful sources, improve the quality of training data, and enhance its authenticity, accuracy, objectivity and diversity.