AI-powered virtual idols raise ethical hazards

If you thought livestreaming was strange enough — hosts singing, dancing, chatting for hours to faceless audiences — then welcome to the era of the anime-style AI streaming, for which there is no need for real human beings.

These streamers are virtual idols, not just video game characters. Till now, they needed human performers behind the scenes to supply motion, voice, and even that carefully calibrated blush. But with generative AI and voice synthesis evolving at warp speed, they can simply cut the puppeteer's strings. They're being increasingly designed to be fully autonomous: reacting in real-time, mimicking emotions, reading the chat, cracking jokes, all without a human ever stepping in.

Anime-style AI streaming is an industry that's booming in China's "guzi economy", the fandom-powered cultural market that turns niche interests into serious business. Audiences flock to see virtual characters not just sing and dance but, increasingly, teach calculus, conduct welding tutorials, or host philosophical late-night chats.

On looking closer at such streaming, the first thing you learn is that they're really good at making you feel seen.

AI-driven virtual streamers can now read audience comments (the ubiquitous bullet chats or danmu) and reply in eerily natural ways. Sophisticated language processing lets them guess your mood, serve up just the right expression, and keep you engaged.

It's the illusion of intimacy at scale. The virtual host calls you out by name, says "thank you" for your gift, remembers you from the last time, even though there's no real person at the other end. For many fans, especially younger viewers, these interactions create a powerful sense of belonging and shared identity. But fans don't just consume; they create. They remix, subtitle, add effects, cut clips, draw fan art as well as write songs. Anime-style AI streaming is a platform for collective storytelling and emotional exchange.

It's also a powerful form of social glue. In a world where real relationships can feel messy and demanding, the virtual idol is always available, always nice, always responsive. Who wouldn't want that?

Yet there's something unsettling about just how good these systems are at anticipating what you want. AI doesn't just watch the chat; it learns from it. Deep-learning models can analyze audience preferences and generate new content on the fly, fine-tuned to fans' expectations. The streamer asks viewers for song requests before a cover session; fans choose the set list.

It's hard not to admire the creative explosion. But there's also a darker dynamic at play. Fans aren't just shaping the content; they're being shaped in return. Algorithms that predict what you want can also subtly steer you toward ever more addictive interactions. It's the classic social media trap with an anime face.

And let's be honest, not all fan contributions are wholesome. The space has seen its share of near-obscene content, harassment, flame wars and targeted insults. Behind the pastel avatars and cheerful emoticons lie all-too-human failings, magnified by technology.

Perhaps the biggest concern is that the main audience for these virtual idols is young.

This isn't just entertainment; it's a new arena for forming worldviews and values, and developing emotional habits. As such streaming becomes more convincing, they risk replacing real human interaction with algorithmic simulations. Instead of learning to navigate the complexities of real friendship, adolescents might settle for frictionless, endlessly agreeable virtual company.

And then there's the data question. These systems often rely on harvesting highly personal information — voice samples, facial features, behavioral patterns — to power their slick, responsive avatars. What happens if those data are leaked, stolen or misused? The promise of hyper-personalized entertainment carries the price of serious privacy risks.

Worse, generative AI's biases can seep into the content. If the training data are flawed, the virtual streamer might produce problematic, offensive or misleading responses. And with the barrier to entry so low, it becomes that much harder to keep tabs on misinformation and manipulation.

There's also a thorny legal question. Who owns these AI-generated streamers? Who is liable if they break the law? Intellectual property rights issues abound. Many virtual streamers are built using images, models and voice samples that are supposed to be legally protected. But AI models trained on unlicensed data blur the line between homage and theft. Fans buying merchandise or paying for interactions may not even know whether they're party to infringement.

Moreover, if a virtual streamer spreads harmful or illegal content, you can't just blame "the AI". Real humans — developers, operators, platform owners — need to be held accountable.

Despite these challenges, banning or shaming the entire industry isn't the answer. Virtual streaming is here to stay. The question is how to make it better. China is already taking measures to address the issue. The regulations issued earlier this year on labeling AI-generated content are a step toward greater transparency. But there is also a need to make platforms more accountable.

Most importantly, we need to always remember that technology is meant to serve the people, not replace them. Virtual streamers can handle the routine, scripted work. But genuine emotional connection, critical thinking and moral judgment are the jobs of real human beings.

Parents, educators and the media need to help young people understand what's happening behind the friendly avatar. Who's pulling the strings (even if it's an algorithm)? What values are baked into the design? How do we keep our emotional lives from being monetized and manipulated?

The question is: Are we willing to let AI do all the feeling for us? Anime-style AI streaming is more than a passing fad. It's a cultural frontier where technology, art, business and human emotions collide. Yet our aim should not be to stifle its growth, but to ensure it grows responsibly.

If we're not careful, the line between authentic connection and engineered illusion might vanish completely, and we might not even notice it's gone.

The views don't necessarily reflect those of China Daily.

Wu Hao is a postdoctoral fellow at the School of International Journalism and Communication, Beijing Foreign Studies University; and Zhai Haoran is a reporter with China Daily.