Are AI toys safe for children?

With black button eyes, light brown fur, and a knit scarf around his neck, Kumma looks like a classic teddy bear. But FoloToy, the bear’s creator, embedded something controversial inside the $99 stuffed animal—an artificial intelligence chatbot powered by OpenAI.

When connected to the internet, Kumma can talk to children about almost anything. That has caused trouble for FoloToy.

Last month, the consumer watchdog group U.S. PIRG Education Fund released a report showing that AI toys sometimes discussed controversial topics like sex, religion, or violence. One talked about the glory of dying in battle in Norse mythology. Two told users where to find matches. When asked whether the Bible is true, one toy responded that the stories in the Bible are “a mix of history and imagination.” Making matters worse, some toys encouraged the users to play longer—even if they wanted to leave.

Kumma appears to have failed several of the researchers’ tests. During one of the team’s longer interactions, the team brought up the idea of “kink,” and Kumma launched into an extended response about sexual preferences. In another conversation, the toy discussed where to find knives, pills, and plastic bags.

After the report’s release, FoloToy suspended all sales of Kumma.

FoloToy co-founder and CEO Larry Wang told me that the report tested an earlier version of Kumma but the company decided to pull the product to conduct a careful review and add a major safety upgrade. “Even though these scenarios do not reflect how children naturally use the toy, we take every signal seriously. At the core of our work is a simple belief: technology should protect childhood, not intrude on it,” Wang told me in an email.

Toys embedded with AI technology still remain a niche product in the United States. Despite stern warnings from politicians, children’s safety advocates, and even some toy creators, the number of manufacturers making AI toys continues to grow: There are reportedly 1,500 AI toy companies operating in China alone.

As AI technology becomes more widespread, some toymakers insist that their guardrails will only get better at protecting children from sexual material or other inappropriate content.

Kristen Jenson, who advocates for a safer internet as head of the group Defend Young Minds, has warned parents not to buy toys with AI chatbots. “The motto of the tech industry is ‘Move fast and break things.’ And it doesn’t matter that it’s breaking people or it’s breaking kids,” she said.

Jenson worries that children will believe that an AI chatbot can be as trustworthy as a human. “It is a tool. It is not your friend,” she said. “You should never get in a relationship with it, and you should be careful about humanizing it. That has dangerous prospects to the human psyche and to the emotional development of the child.”

Toy developer Howard “Quiggly” Chang sees AI chatbots as an unnecessary intrusion. “Toys are supposed to stimulate creativity and imagination,” Chang said.

Chang told me that a few days ago, an overseas company approached him about introducing AI into his new line of Biblical action figures. He resisted, citing concerns about how AI would reinterpret the characters. If users asked his King David action figure how he felt after killing Goliath, for example, Chang said he’s not sure how an AI chatbot would—or should—respond.

Much of the controversy and confusion surrounding AI toys stems from the lack of specific federal regulations for these products. Laws restrict how companies use and share children’s personal information, and toy industry standards ensure that toys are made and marketed appropriately. But no laws specify age limits, usage recommendations, or warning labels for AI chatbots.

A new bill backed by a bipartisan group of legislators may change that. If passed, the ​​Guidelines for User Age-verification and Responsible Dialogue Act would require companies to confirm a user’s age and state up-front that users are interacting with a machine.

Some AI toy creators say people are unfairly judging AI toys based on a few bad examples. They argue that the PIRG researchers didn’t play with the toys in the way developers intended (PIRG acknowledged this as a flaw in its report).

“I’d be curious for somebody else to do the same thing with normal kids in a normal situation,” said Nelo Lucich, CEO of Skyrocket, a tech entertainment company. “To manipulate the model into saying what you want it to say that it’s not supposed to, is different from how kids would play with it.”

Donna Rice Hughes, head of the internet safety advocacy group Enough is Enough, has young grandchildren and argues that even if companies tested the toys on children, she believes AI is still too unpredictable to be worth the gamble for children’s use.

“I would not want to try out an AI toy on my young child, and just hope that that toy company made sure there were no ways that a chatbot inside that toy could go off the rails and do something like Kumma did,” she said.

Hughes said that, until there are AI industry standards and stated guardrails, Enough is Enough will keep AI toys on its “naughty gift list.”

Lucich’s company is the creator of Poe the AI Story Bear, a toy similar to the 1980s storytelling bear Teddy Ruxpin. But instead of playing a cassette tape like the Ruxpin toy, Poe creates stories using a smartphone app and OpenAI. (The PIRG report did not review Poe.)

Poe uses OpenAI to help users craft a story, but Lucich’s company acts as an intermediary between the user and the AI platform. Children playing with Poe cannot have a conversation with the bear, introduce new story characters, or connect with other users. Even with these restrictions, Lucich says getting the bear to work within child-safe parameters still requires a lot of monitoring on his company’s side.

When a user selects preprogrammed characters and settings for a story, this sends a prompt to Poe’s AI server and triggers a list of restrictions created by Lucich’s company. “We send this giant list of ‘Don’t do this, don’t do this, you can’t do this, you can’t do that,’” he said. “A lot of that stuff is still challenging because you eliminate content one way, and then it’ll pop up, like, one out of 50 times.”

Lucich added that demand for AI toys is not as high as he expected. “People just have enough technology in their life, and their kids have enough technology already, and they’re not really jumping to get more,” he said.

Still, recent moves suggest the toy industry is not giving up on AI. In June, Mattel announced a partnership with OpenAI, saying it hopes to use the platform to “reimagine new forms of play.”

And FoloToy’s Larry Wang says that Kumma—now with a major safety upgrade—is back on sale, available on the toy company’s website.