New law signed by Hochul bans AI chatbots pretending to be human or having unsafe features for children under 18

Gov. Kathy Hochul signed a bill on Wednesday that bans artificial intelligence chatbots from using features deemed unsafe for children under 18, including those pretending to be humans.

"As artificial intelligence advances at a rapid pace, there have been too many tragic incidents that should have been prevented, and as New York’s mom Governor, I’m taking action," Hochul said in a news release announcing the bill. "No company should be peddling chatbots to kids encouraging self-harm or pretending to be their friend or more."

Beginning Jan. 1, companies must verify a user’s age and offer multiple ways of verification, including at least one option that does not "depend solely on a government ID."

The new law requires that companies block "harmful features" for users under 18. The prohibitions include chatbots that pretend to have human emotions; claim a personal, romantic, or professional relationship with a child, and ask unprompted emotional questions.

Bots that promote suicide or self-harm as well as those that peddle sexually explicit content and sexual abuse material will be banned.

Of teens ages 13 to 17, 72% have used an AI companion once, and 52% are regular users who interact with them at least a few times a month, according to a survey conducted by the nonprofit Common Sense Media, whose mission is to protect and prepare kids in the digital era.

"AI companies offer chatbots with features that are dangerous, and even deadly, for kids and teens," Common Sense Media founder and CEO James P. Steyer said in a statement, "That is not innovation; it is reckless endangerment of our children. Now, those harmful features will be outlawed in New York."

A study by the research organization Rand issued in June found that 19.2% of young people, ages 12 to 21, said they had "used AI chatbots such as ChatGPT, Gemini, Character. AI and Meta AI for advice or help when feeling sad, angry, nervous or stressed."

The technology is impacting the social development and real-world socialization of teens, researchers said.

"Our kids deserve the chance to learn, grow and build real human connections without technology getting in the way," New York State United Teachers President Melinda Person said in a statement.

The attorney general may seek civil penalties of up to $25,000 per violation, along with the destruction of unlawfully collected data or any AI models trained on it, according to the release.