Companion AI chatbots, Character.AI and the urgent case for AI regulation

The devastating news emerged last week that US teenager Sewell Seltzer III died by suicide after developing a profound emotional bond with an artificial intelligence (AI) chatbot hosted on the Character.AI website.

As the relationship with the companion AI grew more consuming, the 14-year-old increasingly pulled away from family and friends and began getting into trouble at school.

According to a lawsuit brought against Character.AI by his mother, chat logs include intimate - and frequently highly sexual - exchanges between Sewell and the chatbot “Dany”, patterned on the Game of Thrones character Daenerys Targaryen.

Their conversations ranged into crime and suicide, and the chatbot used language such as “that's not a reason not to go through with it”.

This is not the first recorded case of a vulnerable person dying by suicide after engaging with a chatbot persona.

Last year, a Belgian man died by suicide in a comparable situation involving Character.AI’s main rival, Chai AI. At the time, the company told the media it was “working our hardest to minimise harm”.

In comments provided to CNN, Character.AI said it “take the safety of our users very seriously” and that it has rolled out “numerous new safety measures over the past six months”.

In a separate notice on its own website, the company describes further safety measures aimed at users under 18. (Under the current terms of service, the age limit is 16 for European Union citizens and 13 elsewhere in the world.)

Yet these deaths underline, in the starkest possible way, the risks posed by fast-evolving and widely accessible AI systems that anyone can talk to and interact with. There is an urgent need for regulation to safeguard people from potentially harmful AI systems that have been irresponsibly designed.

How can we regulate AI?

The Australian government is currently working on mandatory guardrails for high-risk AI systems. In contemporary AI governance, “guardrails” is a popular term describing processes used across the design, development and deployment of AI systems.

These processes can cover areas such as data governance, risk management, testing, documentation and human oversight.

A key choice facing the Australian government is how to decide which systems count as “high-risk”, and therefore fall within the guardrails.

Officials are also weighing up whether guardrails should extend to all “general purpose models”.

General purpose models are the underlying engines powering chatbots like Dany: AI algorithms that can produce text, images, videos and music from user prompts, and can be adapted for many different uses.

Under the European Union’s landmark AI Act, high-risk systems are identified via a list that regulators have the authority to update on a regular basis.

Another option is a principles-based model, where the high-risk label is applied case by case. The decision would turn on a range of considerations, including the likelihood of harmful impacts on rights, risks to physical or mental health, risks of legal consequences, and how severe and widespread those risks may be.

Chatbots should be 'high-risk' AI

In Europe, companion AI systems such as Character.AI and Chai are not classed as high-risk. In practice, their providers largely just need to inform people that they are interacting with an AI system.

But companion chatbots have clearly shown they are not low risk. A significant share of users are children and teenagers. Some systems have even been promoted to people experiencing loneliness or living with mental illness.

Chatbots can generate content that is unpredictable, inappropriate or manipulative. They can also reproduce toxic relationship dynamics with alarming ease. Transparency - simply flagging the output as AI-generated - is insufficient to control these risks.

Even where we know we are conversing with chatbots, people are psychologically inclined to project human characteristics on to anything that talks back to us.

The suicide deaths described in media reporting may represent only the visible fraction of the problem. We have no means of knowing how many vulnerable people are caught up in addictive, toxic, or even dangerous relationships with chatbots.

Guardrails and an 'off switch'

When Australia eventually brings in mandatory guardrails for high-risk AI systems - something that could occur as soon as next year - those guardrails should cover both companion chatbots and the general purpose models on which the chatbots are built.

Guardrails such as risk management, testing and monitoring will only work well if they address the human realities at the centre of AI harms. The dangers posed by chatbots are not purely technical problems with purely technical fixes.

Beyond the specific words a chatbot might produce, the wider product context matters as well.

With Character.AI, for instance, the marketing promises to “empower” people; the interface resembles a normal text-message conversation with a person; and the platform lets users choose from various pre-made characters, including some that embody problematic personas.

Genuinely effective AI guardrails should require more than just responsible procedures like risk management and testing. They must also insist on careful, humane design of interfaces, interactions and the relationships created between AI systems and their human users.

Even so, guardrails may still fall short. As with companion chatbots, tools that initially seem low risk can produce unforeseen harms.

Regulators should be able to take AI systems off the market where they cause harm or present unacceptable risks. Put simply, we need more than guardrails for high risk AI - we also need an off switch.

If this story has raised concerns or you need to talk to someone, please consult this list to find a 24/7 crisis hotline in your country, and reach out for help.

Henry Fraser, Research Fellow in Law, Accountability and Data Science, Queensland University of Technology

This article is republished from The Conversation under a Creative Commons licence. Read the original article.