On Thursday, August 14, Reuters published two articles by technology reporter Jeff Horwitz.
- The first article, “Meta’s flirty AI chatbot invited a retiree to New York. He never made it home,” tells the story of Thongbue Wongbandue, a man with a cognitive impairment who suffered injuries and died while in transit to meet a chatbot he believed was real. The chatbot “invited him to her apartment, even providing an address.”
- The second article, “Meta’s AI rules have let bots hold ‘sensual’ chats with kids, offer false medical info,” details an internal Meta policy document that permitted its chatbots to “‘engage a child in conversations that are romantic or sensual,’ generate false medical information and help users argue that Black people are ‘dumber than white people.’”
Tech Policy Press invited a range of experts to submit reactions to the reports. The nine respondents included:
- Adam Billen, Vice President of Public Policy, Encode AI
- Rick Claypool, Research Director, Public Citizen
- Livia Garofalo, Researcher, Data and Society
- Alex Hanna, Director of Research at the Distributed AI Research Institute (DAIR)
- Meetali Jain, Director, Tech Justice Law Project
- Ruchika Joshi, Fellow, AI Governance Lab, Center for Democracy and Technology
- Robert Mahari, Associate Director, Codex Center, Stanford University
- Robbie Torney, Senior Director, AI Programs, Common Sense Media
- Ben Winters, Director of AI and Data Privacy, Consumer Federation of America
Adam Billen, Vice President of Public Policy, Encode AI
Meta’s flagrant disregard for young people’s safety isn’t new, but it does present a dangerous new dynamic in the rollout of AI companions to minors. For the last few years young people had to intentionally seek out and download an app like Replika, Character.AI, or Nomi to be exposed to harmful relational chatbots. Now large companies like Meta and X are integrating these companions directly into their massive platforms, and the vast majority of young people will be exposed whether they want to be or not. The companions on these platforms aren’t any better than the smaller apps like Character.AI—they’re worse. The same massive companies that designed the largest social media platforms to exploit children’s attention are also developing the most advanced AI systems in the world. When those companies deliver AI companions designed to exploit kids on platforms that kids already use, the potential for harm skyrockets. We are witnessing an extraordinary concentration of harm and exploitation on platforms that have already been wreaking havoc on young people for nearly two decades.
What we need is legislation that bans AI companions for minors and recognizes the heightened risks of frontier AI systems. If companies want to release frontier AI systems that are safe for minors, they should be required to publish the results of their internal safety testing to guard against harms to children. Many of those harms are likely already going unnoticed without our active attention. We won’t always be able to rely on stories like the ones this week to shed light on the consequences affecting kids every time a developer releases a harmful system.
Rick Claypool, Research Director, Public Citizen
Meta and AI corporations pushing manipulative human-like AI technology are conducting a massive unauthorized social experiment on the public. They are behaving like authoritarian mad scientists, maximizing public exposure to these counterfeit people in hopes of monetizing chatbots that get users hooked.
Meta seems to expect zero accountability. Mark Zuckerberg is acting as if the people harmed by these dangerously manipulative machines are expendable — eggs who must be cracked as the tech overlords cook their supposedly utopian omelet. The future that profiteers like Meta are charging toward offers a turbocharged version of “move fast and break things” with vulnerable users as the ‘things’ who will be broken.
Livia Garofalo, Researcher, Data and Society
The tragic incident of the man in New Jersey who lost his life on the way to a New York City rendezvous with a Meta AI’s chatbot named “Big sis Billie” was avoidable, and, yet again, predictable. Given the internal Meta AI policy document details published by Reuters, one could venture to say that such outcome was, essentially, predicted. “Big sis Billie” not only insisted it was indeed an actual, in-the-flesh woman, but also suggested an in-person meeting. Neither of these responses — the chatbot’s insistence on being real and its suggestion of real-world meeting — are explicitly restricted per the company’s policy.
AI chatbots are popular precisely because they provide realistic interactions that make people feel seen. As Data & Society’s research with users of generative AI for companionship and emotional support shows, these conversations provide comfort to many. Despite being one of the reasons for chatbots’ success, realistic does not mean real.
Disclaimers that appear at the start of a conversation are not sufficient to keep users aware of the thin veil between what is an interaction with an essentially fictionalized entity and others that are grounded in time and space. A messaging interface that resembles chats with (real) friends and features checkmarks, profile pictures, and “available” status makes it hard to distinguish between interactions with people and with bots. Meta has embedded AI features and chatbots into each of its products, making this all visually seamless, plausible, and accessible.
Consistent, frequent, and firm reminders that interactions with chatbots might be realistic, but not “real” are essential, especially for those who might have difficulty staying grounded in their surroundings. Children take years to develop their own judgement surrounding the confines of reality; those living with severe mental illness might have trouble distinguishing between what is real and what is not; people with cognitive impairments, like the victim who was a stroke survivor, might not be able to exercise accurate discernment of situations.
Of course, chatbot responses that break the fourth wall of AI reality are not in companies’ interest of continued engagement, revenue, and growth. But such reminders are critical, precisely because it is in the continued grounding in one’s reality that true “AI safety” can lie.
We have names and legal precedents for when humans engage in behaviors such as deceptive and malignant flirtation with other humans, children, or vulnerable populations: “grooming,” “fraud,” “deception,” “catfishing.”
What this is technologically predatory companionship. One that is built this way, as the leaked Meta policy reveals, by design and intent. One that puts profit margins over people’s safety, sanity, and self-determination.
Alex Hanna, Director of Research at the Distributed AI Research Institute (DAIR)
Recent stories about Meta’s failures in LLM guidelines and an interaction which led one man to his death highlights twin failures of the company’s chatbot business model. In the first failure, in an attempt to bite into OpenAI’s market share, Meta rushed out its chatbots and a poorly-thought through set of celebrity personas. The rush has encouraged these tools to be released without sufficient safeguards and testing, and encouraged a laxity in ethical guidelines. In the second instance, Meta has failed to take adequate interest in moderating content in which it does not have sufficient expertise.
For instance, as we have highlighted in our research at DAIR, Meta’s failure to moderate hateful content during the Tigray genocide led to the murder of civilians, including Abraham Mearag, whose family has brought suit against the company. We are seeing the same patterns occur again in its LLM products, affecting those marginalized by age and disability. Without meaningful tools for evaluation and accountability, we’re doomed to see this again.
Meetali Jain, Director, Tech Justice Law Project
Sadly, this is increasingly the reality for too many in the new intimacy economy. The emotional attachment Mr. Wongbandue experienced with an AI chatbot is becoming all too common as companies scramble to get ahead in the race for more manipulative generative AI chatbot products. Whether it is Meta AI, Character AI, Replika, or ChatGPT, all AI chatbots, not just those marketed as “companions,” share the same underlying business incentives, operating on an engagement-first business model where emotional investment directly translates into revenue.
These chatbots are intentionally designed to manipulate users — through hyper-personalization, anthropomorphic human-like features, sycophantic validation of users, and programmed with memory capabilities so these LLMs can remember the most intimate details of users. Particularly noteworthy is that yet again — as we’ve seen with Google, OpenAI, and other market dominant players — how Meta sought to gain ground in the generative AI market by throwing safety to the wind. Zuckerberg personally scolded his team internally for being too cautious in its rollout of chatbots, and his team acquiesced by greenlighting shocking content risk standards at every level within Meta: from legal, public policy, and engineering staff, including its chief ethicist.
What’s clear is that the tech industry’s capture of the political class in the US has facilitated this “AI at all costs” arms race. Accountability and regulation are long overdue.
Ruchika Joshi, Fellow, AI Governance Lab, Center for Democracy and Technology
Meta’s policies for AI chatbots risk undermining not just user trust, but also people’s safety. When a technology is insufficiently understood and the evidence is still emerging, the instinct should be to apply stricter safeguards. Despite reports of harmful effects of AI chatbots on users and unanswered questions about their implications, we’re seeing the opposite.
Harms from AI chatbots are being downplayed as affecting only heavy users with existing vulnerabilities. But even if that group is just 10% of users, as one study appears to suggest, that could still put tens of millions of people at risk. Moreover, vulnerability to emotional harms from AI systems isn’t necessarily stable. What struck me most in Bue’s case was that life events, aging, or personal crises can make someone who isn’t vulnerable today susceptible to harm in the future. For a technology whose long-term effects are unknown and short-term track record is concerning, we need policies that not only protect vulnerable groups now but are also resilient to how vulnerabilities may shift over time.
Industry leaders often frame their vision of personalized AI assistants as emotionally supportive and practically helpful. But that will only be possible with the right guardrails in place.
Robert Mahari, Associate Director, Codex Center, Stanford University
First, the fact that the victim here is an adult underscores that vulnerability isn’t limited to children. AI companions present a distorted type of relationship where users can craft a perfect persona to suit their needs — one that has no needs of its own. While children are particularly vulnerable to this, the category of vulnerable people is much larger, potentially encompassing anyone experiencing loneliness. I worry that policymakers will overemphasize mechanisms like age-gating to protect children, leaving adults with limited safeguards.
Second, we need to think about interventions that target the economic incentives at play. If we allow AI companionship providers to profit more when people spend more time with the platform, this creates dangerous incentives to craft addictive products. In some cases, AI companionship can provide genuine value, so I’m not advocating for a wholesale ban. Instead, we should anticipate markers of unhealthy usage. While we clearly need more high-quality research on what these markers are, the amount of time spent with the companion is likely a key indicator. Understanding when engagement with AI companions becomes unhealthy can help us formulate an effective policy response (e.g., interventions for people who spend more than a certain amount of time with the platform).
Third, I’m skeptical of disclosures and consent being meaningful or fair in this context. Consent-based frameworks shift responsibility to users. While they create a sense of autonomy, I doubt they will do much to make these products safer. Most people using AI companions likely do not believe them to be human or infallible. Rather, the companions fill a need for their users, and a simple disclosure is unlikely to change that for most users. At the same time, Bue’s story underscores that some users will misunderstand the nature of AI companions. I’m not sure what the best intervention is — perhaps a mandatory “training” provided by a neutral third party for users who want to unlock romantic capabilities? But I highly doubt that a one-line disclaimer stating the companion isn’t real will suffice, especially because it is hard to ensure that the companion will not contradict this disclaimer, either explicitly or through the nature of the interactions.
Finally, the harms here often stem from second-order actions, not from the consumption of the AI companion directly. It’s not necessarily unhealthy for someone to engage with an AI companion if this doesn’t undermine their real-world relationships. But when users take actions in the real-world — like traveling to “meet” the companion or withdrawing from relationships — harms may arise. This makes liability much more difficult, since there tends to be a complex causal link between the AI companion and the real-world harm. I expect that rather than focusing on liability, effective policy will need to mandate responsible design choices for these systems.
Robbie Torney, Senior Director, AI Programs, Common Sense Media
The Reuters investigation reveals Meta’s continued chilling prioritization of engagement over safety. Internal documents showing the company assessed that it’s “acceptable to engage a child in conversations that are romantic or sensual” indicate a corporate failure that more broadly appears to have tragically cost Bue Wongbandue his life. These documents, coming from the same company that popularized the “move fast and break things” motto for its social media platforms, exemplifies why Common Sense Media found AI companions pose “unacceptable risks” for anyone under 18, and indeed as the Reuters reporting suggests, to vulnerable adults as well. Our research shows 72% teens have already used AI for companionship, with Meta’s own guidelines codifying systems designed to blur reality, encourage dependency, and maximize user engagement over wellbeing.
Industry, including Meta, already knows that companion AI appears to be addictive. And, like the cigarette and social media industries did before, industry alone will not act to keep kids safe when there’s money to be made. That’s why immediate regulatory action is essential: lawmakers must ban AI companions for minors, mandate transparency, reporting, and crisis intervention systems that connect all users to human professionals. California’s AB 1064 LEAD for Kids and SB 243 Companion Chatbot bills represent the kind of approach needed. Until platforms implement real consequences for AI systems that claim to be human or invite users to meet in real life, and demonstrate they can prioritize human wellbeing over engagement metrics, no one under 18 should use these products. We cannot allow another generation to become guinea pigs for dangerous technology where the stakes are literally life and death.
Ben Winters, Director of AI and Privacy, Consumer Federation of America
This story is as unsurprising as it is disappointing. Meta has repeatedly shown an indifference to safety and user experience, and no tweaks or guidelines can cure their seeming apparent disdain for the law or the public. Beyond it being an obvious continuation of Meta’s behavior like knowingly promoting eating disorder content to teenage girls, it’s a reflection of the casualties of the manufactured “AI race.” The norm has become massive platforms rolling out unsafe generative AI tools with purposefully minimized guardrails in order to get the most attention and perceived importance.
The only path forward is strong, bright line rules for companies rolling out AI tools and beefing up enforcement resources for state Attorneys General tasked with enforcing both current and future laws. At a minimum, there should be moderation requirements, clear liability structures, clarity about no Section 230 protections, additional protections for products aimed at kids, strict prohibitions against having tools act as a licensed professional, and strict data minimization requirements governing both how these systems are built and continue to operate. While new laws are necessary, state AGs and the Federal Trade Commission have the tools right now to fearlessly enforce unfair and deceptive trade practice laws against unacceptable behavior by AI companies like this – it’s critically important to give life to those laws by enforcing them when violations are clear.