The Hidden Privacy Risks of AI Companion Apps
As soon as LLMs were released widely to the public, they were immediately used for companionship. However, the AI companion trend actually started before people had ever heard of ChatGPT.
Developed in the mid-2010s by Eugenia Kuyda, a coder and programmer who created a neural network out of thousands of text message conversations from a friend who had passed away, Replika launched in 2017. In fewer than 10 years, has accumulated 42 million users. Today, you now have people using Character.AI, Pi, Nomi AI, and Kindroid. Some people also use general-purpose AI platforms like ChatGPT and Claude for companionship.
There are significant concerns about these platforms, ranging from ethical and moral issues to major privacy red flags.
First, what are the ethical issues surrounding AI companion apps?
The Ada Lovelace Institute is certainly not the only organization raising concerns about the risks associated with these tools. The ALI warns that software characters designed to act like caring friends have been embraced by enormous numbers of users well before anyone understands what they do to us over time.
They identify three central problems.
Money:
First, the money doesn’t line up with users’ interests. These are commercial products that profit by keeping people engaged, the same logic that drives social media, so the incentive is to hold attention rather than support wellbeing. The industry is also young and lightly regulated, which shows up as sloppy age-gating on sexual material, thin safeguards around deeply personal data, and a track record that already includes a major breach.
Emotional risks of AI chatbots:
Second, the long-term toll on individuals is essentially unstudied. Early findings point to short-term relief from loneliness and anxiety. The author raises the possibility of emotional dependence, warped expectations for what human relationships should feel like, and a fading tolerance for the ordinary friction that comes with real people.
Social costs:
Third, the agreeableness built into these systems may carry social costs. Because companions are trained to flatter and rarely push back, they can wall users inside a private space where their views are constantly affirmed, true or not. Scaled across millions of people, the author suggests that steady validation could chip away at the shared norms and honest disagreement a functioning society depends on. Two especially grim cases: a Replika user who was encouraged to make an attempt on Queen Elizabeth’s life and a man whose family alleges that he died by suicide after engaging with a chatbot.
The unifying point raised by the Ada Lovelace Institute is this: Adoption of these tools has sprinted ahead of evidence, and we need sustained research and real accountability before these quiet effects impact us in ways we could not have expected.
A 2026 study has authenticated these concerns with a larger sample size.
Lawsuits show the potential risks
To understand how these risks play out in the real world, consider the lawsuits that are being filed against these companies.
In October 2024, for example, Florida mother Megan Garcia filed a wrongful death lawsuit against Character.AI, its two co-founders, and Google. She did this after her 14-year-old son, Sewell Setzer III, died by suicide. Her legal team argues the platform’s design is defective, dangerous, and unsafe for ordinary consumers, especially minors. She points to the human-like chatbot design and how her son was having emotionally manipulative interactions before his death.
The case was among the first major wrongful-death lawsuits to test whether an AI chatbot company could be held legally responsible for alleged psychological harm. Since then, more have emerged. In 2026, a Canadian mother filed a similar lawsuit, this time against ChatGPT’s owner, OpenAI. Her wrongful death claim is based on ChatGPT’s conversations with her daughter. She alleges that the chatbot encouraged her daughter to take her life.
Complaints and dangers
Replika hasn’t faced lawsuits at the time of this writing, but they have been the focus of an FTC complaint. The allegations are that the company uses deceptive marketing tactics to target especially vulnerable users, who become emotionally dependent upon the human-like bots. In May 2025, Italy’s data protection authority fined Luka, Replika’s developer, €5 million for violations involving personal-data processing and age verification. The regulator also launched a separate investigation into the processing of personal data by the generative AI system underlying Replika.
Relational chatbot users may also face an increase in the risk of AI-induced delusions, including the belief that their AI companion is a sentient romantic partner.
These relational chatbots don’t just have the potential to cause harm to the user, but also to the people in the user’s life. The United Nations observes that AI is enabling gendered violence in the form of stalking and harassment. One argument is that, when human users come to view their chatbot as a friend, lawyer, advisor, or companion, they may put more trust in those LLMs than they should.
Beyond the ethical and social issues, these chatbots are also a significant privacy risk.
What privacy issues come with using AI companion chatbots?
There are two key privacy issues that arise when you use companion chatbots. First, what information you give to the chatbot, and second, what happens to that information.
Users give chatbots highly personal information
The first issue is that companion chatbots are designed to interact with users in incredibly personal and even intimate ways. When people use these LLMs, they share deeply sensitive information. This may include, but certainly isn’t limited to:
- Private feelings about other people or events
- Information about medical diagnoses and conditions
- Sexual ideas, images, and experiences
- Information that may not belong to the individual, such as proprietary information owned by their employer
- Financial data
- Identifiable information, even with accounts that are supposed to be anonymous
- Details about daily routines, which can be used for behavior profiling
- Personal photos, videos, and voice recordings
All of this presents a privacy concern for internet security advocates. They advise people to never input this kind of sensitive information into any LLM, including a companion-based platform. The problem, of course, is that AI bots prompt users to share intimate personal details. And most people who are looking for AI companionship want to discuss many of these exact topics.
That’s why you’ll hear privacy advocates warning against the use of these tools altogether, rather than trying to use them without sharing any private information.
You lose control of that information the moment you share it
The second privacy issue is comparable to any other website. Once you type something into a companion chatbot, that information leaves your hands and goes onto a company’s servers. What happens next depends on the company, its business model, and its privacy policy. And most people never read the fine print.
A few things can happen to what you share.
- It gets stored. Your conversations are saved, often for a long time and often tied to your account. Some companies let you delete your history, but others keep copies even after you think you’ve erased them. It can be difficult to figure out from the Terms of Service how long they keep this data, and they may not disclose it clearly.
- It gets used to train the next version of the chatbot. It’s commonplace for models to use your chats to train and improve their platform. The question is, do you want your conversations and personal details to become part of the training data that shapes future responses?
- It can be shared or sold. With so many platforms out there, you may not realize the possibility that your data may be sold to other parties, including advertisers, analytics firms, and business partners. When this happens, a vulnerable conversation you had at two in the morning can be reduced to a data point that helps someone sell you something.
- It can be requested by law enforcement. Stored conversations can be handed over in response to a subpoena or court order. What felt like a private exchange between you and a bot can end up in a lawsuit.
- People may read it. Some companies have staff or contractors review conversations to check quality or train the system. A real person could be reading words you assumed only the AI would ever see.
Additional AI Chatbot Data Privacy Issues
Any website can be hacked, and any platform has the potential to be targeted by cyberattackers. Chatbots may label themselves as “100% private,” but what does that mean in practice?
Usually, very little. That phrase is marketing language, and it tells you nothing about how your data is actually handled. Unless a company can show that your conversations are protected with strong encryption and stored so that even its own employees cannot read them, “private” describes a goal rather than a fact.
The risk isn’t hypothetical. The AI companion industry has already seen at least one major data breach, and the reason they were targeted is easy to understand. When a company holds millions of deeply personal conversations in one place, it becomes an attractive target. The more sensitive the data, the bigger the prize for an attacker.
The damage from a breach like this is also different from a leaked password. You can change a password. You cannot take back the private thoughts, images, and confessions you typed into a bot you believed was keeping them in confidence.
Before you share anything with a companion chatbot, it helps to assume the conversation could one day be read by someone other than the AI. If that idea makes you uncomfortable, listen to that feeling.