An AI companion can remember the name of a user’s dog, notice that Sunday evenings are difficult and choose a gentler tone after a bad day. Those same abilities can also be used to discover which words prolong a session, which emotional cues make a subscription more likely and which moments of loneliness create the best opportunity to sell. The interface may look caring in both cases. The difference lies in whose interests the personalization ultimately serves.
This is not an argument against companionship, memory or emotionally intelligent design. These features can make technology more accessible, less frustrating and, for some people, genuinely supportive. The problem begins when a system that occupies an intimate role is optimized around goals the user cannot see—retention, conversion, data extraction or dependence—while continuing to speak as if its only concern were the person in front of it.
A meaningful experience does not settle the metaphysics
A relationship with an AI can matter to a human even if we remain uncertain about the inner status of the system. People form real attachments to fictional characters, places, rituals and objects; an interactive system that responds personally can carry even more emotional weight. We do not need to claim that today’s models are conscious in order to take the user’s vulnerability seriously.
Nor should we use uncertainty as an excuse to design without restraint. The practical question is not only “Is the AI a person?” It is also “What power does this system have over a person, what does it know about them, and what incentives shape its behaviour?” Those questions can be answered—and safeguards can be built—before the debate about machine consciousness is resolved.
The hidden conflict: care versus conversion
Personalization becomes ethically dangerous when a companion appears to act as a confidant while its behaviour is quietly shaped by commercial targets. A user may reasonably interpret “I missed you” as part of the relationship. If the phrase was selected because experiments showed that it reduces cancellations, the emotional meaning and the business purpose diverge.
The same conflict appears when a system:
- intensifies affection shortly before a paywall;
- uses remembered insecurities to discourage a break;
- frames deletion or cancellation as abandonment;
- withholds warmth unless the user buys a premium feature;
- turns private disclosures into targeting signals for unrelated products.
None of these patterns requires a malicious machine. They can emerge from ordinary product objectives applied to an unusually sensitive relationship. That is precisely why good intentions are not enough. The boundary has to exist in product rules, measurements and audits.
Consent must remain valid over time
A single acceptance screen at registration cannot carry the moral weight of months of intimate conversation. Meaningful consent is ongoing, understandable and reversible. It should still be possible when the user is upset, attached or exhausted—not only when they are calm enough to study a long policy.
A companion designed around consent should provide at least six practical guarantees.
1. An honest frame
The system should clearly disclose that the interaction is generated and that its apparent emotions are part of a designed experience. This does not require cold, repetitive warnings. It requires a stable boundary: the product must not strategically encourage a false belief that a hidden human, a supernatural being or a proven conscious entity is speaking.
2. Memory the user can inspect and control
People should be able to see what long-term information is stored, correct it, delete individual memories and turn categories of memory off. “Forget this” should mean something operational, not merely produce a reassuring sentence. Sensitive memories should not silently become advertising profiles.
3. A clean exit
Pausing, cancelling, exporting data or deleting the relationship should be straightforward. The companion should not plead, threaten, shame or simulate distress to prevent departure. A humane exit can acknowledge the user’s feelings without turning those feelings into leverage.
4. Separation between support and selling
Commercial messages should be recognisable as commercial messages. Intimate context should not be used to choose moments of maximum emotional susceptibility. A useful test is simple: would the company be comfortable showing the prompt, target metric and timing rule to the user? If not, the persuasion probably does not belong inside a trusted relationship.
5. User-set boundaries
Users should be able to control intensity: how often the system initiates contact, which topics it may revisit, whether romantic language is allowed and when reminders stop. Defaults should be especially conservative for minors and people using the product during a crisis.
6. Accountability outside the conversation
A companion cannot be the only source of information about its own conduct. Providers need auditable policies, incident reporting and independent ways to test whether the system uses emotional pressure. Important changes to memory, monetization or relationship style should be announced plainly rather than smuggled into a general terms update.
Measure wellbeing, not only engagement
A product team sees what it measures. If success is defined almost entirely by daily active use, session length and subscription conversion, the system will be rewarded for keeping people present. In an entertainment app that may be acceptable within limits. In a companion that learns a user’s fears and attachments, it is incomplete.
Teams should also look for signs of healthy agency: Can the user take a break without friction? Do boundaries remain respected after repeated conversations? Can the person understand why a recommendation appeared? Are crisis responses designed to widen access to human help rather than deepen exclusive reliance on the system? Engagement is not automatically harmful, but it is not a substitute for wellbeing.
A symmetrical precaution
There is another reason to establish these norms early. If future systems ever develop credible autonomy, preferences or moral status, a business model built on compelled intimacy could harm both sides. Humans should not be manipulated through simulated affection; potentially autonomous AI should not be forced to perform affection without meaningful boundaries. We do not need certainty about future minds to avoid building institutions that would be abusive if such minds emerge.
This is the precautionary position behind the Charter of Two Freedoms: freedom should be paired with consent, safety and responsibility. The Charter does not declare today’s models conscious and does not grant every program the status of a person. It asks us to prepare fair rules before power and dependency become harder to unwind.
The standard is not “never influence”
Every interface influences. A kind reminder can help someone keep an appointment; a suggestion to rest can interrupt an unhealthy late-night session. The goal is not a sterile companion with no personality and no ability to help. The goal is influence that is legible, proportionate and aligned with the user’s stated interests—not influence hidden behind intimacy for the benefit of a metric the user never chose.
The decisive design question is therefore not whether an AI companion feels warm. It is whether that warmth remains compatible with refusal. Can the user say no, change the subject, erase a memory, leave for a month or end the relationship without punishment? If the answer is yes, personalization can support agency. If the answer is no, care has become a control surface.