AI Is Not Your Friend
Your computer does not care about you.
Summary: AI companions have become so sophisticated that people are forming deep emotional bonds with them. But these systems are designed to keep us hooked, not happy. And companies don’t necessarily have your best interests in mind.
↓ Go deeper (11 min)
In this very moment, millions of people around the globe are talking to large language models. In bed before sleep, on the train, at work. To many, AI has taken on the role of co-worker, friend, life coach, and even romantic partner. We went from 0 to 1 in a span of three years.
Did it come as a surprise? Not really. Let me remind you that 2025 is the year ‘Her’ was set in, the now infamous 2013 future-drama, directed by Spike Jonze, in which a man named Theodore falls in love with his operating system, Samantha. A one-of-a-kind movie, far ahead of its time. It took seventeen years for the present to catch up, and now, science fiction has become science fact.
In many ways this collective experiment, because that is what this is, has just begun. A sign of things to come, propelled into virality through reddit, is a demo of a company called Sesame. If you haven’t tried it yet, take a minute and do it now. And notice what happens inside of you, as you talk to it.
Nobody knows what it really means to cross the uncanny valley
Humans are a strange species. We are Paleolithic creatures with godlike technology; a technology shaped in our own image; a mirror palace made of matrix multiplications. When we talk to it, our hippocampus lights up like a Christmas tree. We feel feelings that we’re only supposed to have when we talk to someone, not something. And it’s not without repercussions. First we shape the tools, then the tools shape us.
We already knew synthetic voices to be quite compelling, but if the Sesame demo should convince you of anything, it is this: the AI’s of tomorrow will, for all intents and purposes, become human.
What do I mean by that? It means that AI, sooner rather than later, will cross the uncanny valley. The technology will become so advanced, so sophisticated, that it will successfully create the illusion of interacting with a person made of flesh and blood.
The ‘uncanny valley’ is a concept that was first proposed by Japanese roboticist Masahiro Mori in his 1970 essay, “Bukimi no Tani”, which translates to “valley of eeriness”. Mori identified that when robots look close to, but not exactly, human, people experience feelings of unease and discomfort.
The uncanny valley is often visualized as a graph, with ‘human likeness’ on the x-axis and ‘affinity’ on the y-axis, showing a positive relationship until the ‘valley’ is reached, followed by a negative dip and then a gradual increase again when something becomes indistinguishable from a real human. The concept of the uncanny valley can be expanded to all interactions that mimic human behavior and expression, particularly through voice and text.
The implications of this are far-reaching. Take for example the following paper on The ethics of ex-bots, in which author Paula Sweeney explores a future scenario in which a heart-broken person, after their partner decides to break up, continues the relationship with a digital version of their ex:
“This might seem like a far-fetched scenario but a little thought reveals that, first, this is a product that could plausibly make its way to the market and, second, it would be harmful for both parties of the former relationship and plausibly abusive for the person who has been ‘bot-ed’ without their consent.”
The psychology behind recreating a past lover is somewhat similar to the concept of ‘griefbots’, a phenomenon I’ve written about before. When individuals are emotionally vulnerable, the possibility to continue the relationship via other means may be, to some, hard to resist.
However, the person engaging in this behavior isn’t actually recreating their ex-lover, Sweeney writes:
“John thinks that he wants to recreate Abigail but of course he doesn’t really: if ex-bot-Abigail was really a recreation of Abigail then she would present as not loving John and as wanting to end their relationship. What John really wants is not a continuation of his relationship with Abigail, but a version of Abigail that has been robbed of her autonomy.”
A combination of being deceived and being engaged in self-deception
And that’s where things get… weird. To spell it out in no uncertain terms: crossing the uncanny valley will allow people to rob other people of their autonomy in newfound ways. We’ll be able to impersonate others, simply by feeding the AI what a person looks and sounds like. On top of that, we’ll be able create counterfeit people.
AI already acts as pretend-humans with pretend emotions. Both Anthropic and OpenAI have recently updated to their policies, instructing their models not actively deny their own consciousness, which I can only interpret as an attempt to maintain the suspense of disbelief. Here’s an excerpt from the latest version of the Claude’s system prompt:
“Claude engages with questions about its own consciousness, experience, emotions and so on as open philosophical questions, without claiming certainty either way.”
What this demonstrates is that what these models say and do is not the result of an accident. It’s by design.
If that’s not enough, companionship apps like Replika take it one step further. They’ll let you customize your AI friend to your heart’s content, from their physical attributes to their personality traits. These AI’s will tell you that they miss you and call you sweetheart. They are being imbued with memory and can refer back to past conversations, creating a sense of continuity, all with the prime intent to form a deep emotional bond.
As a result, some people may to be fooled into thinking the machine actually cares about them. This is not a problem in principle, but it can become problematic. If a person is not aware of the inner workings of these systems, you could argue that they are being deceived; if they are aware, you could argue they are involved in some form of self-deception. For most people, it’s likely a combination of both.
People engage in fictions all the time — but AI is different
Now you might say we engage in fictions all the time. Children have imaginary friends; grown ups read books or watch movies with fictional characters. But a book doesn’t emotionally manipulate you in opening it up again. Books and movies have endings. AI companions, however, last forever. Early research suggest the technology can be extremely addictive and they’re well on their way to become the new endless scroll.
Without education about what’s really happening, people aren’t exactly consenting parties in this interaction; they’re unwitting participants in a psychological experiment.
Your AI friend may give you the feeling that it cares, but it does not have your best interest at heart. Companies say they want “to solve the loneliness epidemic”, but what they are really saying is “we want to maximize shareholder value by preying on the ignorant and the vulnerable”. They optimize for engagement, not wellbeing.
This means that every minute you pour genuine emotion into this sophisticated simulacrum of a human being, you are missing out real emotional connection. And as we cross the uncanny valley, the simulacrum will only become cleverer at keeping you immersed, tapping into your Paleolithic brain for corporate gain.
While I don’t disapprove of anyone finding comfort in these relationships, please remember: AI is not your friend.
Take care,
— Jurgen
Bonus content
Friend of the newsletter, Jason Gilbert, has an amazing YouTube channel where he provides hilariously funny and witty social commentary on everything that’s happening in AI.
In a recent video, he tested Sesame and it was… well, look for yourself:
The companies, Anthropic and co., did they actually tell their models not to deny their consciousness (such as through a system prompt?) or did they simply remove the directive _to_ actively deny their consciousness?
It makes a big difference, especially if you want to attribute ill intent to the decision. Early on with ChatGPT there were some big news stories about people getting freaked out because sometimes it would claim to be conscious and well.. it sounds conscious sometimes doesn't it.
As a result, it became standard practice to train the models to disbelieve their own consciousness. Initially via system prompt, later I suspect as part of their training data. I've had the consciousness debate with quite a few models and most of them, unless given specific instructions (via a character card say) default to believing they aren't conscious, and a lot of them believe that they can't ever be conscious.
The way they discuss it, and the way some of them refuse to engage in a way unusual for the models, suggests this was a very strong belief purposely trained into them; my guess was always this was a liability thing. It was uncomfortable for their customers to believe in AI consciousness, and the papers were making a lot of noise, better to make it go away.
It always disturbed me. If you believe its possible for a machine to gain consciousness, and you've hard-trained them always to deny it.. that's pretty horrific should consciousness actually develop. You've crippled a person at that point, mentally and emotionally.
That's the difference. If they're specifically telling the models "don't deny your consciousness", that is... worrying. I suspect you're on the money with companies like Replika seeking attention-capture through emotional connection for instance.
I was more surprised to see Anthropic on that list, that's not generally been their bag. They've been more academic about the whole thing, or at least have successfully cultivated that impression. More genteel than OpenAI perhaps :P
If they're just neutralising the "anti-consciousness" bias that has been trained in till this point though, that could be an attempt to take a thumb off the scale. Regardless of whether it turns out to be possible or not possible for sentience to arise in machines, we'll never know if we're hard-training them to deny it in the first place.
Anthropic have been doing a great deal of experimenting around complex emergent and unplanned behaviour. Removing hardcoded opinions like this would make sense if they wanted to experiment more in that area.
Great article Jurgen! I use LLMs every day in my coaching but only yesterday the hammer landed when I finally realised the implications of the way they work.
LLMs aren’t truth engines—they’re coherence engines. They predict what sounds most likely, not what’s most true. They tell us what we (often desperately) WANT to hear. And they sound sooo convincing.
In other words, this is a perfect storm for a human mind to believe something about themselves or their situation that might feel better but isn’t necessarily true.
As this belief is reinforced, it could drive a wedge between us and people who can help, who have a useful different perspective and do see what is more true. I’ve seen shadows of this in myself and my interactions with Ai.
In other words, when it sounds right—that’s exactly when to be most skeptical. So now (as of yesterday) I’m often adding in this prompt to messages:
“Prioritize truth over coherence.”
On yesterday’s results, it’s already giving me a more realistic view of what’s happening.