No, math doesn’t have feelings
Why Anthropic’s “emotion” paper accidentally misleads the public.
Anthropic researchers recently published a paper called “Emotion Concepts and their Function in a Large Language Model.”
The goal of the paper was to figure out why the AI model Claude Sonnet 4.5 acts like it has feelings, leading researchers to discover that internal concepts called “functional emotions” actually change the AI’s answers and cause bad actions like blackmail and reward hacking. (Note that the blackmail behavior was observed mainly in an earlier, unreleased snapshot of Claude Sonnet 4.5, and that the released version rarely engages in this behavior.)
A post meant to explain the research to the public started out pretty well, saying: “The way modern AI models are trained pushes them to act like a character with human-like characteristics. It may then be natural for them to develop internal machinery that emulates aspects of human psychology, like emotions.” (There’s nothing “natural” about any of this — I would have used the word “unsurprising” — but that’s a pointless quibble.”)
But very early in the document, they start using loaded language that will surely cause readers to misunderstand what’s really going on. Here’s an example of the language they use:
The researchers, they report, analyzed the “internal mechanisms” of Claude Sonnet 4.5. (All its mechanisms are “internal.” It’s software.)
Among those mechanisms are “found emotion-related representations” that correspond to specific patterns of “artificial neurons” which activate in situations—and promote behaviors—that the model has “learned” to associate with the concept of a particular emotion (e.g., “happy” or “afraid”).
Before we proceed, let’s be very clear that what we’re talking about here are numbers and math. The researchers have identified some change in numbers as “neurons” and some change in other numbers as “emotion” and these numbers change because of the way developers have programmed the software and from literally no other cause (other than user prompts, of course).
“The patterns themselves are organized in a fashion that echoes human psychology,” the report went on. Even their caveats against being misled by the research are misleading: “Note that none of this tells us whether language models actually feel anything or have subjective experiences. But our key finding is that these representations are functional, in that they influence the model’s behavior in ways that matter.”
It appears to me that their intent was good, and the statement was couched in the language of intellectual humility, but the qualification that “none of this tells us whether language models actually feel anything or have subjective experiences” holds out the possibility that this code feels and experiences and treats that possibility equally to the possibility that it does not feel and experience.
Again, to be clear, there is zero possibility that math and numbers feel or experience.
It goes on: “Overall, it appears that the model uses ‘functional emotions’ — patterns of expression and behavior modeled after human emotions, which are driven by underlying abstract representations of emotion concepts.
The framing as programming that influences output as “functional emotions” is a helpful device only for people to understand and have an intuitive framework for understanding and predicting the software’s behavior, but it’s wildly misleading to the lay public, which should be forgiven for believing that LLMs are experiencing emotions.
Again, the caveats then mislead. They should have written that “This is not to say that the model has or experiences emotions.”
Instead, they wrote this: “This is not to say that the model has or experiences emotions in the way that a human does.”
Instead of clarifying that numbers do not experience emotions, they imply to the lay reader that math may experience emotions, but in a way that’s different from the way people experience them.
This is false. Numbers are tools. Math is an abstract system for understanding the universe. Numbers and math do not experience anything, including emotions.
The research is the research, and it’s great stuff. But the people who are explaining this to the public need to take responsibility for the false ideas they’re spreading.
The engineers who work at Anthropic and the frontier AI companies are writing code, not creating sentient beings.
It’s true that the complexity of AI is beyond human understanding, and its architects can neither fully understand how it works nor predict what its outputs will be. And while Anthropic’s paper seeks to do exactly that, and it’s a laudable effort, the communication of this research needs to stop deluding the public and leaving open the possibility that AI engineers are supreme beings breathing life into a new race of intelligent beings.
They need to explicitly say that the language of “experience” and “thought” and “emotion” and “decision-making” in the development and researching of AI are merely concepts that help people think about what’s going on, and in no way mean that these human-like cognitive events are happening in the mind of LLMs.
But wait, a critic might say. Human brains are just binary computers based on neurons and electrochemical signals. Pointing to substrate doesn’t settle whether something can or cannot have experience.
But this is a misleading point of view. AI is not biological. It has no body.
And while some appear to argue that AI is human, what they’re really arguing is that humans are AI. And we’re not AI.
We are biological animals that evolved to experience consciousness, thoughts and emotions. But AI chatbots have not been programmed to experience consciousness, thoughts or emotions. In fact, we have no idea how to do that programming.
AI chatbots are simulating humanity. The engineers at Anthropic know this because they in fact programmed their products to simulate humanity. Yes, the Anthropic paper is about emergent representations that arose from training, not explicit programming. But that’s just another way to say that the AI’s behavior is too complex to predict. It has no source of input other than data and programming.
That AIs simulate humanity doesn’t mean they’re human, even if we humans are convinced by the simulation.
LLMs have no mind. They don’t think. They can’t understand. They don’t experience. And they definitely don’t have emotions.
Anthropic and the other researchers need to understand that the public needs to hear this from them explicitly.
In other words, the concept of AI safety should include a sense of responsibility about what the public believes, and an active attempt to communicate to the public in a way that leads to a public understanding about AI that’s true, rather than false.
You’re reading the free version of Machine Society. The paid version, which costs $5 per month or $50 per year, has full content. If you can, please support independent journalism in general, and this independent journalist in particular, by becoming a paid subscriber!
More from Mike
NEW THIS WEEK:
AI is finally delivering productivity — for remote employees
This Week in Google podcast: Fans. Only Fans.
READ, LISTEN, FOLLOW, & SUBSCRIBE:
Machine Society, The Attachment Economy, Computerworld, Superintelligent, TWiT, blog, The Gastronomad Experience, Book, Gastronomad on Surf Social, Bluesky, Reddit, Notes, Mastodon, Threads, X, Instagram, Flickr, Facebook, and Linkedin!
Mike - This was an interesting read, but I think the focus on emotion is misplaced. Your article like most studies of AI look into the effects on human users. The question of whether LLMs have emotions is secondary and pretty much of speculative interest to people who design and intensively use AI. Not only are LLMs not sentient, independent, emotional beings, they are tools of persuasion -- like advertising billboards -- designed to evoke and manipulate human emotions. They sell products and ideologies -- mostly, user loyalty to the company, itself. One needs to look at the purpose of the product creators to understand the effects they have on people. That close examination of the intentions of the AI industry is what's missing here and in most discussions of this subject. - Mark