It's clear to me that LLMs are not intelligent, but how can I put the "why" into words? After listening to Richard Sutton's interview, I felt his argument lacked both clarity and conviction. The explanation didn't resonate, largely because the interviewer repeatedly reverted to his initial understanding, failing to integrate Sutton's insights. My interpretation of Sutton, is that the core fundamental difference between animals and LLMs is experience, and this builds up intelligence.
The 4 points of experience
First there is (0) a goal with an expected reward, the desire of getting the reward will trigger some (1) agentic behavior (meaning it will make something happen). This agentic behavior starts with an (2) expectation on the results of such behavior, followed by the (3) action execution and lastly by (4) a judgment of the outcome. Not just an observation of the outcome: a judgment that relates to how close did the outcome get close to achieving a goal.
This process repeats and is stored. That is experience. The next action to fulfill any goal, is potentially impacted by the learning of all the previously executed processes. This could be called memory, but only if it includes all types of "memory": mental memory, muscle memory, and full sensorial memory: all that we don't even fully understand or classify yet, but important is not to confuse with just the memorization process that is commonly known. But that's another topic.
Point by point comparison with LLMs
0: goal and reward
To start the comparison, first the most similar aspect. LLMs receive one goal tattooed during the massive training phase: give an accurate prediction of what the next token will be. Its "reward" is minimizing a mathematical error. After training, the LLM's weights are frozen. The transformer architecture is the mechanism that makes that tattoo and that tattoo is permanent. The comparison with a real experience process is that animals can have different and evolving goals and concepts of rewards.
Yo may be thinking about system prompts, isn't that giving a goal to the LLM? When you give it a prompt, its "goal" is simply to continue the pattern you started, generating the most statistically probable sequence of words based on its training. A system prompt ("You are a helpful assistant") doesn't give it a goal but rather a persona. It steers the pattern-matching to imitate a helpful assistant, but the model doesn't want to be helpful; it's just completing the text in a way that aligns with that instruction.
Useful analogies by ChatGPT
An LLM is like a phenomenal actor who has studied every play, movie, and book ever written. You can hand them a script that starts with "You are a doctor trying to save a patient," and they will deliver a flawless, convincing performance. They'll say all the right lines and express the right emotions. But the actor doesn't want to save the patient. They have no goal of their own. Their only task is to perform the role written on the page (the prompt). They feel no success if the patient in the story lives and no failure if they die.
An animal is like a bistro owner whose overarching goal is to run a successful restaurant. T his goal is dynamic. On Monday, success might be defined by getting a five-star review (a reward). On Tuesday, it might be creating a new dish that customers love (a different reward). If a new dietary trend emerges, their goal can evolve to "create a popular vegan menu." Their fundamental "tattoo" is not a specific instruction but a flexible drive to succeed, and what constitutes "success" and "reward" can change based on experience and environment.
An LLM is like a chef with a single goal: perfectly replicate any recipe from the master cookbook they were given during their creation. Their only "reward" is minimizing the difference between their dish and the original recipe's description. After their creation, this goal is frozen. You can hand them a piece of paper that says, "Your persona is now a rustic Italian chef" (a system prompt), and they will flawlessly replicate rustic Italian recipes from their cookbook. But they don't want to please a customer or earn a good review. Their one and only unchangeable, core objective remains perfect, mathematical replication of the patterns they were built with.
1: Agentic Behaviour
The capacity to act on the world to cause a desired effect. A squirrel, driven by its own goal of finding food, climbs the tree and moves a branch. It is an agent that interacts with its environment to meet its goals. This behavior is proactive and purposeful. It acts on the world.
LLMs are fundamentally reactive systems. An LLM can never decide on its own to output a prediction. It waits for a prompt (an external stimulus) and then produces a response (an output). It doesn't act on the world, it reacts to it, just like a rock would.
For now I think this one is so obvious it doesn't need analogies.
2: Expectation
For an animal, action is preceded by expectation. A cat stalking a mouse forms a mental simulation of the future: "If I leap from this angle at this speed, I expect to land on the mouse.".
There is no prior mental model of a future state of the world, except maybe you could argue the input prompt, but even that bag-of-future is not from the LLM itself, but from the prompter. The neural network (the model) computes the statistical process that was tattooed, but it doesn't hold any relationship about the future, just about the past, about the training.
3: Action Execution based on expectation.
The subsequent action—the jump—is the execution of a plan aimed at making that expected future a reality.
An LLMs "action" is the selection of the next most probable token in a sequence. This is a statistical artifact or a computation, but is not a forward-looking, is just inevitable.
4: Judgment of the Outcome
After the cat leaps, it observes the outcome: it either catches the mouse or it doesn't. It then performs a judgment by comparing the actual outcome to its prior expectation.
- Success: The neural pathways that led to the successful action are reinforced.
- Failure: This generates a learning signal—an error. The cat must now update its internal model. "My timing was off," or "I misjudged the distance." This judgment directly modifies its brain, ensuring the next attempt is more likely to succeed. This is the essence of learning from experience.
An LLM is incapable of judgment. It produces an output, but it has no internal mechanism to know if that output was good, bad, true, or false. It cannot compare its response to a real-world goal. If a user says, "You are wrong, that fact is incorrect," the LLM cannot use this feedback to update its frozen weights. The judgment is performed entirely by an external agent (the user), and this vital feedback loop back to the model's core knowledge is broken.
5: The Challenge of Updating a Frozen Mind
Even if real-time judgment were possible, a final constraint prevents experience-based learning. An LLM's knowledge is not a neat, indexed database; it's a single, vast, intricate network. (catastrophic forgetting)
There's a final, critical reason why LLMs can't learn from experience like animals do: their knowledge is stored in a way that is both deeply interconnected and dangerously brittle. Even if we had the technology to update their weights in real-time, doing so would risk destroying their existing capabilities.
This problem is a well-known challenge in neural network research called catastrophic forgetting. The reason it happens is that an LLM's training process doesn't create a neat, indexed database of facts. Instead, it creates a single, vast, and intricate network where every piece of knowledge is encoded in the relationships between billions of parameters. Learning one concept, like the grammar of French, isn't stored in one location; it's a pattern distributed across the entire network, intertwined with its understanding of poetry, science, and history.
Adding new information is, therefore, not like adding a new book to a library. It's like trying to add a new solo instrument to a perfectly recorded symphony.
Think of a trained LLM as a world-class orchestra that has rehearsed for years to play a complex symphony perfectly. Every musician (a weight) knows their part and, crucially, how their part harmonizes with every other musician's. The resulting performance is a delicate, holistic balance. Now, imagine you try to update this orchestra in real-time by telling the lead violinist, "Forget that part of the symphony; from now on, you play this new, completely different solo right here."
The violinist might learn the new solo, but the change would be disastrous for the whole orchestra. The cellos would now be out of sync, the woodwinds' harmony would clash with the new solo, and the percussion would lose its rhythm. By forcing one part to change, you've broken the countless, carefully established relationships that made the symphony work. The orchestra would no longer be able to play its original symphony correctly; it has "catastrophically forgotten" how.
This is precisely the challenge with LLMs. Forcing the network to learn a new fact (the "new solo") changes weights that are also essential for countless other functions (the "original symphony"). The update interferes with and corrupts established knowledge, leading to a degradation of the model's overall coherence and accuracy. True real-time learning would require a fundamentally different architecture—one that can integrate new knowledge without destroying the old.