Models Don't Reason, Patterns Do
Large language models achieve remarkable performance on tasks that appear to require reasoning: mathematics, logic puzzles, code debugging, medical diagnosis.
This leads most people to conclude: "The model reasons."
Then the same model fails on a trivial substitution cipher, or cannot distinguish between a cat and a dog when the image is slightly rotated. If the model truly reasoned, it would be consistently logical across all domains.
What it does instead is powerful pattern matching at a scale humans can't inspect — and it looks like reasoning because the patterns it learned happen to map well to our tasks.
What Actually Happens: Statistical Pattern Recognition
A transformer model processes text by converting every token into a high-dimensional vector, then applying layers of attention-weighted transformations. Attention learns which tokens in context are statistically relevant to each other. Feed-forward layers compress patterns across millions of examples. The learned weights are not rules — they are statistical regularities encoded in numerical form.
When you ask "What causes type 2 diabetes?", the model isn't consulting a causal knowledge graph. It's generating the token sequence that best matches the statistical patterns it learned from billions of documents written by humans who were explaining diabetes.
The chess analogy:
A chess engine doesn't "think" about each move. It evaluates a vast position tree using learned evaluation functions refined over millions of games. It plays like a grandmaster because the patterns it learned happen to match optimal play — not because it understands chess theory.
We mistake performance for comprehension.
Why This Matters for Safety-Critical Applications
In medical, legal, and safety-critical contexts, the pattern-matching nature of LLMs creates specific risks:
- Confident wrong answers: The model produces fluent text that matches the pattern of correct explanations — but for the wrong domain or wrong patient.
- Novel case failures: Every patient is unique. Training patterns may not cover their specific combination of conditions.
- Distribution shifts: A model trained primarily on US-centric medical literature may perform poorly for populations with different disease profiles.
Today's Lesson
The model doesn't reason. It matches patterns at scale. Understanding this doesn't diminish what LLMs can do — it makes you a better builder, evaluator, and user. Build for what it actually is. Not for what it looks like when it performs well.