I Finally Understand What 'Attention' Actually Does — and It Changed How I Scope AI Features

After Karpathy's lectures gave me the big-picture mental model of LLMs (pretraining, fine-tuning, why hallucinations are structural — I wrote about that separately), I went one level deeper: Grant Sanderson's 3Blue1Brown series on neural networks and transformers.

Fair warning: this series has math in it. But it's visual math, and that's not too hard to understand. I don't need to implement backpropagation. I need an accurate picture in my head of what's happening when my product calls a model. At least accurate enough to challenge an engineer, scope a feature, or explain to a stakeholder why something is hard. This series builds that picture better than anything else I've found.

Here's what stuck, translated into product language.

1. A neural network doesn't follow rules — it has dials

The foundational reframe: a neural network is millions (or billions) of numbers, called weights, that get gradually adjusted until the whole system produces good outputs. Nobody programs the behavior. Training is turning billions of tiny dials via gradient descent: measure how wrong the output is, compute which direction each dial should move to be slightly less wrong, repeat a few billion times.

Once you see this (and the series literally shows it), several PM realities stop being frustrating and start being obvious:

Why can't the engineers just fix that one wrong answer? Because there's no line of code that produces it. The behavior is smeared across billions of weights. Why is model behavior evaluated statistically instead of tested like normal software? Same reason. Why do small prompt changes sometimes flip the output? You're steering a learned statistical system, not calling an API with a contract.

This is the deepest justification for evals I've encountered: when you can't inspect the logic, measuring the behavior is all you have. Quality assurance for AI products isn't QA with extra steps. It's a whole different discipline on its own.

2. Words live in a space where meaning is geometry

Before a transformer does anything, each token becomes an embedding, a point in a space with thousands of dimensions so to say, where direction encodes meaning. The classic example: the vector from "man" to "woman" is roughly the same as from "king" to "queen." Meaning becomes something you can measure and do arithmetic with.

For a me, this is the concept that unlocks half of the modern AI stack. Semantic search, RAG, recommendations, clustering user feedback. Under the hood, they're really all just "find things that are nearby in embedding space." When I scope a feature like "match a user's product picture to the relevant data of it" I'm really specifying an embedding-similarity problem. Knowing that changes what I ask for, what I expect it to cost, and how I'd measure whether it works.

3. Attention is context updating meaning — and it's why context windows cost money

The transformer chapter finally made "attention" concrete for me. An embedding for the word "model" starts out generic. Attention is the mechanism that lets other words in the context: "fashion model" vs. "machine learning model". Pull that embedding toward its actual meaning in this sentence. Every token gets to ask questions of every other token ("is there an adjective in front of me?") and update itself with the answers.

Two consequences:

First, why long context is expensive: every token attending to every other token means cost grows roughly with the square of the context length. When engineering pushes back on "just feed the whole document in," that's not laziness, but sadly math. It's arithmetic. It's also why "RAG vs. long context" is a genuine architectural tradeoff.

Second, why order and placement in a prompt matter: attention decides what influences what. Context engineering stops being folklore ("put instructions at the top, I heard") and starts being something with a mechanism behind it.

4. Facts live in the layers between attention

One more insight I hadn't seen anywhere else: the MLP blocks, the layers between attention steps, which hold most of a model's parameters, are plausibly where a model stores its factual associations, like "Michael Jordan plays basketball." And research suggests features are stored in superposition: packed on top of each other, more concepts than there are dimensions.

That's a beautiful explanation for a behavior every PM has to design around: knowledge in an LLM is dense, entangled, and not cleanly editable. You can't surgically update one fact. Which is, again, why architectures that keep volatile knowledge outside the model, like retrieval, tools, databases, keep winning for real products.

The meta-lesson

Going one level deeper technically has immediately improved my judgment. And it has demystified a lot what was called a "black box" by many of the people I talk to. Spending just a few hours on the topic will definitely help you understand the concepts of the magic that is called AI.

---

Both series are free on YouTube. If Karpathy is the "what and why" of LLMs, 3Blue1Brown is the "how" — watch them in that order.