We never traced each other either

The reason that arrives in a second

Ask me why I thought something, and I'll give you a reason in a second. Ask an AI, and it'll do the same.

I'm not sure either of us actually knows.

The assumption I walked in with

I had a tidy theory. Humans think in a web: several streams at once, forward and backward, all the time. Machines think in a line: one fixed path, one direction. So we can trace our thoughts back to the root, and they can't.

It's a comfortable theory. It puts me on the right side of it.

Then I read two experiments, one for each half.

The machine and 36 + 59

Nobody writes a model's strategies by hand. They form during training and end up spread across billions of calculations, so even the builders can't read them. Dario Amodei, who runs Anthropic, wrote it plainly in 2025: "we do not understand how our own AI creations work." These systems, he says, are "grown more than they are built". He also sells them, so weigh that as you like.

In 2025 his interpretability team built what they call an AI microscope, borrowing the idea from neuroscience, to see what a model does rather than what it says it does. One of their tests was simple addition. A model trained to predict the next word somehow adds 36 and 59 correctly, so how? They expected one of two answers: it memorised the sums, or it learned the school method.

It was neither. One path estimated roughly how big the answer should be. Another worked out that it ends in 5. Together they gave 95.

Then they asked the model how it got there. It described carrying the 1, the way we learn at school. That wasn't what they had just watched it do. Their suggested reason is that it learned to explain maths from text humans wrote, and learned to do maths on its own.

Parallel paths were on my list of things only we do.

Two things to keep in mind. This is a lab studying its own model. And they say their tool captures only a fraction of what's happening, took hours of work on a short prompt, and may itself distort what it shows.

The people and the swapped faces

So my theory was doing fine on the machine side. Then I checked my own.

In 1977, Nisbett and Wilson argued that when we explain our own decisions, we may have little direct access to what caused them. The debate stalled on one problem: to catch a false explanation, you need to know the true cause, and normally you can't.

Researchers at Lund University built a situation where they could know. People looked at two faces for about four seconds and picked the more attractive one. Sometimes a card trick swapped their choice for the face they had just rejected. Then they were asked why they chose it.

In the study I read, with 80 people, only 27.5% of the swaps were noticed. The rest explained a choice they had never made. One said she looked pleasant and reminded them of a good friend. The researchers predicted huge differences between reasons for real choices and swapped ones. They found almost none that held up.

I watched a video of the experiment and thought: I'd catch that straight away. I'd say, "this wasn't my choice." Maybe I would. About a quarter of the time, people did. Maybe it's memory; four seconds isn't much. But memory only explains missing the swap. It doesn't explain why the others had a reason ready.

These were low-stakes choices made quickly, and critics argue it's an unusual case. It shows our explanations can be made up without our knowing. It doesn't show they always are.

I still think I was right about one thing: the brain is a web. I'm just less sure the web makes me traceable.

A confident reason on top, and something else underneath. That now seems to fit both of us. I find this more unsettling than my original theory.

What still feels different

I don't want to overcorrect, so here is what I still believe. Treat it as a hunch, because I can't measure it.

A model only runs when called. It doesn't learn from this conversation while having it. It has no body, and Yann LeCun's point about the four-year-old who knows a pushed glass will shatter still stands. And my "forward and backward" instinct may have something in it: the brain loops back on itself while it works, while a model computes each word in one pass forward, then feeds the result back in and goes again.

Those are real differences. They just aren't the difference I walked in with. Being able to explain ourselves was never one of them.

What we've been trusting

If we never traced each other, what have we been trusting all this time?

Mostly, checking. When a colleague says something will work, I don't ask to see inside their head. I wait and see if it works.

Paracetamol has been in clinical use for over a century. A 2021 review still calls it a familiar drug with an unexplained mechanism. Nobody asks the pill for its reasoning. We look at whether the pain went away.

That's the actual bottleneck. Answers got cheap. Knowing whether to believe them didn't.

The cure that arrives first

Dario's father died in 2006 of an illness whose cure rate went from about 50% to about 95% three or four years later. He has said that loss gave him his sense of urgency. It's hard to argue with. Three or four years is nothing, unless it's yours.

So take the dream seriously. A machine that gets there first. A cure before the illness arrives.

Then put the two ideas together. An answer from a system nobody can follow is fine, as long as we can check it. You can test a cure on the sick.

What do you test it against when the sick don't exist yet?

Prevention has always had this problem. Nobody thanks you for the flood that didn't happen. But so far the person proposing the flood wall could at least explain the river.

Last week I got a very small version of this. I had an AI audit my portfolio, and it reported five critical bugs. One was the ← Work link on my case studies, which steps back through browser history so you land where you were on the home page. The report called it an "Inescapable Two-Page Loop": go from one project to another, and the link keeps sending you back to the first.

It came with a clean fix. The fix wouldn't have hurt anything. It wouldn't have done anything either, because no project on my site links to another. It was a cure for an illness my site couldn't catch.

So I wrote down why: the page before a case study is always the home page. That was wrong too. Click a source link on a case study, then ← Work, and you stayed on the same page. I think I'd even hit it while testing and taken it for a glitch. The report missed it and so did I, and we both sounded sure. I've fixed that one.

Another finding in that report was real: my www redirect was dropping everything after the ? in a link. It came in the same confident voice as the wrong one. That's what a wrong answer really costs. Not the fix. Every other answer next to it is now worth a little less, until someone checks.

That was a website, and I could read the whole map.

I. J. Good saw part of this in 1965. He called the first ultra-intelligent machine the last invention we'd need, "provided that the machine is docile enough to tell us how to keep it under control". Everyone quotes the first half.

I don't have an answer. I'm not sure there's a wall, or that we'd feel it if we walked past it. I just notice that most of what I trust, I found a way to check, and I'm not sure how we check the things we're about to ask.