Which Bucket Is This In

A few weeks back I was ready to put a client’s site behind a content delivery network.

The reasoning was solid. A batch of sites was stuck under a mobile performance score we couldn’t get past. Everybody who looked at it said the same thing. That’s an infrastructure problem. The server’s too far from the user. Put a cache at the edge and the number moves.

It’s cheap. It’s a known fix. It was sitting in the recommended column of a real decision with a real dollar figure next to it.

Then somebody actually measured it.

We put a sixty-second cache in front of the server and ran it again. The score moved zero. Not a little. Zero. Then we traced the response time from the origin and it came back between fifty-two and ninety milliseconds, which is already fast, which means the thing we were about to buy was solving a problem the site didn’t have.

Worse. When we went back through that batch review, one of the sites we had labeled static was nothing of the sort. It renders on the server. Nobody had checked which mode it was even running in before drawing a conclusion about the whole fleet.

The diagnosis was confident. It was unanimous. It was wrong.

I’ve been doing this long enough to know the failure mode. You gather answers, they all sound equally solid, you build a plan on top of them, and eighteen months later you find out which ones were load-bearing.

So now everything I believe about a project gets one word next to it.

Verified means somebody published it and I can point at it. A measured response time. A number in a filing. Something with a source I can hand you.

Claimed means somebody said it. They might be completely right. I can’t produce the paper. “Everyone knows the ceiling is infrastructure” lived here the whole time and I was treating it like it lived one bucket up.

Inference means I worked it out myself. Every estimate I’ve ever written down is in here, including the ones I like.

That’s it. That’s the whole discipline. It isn’t sophisticated and it doesn’t need a tool. It needs you to stop and write one word next to each thing you believe.

The cost isn’t being wrong. Everybody’s wrong constantly.

The cost is being wrong with confidence and a budget attached. That kind takes a lot longer to find, because a number in a spreadsheet doesn’t come with a label saying where it came from. It was an inference wearing a verified jacket.

I’ve watched this from the other direction too. A while back one of my own tools told me a failed production deploy was caused by memory pressure. Confident. Specific. The box was six gigs into swap at the time, so it even had supporting detail.

I asked one follow-up question. What’s eating the memory?

We went and read the kernel log for that window. Zero out-of-memory events. Not during the failure. Not ever, on that machine. The claim had nothing under it.

The honest answer to what killed that deploy is that I don’t know. The build finished every step and died at the last one with no captured error, and that’s as far as the evidence goes. “I don’t know” is a real answer. It’s usually better than the confident thing that turns out to be furniture.

And this is the conversation I have with every client putting AI into their business. I’ve had it a lot lately.

The model gives you a confident answer. So does Google. So does the guy at the counter who’s done this thirty years. All three sound identical coming out.

The useful question was never whether it’s right. You usually can’t tell, and that’s the whole problem. The useful question is which bucket it’s in. Did somebody publish it. Did somebody say it. Or did something work it out.

People think AI’s problem is that it makes things up. That’s not really the problem. The problem is that it makes things up in the same voice it uses for things that are true, and so does everyone else, and so do you when you’re repeating something you heard three times. The machine didn’t invent confident nonsense. It industrialized it.

I’m not telling you the crowd is always wrong. The infrastructure fix might be exactly right on some other site. I’m telling you I couldn’t verify it on this one, which means I don’t get to spend money as though it’s settled.

So the fix isn’t skepticism. Skepticism is just being difficult.

The fix is a label. Go through whatever you’re about to bet on. The plan, the budget, the architecture, the thing the model told you. Put one word next to every line. Verified. Claimed. Inference. Don’t argue with any of them yet. Just sort the mail.

You’ll find one wearing the wrong jacket. There’s always one.

Half of doing this well is refusing to let a claim get promoted to a fact because you heard it three times.