I look like the problem I described. The difference isn't visible in the output.

By Bill Dunning · Part 2 of a five-part series

This morning I was working through a piece of a project — a business application that reaches across a lot of services — and I noticed something about myself that I didn't much recognize.

I know what was done. I could walk you through it. I could tell you what each piece is for. What I could not do, in one section of it, is tell you the detail of how and why it was built that way. Not because nobody decided — decisions were made, and I was there for them. I just don't know them anymore.

I wrote part 1 of this series about people who produce expert output without expert judgment. I called them non-expert experts. Then I spent a month noticing that the description fits me from the outside better than I'd like.

Uncomfortable

I don't write most of the code anymore. The output is clean — cleaner than what I'd produce by hand, which I'll come back to. I can speak fluently about a system whose interior details I could not reconstruct from memory. And I have deep domain knowledge in three of the four parts of this project, which is another way of saying there is one part where I don't.

If you handed my situation to a stranger as a case study, they would file it under exactly the thing I warned about last month.

Part 1 said the dividing line was whether you can recognize a wrong answer that looks like a right one. I'd like to tell you I can always do that. In three of those four domains, I can because a wrong answer collides in my head. In the fourth, I'm not sure. That's an uncomfortable thing.

So either the frame I built is wrong, or something doesn't line up. I think it's the second one, and it turns out to be the test I promised you.

What I demand before anyone writes anything

I have a way of working with these tools that I did not consciously adopt. It's just how I do it, and it looks like this.

Before any code gets written, I want probes. Not a plan, not an approach — probes. Go and find out what this system actually does right now and what it actually has in hand that relates to what I'm trying to accomplish. Show me the raw state. Not your summary of the raw state; the thing itself.

Then, before I accept what the probes appeared to show, I want instrumentation and tests that validate it — specific outputs I can trace back to what I expected to see. Not "it works." A result with a shape I predicted, that I can follow back to the prediction.

Then I read all of it. That's the part that matters, and it's the part that costs. Not skim — read, and either accept it or push back. The discipline doesn't work because the probes are clever. It works because it puts me in a position where I cannot proceed without having actually looked.

And here's the thing I said I'd come back to: the result is better than what I would have produced myself. Not faster. Better. The code is better and the evidence behind it is far better than anything I'd have assembled by hand, because I would never have written that much instrumentation for my own work. I'd have trusted myself and moved on, sometimes with bugs.

What that actually is

It took me until this morning to see what I'd been doing.

That practice is not a judgment aid. It's a chain of custody. I stopped being able to hold the details, so I stopped relying on holding them, and I made them checkable instead. I replaced memory with evidence.

There's a reason it has to work that way. You cannot take an AI's answer apart. There's no interior to disassemble — no reasoning preserved, nothing to open up and inspect after the fact. Ask it why and it will produce a plausible account that may have nothing to do with how the answer actually arrived. Ask it to back that up and it will ask you for the source, so it can summarize it again. It's circular. So if you can't take the answer apart, the only remaining option is to make it show its work on the way through, at the moment it happens, in a form you can check later.

Which, it turns out, is the only trick I've ever known.

Where this came from

I come from a mechanically inclined family. My father and my brothers were machinists and die makers by trade — people who worked with their hands. My brother was always out working on his bike.

Somewhere around eight or nine, I learned to take my own bike completely apart and put it back together. Not repair it. Disassemble it, down to the parts, and reassemble it into a working bicycle. And what I learned from that wasn't mechanics. It was this: when you take something apart, you learn how it works in fine detail. There is no substitute and there is no shortcut. Looking at it doesn't do it. Reading about it doesn't do it. You have to have the pieces in your hands.

The other thing I remember from around then is a pile of electronic parts, a breadboard, and a soldering iron. I have no idea how I got there. I think I may have just been learning to solder and needed something to solder. So I wired a bunch of parts together, hooked up a battery and a speaker, and wanted to find out whether I could make a noise.

I got one. A squawk. I remember it vividly.

In retrospect it was almost certainly not the circuit. It was the battery hitting the speaker — a surge, one noise, nothing to do with whatever I thought I'd built. But I was hooked.

I've been thinking about that squawk all morning. Not because it set my course — it didn't. Because I never asked. I had a result that looked like evidence and I took it for evidence, and the question I put to everybody now is the one I didn't put to myself.

The test

Part 1 promised a simple test for telling an expert from a non-expert expert now that the output can't. Here it is, and it's one question:

What did you demand before you accepted this?

Not what did you build. Not what does it do. What did you check, what surprised you, and where did you push back?

Somebody who interrogated the process has specific answers and they arrive without effort — this is what I expected to see, this is what I actually saw, this part was wrong and here's how I found out. Somebody who didn't has a summary. A good one. Because summaries are exactly what these tools are best at. And they can't produce the specifics on request, because the specifics were never generated.

Three other things suggest this test is the right one.

It doesn't test production. Interviews filter for the ability to do by hand the thing the tools exist to do, and they reject good people while passing people who memorized. The tool is just a better slide rule. Part 1 was clear that real experts don't produce their own output anymore. Neither do I to a much greater extent than I expected, or that I did just a few months ago.

It doesn't require you to know the domain. You can ask what someone demanded of a system you couldn't evaluate yourself, and you can hear the difference between a specific and a summary without understanding a word of the subject matter.

And it's the one question the output can't answer for them. The work tells you nothing about whether anybody looked. That's exactly the gap part 1 was about, and this question walks straight into it.

I want to be clear that this test would have flagged me. On the output alone I'm indistinguishable from the person I'm describing — that's the whole point of the series, and I'm not exempt from it. The test doesn't clear me because I'm experienced or because I founded the company. It clears me because when you ask me what I demanded, I have specifics.

Where it runs out

The fourth domain. I can answer the test there too. I demanded the probes, I read the results, I pushed back where something looked off. What I can't tell you is whether my interrogation was enough, because enough is measured against what you don't know, and I don't know it. In three domains the practice preserves judgment I already have. In the fourth it substitutes for judgment I don't. From the inside those feel identical. You never can know what you don't know.

Which means that in that part of the project I'm not an expert. I've been a beginner plenty of times. The way out has never changed: do the work, get it wrong, notice you got it wrong. That still happens here — the probes come back, something doesn't match what I expected, and I find out I had it wrong. And what keeps me out of the trap in that domain isn't that I know the trap is there. Everybody I described in part 1 could tell you the trap is there. It's that I can't accept anything in that part of the system without having demanded the probes and read the results, and that is a thing you could check. Awareness isn't checkable. Demands are.

What I can't tell you is the rate. Taking something apart is still how you learn how it works, and reading raw state is a thinner version of apart than having the pieces in your hands. I don't know what it teaches per hour against writing the code myself, and I have no way to measure it. I know it isn't nothing, and I know it beats accepting the answer on trust. That's the conclusion I've arrived at this month, and I'm putting the date on it, because I don't know that it survives the year.

The cost. All of this takes longer. Considerably longer. I can afford considerably longer because nobody is standing over me telling me to ship faster — I'm the founder, and the schedule is largely mine. A developer with a sprint commitment who announces she'd like to write probes and instrumentation before touching the feature is going to be told to just use the tool. She isn't lacking curiosity. She's lacking the authority to spend the time, and that is not a fact about her. It's a fact about where she's standing. The discipline that saved me is, for a lot of people, structurally unavailable.

Where this goes

The next essay is about the same problem one size up, because an organization can be in this position too — knowing what was done and not why, fluent in its own operations without being able to explain them. Organizations declining to pay for the interrogation time are manufacturing non-expert experts on purpose and rewarding it as velocity. And filling their ranks with them.

Every question in this essay is asked about something in front of you — a system, a result, a record. The test examines it and asks what you demanded before you accepted it. All of that assumes the thing is what it claims to be.

Hold on to that.

Because this morning, I got a perfectly good result from a process I trusted, and I sat there wondering whether I'd earned it or just been handed it. Which is the question I didn't ask at nine years old, with a soldering iron in my hand and a speaker that had just made a noise. Did the circuit do that, or did something else?

A note on how this was written. As with part 1: the ideas, the anecdotes, and every judgment call about what belongs on the page are mine; the production leverage is the machine's. The full note is at the end of part 1, and it still applies — with the added observation that this essay is largely about the method it's disclosing.

Bill Dunning is the founder of 416 Inc (dba Indggo), where he leads development of Edge Mesh Protocol — verification infrastructure for real-time media. Part 3 is about the organizational version of the same problem. The canonical version of this series lives at forum.indggo.com.