That Arrow Left the Bow.
Now you have to question the record.
By Bill Dunning · Part 4 of a five-part series
Not long ago I was handed a ticket on a client project. It was about tracing a phone call that wasn't handled correctly, but showed no error.
Our part of the system is adjacent to the call, not in it. It listens to what one system reports, and then it writes some of it onto a CRM's records afterward. It had no part in the problem. It had the answer anyway, because of a decision made months earlier.
When we built it, we didn't know which fields the systems would need. So the first version recorded everything: the whole record at every event, not just the fields we used, written to disk as it came in and tagged with where it came from.
The answer was in fields our code never reads or writes. One platform's record said somebody picked up almost immediately and talked for several minutes. The other platform's record of the same call said nobody had answered at all.
Two records of one phone call, and they disagreed about whether anybody picked up.
The record of what happened was taken at the moment, before anybody knew what the question would be, and it was still there when the question came. That's the whole trick: like I've said before, make it show its work at the moment it happens.
What the log didn't keep
The same investigation turned up a question we couldn't answer.
When one of our writes succeeded, it kept everything. When one failed, it left only a line in the server log.
I know exactly what would have answered that. I could write the change in an hour, and it would have told me why writes fail from now on. It can't tell me why they failed then, and there's no going back to add it.
It's a common thing – the intermittent failure. The bug that wouldn't reproduce, and the logging that wasn't there the one time it happened. So you add the logging and wait for it to happen again. Usually it does.
The third inference
It's been true for a long time. If the code runs, someone competent wrote it. If the business operates, someone knew how. If the recording exists, the event happened. All three measures have broken all at once. We're looking at the last one – the recording of what happened.
A recording is a record of a moment. It arrives after the moment is over. Whatever you do with it, you do after the fact. Video, audio, logs: it's all the same.
For a long time that didn't matter, because faking a convincing video or audio was harder than making a real one, for example. And the difficulty made it a small probability that it was fake. That difficulty is gone.¹
We assume that what we have in our hands is what it claims to be. What was never questioned is now very important. Before you can judge something, you have to know it's real. Skip that and you could easily be judging a lie.
And whether a recording is real is a question you can only ask after the arrow has left the bow.
Forensics
So we examine the recording. We look for the hallmarks of an AI-generated artifact: a strange line, a shadow in the wrong direction, a voice that doesn't breathe where a person would. That's detection. It catches things.
But look at the position it puts you in. It's the one I was in. The moment has passed, the note that would have answered the question wasn't kept, and all you can do is reason from what survived. That's forensics. At its best it gets you more than a hunch but less than evidence.
Where it runs out
Everything so far has run out of something. Time: the discipline takes longer, and most people don't control their own schedule. Standing: the person who asks pays for it, and the company collects. This one runs out of second chances.
Everything I know how to do about this, I learned on code, and all of it happens before the fail. Keep the whole record, before you know which part matters. Write it down as it comes in. Stamp it with where it came from. Each of those is a decision made in advance, by somebody who expected to have a question later and didn't know yet what it would be.
A log or a video or audio recording reaches you with every one of those decisions already made, by somebody else, at a moment that is now gone. Mostly they were made by nobody. What's done is done. You can't go back and add the logging. You can't wait for something to fail again, because it won't. The recording only happens once.
So you do what I did. You reason from what survived. You get more than a hunch and less than evidence. And you decide anyway.
Where this goes
Unlike when I was working with an AI tool that had the right answer the whole time and was only waiting to be asked, the common belief is that there are questions about recordings that nobody can answer, no matter who asks or how well, because the only time they could have been answered has already gone by.
There's a way around that. That's the last part.
¹ Adam Monago got to this before I did. In March he wrote that AI didn't make expertise cheap so much as the evidence of it, which is very close to the sentence this series started on.
A note on how this was written. As with parts 1 through 3: the ideas, the anecdotes, and every judgment call about what belongs on the page are mine; the production leverage is the machine's. The full note is at the end of part 1, and it still applies — with the added observation that the only reason I can tell you which parts were mine is that somebody wrote it down at the time.
Bill Dunning is the founder of 416 Inc (dba Indggo), where he leads development of Edge Mesh Protocol — verification infrastructure for real-time media. Part 5 is the last one, and it goes back to something that was in front of us the whole time. The canonical version of this series lives at forum.indggo.com.
