The pilot went well. Something that used to take a week came back in a morning, the room was genuinely impressed, and the follow-up meeting was booked in the good mood that produces. Six months on it is running for two enthusiasts and nobody has switched anything off, because nobody has had to.
This is now the ordinary outcome rather than the embarrassing one. The surveys this year keep landing on the same result: most pilots produce no return anyone can measure, and most agent projects never reach production. That is a large enough share that the interesting question is not why particular projects fail. It is what the successful demonstration was actually evidence of.
State the province first
In 1843 Ada Lovelace published a set of notes on Charles Babbage's Analytical Engine, a machine which was never built. Her notes are longer than the paper they annotate, and the part that survives everything else is the order in which she made her argument. Before describing what the engine could do, she set out what it could not.
Her line was origination. The engine could carry out whatever it could be instructed to carry out, however elaborate the instruction — and it had no power to arrive at anything on its own. Everything it produced traced back to an order somebody gave it. She drew the boundary first, and only then made her claim for the machine's reach, which was far more ambitious than anything Babbage had written down.
That order of operations is the method, and it is the opposite of how a pilot is usually run.
A pilot is designed to demonstrate reach. Take the task with the cleanest inputs, the most tolerant reviewer and the lowest cost of being wrong, and show what comes back. What the demonstration establishes is that the system executes well when the order is well formed. What the business case then assumes, almost always without writing it down, is something else: that it will also handle the cases where the order is not well formed, where the necessary fact was never recorded, where the objective is wrong and somebody has to notice.
The gap between those two is what people are describing when they say the project died in the last mile. It is not really a mile, and it is not really last. It is the part of the work that was the job.
The one page that survives the meeting
The practical form is a page written before the demonstration, not after it, with three columns.
What we order: the instruction, stated precisely enough that somebody else could give it.
What it executes: the work that follows from that instruction, which is the part the pilot will show you.
What remains ours: the decisions that cannot be handed over, because the input needed to make them does not exist anywhere the system can see.
The third column is the useful one. When it is empty, the project is not ambitious, it is mis-specified — someone has assumed the system will supply judgement about a situation that has never been written down. When it is full and honest, the same demonstration means considerably more, because you now know what it was a demonstration of.
Watch out for
The obvious objection is the right one: a bright line against origination, held too firmly, becomes an excuse for not looking.
Lovelace was writing about a machine that did not exist, from a design on paper, and her boundary was drawn in a period when nobody could test it. Whether ordering, carried far enough and applied to enough examples, turns into something that deserves another word is exactly the question modern systems make hard to settle — and the doctrine of provinces gives you no help with it, because it was formulated before the question could arise. Repeating her line as though it settled the matter is its own kind of overclaiming, in the sceptical direction.
There is a fair criticism of her too, and it belongs here rather than in a footnote. Her contribution has been inflated into founding-figure mythology and dismissed as Babbage's work with her name on it, in roughly equal measure, for a century and a half. The defensible position is the narrow one: the notes are hers, the argument in them is unusually careful, and the discipline it recommends is worth adopting on its own merits.
Answer this next
For the pilot on your desk, write one sentence naming what this system must never be trusted to decide.
If it takes you five minutes, you know what you are buying. If you cannot write it at all, the demonstration has not told you anything you can spend money on.
