Shipline is the pipeline I run agents inside: six phases, three gates, and four documents that have to exist before a story counts as done.
When I wrote it down, I left things out on purpose. The design notes say it plainly:
Specialist agent personas, confidence scores and evidence validation protocols read well in a framework document and do no work in practice.
I still believe the first and third of those. A persona is a costume, and an "evidence validation protocol" is a promise that the model will police itself. But the middle one, confidence scores, I got wrong, and I got it wrong for a reason worth writing down.
Why a prose confidence score is decoration
Ask a model "how confident are you, 0 to 100?" and it answers with a sentence. The number arrives as text, in the same channel as everything else it tells you, with no more weight than the sentence around it.
Nothing can consume it. My gates are code, and code cannot branch on prose. The number cannot fail, cannot be logged as a value, cannot be compared against a threshold. So it never changes what happens next. I was right that it did no work, because in that form it cannot do work.
The mistake was concluding the idea was useless rather than the shape.
What changed
System One models return a typed value and a calibrated confidence instead of text. Told to answer with one member of a set, the model returns that member. No parsing, no forgiving a malformed answer, no inventing an option that was never offered. And the confidence is a number that came out of training, not a sentence produced to sound measured.
That is the difference between decoration and a decision, and it is entirely about what reads it. My gate can branch on a typed verdict. It cannot branch on a paragraph.
There is a second reason this matters at a gate specifically. Typed answers are fast, 70 to 500ms in the numbers Typesafe reports, so the check can run on every phase rather than being something I skip because it costs a conversation. A gate that is too slow to run is not a gate, it is an intention.
The shape of the change
The gate asks one question with one permitted answer set:
Given the plan's acceptance criteria and the pasted evidence, is this phase done?
PASS every criterion is met by evidence that was actually run
FAIL at least one criterion is not met
NEEDS_HUMAN the evidence does not settle it
and it returns a verdict with a confidence. The pipeline then decides on two conditions, not one:
| Verdict | Confidence | What happens |
|---|---|---|
| PASS | high | Phase advances, evidence logged |
| PASS | low | Treated as NEEDS_HUMAN. A confident model is not the same as a correct one |
| FAIL | any | Stops with the failing criterion named |
| NEEDS_HUMAN | any | Stops and waits for me |
The low-confidence PASS is the row that pays for the whole change. That is the case I currently catch by reading carefully, usually late, usually after something has already gone wrong.
The boundary I am keeping
A PASS from a typed model is still not permission.
Shipline's first forbidden action is that nothing commits without explicit permission, and this does not touch it. The typed verdict gates attention, not authority. Plan approval stays mine. The test script stays mine to run. The commit stays mine to permit. If the model is sure and I have not said yes, the answer is still no.
This is the same pattern I already run in production. In SnapInvois a typed gate sits in front of the expensive path: before spending an OCR call on free text, a cheap classification decides whether the text is even invoice-shaped. Same idea, narrower scope. Do not spend an expensive call, and do not spend a human, when a typed answer is enough to route.
What I am not putting back
Specialist personas stay out. Evidence validation protocols stay out. Both are still the failure mode the original design note was aimed at: work that looks like rigour on a page.
The difference now is that I can tell the two apart. A persona asks a model to behave like a role. A typed gate gives it a small question with a fixed answer set and a number, and then my code acts on the answer. One of those is theatre. The other changes what the pipeline does next, and I can point at the row in the table where it does.