Waqar Uddin

Building the decision layer of my AI software factory

September 27, 2026 (2w ago)4 views

Disclosure: I work at Jazz as Principal Evangelist Cloud & AI. This article is written as personal industry analysis. Views are my own.

What I mean by a software factory

I keep coming back to the same north star: build an internal software factory that can produce production-grade enterprise software. Not a toy demo that writes a function, but a system that reliably ships code past an enterprise bar without a human watching every step.

The first honest thing to say is that the win condition is not model brilliance. It is reliability at scale. Can the system consistently decide where work should go, and can it refuse to ship anything that does not pass? Most agent setups optimise for the first hundred lines of a green demo. The part almost nobody builds is the enforcement spine underneath: the thing that routes work and the thing that says no.

That spine is what I spent the recent weeks building and, more importantly, measuring.

What a decision layer is

A decision layer has two slots.

Routing takes an incoming request and sends it to the right lane. Which harness runs this task. Which repo owns this. Is this a client engagement or factory work. A security triage or a feature.

Acceptance takes a candidate output and decides pass or escalate. Does this patch actually clear every check, or does a human need to look first.

The promise of small decision models is that both of these can run self-hosted in tens of milliseconds with typed, calibrated answers and an honest "I do not know" output, instead of billing a frontier model for every trivial judgement.

The assumption that failed

I started from the assumption that a capable stock decision model would mostly work out of the box. It did not.

On a ten-case acceptance bench, the stock model scored five out of ten. It was biased toward escalate. It would never have let anything ship, which is safe but useless. I tried improving the prompt, including the specific framing the model was trained on, and it moved to six out of ten. The conclusion was uncomfortable and clear: the stock checkpoint had almost no learned signal for my specific factory semantics. Framing was not the fix. The weights were.

The fix that actually worked

The fix was fine-tuning, and it was cheap. Same model, same base checkpoint, a fine-tuned intent head, maybe twenty seconds on CPU because the model is small and the data is small.

To get there I stopped trusting intuition and wrote a bench first. I collected a corpus of labeled factory cases, split it into train, validation, and a frozen test set, and defined the GO bar before touching anything: accuracy at least 0.85, calibration error at most 0.10, and honest abstention on out-of-scope input.

Then I fine-tuned on around sixty to eighty hand-labeled cases. The acceptance gate went from six out of ten to a perfect ten out of ten on the frozen test, with confidence that finally ordered correctly instead of sitting flat at fifty percent.

The hardening that caught my mistake

A perfect score on an in-style test is encouraging and also a trap. So I wrote a second, out-of-distribution hold-out: twenty-eight cases deliberately varied in wording and in failure mode, none of them resembling the training set.

The fine-tuned gate scored 0.93 out of distribution. Good, but it exposed the one error that matters most. It passed a change that touched a live user-session path with no regression re-run. That is a live-path false-pass, the exact thing a gate must never do.

I tried the obvious fix first: add more training labels in that shape and retrain. It made things worse. The gate dropped to 0.82 and still missed the case. More labels did not teach the concept, they just made the model more conservative.

The working fix was not more training. It was a deterministic guardrail layered on top of the model. If a change touches a live, payment, broker, session, canary, or live-user path and there is no positive re-verification signal, force it to escalate. And the guardrail had to understand negation, because "no regression ran against a live-ish fixture" contains the word regression but is a red flag, not a green light. A naive keyword check would have silently passed exactly the cases that matter.

After the guardrail, the combined hold-out was 0.974 with zero live-path false-passes.

Calibration was a separate piece

The model was accurate but its confidence was still miscalibrated. Temperature scaling fixed that without touching accuracy. One scalar, fit on a held-out set, drove the calibration error from 0.24 down to 0.05. That is the difference between a probability you can branch on and a number you ignore. Always calibrate before you trust the confidence.

Routing took the same medicine

The routing head started at 0.80 on its frozen test, functional but not great and not calibrated. The cure was the same approach with better labels targeting the classes it kept confusing: harness versus repo, work lane versus repo, trading lane versus market research. After enriching twenty of those cases and retraining, routing hit 0.90 on the frozen test, with the hard lane-vs-repo miss now fixed.

The gate that keeps it honest

None of this stays true on its own. So I wired it into a single command that grades both heads over all the hold-outs and fails hard on regression: accuracy below 0.90, calibration error above 0.10, or any live-path false-pass at all.

It passes today. That command is the difference between a blog that tells you what worked last week and a system that can tell you whether it still works after you change something. That is what an evaluation harness is for. You cannot tune what you cannot measure.

The numbers, honestly

All of this runs as one small model with two fine-tuned heads on a single Mac mini. That is the part I find genuinely interesting: a production-grade enforcement spine for a software factory that fits in the corner and costs almost nothing to run.

Why I am publishing this

Because the market is full of claims and thin on measurements. The fastest model for trading. The cheapest model for writing. Almost all of it is someone selling something.

The honest version is more useful and far rarer. Here is the exact thing that did not work, the numbers when it did, and the mistake the out-of-distribution test caught before it shipped anything. The objective skill is not claiming capability. It is keeping the receipts and publishing them, especially the ugly ones.

A decision layer is not going to replace judgement. Its whole job is to make good, calibrated judgement cheap enough to run everywhere, and to abstain or escalate honestly when it cannot. My factory is now a little closer to shipping its own software on top of that discipline, and the measuring happened in public.

Related reading: This post picks up where the stock model fell short in I built a decision layer for my software factory, and walks through the fine-tune, calibration, and guardrail that got it to a passing gate.