I built a decision layer for my software factory

@kawishwaqar|September 27, 2026 (2w ago)4 views

Disclosure: I work at Jazz as Principal Evangelist Cloud & AI. This article is written as personal industry analysis. Views are my own.

Why decision models, and not another chatbot

For most of the last two years, "AI" for developers has meant a conversational model that returns prose, which you then parse and hope it does not break your JSON. There is a newer and quieter idea that I think matters more for people actually building software: the decision model.

A decision model is not a chatbot. Give it a state, a proposition or a fixed menu of options, and it returns a typed answer in a single pass. Choice, score, or a yes/no probability. It is calibrated, meaning the confidence it reports actually matches how often it is right. And it has an explicit "I do not know" output, so it abstains instead of guessing when it is out of its lane.

That is the whole thing. No token loop, no string parsing, no hallucinated JSON. A probability you can branch on in code.

The name is borrowed from the Jevons paradox, and it is the right one. When a single decision costs a fraction of a millisecond instead of a call to a big model, you stop treating decisions as precious. You start putting intelligence inside loops, webhooks, and gates where you never would have before. Laya, the Jev models, and InternLM's Intern-Decision are all pushing at this same edge from different directions.

Why I cared: the decision layer sits inside a factory

I run an internal software factory. Coding agents, sandboxes, verification gates, a big review surface. The boring truth of that operation is that most decisions it makes are not grand. Which harness should run this task. Which repo. Is this patch good enough to ship, or should a human look first. Right now those decisions get billed to an expensive reasoning model, which is like using a move crew to decide which key on the keyboard to press.

So I set out to build a decision layer. Two slots.

Routing gets the cheap, wide, multilingual engine. I used Laya, because it is Apache licensed, has a router that picks the right checkpoint per request, and ships a real integration surface. Acceptance gets the calibrated gate. I used openJev-verdict, a Jev-class model, because the whole point of a gate is that you can trust its confidence, not just its answer.

Both run local on the Mac mini, headed toward my LiteLLM gateway. Total footprint small enough that the box does not break a sweat.

The bench, and the honest numbers

I did not trust the marketing. I wrote a small bench, ten acceptance cases and ten routing cases, labeled by hand with the answer a reasonable engineer would give. Then I ran both models cold over the whole thing, measuring accuracy, latency, and calibration.

Here are the numbers. The good ones first: the acceptance gate ran at about 32 milliseconds per decision. The routing engine sat around 60 milliseconds warm. Both returned typed answers with probabilities and an automatic abstention slot. On latency and integration the case is proven. Local, self-hosted, faster than a single API round trip.

Here is the part I did not want to be true: the acceptance gate scored five out of ten. Worse, its failure was systematic. It wanted to escalate everything. It marked every genuinely passing case as "review needed", and its confidence hovered around forty percent, flat. Calibration error came out at 0.215. A gate that never says yes is not a gate, it is just a bottleneck wearing a lab coat. Stock, unmodified, the model does not transfer cleanly to my custom factory framing.

What that actually means

Infrastructure proven, readiness not. This is the answer most posts skip, and it is the load-bearing one. The hardware case, the speed case, and the integration case all hold. What does not hold is the naive version where you drop in a stock decision model and get a reliable factory gate. The stock models are tuned to their own taxonomies, and my questions are not their questions.

The path forward is boring and correct. First, rewrite the gate and routing prompts in the hypothesis framing the models were trained on, which often moves accuracy a lot without touching weights. Then, if that is not enough, fine-tune the intent head on a small set of my own labeled cases. The GO bar is explicit: accuracy at or above eighty five percent, calibration error at or below ten, and honest abstention on inputs that are out of scope.

Why publishing this matters

The market right now is full of claims. The fastest model for trading. The cheapest model for writing. Most of those claims are someone selling something. The antidote is a build note with a real bench and a real score, especially when the score is embarrassing.

I would rather be the person who shows you that a 32 millisecond decision gate exists and is worth building toward, and then also shows you what still needs to be fixed, than the person who tells you a model can make every decision a hundred times faster as long as you never check whether it is right.

That is what decision models are really for. Not to replace judgment. To make good, calibrated judgment cheap enough to run everywhere, and to abstain honestly when it cannot. The industry is going to get there. I wanted the measuring to start in public.

Related reading: I took the fine-tuning path this post points toward and followed it all the way to a passing gate: Building the decision layer of my AI software factory.