What We Gained by Swapping LLMs for Decision Models
What We Gained by Swapping LLMs for Decision Models
In triage decisions on a real operation, Jev responded roughly 3.5× faster than the LLM we were using, cost between a quarter and a third as much, and got 43 out of 44 cases right, versus 44 out of 44. OpenAI's Decisions API was even faster. Neither is a universal replacement, and knowing where each one fits is what actually matters.
In our previous post, we explained what decision models are: instead of generating a response, they pick from a set of options and report how confident they are. What we hadn't shown yet was what that means in a real operation.
We put Jev and the Decisions API head-to-head with the LLM we were already running, on two decisions that happen the moment a request lands at a property management company: What kind of request is this? and Which property is it about? All three answered the same questions, with the same options, on the same cases, on the same day.
Result 1: What kind of request is this?
Thirty requests, each run three times per model.
| LLM | Jev | Decisions API (OpenAI) | |
|---|---|---|---|
| Accuracy | 30 of 30 | 29 of 30 | 29 of 30 |
| Typical response time | 1.7 s | 0.5 s | 0.3 s |
| Slowest response time | 2.9 s | 0.6 s | 0.5 s |
| Cost per 1,000 requests | $0.087 | $0.020 | $0.024 |
| Requests with 80%+ confidence | not reported | 87%, all correct | 67%, all correct |
The LLM got all 30 right; each decision model missed one, and not the same one. In return, both responded three to six times faster and cost roughly a quarter of what the LLM did. Part of that savings comes down to how billing works: an LLM charges for what it reads and, at a higher rate, for what it writes; decision models only charge for what they read.
The errors tell an even better story. In both cases, the model flagged low confidence, below 40%, on every attempt. In other words: the model was wrong, but it knew it wasn't sure. That's exactly the kind of case that should go to a human.
Every time Jev reported 80% confidence or higher, it got the classification right. That happened on 87% of requests.
In practice, nearly nine out of ten requests can move forward on their own, and the system knows upfront which ones can't. With the LLM, we had an answer. With the decision model, we have an answer and a built-in signal for when to trust it.
The Decisions API also got every high-confidence call right, but it only reached that threshold on two-thirds of requests, meaning more cases would have been flagged for human review under the same criteria. On the other hand, it was completely deterministic: given the same request three times, it returned the same answer with the exact same confidence score each time. Jev's confidence scores varied slightly across runs.
Result 2: Which property is it about?
This one carries more weight. Routing a request to the wrong property is worse than admitting you couldn't identify it.
| LLM | Jev | Decisions API (OpenAI) | |
|---|---|---|---|
| Correct property | 14 of 14 | 14 of 14 | 14 of 14 |
| Wrong property returned | none | none | none |
| Typical response time | 1.8 s | 0.5 s | 0.3 s |
| Slowest response time | 3.1 s | 0.6 s | 0.5 s |
| Cost per 1,000 requests | $0.094 | $0.034 | $0.056 |
All three tied on accuracy, and none returned a wrong property. Jev cost just over a third of what the LLM did; the Decisions API came in at around 60%.
Adding both decisions together, a request that used to take roughly 3.5 seconds to classify and route now takes about 1 second with Jev and 0.6 seconds with the Decisions API.
A bonus test: emails that shouldn't get through
We applied the same approach to a third question: Is this email work-related, a newsletter, advertising, or a scam? Here, the cost of each type of error is different. Discarding a work email is serious; letting a newsletter slip through is just a nuisance.
The dataset had 111 emails, 35 of them work-related, most written specifically to look suspicious. Both models held the line on what mattered most: no work email was discarded. The Decisions API blocked more unwanted messages, up to 99% of them; Jev reached 95% and cost 39% less per email. Purpose-built scam emails remain the hardest case for both, and that's the next challenge.
Jev or Decisions API?
Each one came out ahead in different ways. Jev was cheaper and more often reached the confidence level that allows it to act autonomously. The Decisions API was the fastest, fully consistent across repeated runs, and the only one that can process images. Since both answer the same type of question, choosing between them becomes an engineering decision, measured case by case, rather than a leap of faith.
And neither replaces an LLM when the task requires generating a response. They replace the LLM in the cases where it was being used just to make a choice.
How we measured
- All figures come from a single test run on October 8, 2026.
- Each question was tested on real or realistic cases with known correct answers. For the first two, each case was run three times per model, with the majority answer counting.
- The comparison LLM was the one already in production, a general-purpose OpenAI model.
- List prices as of October 2026, per million tokens: the LLM charges $0.20 for input and $1.20 for output; Jev charges $0.042 and the Decisions API charges $0.10, both for input only.
- Response times were measured from our side and include network latency.
- The datasets are small (30, 14, and 111 cases) and are meant to inform an architecture decision, not to serve as a definitive model ranking.
If you're thinking through where AI fits, and where it doesn't, in your company's workflows, we'd love to talk.
Tags: #ArtificialIntelligence #DecisionModels #LLM #DecisionsAPI #OpenAI #Jev #ProcessAutomation #AICost #AutomatedTriage #AIArchitecture #AIForBusiness #MLOps #TextProcessing #AIInOperations #AIOptimization