AIPublished ·Updated ·14 min read

JEV and System One models: when AI stops talking and starts deciding

JEV does not generate text. It returns typed decisions with a confidence score. What that changes for hallucination, automation and agent design, based on TypeSafe's docs and the first independent evaluation.

NE
Naoufal EL BEKRIChief Scientist, LumenAI
LinkedIn ↗

Key takeaways

  • →JEV returns typed decisions (Choice, Score, Noul) with probabilities, never free text.
  • →A closed answer space kills hallucinations of form. Wrong answers remain, but they become measurable classification errors.
  • →Confidence measures how concentrated the distribution is, not whether the answer is right. Calibration and local validation close that gap.
  • →An independent CMU study finds JEV close to GPT-6 when the verdict can be read from the text, and far behind when it must be derived.
  • →A confidence-gated cascade matched or beat GPT-6 accuracy at a fraction of the cost. The escalation rate tells you how hard your domain really is.

JEV and System One models: when AI stops talking and starts deciding

A language model can explain Bayesian inference to a ten-year-old with remarkable clarity. Ask it to decide, ten thousand times an hour, whether a support ticket concerns billing or a technical fault, and the picture changes.

Each call takes seconds and has a real cost. The format varies from one answer to the next: Billing, then billing., then a full paragraph opening with "Great question!". When the model is wrong, it is wrong with the same assurance as when it is right.

This last point is the real obstacle to automation.

TypeSafe AI's launch post starts from an observation: conversational models have been highly capable for several years, but automation has not scaled accordingly. JEV is the company's response, a model that returns decisions rather than text. The analysis below relies on the company's announcement, its technical documentation and the first independent evaluation of the model, published on arXiv by researchers at Carnegie Mellon University in September 2026. It distinguishes the claims supported by independent measurement from those that remain self-reported, and argues that the model's most significant property is not its speed but its ability to quantify its own uncertainty.

What is JEV?

JEV is a "System One" model built by TypeSafe AI. Text goes in. Typed decisions with probabilities come out. Never free-form text.

TypeSafe is a San Francisco-based AI company founded in 2024. JEV entered limited early access on 15 September 2026. Its CEO, Diogo Almeida, spent about four years at OpenAI working on ChatGPT, GPT-4, and reinforcement learning from human feedback (RLHF), a training method that optimizes models to produce answers that humans prefer.

His diagnosis, as quoted by Forbes, points to the limit of that approach: "We've been optimizing for humans and we're super human at pleasing humans."

Pleasing a human and feeding a decision into a pipeline are two different jobs. JEV is designed for the second task.

System 1, System 2: a different class of model?

TypeSafe borrows the distinction from Daniel Kahneman: System 1 is fast and intuitive, System 2 is slow and deliberate. LLMs are optimized for System 2 style work, while most decisions embedded in software require System 1.

An LLM is an autoregressive model: it generates one token at a time, each conditioned on the previous ones. The process is intrinsically sequential. TypeSafe describes System One models as producing all their outputs in a single parallel pass, and summarizes JEV as "a frontier-intelligence function call: unstructured state in, typed probabilistic decisions out."

Its scope is deliberately narrow: its does not write replies, produce code, or explain its reasoning. Input is currently limited to text: strings, JSON objects and arrays of text. Images and audio are not yet supported.

LLMs remain the appropriate tool when the output is text intended for a person, as in chatbots or coding assistants. System One models target decisions embedded in software: classifying, routing, scoring, extracting and branching.

What does a JEV call look like?

A JEV request contains a state and a set of questions. JEV answers all of them in a single call, each with a probability distribution over the allowed answers.

The state is the information to evaluate: a string, a JSON object or an array of text. Each question uses one of three primitives:

Primitive What it answers What you get back
Choice Selects one option from a defined set (up to 255 options) The selected option, a probability per option, a confidence
Score Places the state on an ordered scale A score, a probability per level, a confidence
Noul Evaluates a yes/no statement A single probability between 0 and 1

The example below combines the three primitives in a single call. The structure follows TypeSafe's documentation, and the probabilities are for illustration purposes only:

Input state. A customer support message:

"I was charged twice for my subscription. This is the second time this happens. Please refund the duplicate charge."

Questions.

Name Primitive Question Possible answers
category Choice What is the ticket about? billing, technical, account
frustration Score How frustrated is the customer? 0: calm, 1: frustrated, 2: very frustrated
refund_requested Noul Does the customer request a refund? Probability of "yes"

Response.

Question Probabilities Answer Confidence
category billing 0.91, technical 0.06, account 0.03 billing 0.87
frustration calm 0.05, frustrated 0.50, very frustrated 0.45 1.4 0.25
refund_requested yes 0.95 0.95 0.90

The category is identified with high confidence. The frustration score of 1.4 lies between "frustrated" and "very frustrated", and its confidence is low because the probability is split between these two adjacent levels.

Using the response. Because every answer is typed, application code can consume it directly, without parsing any text (pseudo-code):

if (answers["refund_requested"].probability > 0.9
        and answers["category"].choice == "billing"
        and answers["category"].confidence > 0.8):
    route_to("refunds_queue")
else:
    route_to("human_agent")

Does JEV hallucinate?

Not in form, but it can be wrong in substance.

Two very different failures are often grouped under the term "hallucination".

The first is a failure of form: an invented field, a label that is nott in your list, a Broken JSON, or a call to a function or tool that does not exist. With JEV this cannot happen, because the answer space is fixed before the model ever sees the input. TypeSafe specifies that the 0% type-error figure is guaranteed by schema matching rather than measured. It is a property of the design.

The second is a failure of substance: the model chooses a listed label, but it happens to be the wrong one. JEV is subject to this type of error. The CMU evaluation finds it 27.6 points behind GPT-6 on logic puzzles. No schema can prevent this type of error.

The claim of "zero hallucination" therefore needs to be stated more precisely:

The schema does not remove errors, but it turns them into classification errors.

And classification errors can be handled: they can be counted, structured through a confusion matrix, tracked by class, over time or by customer, and controlled with a threshold. These methods cannot be applied directly when the output is a free-form text that sounds confident whether it is right or wrong.

One trap to know about. If the right answer is not in your list, JEV will still pick something from the list. It has no other option. Give it an exit every single time: insufficient_information, other, not_applicable.

Can an LLM report its own confidence?

Only indirectly, and less reliably after post-training on human preferences.

Probabilities over tokens, not decisions. An LLM assigns probabilities to tokens. The same answer can take many forms ("Billing", "billing", "This looks like a billing issue"), so its probability is spread across variants, and not every API exposes token probabilities.

Self-reported confidence is generated text. A statement such as "I am 90% confident" is generated text like any other output and can be just as wrong.

Post-training degrades calibration. According to the GPT-4 Technical Report (OpenAI, 2023), the pre-trained model was highly calibrated on a subset of MMLU, but "the post-training hurts calibration significantly": the expected calibration error, the average gap between stated confidence and observed accuracy, rose from 0.007 to 0.074.

The signal still exists. Kadavath et al. (2022) show that larger models are well calibrated on multiple-choice and true/false questions asked in the right format. Chat interfaces simply do not expose it. JEV does, with a probability distribution attached to every answer.

How does JEV measure its confidence?

Confidence is a number between 0 and 1 derived from the answer's probability distribution: 1 when all the probability is on one outcome, 0 when it is spread evenly.

Choice.

Confidence = pmax − 1/n1 − 1/n

where n is the number of options. Only the top probability counts: (0.6, 0.3, 0.1) and (0.6, 0.2, 0.2) both yield 0.4.

Score. The formula accounts for the order of the levels. On a three-level scale, hesitating between two adjacent levels (0, 0.5, 0.5) yields 0.25, while hesitating between the two extremes (0.5, 0, 0.5) yields 0, a more fundamental uncertainty.

Noul. The probability itself expresses the uncertainty: 0.5 means complete indecision. An equivalent confidence is |2p − 1|.

The documentation presents this as "one reasonable way to summarize a distribution, not the only one". Since full probabilities are returned, other measures, such as the ratio between the two highest probabilities, can be used instead.

Is a confident answer a correct answer?

No. Confidence measures how concentrated the distribution is, not whether it is concentrated on the right answer.

A model can put all its probability on the wrong label, and still report a confidence of 100%. What links confidence to correctness is calibration: a calibrated model that reports 80% should be right in about 80% of those cases.

TypeSafe trains JEV with what it calls RLCD (Reinforcement Learning for Calibrated Decisions), which optimizes probabilities against real outcomes instead of human preference (RLHF) or verifiable rewards (RLVR). The architecture is not published. TypeSafe describes it as transformer-based and trained on synthetic data only.

The documentation sets the limits of this guarantee: "Calibration is measured across groups of predictions; it does not guarantee that an individual answer is correct."

The documentation then describes three paths:

  • High confidence: act automatically.
  • Medium confidence: request confirmation, flag for review or gather more information.
  • Low confidence: do not act. Route to a human or another system.

Thresholds should reflect the cost of being wrong. In TypeSafe's example, a read-only action requires no extra check, while a destructive action proceeds only above 0.9. Their recommendation is: "Start with conservative thresholds, test with your own data, and adjust as you observe results."

The launch post summarizes the whole idea:

"If a model can do a task 95% of the time but doesn't say when it's in the 5%, it can't automate that task."

What does independent testing say?

JEV is close to GPT-6 when the verdict can be read directly from the text, and well behind when it must be derived through reasoning. It is about 277 times cheaper per judgment.

TypeSafe's figures are self-reported: its workflow evaluations were built by its own team, and the company acknowledges that it "can't prove [its pricing] isn't subsidized."

The first independent evaluation is "JEV-as-a-Judge: Accept When Confident, Escalate When Unsure" by Li, Miao, Krishnan and Padman at Carnegie Mellon University (arXiv:2609.26550). The authors use JEV as an LLM judge, which decides which of two answers is better or whether an answer is correct, and compare it with sixteen other judges.

Claimed by TypeSafe Measured by CMU
Latency 70 to 500 ms 0.15 s median, 0.24 s for 95% of calls (GPT-6: 1.89 s and 3.86 s)
Cost Up to 444.6x cheaper About 277x cheaper: $0.044 vs $12.18 per 1,000 judgments
Accuracy "Frontier intelligence" Mid-ranking, 1.7 points behind GPT-6 over 4,850 judgments

Accuracy gaps with GPT-6 vary widely by domain:

Domain Gap
Chat quality, refusals, evidence-based factuality Within about 2 points
Expert knowledge −7.0
Code −12.9
Math −14.3
Logic puzzles −27.6
Style-adversarial pairs (RM-Bench hard) −13.5

JEV performs well when the answer is present in the text and only needs to be recognized. It performs poorly when the answer requires several steps of reasoning. A plausible explanation is that a single-pass model, unlike reasoning models, cannot work through a problem step by step. This limitation shapes the use case presented later.

The authors note several limits: a single version of JEV was tested, the judges were not compared at equal computing budget, most costs and latencies were simulated from price lists, latency was measured from a single location, and no specialized professional domain was evaluated.

How to build "accept when confident, escalate when unsure"?

JEV answers first. If its confidence is higher than a threshold τ, the answer is kept. If not, the item goes to a reasoning model or a human, whose verdict replaces JEV's.

The idea of a model cascade is not new. FrugalGPT (Chen, Zaharia, Zou, 2023) and RouteLLM (Ong et al., 2024) both explored sending easy queries to cheap models and complex ones to more capable models. What JEV adds is a low-cost model whose own confidence is reliable enough to make the routing decision.

The CMU study evaluates a cascade from JEV to GPT-6 in three settings:

Setting Items escalated Accuracy (cascade vs GPT-6 alone) Cost (share of GPT-6)
All benchmarks, τ = 0.9 (1,610 pairs) 31.5% 93.4% vs 92.5% 41.4%
JudgeBench, reasoning-heavy (same τ) 64.8% 0.7 points below GPT-6 66.8%
Two new workloads, τ set per workload 74.2% 90.2% vs 90.2% 75.5%

On the full benchmark set, the cascade is both more accurate and less expensive than GPT-6 alone: JEV's high-confidence answers are mostly correct, and the uncertain cases go to the model best suited to them. On reasoning-heavy tasks, escalation increases and the cost advantage narrows.

The escalation rate therefore indicates the nature of the task. A low rate means the task relies mainly on recognition, and JEV handles most of the volume. A high rate means it relies mainly on reasoning, and JEV acts as a first-level filter. This can only be established by measurement on one's own data.

The authors propose a simple procedure to set τ:

  1. Label about 100 examples from the target workload.
  2. Select the most permissive threshold for which the accuracy loss, compared with the stronger model, stays within 2 points with 95% confidence (one-sided bound).

Using the point estimate instead of the confidence bound exceeded the 2-point tolerance in 40 to 50% of simulations. The confidence bound reduced this to about 5%.

This procedure sets τ according to an accuracy tolerance, not the cost of errors, a limitation the authors acknowledge. TypeSafe's documentation recommends the latter approach: the higher the cost of an error, the higher the threshold. In an operational setting, thresholds should be set per action according to that cost.

Use case: a monitoring agent for renewable energy fleets

Prospective use case. This section describes a possible design based on JEV's documented strengths and limitations; it contains no measured results.

Monitoring a solar or wind fleet involves many small, recurring decisions: is a production drop a fault or a weather effect? Is it soiling or a tripped string? Does it justify a site visit? These are fast, structured decisions of the kind System One models are designed for.

Design principle. JEV performs poorly when the answer must be derived through calculation, and it only reads text. Raw time series are therefore processed beforehand by deterministic code, and JEV evaluates a summarized state.

The workflow has five steps:

Step Role
1. Physical modeling Compute expected production (physical or deep learning model), the gap with actual production, and its profile: morning and evening losses, constant offset, slow drift, sudden drop.
2. State construction Build a compact JSON record per asset and per day: production gaps, weather context, sensor status, recent maintenance, behavior of neighboring assets.
3. JEV evaluation Ask all questions in a single call (see below).
4. Routing Act, escalate or refer to a human, depending on confidence and the cost of the action.
5. Feedback Use each human decision as a labeled example to recalibrate thresholds by site or asset type.

Questions asked to JEV:

Question Primitive Possible answers
Is the gap an anomaly rather than a weather effect? Noul Probability of "yes"
Probable cause Choice Soiling, tripped string, stuck tracker, inverter clipping, grid curtailment, faulty sensor, insufficient information
Production dynamics Choice Stable, slow drift, sudden drop, intermittent, insufficient information
Severity Score 0 to 3
Are the sensor data reliable? Noul Probability of "yes"

The "insufficient information" option prevents JEV from assigning an unlisted fault to one of the listed categories.

Thresholds by action. The same confidence level does not justify the same action:

Action Cost of an error Condition
Log and monitor Negligible Low threshold, automatic
Open a maintenance ticket Limited Intermediate threshold
Escalate to a reasoning model Latency and cost Uncertain confidence
Send a team on site High (travel, downtime, safety) High threshold and human approval

Role of the engineer. The human operator handles ambiguous cases (soiling or persistent haze?), costly decisions (site visits) and patterns never observed on the fleet before. The purpose of the architecture is to direct these cases, and only these, to the engineer.

The same approach applies to wind power, using different indicators: gap with the power curve, curtailment, icing conditions, yaw misalignment.

Conclusion

JEV illustrates a shift in how AI can be integrated into software: from generating text to producing structured decisions with an explicit level of confidence.

Its value lies less in its speed than in its ability to indicate when it is uncertain, which makes it possible to automate the clear cases and refer the ambiguous ones to a human expert.

Frequently asked questions

Is JEV a large language model?

TypeSafe describes it as transformer-based but positions it as a different class, "System One models". It does not generate text: it returns typed answers (Choice, Score, Noul) with probabilities in a single parallel pass.

Can JEV hallucinate?

Not in form: by construction, it cannot return a value outside the defined schema. It can, however, pick the wrong value inside the schema. These errors are measurable, and its confidence score helps flag them.

How is JEV's confidence different from asking an LLM how sure it is?

JEV's confidence is computed from a probability distribution over the defined answers. An LLM's stated confidence is generated text, and its token probabilities do not map directly to decisions. Post-training on human preference is also known to degrade LLM calibration.

Where is JEV weak?

On tasks where the answer must be derived through reasoning. In the CMU evaluation, it trailed GPT-6 by 12.9 points on code, 14.3 on math and 27.6 on logic puzzles, while staying within about 2 points on chat quality and evidence-based factuality.

How should a confidence threshold be chosen?

Use around 100 labeled examples from the target workload and select the most permissive threshold for which the accuracy loss stays within the accepted tolerance with 95% confidence. Thresholds should then increase with the cost of the action: low for reversible actions, and human approval for the most costly ones.

Sources

  1. Introducing System One Models & Jev, TypeSafe AI blog, 2026. link ↗
  2. System One, TypeSafe AI documentation, 2026. link ↗
  3. Confidence, TypeSafe AI documentation, 2026. link ↗
  4. JEV-as-a-Judge: Accept When Confident, Escalate When Unsure, Carnegie Mellon University (arXiv), 2026. link ↗
  5. GPT-4 Technical Report, OpenAI (arXiv), 2023. link ↗
  6. Language Models (Mostly) Know What They Know, arXiv, 2022. link ↗
  7. On Calibration of Modern Neural Networks, ICML, 2017. link ↗
  8. FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performance, arXiv, 2023. link ↗
  9. RouteLLM: Learning to Route LLMs with Preference Data, arXiv, 2024. link ↗
AILLMAgentic AICalibrationModel cascades

See where your portfolio is losing energy

Book a Demo