When Not to Trust Jev: 9 Documented Failure Modes

Last verified September 20, 2026. The failure modes below are TypeSafe's own, published in its model documentation.

The unusual thing here is that the vendor wrote this list

Most model launches give you benchmarks. TypeSafe publishes a page describing where Jev is weak, which it calls model jaggedness. That is a genuinely good practice and it deserves more attention than it has had — as of this writing, essentially no launch coverage walks through it.

So this is not a takedown. It is the vendor's own list, read carefully, with what each item means for a system you were planning to build.

1. It reads literally

Jev takes the state at face value. Sarcasm, understatement, implication and politeness that inverts meaning are all places it can land on the wrong side.

"Great, another outage" is not praise. "I suppose I could wait another week" is not patience. Customer messages are full of this, and a routing layer that misreads tone will quietly send the angriest people to the slowest queue.

What to do: ask about observable facts, not inferred states where you can. "Does this message mention a missed deadline?" is more reliable than "is this customer angry?"

2. It is weak at math and counting

It is a decision model, not a calculator. Asking it to total a column, count occurrences or compare quantities is asking for the one thing deterministic code does perfectly and free.

What to do: compute the number yourself, then ask Jev about the result. "Is this figure above the approval limit?" is a decision. "What is 12% of this invoice?" is arithmetic.

3. It is weak at date ordering

Sequencing dates and reasoning about elapsed time are documented weak points. Anything that turns on which of two things happened first, or how long ago something was, is on shaky ground.

What to do: resolve dates to a computed field before the model sees them. Pass a computed "days_since_last_contact: 47" and not two timestamps and a hope.

4. It is weak at multi-hop indirection

Chains of inference — this refers to that, which relates to this other thing, therefore — are not what parallel evaluation against a flat state is good at. The architecture that makes it fast is the architecture that makes it shallow.

What to do: this is the clearest signal that a task belongs a rung higher. If the hard part is connecting facts, not judging one, that is reasoning, and it wants a reasoning model.

5. It degrades with irrelevant context in the state

More context is not better. Padding the state with material that has nothing to do with the question measurably hurts the answer.

This one catches people out, because the instinct with a large context budget is to fill it. Dumping an entire ticket history, the customer record and the last three orders into the state "so it has everything" is likely to make the decision worse, not better.

What to do: send the minimum state that could answer the question. Treat context as a cost, not a courtesy.

6. It does not treat adversarial input as hostile by default

This is the one with security consequences, and it deserves the most attention of the nine.

If your state contains text a stranger wrote — a support ticket, a form submission, a comment, an inbound email — then someone can put instructions in it. Jev is not, by default, treating that content as an attack surface. A message engineered to read as a refund request may be classified as one.

What to do: treat any decision made on user-supplied state as advisory when the consequence is real. The mitigations are the ordinary ones — separate untrusted content from instructions, keep the consequential actions behind a check that does not depend on the classification, and do not let a single model output authorize something irreversible.

7. It breaks under contradictory instructions

If your questions conflict with each other, or a question conflicts with something in the state, behavior degrades. This usually shows up as categories that overlap — two options where a reasonable person would also hesitate.

What to do: the test is whether two competent people would assign the same label. If they would not, the taxonomy is the problem and no threshold will fix it.

8. Complementary probabilities are not guaranteed to sum to 1.0

A technical point with a practical edge. If you ask "is this urgent?" and "is this not urgent?" as separate questions, the two probabilities are not structurally guaranteed to add to one.

What to do: do not build logic that assumes they will. If you need mutually exclusive options, use a single Choice question, where the distribution is computed across the set, not several Nouls you then try to reconcile.

9. It is not trained to generate text

Listed last because it is the least surprising and the most frequently ignored. Jev decides. It does not write the reply, summarize the thread, or explain its reasoning in a sentence you can show a customer.

What to do: plan for a second component from the start. Every production design that works has something else producing the language.

The tenth item, which is not on the list

TypeSafe trains Jev with a method it calls Reinforcement Learning for Calibrated Decisions. Calibration has a precise meaning: a well-calibrated model that says 80% should be right about 80% of the time. That property is the entire basis for a confidence threshold — "automate above 0.9" only means something if 0.9 is stable.

As of this writing, no calibration curve and no expected-calibration-error figure appear in TypeSafe's public documentation. The threshold guidance that exists is a comment in a code example noting that 0.5 is a starting point and the right number depends on your use case.

That is a measurement rather than an accusation — the model is days old, and documentation follows launches. But it changes where the work sits. If you are going to automate on a threshold, you are choosing that threshold without a published curve behind it, and the only way to know what it costs you is to measure it against data you labeled yourself.

What "zero hallucinations" actually covers

Since it is the claim most likely to travel further than it should: Jev can only return values from the schema you defined, so it cannot invent an answer that was not on your list. That class of failure is genuinely gone.

It can still choose a permitted answer and choose wrong. TypeSafe's own wording is that schema matching is guaranteed and the zero figure is "not empirical." A validly structured decision can be a mistaken one, and the nine items above are a list of the circumstances in which it is most likely to be.

The honest version of all this

None of the nine is a reason not to use a decision model. They are a map of where the edges are, published by the people who built it, and the failure they describe is quieter than the one they replace.

A language model that hallucinates produces visible nonsense and somebody notices. A decision layer that misclassifies four percent of tickets produces nothing at all — no error, no alert, just a small steady stream of things going to the wrong place. That is the failure mode to design for, and it is why a threshold you measured, an escalation path that exists, and somebody actually checking the output are not optional extras. They are the product.