Choice answers include a probability distribution over the options you supplied. The selected option’s probability estimates how likely that answer is, while confidence summarizes how concentrated the distribution is. These values describe different properties of the result and shouldn’t share a threshold without evaluation.

AI SDK calls a yes/no question Boolean, while TypeSafe’s native API calls it Noul and returns its probability without a separate confidence field. Values near zero indicate support for false, while values near 0.5 indicate uncertainty about whether the statement is true.

What can the full distribution tell you?

These hypothetical results assign the same probability to the leading destination, but the remaining probabilities suggest different follow-up questions.

In example A, most of the probability is split between editorial and legal review, so checking for contractual language could help distinguish them. Example B leaves both legal and design review plausible, meaning that the same check might still leave uncertainty about whether the document needs design review.

Both results would receive the same treatment under a threshold based only on the leading probability. Examining the alternatives helps your application choose what to investigate next, while the additional evidence determines whether the proposed route is appropriate.

Why does a high probability still need evaluation?

Calibration describes how predicted probabilities compare with observed outcomes across many cases. As TypeSafe’s AI primer explains, events assigned a probability near 0.8 should occur roughly 80% of the time in a well-calibrated set, without guaranteeing that any individual prediction is correct.

For a document router, build an evaluation set that reflects the decisions your application will face:

• Ask reviewers to independently identify the correct destination, rather than treating acceptance of the model’s suggestion as a label.
• Include ambiguous documents that could belong to multiple teams, as well as cases missing information needed for routing.
• Reserve separate examples for the final evaluation so you aren’t judging the policy on the same cases used to refine its questions.

Group predictions into probability ranges and compare them with observed outcomes. For example, suppose 100 held-out cases with predicted probabilities near 0.9 produce only 70 correct routes. That gap would warrant investigating the labels, task definition, and calibration before adopting a 0.9 cutoff.

Keep the number of cases alongside each rate, since results from a small group can change substantially with only a few additional errors.

The higher cutoff avoids 34 incorrect routes in this example, but sends 360 more documents to review. Whether that is worthwhile depends on the cost of rerouting and the team's review capacity.

Record the denominator when reporting accuracy. Four errors among 400 automatic routes is a different measure from four errors among all 1,000 documents. Also count timeouts and failed evaluations; they need a defined destination even though they produce no usable prediction.

After selecting a candidate policy, run it without changing existing routing. Compare its proposed actions with the decisions your team makes. Revisit the evaluation when the document mix or routing criteria change.

Why can't you treat a rubric score as a probability?

Jev’s Score output represents a probability-weighted position on the ordered levels you define. Dividing it by the highest level rescales the result to between zero and one, without changing what it measures.

On an editorial rubric, 0 could represent a draft needing substantial revision and 4 one ready for copy editing. Dividing a score of 3.2 by 4 produces 0.8, but that value still describes the draft’s position on the rubric. It does not mean there is an 80% chance the draft is ready.

Define the levels around distinctions your application needs to make, and check that editors can apply them consistently. If adjacent levels are difficult to distinguish, clarify their descriptions before choosing a score cutoff.

What changes when you switch evaluation providers?

Before reusing a threshold with another provider, check which outputs it supplies. AI SDK makes Choice and Score distributions optional, while TypeSafe’s separate confidence values live in result.providerMetadata?.typesafe?.confidence, keyed by question ID. That provider-specific statistic shouldn’t be treated as a shared measure across models.

Routing code should check for a distribution before reading the selected option’s probability and use a defined fallback when it’s missing. Substituting 0 or 1 would turn an absent value into an apparent prediction, distorting both routing decisions.

Record which model produced each result, then evaluate any replacement on the same labeled cases before choosing its cutoff. Models that share an API can still produce probability estimates that require different thresholds.

When should you revisit a production threshold?

Independently review a sample of automatically routed documents so you can measure errors among accepted predictions, alongside cases already sent for review. Track corrections by destination to identify categories whose mistakes might otherwise be obscured in the overall error rate.

Re-run the labeled evaluation whenever you change category descriptions or switch models, and check whether incoming documents still resemble your test set. If long customer submissions begin to replace short internal briefs, evaluate the new traffic before deciding whether to adjust the cutoff or revise the routing question.

What can't a threshold establish?

Before relying on a threshold, check whether Jev fits the task and whether it has enough current information to make the decision. If the document’s intended audience is missing or the supplied policy is out of date, update the context or refer the case to someone who can resolve the uncertainty.

When Jev is part of an agent loop, crossing a threshold can inform the next action, but application code must still check permissions before executing it.

Frequently asked questions

Is Jev's confidence the probability that its selected answer is correct?

No. Confidence summarizes the spread of the returned distribution. Use labeled outcomes to determine how that statistic relates to errors in your application.

Which probability threshold should I use for automatic decisions?

Choose a cutoff by measuring mistakes and review volume on representative labeled cases. A policy for reversible internal routing can tolerate different errors from a policy that discloses a document externally.

Does a low Boolean probability mean Jev is uncertain?

A value close to zero indicates support for false. Uncertainty about a yes/no statement is greatest around the midpoint, so checking only for a high value would miss confident negative answers.

Is a normalized Jev Score a success probability?