Product / Evals

Check the models.
Switch when a cheaper one meets the bar.

A judge checks a sample of the model calls on your graph. Your corrections become ground truth. Trials test another model against it.

[01] In the console

Test another model on your calls.

A trial answers checked calls again with another model. It shows whether that model meets the bar.

The judge.

Claude Opus 5.5 checks answers, routing, reranking and ontology fits.

Your corrections.

A correction becomes the ground truth for that call.

A trial before a switch.

Another model answers the checked calls. The trial compares it with the bar.

The Models page: search embeddings by OpenAI, rerank by Jev, the eval judge Claude Opus 5.5, and question answering by Jev and Claude Sonnet 5.5 — The models behind each use, on the demo graph.

[02] Models

We run the models.
You see which one does what.

The Models page lists each use with its model. It shows calls, tokens and cost by day.

Embeddings: an OpenAI embedding model.
Rerank and routing: Jev, from TypeSafe.
Answers: Claude Sonnet 5.5. The judge: Claude Opus 5.5.

Check model quality

[03] What is included

Checks, trials and switches.

You find them in the console and in the API.

Agree · Correct

Review a call. Agree, or write the right answer.

trials

A candidate model answers the checked calls again.

the bar

At least 20 calls. At most 5 points worse at the low end of the 95% interval.

switch

Switch question answers or query routing. Revert at any time.

search tuning

Scores search settings on your golden questions. Nothing changes until you apply.

Model choice

Cards with score against cost, and a view over time.

[04] How it works

You approve the switch.

Step 01

Check.

The judge checks a sample of calls. You agree or correct.

Step 02

Trial.

A candidate answers the checked calls again and is compared with the bar.

Step 03

Switch.

Switch when the candidate meets the bar.

[05] FAQ

Questions about evals.

Which model calls are checked?

Reranking, routing, answers, query rewriting, ontology fits, ontology proposals and eval labels.

Does a trial switch models by itself?

No. A trial changes no model. You switch, and you can revert.

Which uses can switch?

Question answers and query routing. The other uses are tested only.

What does a trial cost?

A trial spends at most $3.00.

Next step

Check the models.Switch when a cheaper one meets the bar.