Ori Eval: Find the Best Model for Your Project
Summary
OpenRouter's Ori Eval is an agent that writes and runs evaluations against your codebase so you can pick a model with scored proof instead of guesswork.
Pick the model for this project.
There is no universal winner; Ori Eval compares models against your surfaces, budget, and gold answers.
Ori Eval helps you find that model, with quantitative proof.
Pin the harness or the score moves.
Ori pins model and effort during a run so a later difference comes from the agent, not a swapped provider.
Ori Eval's agent harness and model are pinned, so every run scores the same setup.
Evals are ordinary TypeScript tests.
After an interview it writes .eval.ts files and runs them with bun test; CI can fail the build.
Grade tools and answers.
An eval asserts tools called, tools avoided, and answer quality; open-ended answers use an LLM judge and a minimum score.
An eval checks three things: the tools the agent called, the tools it avoided, and the quality of the answer
Use your data and a price ceiling.
Compare new OpenRouter models to the incumbent on every gold row, and keep the current model in the mix if you want.
The majority of harnesses that people use are limited to the company's AI models.
Read the original ↗Save to your Sponge
Summarized with Sponge. Quotes are the author’s own words.