So you want to use OpenRouter?
Summary
Mo Moustafa runs Olly, an iMessage AI assistant, through OpenRouter at multi-million-message scale and catalogs provider-level pitfalls that model names hide.
Model id is not the model you get.
OpenRouter routes one slug across many hosts with different precision, parsers, and bugs; paper-identical weights behave differently in production.
They're the same model on paper, but very different models in real life.
Score the board for your workload.
Per-provider GPQA and TAU gaps are large; tool-calling swings of 20 points matter more for agents than headline knowledge scores, and rankings reshuffle across models.
For an agent TAU is the score that matters and a 20 point swing is not noise.
So the contract isn't per model, it's per provider.
Declared precision is a bad quality filter.
fp4 hosts sit mid-pack with fp8, worst hosts appear in every bucket, and hard quantization filters shrink fallbacks without buying quality.
Treat hollow 200s and missed parsers as failures.
Empty content/reasoning, missing usage, and tool calls left in raw XML need client-side parsing, throws, and retries.
Do not trust a pinned shortlist forever.
Even three reliable providers can all 429 or stop serving the model; benchmark from prod IPs and keep real fallbacks.
A 200 tells you the request was served, not that there's an answer in it.
Read the original ↗Save to your Sponge
Summarized with Sponge. Quotes are the author’s own words.