← @bgSponge

So you want to use OpenRouter?

Summary

Mo Moustafa runs Olly, an iMessage AI assistant, through OpenRouter at multi-million-message scale and catalogs provider-level pitfalls that model names hide.

  1. Model id is not the model you get.

    OpenRouter routes one slug across many hosts with different precision, parsers, and bugs; paper-identical weights behave differently in production.

    They're the same model on paper, but very different models in real life.

  2. Score the board for your workload.

    Per-provider GPQA and TAU gaps are large; tool-calling swings of 20 points matter more for agents than headline knowledge scores, and rankings reshuffle across models.

    For an agent TAU is the score that matters and a 20 point swing is not noise.

    So the contract isn't per model, it's per provider.

  3. Declared precision is a bad quality filter.

    fp4 hosts sit mid-pack with fp8, worst hosts appear in every bucket, and hard quantization filters shrink fallbacks without buying quality.

  4. Treat hollow 200s and missed parsers as failures.

    Empty content/reasoning, missing usage, and tool calls left in raw XML need client-side parsing, throws, and retries.

  5. Do not trust a pinned shortlist forever.

    Even three reliable providers can all 429 or stop serving the model; benchmark from prod IPs and keep real fallbacks.

    A 200 tells you the request was served, not that there's an answer in it.

Read the original ↗Save to your Sponge

Summarized with Sponge. Quotes are the author’s own words.

More from @bg

All readings →