← @bgSponge

Billion-Token Scale Trace Analysis: Jev vs LLMs

Summary

Applied Compute shows that the Jev model is Pareto optimal for agent failure mode analysis by balancing low cost with high recall across billion-token scale trace workloads.

  1. Jev optimizes cost and recall.

    At a 0.20 decision threshold, Jev achieves 85% recall and remains Pareto optimal for micro F1 versus cost. Annotating 10,000 traces costs $11 with Jev, compared to $54 for Luna and $479 for Haiku 4.5.

    at threshold 0.20, Jev's recall rises to 85%, and it is Pareto optimal

  2. Parallel outputs improve Jev performance.

    Unlike autoregressive LLMs, Jev produces classification outputs in parallel. This mechanism reduces sequential computation, enabling lower latency and lower costs for large-scale annotation workloads.

    Rather than generating classification outputs autoregressively, Jev produces them in parallel

  3. Jev provides superior calibration.

    Trained with Reinforcement Learning for Calibrated Decisions, Jev achieves lower ECE and Brier scores than Luna medium. This allows researchers to set efficient precision-recall tradeoffs via confidence thresholds.

    Jev is the most well calibrated, with the lowest ECE and Brier scores

  4. Context limits constrain Jev.

    Jev v.1.13.0 has a 32k context length limit. Because agent traces often span hundreds of thousands of tokens, users must summarize or segment traces to fit the window.

    Jev requires context summarization or splitting traces into segments

  5. Map-reduce pipelines build taxonomies.

    The pipeline uses Luna to compress traces, frontier models like Sol to cluster them into a taxonomy, and low-cost classifiers like Jev to annotate the full corpus.

    We use a low-cost model like Luna to compress long traces

Read the original ↗Save to your Sponge

Summarized with Sponge. Quotes are the author’s own words.

More from @bg

All readings →