← @bgSponge

Language Models Can Control Their Own Attention

Summary

Namgyu Ho and coauthors introduce Declarative Attention, a protocol that lets models declare which context to attend while decoding.

  1. Declarative Attention makes the model name its attention span.

    Generation partitions into global (full context), focus (named ~2K-token magic chunks), and local (response-only) modes that the inference engine parses like tool calls.

    wouldn’t the model already know which parts of the context are relevant?

    turning selective attention from a pattern inferred inside the network into one the model states out loud

  2. Savings come from the mask, not shorter generation.

    Ablations show DA-nm (same prompt, full attention) matches vanilla accuracy but attends more tokens; the dynamic mask drives reductions up to 71.1% versus that ablation.

  3. Cost falls while accuracy stays close.

    On Gemma-4-31B and Qwen-3.6-27B, average attended tokens drop 52.0% and 31.1% with 1.27pp and 2.75pp accuracy drops across 15 tasks.

  4. Scale closes the accuracy gap.

    From 4B to 31B, focus-call parsing succeeds more often and the accuracy gap to vanilla shrinks; absolute token savings grow with context length, up to about 21M tokens per response.

  5. vLLM can apply the mask without new kernels.

    Block-aligned KV masking keeps FlashAttention unchanged; roofline analysis projects decode wall-clock at 0.71× and 0.77× of vanilla on the two headline models when serving is well optimized.

    DA needs no auxiliary scorer and runs on off-the-shelf models with no training.

Read the original ↗Save to your Sponge

Summarized with Sponge. Quotes are the author’s own words.

More from @bg

All readings →