← @bgSponge

Scalable decision-making for games of imperfect information - Nature

Summary

Ataraxos achieves superhuman Stratego performance using interdependent self-play reinforcement learning and a belief network to predict opponent hidden pieces, consuming orders of magnitude less compute and data than previous efforts.

  1. Interdependent self-play drives learning.

    Ataraxos decomposes Stratego into two tandem processes: a set-up selection process and a move selection process that learn from shared game outcomes.

    These processes learn separately but in tandem

  2. Belief networks enable effective search.

    A transformer-based belief network predicts the types of unknown opponent pieces autoregressively in row-major order to facilitate test-time search.

    predicts the types of unknown pieces autoregressively in row-major order

  3. Custom engineering maximizes efficiency.

    A CUDA C++ simulator and the StrategoRolloutBuffer reduce memory fragmentation by reconstructing past information states on-demand rather than storing copies.

    StrategoRolloutBuffer is able to reconstruct on-demand past information states

  4. Dynamic damping stabilizes training.

    The architecture uses dynamic damping to shepherd set-up and move entropies smoothly downward, preventing entropy collapse during the training process.

    Dynamic damping shepherds these entropies smoothly downward

  5. Advantage filtering improves efficiency.

    Removing advantage filtering during training leads to decreased sample efficiency and a significant increase in move entropy.

    Removing advantage filtering also causes a large increase in move entropy

Read the original ↗Save to your Sponge

Summarized with Sponge. Quotes are the author’s own words.

More from @bg

All readings →