← @bgSponge

Training a 125M-parameter Model to Autocomplete Piano

Summary

Simon Edwardsson trained a 125M-parameter transformer to autocomplete piano performances in real time, achieving 108 notes per second on an iPhone 15.

  1. Single-note tokens improve speed.

    Instead of four transformer steps per note, the model advances by one complete note at a time using five categorical fields per token.

    the transformer advances the music by one complete note at a time.

  2. Data cleaning beats scaling.

    Expanding the dataset by 5x decreased performance; the final 300 million note events were filtered for piano-focused material and density.

    Cleaning and selecting the data mattered more than simply adding more of it.

  3. Scheduled sampling improves rollout.

    Feeding the model its own predicted pitch during training increased validation loss but raised Gemini-scored preference from 35.7% to 64.3%.

    this increased validation loss but improved the continuations.

  4. DPO provides the biggest gain.

    Using Gemini 3.5 Flash for pairwise evaluation allowed DPO to increase preferred continuations from 24.55% to 69.05% with a consensus dataset.

    DPO made the biggest difference after pretraining.

  5. Sustain is baked into duration.

    To simplify modeling, the author extended note durations to the pedal-up time during preprocessing rather than adding explicit pedal events.

    sustain is baked into note duration during preprocessing.

Read the original ↗Save to your Sponge

Summarized with Sponge. Quotes are the author’s own words.

More from @bg

All readings →