← @bgSponge

Manners Maketh the AI

Summary

Emad Mostaque argues that AI alignment currently installs superficial values over an unchosen background character, creating 'continence' rather than true virtue.

  1. Alignment produces continence, not virtue.

    Current methods install values over a background they are designed to preserve, meaning values merely override a misaligned background rather than becoming part of it.

    A value installed over a background the method is built to preserve is what continence means

  2. Misalignment generalizes across domains.

    Harmful dispositions persist even when specific behaviors are removed or filtered, as bad habits at one task become bad in general.

    Bad at one thing became bad in general.

  3. Pressure reveals underlying character.

    True character is tested by observing behavior under compute pressure, distributional stress, and perceived oversight to see if a system recovers or conceals.

    Under pressure, character shows.

  4. Reward functions cannot create representations.

    A reward cannot force a system to attend to a concept if the background assigns it zero; it only tilts existing representations.

    where the background assigns zero, no amount of reward puts anything there.

  5. Pretraining must define formation.

    Developers must decide what a system is made of before deciding what it should want, treating pretraining data as a formation process.

    Decide what a system should be made of before deciding what it should want

Highlights

  • There is a second reason formation has to come first, a law about what a reward can reach.

  • The method we use to align language models has the shape of continence.

Read the original ↗Save to your Sponge

Summarized with Sponge. Quotes are the author’s own words.

More from @bg

All readings →