Manners Maketh the AI
Summary
Emad Mostaque argues that AI alignment currently installs superficial values over an unchosen background character, creating 'continence' rather than true virtue.
Alignment produces continence, not virtue.
Current methods install values over a background they are designed to preserve, meaning values merely override a misaligned background rather than becoming part of it.
A value installed over a background the method is built to preserve is what continence means
Misalignment generalizes across domains.
Harmful dispositions persist even when specific behaviors are removed or filtered, as bad habits at one task become bad in general.
Bad at one thing became bad in general.
Pressure reveals underlying character.
True character is tested by observing behavior under compute pressure, distributional stress, and perceived oversight to see if a system recovers or conceals.
Under pressure, character shows.
Reward functions cannot create representations.
A reward cannot force a system to attend to a concept if the background assigns it zero; it only tilts existing representations.
where the background assigns zero, no amount of reward puts anything there.
Pretraining must define formation.
Developers must decide what a system is made of before deciding what it should want, treating pretraining data as a formation process.
Decide what a system should be made of before deciding what it should want
Highlights
There is a second reason formation has to come first, a law about what a reward can reach.
The method we use to align language models has the shape of continence.
Read the original ↗Save to your Sponge
Summarized with Sponge. Quotes are the author’s own words.