← @bgSponge

Go CEV or Go Home

Summary

Oliver Habryka argues Coherent Extrapolated Volition should be the bar for whether building ASI is worth it.

  1. CEV is the threshold, not a nice-to-have.

    Without extrapolating human values into the future, most cosmic value is lost.

  2. Vague pointers are not enough.

    Hoping models "kind of care" like current assistants underestimates value divergence.

    So there's a very decent chance I'm like strawmanning the opposition here.

    My current guess is we're really quite far away from figuring that out.

  3. Concepts are evolution-grounded.

    AIs lacking that history will not naturally share human evaluative structure.

    A post-AGI future that doesn't involve moral deliberation grounded in human-like cognition is very likely close to valueless.

  4. A 1% CEV gamble can beat sure mild alignment.

    Habryka prefers a small chance of human-guided CEV to a certain weakly pointed future.

Read the original ↗Save to your Sponge

Summarized with Sponge. Quotes are the author’s own words.

More from @bg

All readings →