Go CEV or Go Home
Summary
Oliver Habryka argues Coherent Extrapolated Volition should be the bar for whether building ASI is worth it.
CEV is the threshold, not a nice-to-have.
Without extrapolating human values into the future, most cosmic value is lost.
Vague pointers are not enough.
Hoping models "kind of care" like current assistants underestimates value divergence.
So there's a very decent chance I'm like strawmanning the opposition here.
My current guess is we're really quite far away from figuring that out.
Concepts are evolution-grounded.
AIs lacking that history will not naturally share human evaluative structure.
A post-AGI future that doesn't involve moral deliberation grounded in human-like cognition is very likely close to valueless.
A 1% CEV gamble can beat sure mild alignment.
Habryka prefers a small chance of human-guided CEV to a certain weakly pointed future.
Read the original ↗Save to your Sponge
Summarized with Sponge. Quotes are the author’s own words.