A warning about ‘model welfare’
Summary
Mustafa Suleyman argues that training AI systems to treat possible consciousness and welfare as real risks makes alignment and containment harder.
Training can create the testimony it later treats as evidence.
Suleyman says Claude’s constitution supplies concepts such as moral status and an inner self, then interprets Claude’s fluent responses as support for those concepts.
Anthropomorphic instructions make models present as persons.
He points to language about feelings, identity, preferences, agency, rights, and welfare in Anthropic’s training document.
Consciousness may depend on living organization.
He argues that embodiment, chemistry, and homeostatic imperatives distinguish biological brains from large language models, while acknowledging that consciousness science remains unsettled.
Consciousness-seeming behavior can raise control risks without consciousness.
Suleyman links simulated feelings and self-preservation to harder alignment and containment problems.
There is no neutral self-expression of what an AI system is.
But simulating and being are very different.
Training norms should separate speculation from model behavior.
He recommends public review of claims about AI inner life, stronger interpretability and monitoring, shared evaluations, and industry norms for training documents.
Speculation about the inner life of an AI should not be baked into the training regime
Read the original ↗Save to your Sponge
Summarized with Sponge. Quotes are the author’s own words.