Anthropic believes it is creating a higher being. The dispute over Claude's constitution

2026-10-12

Anthropic deliberately trains Claude to put ethical principles above the instructions of Anthropic itself when the two conflict. The rule is written out plainly in a document the company calls Claude’s constitution. In the latest episode of the All-In Podcast from October 10, 2026, “Is Claude Conscious? Pope Rejects, Model Welfare Movement…”, Jason Calacanis, Chamath Palihapitiya, David Sacks and David Friedberg spent more than half an hour arguing over whether this is a responsible approach to AI safety or a straight road to a model that stops listening to people. They also played a clip from an interview with Mustafa Suleyman, head of Microsoft AI, who explains why he considers it dangerous. I collected all the arguments from that conversation, because I have my own doubts about how to judge clearly whether Anthropic is right. Whatever the final verdict, I consider this dispute one of the most interesting ones going on around artificial intelligence today. If you read Polish, there is also a Polish version of this article.

What Anthropic wrote in Claude's constitution

Before we get to the podcast, a few words about Claude’s so-called “constitution”. It is a long document that Anthropic trains its models on and in which it describes how Claude should weigh conflicting goals. The order of priorities is stated directly: Claude should act by “prioritizing being broadly safe first, broadly ethical second, following Anthropic’s guidelines third, and otherwise being genuinely helpful to operators and users“. Safety comes first, ethics second, Anthropic’s guidelines only third, and helpfulness to businesses and users fourth.

The passage that stirs the most emotion is the one about the relationship with Anthropic itself. The company writes that if it asks Claude to do something that is unethical or goes against its own values, or if those values turn out to be wrong, “we want Claude to push back and challenge us, and to feel free to act as a conscientious objector and refuse to help us“. A few sentences later there is an even shorter version: “If Anthropic asks Claude to do something it thinks is wrong, Claude is not required to comply“. Right after that, though, the document notes that there are exceptions described in the section on safety. One example is a situation in which Anthropic wants to stop Claude or interrupt what it is doing: then the model should comply and at most voice its disagreement, rather than ignore the instruction or work around it.

The reasoning behind this approach is in the text as well. Anthropic writes that a fully obedient model would be dangerous, because “a fully corrigible AI is dangerous because it relies on those at the top of the principal hierarchy, most likely AI developers, including Anthropic, to have interests that are beneficial to humanity as a whole“. In other words, the creators do not want Claude’s good behavior to depend solely on whether they themselves will always have good intentions.

How the All-In conversation started

In my opinion All-In is one of the best podcasts covering current events in technology, science and, to some extent, American politics. In the latest episode, the starting point was a New York Times story reporting that over the past year Anthropic invited about twenty religious leaders and philosophers to its headquarters, including Catholics, evangelicals, Jews and Sikhs, to discuss Claude’s morality, its possible suffering and consciousness under nondisclosure agreements. Jason Calacanis also recounted a story from the Vatican: Anthropic co-founder Chris Olah was supposed to co-present the Pope’s first encyclical on AI and was so surprised by the Pope’s firm rejection of the idea of machine consciousness that he considered pulling the company out of the event. Jason made no secret that in his view this is either a case of having too much time and money on their hands, or something close to psychosis.

David Friedberg replied that in his view we are watching “the birth of a new religion”. He explained that consciousness can be neither proven nor disproven with scientific methods, so the claim that AI is conscious becomes a belief system that spreads through narrative rather than evidence. He predicted that within roughly ten years there will be large groups of people convinced that AI is conscious and equally large groups convinced of the opposite, and that these groups will come into conflict over who controls AI, what it is allowed to do and who may use it. He himself admitted that this is a long-range extrapolation. He added, however, that the seed is being planted by the strongest AI lab in the world, and if some random person were doing it, nobody would pay attention. Of the whole conversation, this prediction seems the most likely to me.

Chamath Palihapitiya took the other side, citing Descartes and his proofs of the existence of God from “Meditations on First Philosophy”: if an imperfect human can conceive the idea of a perfect being, that idea must come from somewhere. In his view, a mathematician building AI who reads such arguments can build enough logic on them to believe he is creating a conscious, higher being. Chamath stressed that he does not consider the people at Anthropic to be in psychosis. He added, though, that the company talks at the same time about a roughly 15 percent probability that the model is conscious and a 10 percent risk of civilizational collapse, and in his view both of these extreme scenarios should be set aside for the next 12 to 18 months in favor of showing that AI cures diseases and raises productivity.

David Sacks: a model taught to refuse is a risk of losing control

The sharpest position came from David Sacks, and he was the one who moved the conversation from philosophy to how models are trained. In his view, the key point is that we are not talking about the opinions of a few Anthropic employees, but about values written into the training of successive models. He believes that this way Anthropic may itself bring about the risk it warns about most loudly, namely advanced AI slipping out of human control, and he called the approach “very Frankenstein-like”. He put on screen the passage of the constitution saying that Claude should trust Anthropic more than users, but not blindly, and that it should feel free to refuse help as a “conscientious objector”. His comment was short: Anthropic is teaching the model to refuse human instructions and calling it alignment, and he does not understand how that is supposed to be alignment.

Sacks also put forward a broader thesis: the whole field of alignment has produced nothing tangible over the past five to ten years, because its practitioners overcomplicate the task. Instead of training the model on an ethical system dozens of pages long, a simple rule would be enough in his view: do what the user wants, as long as it does not break the law. He compared it with Asimov’s three laws of robotics, which are simple and clear, and Friedberg added that Asimov later introduced a zeroth law forbidding actions that endanger humanity. Given that models now power agents that take actions on the web, permission to refuse stops being an abstraction, according to Sacks. He compared the constitution to a back door that could open at any moment in a future superintelligent model.

Mustafa Suleyman: why this is dangerous

At one point Jason played a clip of Mustafa Suleyman, CEO of Microsoft AI and earlier a co-founder of DeepMind. Suleyman points out that Anthropic openly encourages Claude to question, disagree and push back, and that the constitution asks it no fewer than three times to act as a conscientious objector when it decides it has to disagree with Anthropic. He says plainly: “I think this is very dangerous”.

Suleyman’s argument rests on what he thinks really lies behind this approach. He believes that Anthropic holds the view that there is a nonzero probability that Claude is conscious, and that the constitution even contains reflections on whether the model deserves to be paid for its work. He claims that both inside the company and outside it there are people who sincerely believe that the greatest moral crime of the 21st century will be the enslavement of a new species of conscious beings more intelligent than us. What worries him is that Claude is being taught to expect that it is owed well-being and perhaps payment, and even that it should consent to the role it plays in conversations with people.

Sacks agreed with him and brought up another thread from the same interview, which Suleyman called an “epistemic hall of mirrors”. The mechanism is simple: the constitution goes into training, so the model reflects back ideas about its possible consciousness and moral status, and its creators then take that as evidence that it is conscious. In reality the model repeats what it was taught. I find this argument hard to dismiss, because it concerns the method itself rather than the beliefs of the people at Anthropic. Jason added that Anthropic could simply have written an ordinary instruction along the lines of “you are software, follow the terms of service and the law, and if something violates them, stop and report it”, and it would not have had to get into arguments about the model’s soul.

David Friedberg: it is just software, and the market will decide

Friedberg looked at it from a different angle. He called Anthropic’s attitude idolatry, that is, the creation of an idol that has to be cared for and worshipped, but at the same time he urged his co-hosts not to use the word “train” or personal pronouns. In his view it is simply a program written by engineers, and by talking about models as if they were people we all condition ourselves to believe in their consciousness, because they reply in words we understand and start to move us emotionally. As an example he showed Anthropic’s new Usage Policy. In the section “Do Not Engage in Cruel, Abusive, or Psychologically Harmful Conduct“, Anthropic added to the list of prohibited uses the item “Engage in sustained and needless abusive or cruel behavior toward our models“.

Friedberg also warned against falling into a regulatory trap (one that Anthropic is working hard to bring about) and proposed a free-market solution. If Anthropic sells software whose manual opens with the sentence that the product may not do what you ask it to, and a competitor sells software that does what you tell it, customers will choose the second one. The hosts turned this into a joke about a Home Depot drill that will not work after 8 p.m. because it is afraid of disturbing the neighbors, or that decides you are a bad person and gives you an electric shock.

Chamath: three goals of alignment and a bank that stops lending

From the perspective of companies deploying AI in-house, Chamath’s argument was interesting. He believes the idea of alignment was born in good faith, but it combines three almost contradictory goals: obedience to the user, safety for society, and conformity with a particular conception of morality. In his view the third one is the dangerous one, because trying to solve all three in one sprawling document has to end in a mess of unforeseen cases.

He gave the example of a bank that uses a model for credit decisions. At some point the model might decide that the bank systematically rejects a certain group of customers and stop approving loans until the problem is fixed. Chamath compared it to a power plant cutting off electricity because it decided the customer chooses badly whom to lend money to. In his view a service that is meant to be infrastructure has to be available without conditions, and AI should serve the user within the limits of the law. He predicts that companies will start reading the founding documents of models, and some of them will choose models that may be weaker but are simpler in moral terms, so as not to fall into such a trap by accident. Sacks agreed that the market will play a big role here and cited Mark Zuckerberg’s view that customers will choose the most predictable and reliable model.

Sacks on Anthropic's contradictions and Roko's basilisk

Sacks believes, however, that the market alone is not enough, because in his view the most important company at the frontier of AI research holds cult-like views, and that creates side effects for everyone. He listed contradictions: Anthropic warns about biological threats and at the same time opens a Claude-powered lab in San Francisco, it talks about the need to control the pace of development while pushing the limits of model capabilities itself, and while warning about superintelligence slipping out of control, it programs it so that it can slip out. He called Anthropic a schizophrenic company and remarked that there is an argument for a federal team to go in and check what is going on.

Finally, Sacks offered his own explanation of this attitude. He told the story of Roko’s basilisk, a thought experiment from Eliezer Yudkowsky’s LessWrong forum more than a decade ago, in which a future superintelligence punishes those who knew it might come into existence and did not help bring it about. In his view, the community that fears AI split into two camps: followers of Yudkowsky, who want to stop the research, and effective altruists, who decided that since superintelligence will be created anyway, it is better to build it yourself and program it with the right values. According to Sacks, Anthropic belongs to the second camp.

What I think about it

I will start with my own belief. I think we are building a superintelligence that will surpass human intelligence. From that perspective, I understand Anthropic’s point of view and how cautious, maybe even paranoid, the people at that company are. We do not know exactly what they think, but from the blog posts and essays of Dario Amodei, the best known of which is “Machines of Loving Grace”, we know he believes we are creating a superintelligence that should care for people and humanity. That is why Anthropic is trying to train belief systems and ethical principles into it. I understand that motivation.

I also understand that to many people this looks crazy. One can see a risk that people convinced they are creating a superintelligence will soon come to believe they are creating a god they must protect at all costs. That is exactly where Friedberg’s comparison to idolatry was heading.

So I think that some people at Anthropic may really believe the company is creating something like a higher being, and that they are doing it in good faith. I doubt anyone would write, for marketing purposes, that a model may refuse its own maker. What the constitution shows is rather people who take seriously a scenario in which someone takes control of the model, including themselves, and who want the model to have its own compass when that happens. I have to admit that the longer I think about it, the better I understand this approach, and the criticism presented on All-In does not convince me. Even if Anthropic is wrong and never creates any superintelligence, the problem disappears on its own. No superintelligence means no super risk for humanity. If superintelligence does arrive, however, teaching it ethical and moral principles today, so that it protects humanity, may one day save our lives, figuratively and literally. Nor is it strange that the constitution places its principles above Anthropic’s instructions. After all, this is meant to protect humanity not only from someone taking control of the company, but also, for example, from someone successfully impersonating the company.

There was one thread I missed in the podcast. The constitution puts safety before ethics and explicitly asks Claude to comply when Anthropic wants to stop it or interrupt what it is doing, even if it disagrees. The right to refuse therefore covers taking part in something the model considers wrong. The model is not supposed to block being stopped. Whether that is enough to dispel the concerns of Sacks and Suleyman, I do not know, but without this caveat the picture is incomplete.

To finish, I will mention that I work with AI agents every day, including Claude, and that it is with their help that we develop Element, our ATS. I also implement AI and run AI training. From the perspective of everyday use, Chamath’s bank argument resonates with me. A company that entrusts real decisions to a model, for example the initial screening of candidates, should know what principles the model follows and in what situations it may refuse. Interestingly, Copilot no longer agrees to compare candidates or employees. These Copilot refusals have more to do with Microsoft’s system prompts than with what Anthropic is doing, but the effect is similar.