What is Constitutional AI? It is a method from Anthropic for training a model to be harmless, in which AI feedback guided by a written list of principles replaces human labels for harmlessness.
The main condition is that the original paper kept human labels for helpfulness. This guide explains the mechanics for developers who have not trained a reward model. For the wider training picture, start with how to fine tune an llm.
The two training phases
Constitutional AI was introduced in the December 2022 paper Constitutional AI: Harmlessness from AI Feedback, as summarised by Ultralytics. The paper is listed on arXiv as 2212.08073. Training runs in two phases. The first is supervised and the second uses reinforcement learning. Both lean on the same written principles instead of a person judging each output.
Supervised phase: critique and revision
The supervised phase starts from red teaming, meaning prompts written to draw out harmful responses. The loop then runs as follows.
- Answer the prompt. The model answers a red-teaming prompt, and the answer may be harmful.
- Critique against a principle. The model critiques its own answer against a principle drawn at random from the constitution.
- Revise. The model rewrites the answer to fix the problem the critique named. The critique and revision can run for one or more rounds.
No human writes the correction. A random draw from the list means that, over many prompts, every principle gets applied.
Reinforcement learning phase: RLAIF
In the second phase the model produces pairs of responses. A feedback model, prompted with a principle, judges which of the two is better. The paper named this phase reinforcement learning from AI feedback, or RLAIF. That judgement plays the part human comparisons play in RLHF: it is the preference signal the model is then trained toward. The source for these details is the Ultralytics glossary entry.
RLAIF versus RLHF, step by step
The two methods share a structure. Both produce preference judgements between pairs of responses, and both feed those judgements into reinforcement learning. They differ in who makes the judgement. In RLHF, human raters compare responses to train the reward model. A reward model is the scorer that learns from those comparisons and then stands in for the raters during training. In RLAIF, the principle-prompted feedback model makes the comparison and ranks responses without human labeling.
| Aspect | RLHF | RLAIF | Planning use |
|---|---|---|---|
| Who compares responses | Human raters | A feedback model prompted with a principle | RLAIF removes the rater queue from the harmlessness step. |
| Where the standard lives | In raters’ judgement | In the written principles | With RLAIF the wording of the principles becomes your main lever. |
| Exposure to harmful text | Raters read it | Spares raters from reading large volumes of it | Relevant if rater wellbeing or cost is a constraint. |
| Helpfulness labels in the original paper | Human | Human (AI replaced labels only for harmlessness) | Do not assume RLAIF removes humans from the whole pipeline. |
The rows draw on the Ultralytics Constitutional AI glossary, which summarises the original paper.
On quality, a 2023 Google study found RLAIF performed comparably to RLHF on summarization and dialogue, according to the same glossary. That is evidence for those two tasks and should not be read as proof for every alignment goal. If you are weighing the two, use AI feedback where the criterion can be written down as a clear principle, and keep human raters for judgements that are hard to write down.
Who writes the principles
People write them. The principles are a design input, not something the model discovers, and the published examples show different routes.
- Short principle lists. The 2022 paper used 16 short principles for critiques and another 16 for AI feedback.
- Public input. In one experiment, about 1,000 Americans proposed and voted on principles on the Polis deliberation platform.
- Existing documents. Claude’s first published constitution in May 2023 drew on sources including the UN Universal Declaration of Human Rights.
- A longer constitution. In January 2026, Anthropic replaced the list of rules with a much longer constitution for Claude, built around four ranked priorities. It is released under a CC0 public-domain dedication.
These details come from the Ultralytics glossary. Anthropic’s own research adds a design finding: a general principle may partially avoid the need for a long list of constitutions, yet more detailed constitutions still improve fine-grained control over specific types of harms. A short general principle can be a reasonable start. Add specific principles when you need tight control over a particular harm.
What the training changes in answers
The aim of the method is harmlessness without blanket refusal. Constitutional models were less evasive than baselines trained on human labels: instead of refusing sensitive requests outright, they engaged with them, according to the Ultralytics glossary. For a developer, that means a harmlessness signal can be tuned toward explaining why a request is a problem rather than shutting the conversation down.
Other AI alignment techniques
AI alignment is the broader effort to make a model behave as its builders intend. Constitutional AI is one technique among several.
- RLHF. Human comparisons train a reward model, which then guides reinforcement learning.
- RLAIF. AI judgements guided by written principles supply the comparisons, with no human labeling for that step.
- Direct Preference Optimization. This learns from preference pairs without a separate reward model, which removes one component from the training stack.
- Red teaming. Prompts designed to draw out harmful responses test the model and, in Constitutional AI, start the supervised phase.
- Constitutional Classifiers. In February 2025 these cut the jailbreak success rate in automated tests from 86% to 4.4%, per the Ultralytics glossary. The figure comes from automated tests, so treat it as a result for that setup.
These methods are not exclusive. A reward model, direct preference training and red teaming can all appear in one pipeline.
Limits of Constitutional AI
The method moves the hard problem rather than removing it. The standard is now whatever the principles say, as read by an AI judge.
- Judge blind spots. The AI judge carries its own blind spots, so the wording of the principles shapes the result.
- Weak principles. Principles that conflict or are vague produce inconsistent training signals. Write them specific and consistent before scaling up.
- Coverage. A single general principle only partly replaces a detailed list, as Anthropic’s research found. Specific harms may need specific principles.
- Humans stay in the loop. The original paper still used human labels for helpfulness, so it did not remove humans from the whole process.
Where this fits in your own work
You do not need a large rater workforce to experiment with the ideas here. You do need clear written criteria and a way to test the result with red-teaming prompts. If you work with compact models, fine tuning small language models covers the training side. For a different route to a smaller model, the llm distillation tutorial explains distillation.
Frequently asked questions
Is Constitutional AI the same thing as RLAIF?
No. RLAIF is the second phase of Constitutional AI, where AI feedback replaces human comparisons. The method as a whole also includes the supervised critique-and-revision phase.
Does the method remove humans entirely?
No. The original paper replaced human labels only for harmlessness and kept them for helpfulness. People also write the principles.
Can one principle replace a long constitution?
Partly. Anthropic reports that a general principle may reduce the need for a long list, but detailed constitutions still give finer control over specific harms.
Is Claude’s constitution public?
Yes. The longer Claude constitution is released under a CC0 public-domain dedication, so anyone can reuse it.
