| Criterion | RLHF (reward model + PPO) | DPO | Planning use |
|---|---|---|---|
| Reward model | Built from preference data first | No explicit reward function; reward-free | DPO removes one model to train and maintain. |
| RL loop | Actor-critic training with PPO | Avoided; only the policy is optimized | Fewer moving parts to debug with DPO. |
| Reference model | Reference model updated during PPO training | Frozen copy of the model before fine-tuning, used as a baseline | Plan memory for the extra model copy either way. |
| Reported results | PPO outperformed DPO in all experiments of one arXiv paper | 61% summarization win rate against PPO’s 57% in another report | Results vary by task and setup, so test on your own task. |
The comparison draws on an arXiv paper on reward-based and reward-free alignment and the Turing Post overview of RLHF variants.
DPO vs RLHF comes down to simplicity versus ceiling: DPO drops the separate reward model and the RL loop, while PPO-based RLHF outperformed DPO in every experiment of an arXiv comparison.
This page sits inside the broader guide on how to fine tune an llm. If the older pipeline is unfamiliar, a separate walkthrough explains how does rlhf work stage by stage.
What each method removes from the pipeline
The difference between DPO and RLHF is the route from human preferences to a better policy. Reward-based RLHF constructs a reward model from preference data and then applies actor-critic algorithms such as Proximal Policy Optimization (PPO). DPO is a reward-free alternative that eliminates the explicit use of a reward function.
In practice that means DPO skips the separate reward model and avoids RL altogether. The code change is also small: one write-up on scaling DPO describes adding 4-6 lines of code for the loss function to standard LLM training code.
Both methods still start from preference data, meaning responses ranked by people or a proxy for people. What changes is how much machinery sits between that data and the final model.
Choose DPO when you have preference pairs and limited infrastructure
DPO suits teams that already hold a preference dataset and want to avoid training and serving a reward model. One cost argument comes from open replication work, which called a general reward-model dataset a big investment (order of millions of dollars). DPO does not remove the need for preference data, but it removes the reward-model stage built on top of it.
It also scales. The same source reports that the Tulu-2-DPO model was the first 70B parameter run that converged. On quality, Turing Post reports that DPO achieved a 61% win rate in summarization, beating PPO’s 57%.
- Fewer stages: no reward model to train, validate or keep in memory, and no PPO loop to stabilize.
- Smaller code change: the method plugs into ordinary training code rather than needing a separate RL setup.
- Dependence on your data: results hinge on how well the preference pairs match what your model actually produces, as covered below.
- Possible quality gap: the simpler route does not always match PPO, as the next section shows.
Choose RLHF when quality ceiling matters more than simplicity
PPO-based RLHF remains the stronger option when you can afford the extra stages and need the best result. The arXiv comparison reports that PPO consistently outperforms DPO across all its experiments. On a code-generation benchmark, its 34B PPO model showed a CodeContest 10@1k improvement from 16.4% to 22.4%.
The paper also gives a theoretical reason for caution. Its analysis finds that DPO might find biased solutions that exploit out-of-distribution responses, and that DPO’s performance is significantly affected by the distribution shift between the model outputs and the preference dataset. That risk grows when your preference pairs come from a different model than the one you are tuning.
RLHF is not effortless, though. The same paper names advantage normalization, a large batch size and an exponential moving average update for the reference model as key factors for the best PPO performance. Choose RLHF if you have the engineering capacity to tune those, and DPO if you do not.
Reported results do not all point the same way, since one summarization report favours DPO while the paper’s experiments favour PPO. Treat published numbers as hints about the task and setup, not a ruling for yours.
Preparing preference data and the reference model
DPO training data in TRL has three fields: prompt, containing the context inputs, plus a chosen and a rejected response. Mapping your own columns to that triple is the main data task, and the format is the common ground if you later compare methods on the same dataset.
The reference model is the other input. In DPO it is a frozen version of the model before fine-tuning that serves as a baseline. The DPOTrainer expects it as a copy of the original checkpoint, described in the Argilla walkthrough as the non-fine-tuned version of the model that’s being fine-tuned.
Running DPO with TRL
The TRL library wraps DPO in a single trainer, so a run is mostly configuration. The steps below follow the Argilla walkthrough and the Interconnects scaling notes.
- Run supervised fine-tuning first. The pipeline begins by running supervised fine-tuning (SFT) on the dataset(s) of interest, and preference learning follows.
- Format the dataset. Provide prompt, chosen and rejected fields, as described above.
- Pass the reference model. Supply the original checkpoint as the reference copy alongside the SFT model.
- Set beta. Beta is a temperature hyperparameter that controls the sharpness of the model’s updates. Argilla’s guidance puts it typically in the range of 0.1 to 0.5.
- Choose a low learning rate. Zephyr and Tulu used a learning rate of 5e-7, far below the 3e-4 that is generally the default in other ML work.
- Start training. Once the trainer is built, the only command still to run is dpo_trainer.train().
Where ORPO fits in the comparison
ORPO is another preference-tuning method, and TRL provides an ORPOTrainer alongside DPOTrainer. If your goal is to cut the RLHF pipeline down further, it is worth running on the same prompt, chosen and rejected dataset so the comparison is fair.
Read ORPOTrainer’s parameters in the TRL documentation before assuming that the DPO settings above, especially beta and the learning rate, carry over. Judge both by the same held-out checks rather than by training curves alone.
Checking the result
A finished run still needs evaluation on your own task. Because DPO’s performance is significantly affected by the distribution shift between model outputs and the preference dataset, test on prompts that resemble real use and compare the tuned model against the SFT starting point.
For scoring outputs at scale, llm as a judge is one practical approach to keep the comparison consistent across methods.
Frequently asked questions
Does DPO need a reward model?
No. DPO eliminates the explicit use of a reward function. It expresses the reward in a logarithmic form of the policy and focuses solely on policy optimization.
Is DPO reinforcement learning?
Not in the PPO sense. DPO avoids the RL loop and treats the problem as a classification problem on human preference data.
Do deployed assistants use RLHF or DPO?
The arXiv paper notes that applications such as ChatGPT and Claude leverage reward-based methods that first learn a reward model, so reward-based RLHF remains in production use.
