Supervised learning and reinforcement learning are the same objective

Note (2026): This started as a 2017 note on a Berkeley lecture by Mohammad Norouzi. I held it because the 2017 RL literature was fragmented and the unification felt premature. Nine years later the methods that actually scaled to frontier models (RLHF, DPO, GRPO, reasoning-model fine-tuning) turned out to be instantiations of exactly this framing, so here it is, rewritten.

Supervised learning and reinforcement learning look like different subjects in a textbook. They are the same objective written two ways. Both fit a distribution over outputs conditioned on an input. Both can be cast as minimizing a KL divergence to an implicit “optimal” target distribution. The only differences are which distribution you sample from and which direction of the KL you use. Entropy regularization is the knob that turns one into the other. (This post is a close cousin of the similarity post: similarity-between-distributions is another name for what we are doing here.)

The shared setup

Both paradigms want to learn a mapping from inputs $x$ to outputs or actions $a$, parameterized as a conditional distribution ${\pi }_{\theta }(a\mid x)$. The output $a$ may be a single label, a caption, a sequence of tokens, or a trajectory of actions; the shape of the output is irrelevant to the argument.

What does change is the training signal.

• In supervised learning, every input $x$ comes with a target output $a^{\star }$, and you maximize $\log {\pi }_{\theta }(a^{\star }\mid x)$ averaged over the dataset.
• In reinforcement learning, every output $a$ gets a scalar reward $r(a\mid x)$, possibly sparse or delayed, and you maximize the expected reward ${\mathbb{E}}_{a\sim {\pi }_{\theta }}[r(a\mid x)]$.

The objectives read as two separate problems. They are not.

The optimal policy is a Boltzmann distribution

Fix the reward function $r(a\mid x)$. The optimal entropy-regularized policy at temperature $\tau$ is

$${\pi }^{\star }(a\mid x)=\frac{1}{Z(x)}\exp \left(\frac{r(a\mid x)}{\tau }\right).$$

This is the softmax / Boltzmann distribution over outputs, with high-reward outputs getting probability mass in proportion to $\exp (r/\tau )$. Two limits make it intuitive. As $\tau \to 0$, ${\pi }^{\star }$ concentrates on the argmax (greedy exploitation). As $\tau \to \infty$, ${\pi }^{\star }$ becomes uniform (pure exploration). In between, $\tau$ trades off exploration against exploitation in a principled way.

Supervised learning has a similar implicit target. Given a labelled dataset $\{(x,a^{\star })\}$, define $r(a\mid x)=1[a=a^{\star }]$. The optimal policy at $\tau \to 0$ is a delta on $a^{\star }$. This is just saying: the “correct answer” is the Boltzmann distribution at zero temperature, with reward being the indicator of correctness.

Once both paradigms have an optimal target distribution, the question is how the model gets close to it.

Both are KL divergences, in opposite directions

Supervised learning’s cross-entropy objective, written out, is

$${\mathcal{L}}_{\text{SL}}(\theta )=-{\mathbb{E}}_{a\sim {\pi }^{\star }}\left[\log {\pi }_{\theta }(a\mid x)\right]\propto \mathrm{KL}\left[{\pi }^{\star }\Vert {\pi }_{\theta }\right].$$

You sample from ${\pi }^{\star }$ (the data distribution) and push ${\pi }_{\theta }$ toward it. This is the mode-covering direction of the KL: the model is penalized whenever ${\pi }^{\star }$ puts mass somewhere that ${\pi }_{\theta }$ does not, so ${\pi }_{\theta }$ learns to cover every mode of the data.

Reinforcement learning’s entropy-regularized objective ${\mathbb{E}}_{a\sim {\pi }_{\theta }}[r(a\mid x)]+\tau \mathbb{H}[{\pi }_{\theta }]$, written with the same Boltzmann ${\pi }^{\star }\propto \exp (r/\tau )$, becomes

$${\mathcal{L}}_{\text{RL}}(\theta )=-{\mathbb{E}}_{a\sim {\pi }_{\theta }}\left[r(a\mid x)\right]-\tau \mathbb{H}[{\pi }_{\theta }]=\tau \mathrm{KL}\left[{\pi }_{\theta }\Vert {\pi }^{\star }\right]-\tau \log Z(x).$$

The $\log Z$ term does not depend on $\theta$, so minimizing this objective is equivalent to minimizing the KL. Plain expected-reward maximization (without the entropy term) is not a KL by itself; the entropy term is exactly what turns the inner product $⟨{\pi }_{\theta },r⟩$ into a divergence to a target distribution. Almost every RL method that works at scale (RLHF, maximum-entropy RL, soft actor-critic) carries this term, which is why the unification is practically, and not just nominally, useful.

You sample from ${\pi }_{\theta }$ (the policy) and push it toward ${\pi }^{\star }$. This is the mode-seeking direction: the model is penalized whenever it puts mass where ${\pi }^{\star }$ does not, so ${\pi }_{\theta }$ learns to concentrate on high-reward regions.

Once entropy regularization is in the picture, the only structural differences between supervised learning and reinforcement learning are:

1. Which distribution you sample from at training time, ${\pi }^{\star }$ (data) for SL, ${\pi }_{\theta }$ (policy) for RL.
2. Which direction of the KL you optimize, mode-covering for SL, mode-seeking for RL.

Everything else (sample efficiency, variance, off-policy corrections, actor-critic, baselines) is engineering around those two choices.

Entropy regularization bridges them

Once you see that both are KL objectives, a family of intermediate methods falls out.

Reward-augmented maximum likelihood (Norouzi et al., 2016) samples proposals from ${\pi }^{\star }$ at a positive temperature and treats them as soft targets. You get supervised-style stable training with access to the full reward landscape, not just the argmax.

Entropy-regularized policy gradients add an $\mathbb{H}[{\pi }_{\theta }]$ term to the RL objective. This is exactly the KL-to-${\pi }^{\star }$ at positive temperature, which prevents the policy from collapsing to a narrow mode and keeps exploration alive.

UREX (Under-appreciated Reward Exploration) mixes the two KL directions so the model benefits from both mode-covering (to avoid forgetting good solutions) and mode-seeking (to concentrate on the best ones).

In all three cases the knob is the same. It is the temperature $\tau$ of the Boltzmann target, or equivalently the coefficient of the entropy regularizer.

What this predicted about 2024-2026

This framing sat in a Berkeley lecture in 2017 and mostly waited. What happened over the next eight years was that the methods that actually scaled to frontier models turned out to be instantiations of it.

RLHF is exactly the entropy-regularized RL objective at a positive temperature, with the reward model $r_{\varphi }(a\mid x)$ playing the role of the (learned) reward. The KL penalty against the base model that every RLHF paper includes is the entropy-regularization term, written against the prior rather than the uniform distribution.

Direct Preference Optimization (Rafailov et al., 2023) starts from the same Boltzmann target. The authors noticed that if you plug the optimal policy form ${\pi }^{\star }\propto {\pi }_{\text{ref}}\exp (r/\tau )$ back into the Bradley-Terry preference likelihood, the reward cancels out and you can optimize the policy directly against preference pairs as a logistic regression on log-ratios $\log {\pi }_{\theta }(a)/{\pi }_{\text{ref}}(a)$. The resulting loss is supervised-style (no sampling from ${\pi }_{\theta }$, no gradient-variance concerns) even though it is derived from the same entropy-regularized RL objective, which is why DPO is often described as “RLHF done as supervised learning.”

GRPO and its descendants (the post-training methods behind the 2024-2025 reasoning models) are online entropy-regularized RL with group baselines instead of value functions, but the objective is structurally the same.

Reasoning-model fine-tuning (OpenAI o-series, DeepSeek-R1, and the open replications) computes a verifiable reward on math and code problems and trains against the mode-seeking KL. The verifiable reward makes $\tau$ effectively small, so the optimal policy concentrates sharply on correct traces.

None of these methods required a conceptual breakthrough. The scaffolding was already in place. What was missing was a big enough base model to make the Boltzmann target well-defined on useful tasks, and a cheap enough reward signal (preference labels, unit tests, verifiers) to estimate the gradient of the KL.

Takeaways

A few things are worth remembering from this framing.

The split between supervised and reinforcement learning is about which distribution you sample from, not about what you are optimizing. If your reward is sparse and delayed, you sample from the policy and pay the variance cost. If your reward is dense (a label, a unit test), you can sample from the data and get a lower-variance gradient.

The KL direction is not a cosmetic choice. Mode-covering makes training stable but leaves mass on bad outputs. Mode-seeking sharpens the policy but risks collapse. Real methods that work at scale tend to mix both.

Temperature is the same knob as the entropy coefficient, the KL penalty, and the preference-likelihood scaling. Every post-training paper has it; different papers give it different names.

A method that does not fit in this frame is a signal that something is genuinely new, and worth paying attention to. Most methods fit.

Reading

• Norouzi et al., Reward Augmented Maximum Likelihood for Neural Structured Prediction (NeurIPS 2016).
• Nachum et al., Bridging the Gap Between Value and Policy Based Reinforcement Learning (NeurIPS 2017).
• Rafailov et al., Direct Preference Optimization: Your Language Model is Secretly a Reward Model (NeurIPS 2023).
• Shao et al., DeepSeekMath / GRPO (2024).

---

Cover image: Go stones on a goban, by Dietmar Rabich, CC BY-SA 4.0.