[{"content":"I chanced upon a pretty interesting idea proposed by Peter Hayes (now at Anthropic) in 2025, to more efficiently use feedback data in the standard RLHF pipeline to speed up the whole thing. Basically, given external feedback/critique/verification of generated answers, can we incorporate this feedback in an NLL term in addition to the standard DPO loss; trained on the base model? Most previous efforts either used this feedback to train a better verifier, or used other additional NLL terms in the base model training. I spent couple of months reading around the literature, setting up the HPC infra, and running some experiments. Unfortunately experiments (at least on smaller models) resulted in negative findings - the model was extremely prone to reward hacking\u0026hellip;\nFull paper and code available =D\nThis post is a summary of the background and results. We go through DPO in quite a bit of depth, cover relevant research, and show reward hacking for our proposed method (Critique DPO).\nIntroduction Direct Preference Optimization (DPO) enables fine-tuning language models from pairwise preferences without reinforcement learning, often used in Reinforcement Learning with Human Feedback (RLHF) pipelines to align LLMs with human intent. Instead of human preferences, we may use LLM-as-a-judge workflows, where chain-of-thought reasoning improves judgement quality. Can we leverage the judge\u0026rsquo;s reasoning \u0026mdash; not just its binary label \u0026mdash; to improve training sample efficiency when fine-tuning the base model?\nWe propose Critique DPO (CDPO), which augments the DPO objective with a negative log-likelihood term on the verifier\u0026rsquo;s chain-of-thought critique, training the base policy to reproduce step-by-step verification reasoning alongside learning from pairwise preferences. We evaluate CDPO against DPO and SFT baselines on Qwen2.5-0.5B-Instruct using 120,000 math preference triples from the GenRM dataset. Our results are a negative finding: the critique term exacerbates rather than mitigates DPO\u0026rsquo;s reward-hacking failure mode, with CDPO collapsing faster than vanilla DPO on both GSM8K and MATH-500.\nBackground: DPO For a given prompt $x$, two completions are sampled from the policy $(y_{w}, y_{l}) \\sim \\pi_{\\theta}(y | x)$ and an oracle judges which it prefers. We denote $y_{w}$ as the preferred completion and $y_{l}$ as the other. This process is repeated over $N$ prompts resulting in the pairwise preference dataset $\\mathcal{D} = \\\\{x^{(i)}, y_{w}^{(i)}, y_{l}^{(i)}\\\\}_{i=1}^{N}$.\nBradley-Terry Preference Model We can model the judge preferences using the Bradley-Terry (BT) model, which expresses the probability that completion $y_1$ is preferred over $y_2$ under a latent reward function $r^*$:\n$$p^*(y_1 \\succ y_2 \\mid x) = \\frac{\\exp\\left(r^*(x, y_1)\\right)}{\\exp\\left(r^*(x, y_1)\\right) + \\exp\\left(r^*(x, y_2)\\right)} = \\sigma\\!\\left(r^*(x, y_1) - r^*(x, y_2)\\right)$$A key property is that BT depends only on the difference of rewards, not their absolute values.\nStandard RLHF The standard RLHF pipeline comprises 2 stages. First, we learn a reward model $r_\\phi$ where we find the maximum likelihood estimate (MLE) of $\\phi$ under our preference dataset $\\mathcal{D}$ and BT model - termed reward modelling. Then, we use RL (PPO) to learn an optimal policy $\\pi_\\theta$ for this learned reward model.\nReward Modelling Since $r^*$ is unknown, we learn a parametric approximation $r_\\phi$. Under the BT model, the likelihood of the observed preference dataset is $\\ell(\\mathcal{D} \\mid \\phi) = \\prod_{i=1}^{N} \\sigma\\!\\left(r_\\phi(x^{(i)}, y_w^{(i)}) - r_\\phi(x^{(i)}, y_l^{(i)})\\right)$. Taking the log and writing as an expectation, the MLE is obtained by minimising\n$$\\mathcal{L}_{\\text{RM}}(\\phi) = - \\mathbb{E}_{\\mathcal{D}} \\left[\\log \\sigma\\!\\left(r_{\\phi}(x, y_{w}) - r_{\\phi}(x, y_{l})\\right)\\right] \\tag{1}$$RL Fine-Tuning The learned reward model is then used to fine-tune the policy. The aim is to maximise the KL-constrained objective\n$$\\max_{\\pi_\\theta} \\; \\mathbb{E}_{x \\sim \\mathcal{D},\\, y \\sim \\pi_{\\theta}(y | x)} \\left[r_{\\phi}(x, y)\\right] - \\beta \\, D_{\\text{KL}}\\!\\left(\\pi_{\\theta}(y | x) \\| \\pi_{\\text{ref}}(y | x)\\right) \\tag{2}$$where $\\pi_{\\text{ref}}$ is the reference policy (typically the SFT model), kept frozen. The KL term prevents the policy from deviating too far from the distribution on which the reward model is accurate, and $\\beta \u003e 0$ controls this trade-off. In practice this is optimised with PPO, requiring four models in memory: the policy, reference, reward model, and value function.\nDPO DPO [Rafailov et al., 2024] bypasses the reward modelling and RL steps entirely through a change of variables. The key insight is that there exists a bijection between reward functions and their corresponding optimal policies under the KL-constrained objective $(2)$, and this bijection can be exploited to reparameterise the preference likelihood directly in terms of the policy. We follow the expositions by Wolfe, 2025 and Romero, 2024 in our presentation of the derivation below.\nOptimal Policy For any reward function $r$ (not just the learned $r_\\phi$ or the latent $r^*$), the KL-constrained objective $(2)$ can be rewritten as a minimisation of a KL divergence between $\\pi$ and a target distribution defined by $r$, plus a constant. Specifically, letting $Z_r(x) = \\sum_y \\pi_{\\text{ref}}(y|x)\\exp\\!\\left(\\frac{1}{\\beta}r(x,y)\\right)$ be a partition function that depends on $x$ and $r$ but is constant with respect to $\\pi$, the objective becomes\n$$\\min_{\\pi} \\; \\mathbb{E}_{x} \\left[ D_{\\text{KL}}\\!\\left(\\pi(y|x) \\;\\Big\\|\\; \\frac{1}{Z_r(x)}\\,\\pi_{\\text{ref}}(y|x)\\,\\exp\\!\\Big(\\tfrac{1}{\\beta}\\,r(x,y)\\Big)\\right) - \\log Z_r(x) \\right]$$By Gibbs\u0026rsquo; inequality ($D_{\\text{KL}} \\geq 0$, with equality iff the distributions are identical), the unique optimum is\n$$\\pi_r(y|x) = \\frac{1}{Z_r(x)}\\,\\pi_{\\text{ref}}(y|x)\\,\\exp\\!\\left(\\frac{1}{\\beta}\\,r(x,y)\\right) \\tag{3}$$This defines a bijection: every reward $r$ maps to a unique optimal policy $\\pi_r$, and conversely every policy $\\pi_r$ defines a unique reward 1 $r$ for which it is optimal. Using the latent reward $r^*$ gives the optimal policy $\\pi^* \\equiv \\pi_{r^*}$ for our latent rewards.\nReparameterisation Rearranging $(3)$ for the reward expresses any reward purely in terms of its corresponding optimal policy and a prompt-dependent partition function:\n$$r(x, y) = \\beta \\log\\frac{\\pi_r(y|x)}{\\pi_{\\text{ref}}(y|x)} + \\beta \\log Z_r(x)$$Substituting the latent reward $r^*(x,y) = \\beta \\log\\frac{\\pi^*(y|x)}{\\pi_{\\text{ref}}(y|x)} + \\beta \\log Z_{r^*}(x)$ into the BT model, the $\\beta \\log Z_{r^*}(x)$ terms cancel because BT depends only on reward differences and $Z_{r^*}(x)$ is independent of $y$:\n$$p^*(y_w \\succ y_l \\mid x) = \\sigma\\!\\left(\\beta \\log \\frac{\\pi^*(y_w|x)}{\\pi_{\\text{ref}}(y_w|x)} - \\beta \\log \\frac{\\pi^*(y_l|x)}{\\pi_{\\text{ref}}(y_l|x)}\\right)$$The preference model now has no dependence on a reward model \u0026mdash; it is expressed entirely in terms of policies.\nDPO Loss We replace the fixed $\\pi^*$ with a learnable $\\pi_\\theta$ and maximise the likelihood of the observed preferences, conditioned on our policy parameters $\\theta$. Defining the implicit reward\n$$\\hat{r}_\\theta(x, y) = \\beta \\log\\frac{\\pi_\\theta(y|x)}{\\pi_{\\text{ref}}(y|x)} \\tag{4}$$the DPO loss is\n$$\\mathcal{L}_{\\text{DPO}}(\\theta) = - \\mathbb{E}_{\\mathcal{D}} \\left[ \\log \\sigma \\left( \\hat{r}_\\theta(x, y_{w}) - \\hat{r}_\\theta(x, y_{l}) \\right) \\right] \\tag{5}$$This is identical in form to the reward model loss $(1)$, but conditioned on the policy parameters $\\theta$ instead of the reward model parameters $\\phi$. Traditionally, RLHF performs MLE on $\\mathcal{D}$ conditioned on $\\phi$ to learn a reward model, then uses RL to find the optimal policy for that reward. DPO instead performs MLE on $\\mathcal{D}$ conditioned directly on $\\theta$. Because of the reward\u0026ndash;policy bijection, improving $\\pi_\\theta$ to better fit the preference data simultaneously yields an implicit reward $\\hat{r}_\\theta$ for which $\\pi_\\theta$ is guaranteed to be optimal. This eliminates both the reward modelling step and the RL step, requiring only two models in memory (the policy $\\pi_\\theta$ and the frozen reference $\\pi_{\\text{ref}}$).\nLiterature Review We review research incorporating reasoning and critique data into fine-tuning, building toward the critique-augmented preference optimisation approach of this work. We adopt a unified notation: $\\pi_\\theta$ denotes the learnable policy, $\\pi_{\\text{ref}}$ the frozen reference (usually the base SFT model), $x$ the prompt, $y_w$ and $y_l$ the preferred and rejected responses, $r_\\phi(x,y)$ a learned reward (where $\\phi$ parameterises any judge, verifier, or reward model), and $\\hat{r}_\\theta(x,y) = \\beta \\log \\frac{\\pi_\\theta(y|x)}{\\pi_{\\text{ref}}(y|x)}$ the implicit reward. For chain-of-thought traces, $c_w$, $c_l$ denote the model\u0026rsquo;s own reasoning, while a $v$ subscript $c_{v,w}$, $c_{v,l}$ denotes a verifier\u0026rsquo;s or external oracle\u0026rsquo;s critique for the winning or losing response accordingly.\nThe review is organised around a central question: what kind of signal is being used, and for which model? The narrative moves from binary AI feedback, through enriching DPO with supervised reasoning terms, to methods that improve verifiers by having them reason, and finally to the closest work \u0026mdash; methods that bring verifier critique directly into the base model\u0026rsquo;s training objective.\nBeyond Binary Feedback RL from AI Feedback (RLAIF) replaces human annotators with LLM oracles [Bai et al., 2022, Lee et al., 2024], and instructing the labeller to reason explicitly yields modest improvements [Li et al., 2024]. A shared limitation is that the oracle\u0026rsquo;s reasoning is discarded after labelling.\nCan richer reasoning data help in DPO at all? Gao et al., 2025 study CoT\u0026rsquo;s impact on DPO in text-to-SQL, where preference datasets traditionally contain only final SQL queries without reasoning steps. Vanilla DPO degrades performance across most tested LLMs (severe reward acking observed), but augmenting the same datasets with chain-of-thought solutions achieves consistent improvements via a couple of mechanisms: reward hacking mitigation (without CoT, the implicit reward $(4)$ inflates via superficial correlates; with CoT the implicit reward tracks actual performance), and sample efficiency (CoT-augmented DPO continues improving with more data while vanilla DPO saturates).\nSTaR [Zelikman et al., 2022] shows that even self-generated CoT is a richer signal than outcome labels alone. Given a pretrained model $M$, STaR iteratively generates reasoning traces, retains only those leading to correct answers, and fine-tunes on the filtered set:\n$$\\mathcal{L}_{\\text{STaR}}(M) = -\\mathbb{E}_{(x,y) \\sim \\mathcal{D}} \\left[\\,\\mathbf{1}[\\hat{y} = y]\\cdot \\left(\\sum_{t=1}^{|\\hat{c}|} \\log M(\\hat{c}_t \\mid x, \\hat{c}_{\u003c t}) + \\log M(\\hat{y} \\mid x, \\hat{c})\\right)\\right]$$V-STaR [Hosseini et al., 2024] extends this by repurposing the incorrect solutions STaR discards to co-train a verifier $V$ via DPO on preference pairs $(x, y_w, y_l)$:\n$$\\mathcal{L}_{\\text{DPO}}(V^t;\\, G_{\\text{SFT}}) = -\\mathbb{E}_{(x, y_w, y_l) \\sim \\mathcal{D}_{\\text{VER}}} \\!\\left[\\log \\sigma\\!\\left(\\hat{r}(x, y_w) - \\hat{r}(x, y_l)\\right)\\right], \\quad \\hat{r}(x, y) = \\beta \\log \\frac{V^t(y \\mid x)}{G_{\\text{SFT}}(y \\mid x)}$$Note the verifier provides no training-time gradient to the generator \u0026mdash; it is used only for Best-of-$k$ reranking at inference. This distinguishes V-STaR from later methods that feed verifier reasoning directly into the generator\u0026rsquo;s training.\nDPO with NLL A recurring structural pattern combines DPO with an auxiliary negative log-likelihood (NLL) term on a supervised target $t$:\n$$\\mathcal{L}_{\\text{DPO+NLL}} = \\underbrace{-\\log \\sigma\\!\\left(\\hat{r}_\\theta(x, y_w) - \\hat{r}_\\theta(x, y_l)\\right)}_{\\text{preference term}} \\;+\\; \\alpha \\underbrace{\\mathcal{L}_{\\text{NLL}}(t \\mid x)}_{\\text{supervised anchor}} \\tag{6}$$The DPO term provides a contrastive signal (push apart $y_w$ and $y_l$), while the NLL term anchors the policy to a specific supervised target. The choice of $t$ is what distinguishes these methods.\nIRPO [Pang et al., 2024] sets $t$ to the model\u0026rsquo;s own winning chain-of-thought $(c_w, y_w)$. At each iteration, the model samples reasoning traces, partitions them into correct and incorrect by checking against ground truth, and optimises:\n$$\\mathcal{L}_{\\text{IRPO}} = -\\log \\sigma\\!\\left(\\beta \\log \\frac{\\pi_\\theta(c_w, y_w \\mid x)}{\\pi_{\\text{ref}}(c_w, y_w \\mid x)} - \\beta \\log \\frac{\\pi_\\theta(c_l, y_l \\mid x)}{\\pi_{\\text{ref}}(c_l, y_l \\mid x)}\\right) - \\alpha \\frac{\\log \\pi_\\theta(c_w, y_w \\mid x)}{|c_w| + |y_w|}$$Under standard DPO (i.e. $\\alpha = 0$), the log-probabilities of chosen sequences may actually decrease during training \u0026mdash; the model achieves the required preference margin by pushing rejected probabilities down faster than chosen ones fall. The NLL term prevents this: with $\\alpha \u003e 0$, chosen log-probabilities increase while rejected ones decrease.\nAIPO [Shen et al., 2024] and RPO [Liu et al., 2024] simplify the NLL target to just the winning response $t = y_w$. RPO derives this from first principles \u0026mdash; framing DPO overoptimisation as a minimax problem over adversarially chosen reward models, the objective collapses to:\n$$\\mathcal{L}_{\\text{RPO}}(\\theta) = \\mathcal{L}_{\\text{DPO}}(\\theta) \\;-\\; \\eta\\beta\\,\\mathbb{E}_{(x,y_w)\\sim\\mathcal{D}}\\!\\left[\\log \\pi_\\theta(y_w \\mid x)\\right]$$Both methods improve over vanilla DPO, confirming that the NLL anchor mitigates reward hacking regardless of whether the motivation is empirical or theoretical.\nFinally, DJPO [Wang et al., 2024] sets $t$ to the verifier\u0026rsquo;s CoT critique and judgment $(c_v, j)$, but crucially trains a judge rather than the base policy:\n$$\\mathcal{L}_{\\text{DJPO}} = -\\frac{\\log \\pi_\\phi(c_{v,w} \\mid x)}{|c_{v,w}| + |x|} - \\log \\sigma\\!\\left(\\beta \\log \\frac{\\pi_\\phi(c_{v,w} \\mid x)}{\\pi_{\\text{ref}}(c_{v,w} \\mid x)} - \\beta \\log \\frac{\\pi_\\phi(c_{v,l} \\mid x)}{\\pi_{\\text{ref}}(c_{v,l} \\mid x)}\\right)$$where $c_{v,w}$ and $c_{v,l}$ are correct and incorrect judge evaluations respectively. Their SFR-Judge-8B outperforms GPT-4o on aggregate pairwise evaluation across 7 benchmarks.\nGenerative Verifiers and Critique-Augmented Reward Models A parallel line of work improves the verifiers and reward models that judge responses, by having them produce reasoning before scoring.\nGenRM [Zhang et al., 2024] recasts verification as next-token prediction: instead of a scalar reward head, the verifier is an LLM fine-tuned to answer \u0026ldquo;Is the answer correct?\u0026rdquo; In its simplest form (GenRM-Direct), the score is just the next-token probability of \u0026lsquo;Yes\u0026rsquo;. The key variant, GenRM-CoT, inserts a chain-of-thought verification rationale $c_v$ before the final Yes/No token \u0026mdash; rationales are generated offline by a capable LLM and filtered by majority agreement:\n$$r_{\\text{CoT}}(x, y) = \\pi_\\phi(\\text{`Yes'} \\mid x, y, I_v, c_v), \\qquad c_v \\sim \\pi_\\phi(\\cdot \\mid x, y, I_v)$$where $I_v$ = \u0026ldquo;Let\u0026rsquo;s verify step by step.\u0026rdquo; GenRM-CoT achieves 94.2% Best-of-32 on GSM8K (vs. ~90% for a discriminative RM) with 6.4x better sample efficiency. This dataset \u0026mdash; including the per-solution verification rationales \u0026mdash; is the data source for our experiments.\nMahan et al., 2024 extend GenRM by applying DPO to the judge model directly over rationale-plus-judgment sequences. Given unranked response pairs $(x, y_1, y_2)$ where $I$ encodes which response won, instead of discarding incorrect self-generated critique rationales, they construct preference pairs $(c_{v,w}, I_w)$ vs $(c_{v,l}, I_l)$ from correct and incorrect reasoning traces:\n$$\\mathcal{L}_{\\text{STaR-DPO}}(\\pi_\\phi) = -\\mathbb{E}_{\\mathcal{D}}\\!\\left[\\log\\sigma\\!\\left(\\beta\\log\\frac{\\pi_\\phi(I_w, c_{v,w} \\mid x, y_1, y_2)}{\\pi_{\\text{ref}}(I_w, c_{v,w} \\mid x, y_1, y_2)} - \\beta\\log\\frac{\\pi_\\phi(I_l, c_{v,l} \\mid x, y_1, y_2)}{\\pi_{\\text{ref}}(I_l, c_{v,l} \\mid x, y_1, y_2)}\\right)\\right]$$On-policy rationales are essential: off-policy rationales from a stronger model match in-distribution but collapse OOD, due to distributional mismatch between training and inference reasoning styles. CLoud [Ankner et al., 2024] corroborates this finding in a different setting: training reward models on oracle (off-policy) critiques causes a 5.6% RewardBench accuracy drop versus self-generated critiques, because oracle critiques are systematically different from what the model produces at inference. Yu et al., 2025 extend this with a variational formulation (Critic-RM) that treats self-generated critiques as latent variables.\nTraining the Base Model with Critique The methods above improve judges and reward models. A closely related cluster instead trains the base policy model directly using critique signals.\nCritique Fine-Tuning (CFT) [Wang et al., 2025] asks: what if we use only the NLL term on critique data, without any preference optimisation? The base model is trained on a teacher\u0026rsquo;s critique $c_v$ of a query-response pair:\n$$\\mathcal{L}_{\\text{CFT}}(\\theta) = -\\sum_{t=1}^{|c_v|} \\log \\pi_\\theta(c_{v,t} \\mid c_{v, \\lt t}, [x; y])$$where $[x;\\, y]$ concatenates the query and a (potentially flawed) response, and $c_v$ is generated by GPT-4o. At inference, the CFT-trained model generates direct answers without producing critiques. CriticGPT [McAleese et al., 2024] and CTRL [Xie et al., 2025] provide complementary evidence that LLMs can be explicitly taught to generate high-quality critiques.\nPositioning of CDPO The three defining features of CDPO are: (A) a DPO loss on answer preferences $(y_w, y_l)$; (B) an NLL term on the verifier\u0026rsquo;s CoT critique $c_v$; and (C) applied to the base policy model.\nPaper DPO on answers NLL on verifier CoT Base model CDPO (this work) Yes Yes Yes IRPO [Pang et al.] Yes NLL on own CoT Yes AIPO [Shen et al.] / RPO [Liu et al.] Yes NLL on $y_w$ Yes DJPO [Wang et al.] Yes Yes No (judge) GenRM-STaR-DPO [Mahan et al.] Yes Yes No (judge) CFT [Wang et al.] No Yes Yes CDPO sits at the intersection of three lines of work (1) IRPO establishes (DPO + NLL) but anchors to the model\u0026rsquo;s own CoT rather than an external verifier\u0026rsquo;s critique; (2) DJPO and GenRM-STaR-DPO use verifier reasoning as the NLL target but apply it to train a standalone judge rather than the answer-generating base policy; and (3) CFT applies verifier critique as a direct NLL target for the base policy but omits the DPO preference term entirely.\nMethodology We now describe the CDPO objective, dataset construction, and baseline methods.\nData Construction We use the GenRM dataset [Zhang et al., 2024], which provides math problems from the GSM8K [Cobbe et al., 2021] training set. For each problem $x$, multiple candidate solutions are generated by Gemini 1.0 Pro. Each candidate solution is labelled correct or incorrect by checking the extracted final answer against the ground-truth answer. An LLM oracle generates a step-by-step verification rationale for each individual solution, verifying each reasoning step against the expected answer and concluding with a \u0026ldquo;Yes\u0026rdquo; (correct) or \u0026ldquo;No\u0026rdquo; (incorrect) judgement. We denote the verification rationale for a correct solution as $c_{v,w}$ and for an incorrect solution as $c_{v,l}$.\nFrom this raw data, we construct preference triples $(x, y_w, y_l, c_v)$ by pairing, for each problem, a correct solution $y_w$ with an incorrect solution $y_l$. The critique sequence $c_v$ is then assembled by concatenating both solutions together with their respective verification rationales into a single contrastive narrative:\n$c_v$ = \u0026ldquo;This is a correct solution and preferred:\u0026rdquo; + $y_w$ + \u0026ldquo;Here\u0026rsquo;s why this solution is correct and preferred:\u0026rdquo; + $c_{v,w}$ + \u0026ldquo;This is an incorrect solution and not preferred:\u0026rdquo; + $y_l$ + \u0026ldquo;Here\u0026rsquo;s why this solution is incorrect and not preferred:\u0026rdquo; + $c_{v,l}$\nFrom this data we derive four training methods, described below.\nCDPO: Critique Direct Preference Optimization Each CDPO sample is $(x,\\, y_w,\\, y_l,\\, c_v)$. CDPO augments the standard DPO loss with an auxiliary negative log-likelihood (NLL) term that trains the base policy to reproduce the verifier\u0026rsquo;s chain-of-thought critique:\n$$\\mathcal{L}_{\\text{CDPO}}(\\theta) = \\mathbb{E}_{\\mathcal{D}}\\!\\left[(1 - \\alpha)\\,\\mathcal{L}_{\\text{DPO}}(\\theta) \\;+\\; \\alpha\\,\\mathcal{L}_{\\text{critique}}(\\theta)\\right] \\tag{7}$$where $\\alpha \\in [0, 1]$ controls the trade-off between the preference signal and the critique signal. The DPO term $\\mathcal{L}_{\\text{DPO}}$ is the standard loss from Eq. $(5)$. The critique term is the length-normalised NLL over the composite critique $c_v$:\n$$\\mathcal{L}_{\\text{critique}}(\\theta) = -\\frac{1}{|c_v|}\\sum_{t=1}^{|c_v|} \\log \\pi_\\theta(c_{v,t} \\mid x,\\, c_{v, \\lt t})$$Concretely, the model receives the input sequence $[x;\\, c_v]$ \u0026mdash; the problem followed by the critique \u0026mdash; and the NLL loss is computed only over the $c_v$ tokens. Because $c_v$ embeds both solutions $y_w$ and $y_l$ within its narrative (as shown above), it is self-contained: the model learns to generate the full contrastive analysis given only the problem statement. We hypothesise that reproducing this analysis teaches the model to distinguish correct from incorrect reasoning steps, thereby improving its own solution quality.\nCDPO follows the DPO+NLL pattern of Eq. $(6)$ with the NLL target set to the external verifier\u0026rsquo;s critique $c_v$, but uses a convex combination rather than an additive weight so that $\\alpha$ directly controls the relative contribution of each term. Note that at inference time, the CDPO-trained model generates solutions directly \u0026mdash; it does not produce critiques.\nBaseline Methods We compare CDPO against three baselines, all trained on the same underlying data:\nDPO. Each sample is $(x,\\, y_w,\\, y_l)$. The standard DPO loss Eq. $(5)$, equivalent to setting $\\alpha = 0$ in Eq. $(7)$. This isolates the contribution of the critique term.\nSFT (no verification). Each sample is $(x,\\, y_w)$. Standard supervised fine-tuning on correct solutions only:\n$$\\mathcal{L}_{\\text{SFT}}(\\theta) = -\\sum_{t=1}^{|y_w|} \\log \\pi_\\theta(y_{w,t} \\mid y_{w, \\lt t},\\, x)$$SFT (with verification). Each sample is $(x,\\, y_w,\\, c_{v,w})$. The same SFT objective, but the training target is $[y_w;\\, c_{v,w}]$ \u0026mdash; the correct solution followed by its individual verification rationale. This tests whether the verification signal is useful as a pure SFT target, without the DPO preference term.\nExperimental Setup Training Configuration We use Qwen2.5-0.5B-Instruct [Qwen Team, 2024] as the base model. All four methods share the same base configuration: LoRA with rank 8 applied to all linear modules, learning rate $1 \\times 10^{-6}$ with cosine decay and 10% warmup, 2 training epochs, bf16 mixed precision, FlashAttention-2, and DeepSpeed ZeRO Stage 3 across 8 NVIDIA H100 80GB GPUs. For DPO and CDPO we use $\\beta = 0.1$; CDPO additionally uses $\\alpha = 0.5$ with length-normalised critique. Batch sizes differ across methods because CDPO processes three sequences per example (chosen, rejected, critique), requiring significantly more memory.\nEvaluation We evaluate on two mathematical reasoning benchmarks: GSM8K [Cobbe et al., 2021] (500 test samples) and MATH-500 [Hendrycks et al., 2021]. All evaluations use greedy decoding (temperature 0, top-$p$ = 1) via vLLM for efficient inference. LoRA adapters are merged with the base model before evaluation. We evaluate checkpoints saved at every 10% of training. Note that our training data is drawn from GSM8K training problems, so GSM8K evaluation is in-distribution while MATH-500 is out-of-distribution; this partly explains the more stable GSM8K results for SFT methods.\nResults Main Results The table below summarises the best and final checkpoint accuracy for each method. The baseline row shows the pre-training accuracy of the unmodified Qwen2.5-0.5B-Instruct model (checkpoint 0, shared across all methods).\nMethod GSM8K Best GSM8K Final MATH-500 Best MATH-500 Final Baseline (ckpt-0) 49.6 \u0026mdash; 34.4 \u0026mdash; SFT (no verification) 51.8 51.4 35.4 32.6 SFT (with verification) 50.8 50.8 35.2 33.0 DPO 49.6 4.8 34.4 3.2 CDPO ($\\alpha\\!=\\!0.5$) 49.6 0.6 34.4 0.8 SFT methods provide modest, stable improvements over the baseline, while both preference-based methods (DPO and CDPO) suffer catastrophic performance collapse. Critically, CDPO \u0026mdash; the method designed to leverage richer feedback \u0026mdash; performs worse than vanilla DPO.\nTraining Dynamics The figure below shows evaluation accuracy across training checkpoints. The SFT variants maintain stable accuracy throughout training, with SFT (no verification) reaching 51.8% on GSM8K \u0026mdash; a modest 2.2 percentage point improvement over the 49.6% baseline. Both DPO and CDPO begin at the baseline accuracy (checkpoint 0 is the unmodified model) and collapse rapidly: DPO drops to ~6% within the first 400 training steps, while CDPO collapses even faster to ~1% by step 600.\nGSM8K evaluation accuracy over training checkpoints MATH-500 evaluation accuracy over training checkpoints The figure below reveals the mechanism behind this collapse. Despite the catastrophic drop in downstream accuracy, both DPO and CDPO show smoothly decreasing training loss and reward accuracy approaching 1.0 throughout training. This suggests reward hacking: the model learns to separate chosen from rejected examples in the implicit reward space (by pushing log-probability ratios apart), but this optimisation does not correspond to improved generation quality \u0026mdash; it instead degrades the model\u0026rsquo;s ability to produce coherent solutions.\nDPO training loss CDPO training loss DPO reward accuracy CDPO reward accuracy Discussion The dramatic collapse of both DPO and CDPO, combined with smoothly improving training metrics, is a well-known failure mode of DPO. Gao et al., 2025 identify exactly this pattern in text-to-SQL tasks: the implicit reward $\\hat{r}_\\theta$ can be inflated by superficial correlates that do not reflect reasoning quality \u0026mdash; model performance initially peaks then decreases, while the implicit reward continues to rise. At the 0.5B parameter scale, the model appears to lack sufficient capacity to simultaneously maintain generation quality and satisfy the DPO preference margin. We identify two potential mechanisms behind this collapse:\nDistributional mismatch. The verification rationale $c_v$ is generated by an external oracle whose writing style, reasoning depth, and vocabulary differ substantially from the base model\u0026rsquo;s natural distribution. This echoes findings from GenRM-STaR-DPO [Mahan et al., 2024], where off-policy rationales from a stronger model matched STaR-DPO in-distribution but collapsed out-of-distribution. CLoud [Ankner et al., 2024] similarly reports a 5.6% drop when training on oracle critiques versus self-generated ones, attributing it to distributional shift between training and inference.\nTask conflict. The verification rationale is fundamentally different from the solution-generation task the model performs at inference. Verification text contains meta-analytical language (\u0026ldquo;Step 1: check\u0026hellip;\u0026rdquo;, \u0026ldquo;Expected Answer: Correct\u0026rdquo;, \u0026ldquo;Verification: Is the answer correct? Yes/No\u0026rdquo;) that is absent from problem-solving. Training the model to produce both types of text may create a multi-task conflict that degrades performance on the primary task.\nConclusion We proposed Critique DPO (CDPO), which extends the DPO objective with an auxiliary NLL term on a verifier\u0026rsquo;s chain-of-thought critique. We evaluated CDPO against DPO and SFT baselines on Qwen2.5-0.5B-Instruct using 120,000 math preference triples from the GenRM dataset, with evaluation on GSM8K and MATH-500. Our results are a negative finding: both DPO and CDPO suffer model collapse at this scale, with CDPO collapsing faster and more severely than vanilla DPO. The critique NLL term, intended to transfer the verifier\u0026rsquo;s analytical capabilities to the base model, instead exacerbates the well-documented DPO reward-hacking failure mode. Meanwhile, simple SFT on correct solutions provides modest but stable improvements, and adding verification rationales to SFT yields no additional benefit. Future work should explore: (1) on-policy critique generation (STaR-style self-generated rationales instead of off-policy oracle critiques), (2) larger model scales where the multi-task learning implicit in CDPO may become tractable.\nReferences Rafailov, R., Sharma, A., Mitchell, E., Ermon, S., Manning, C. D., \u0026amp; Finn, C. (2024). Direct Preference Optimization: Your Language Model is Secretly a Reward Model. arXiv:2305.18290. https://arxiv.org/abs/2305.18290\nWolfe, C. R. (2025). Direct Preference Optimization (DPO). Blog post. https://cameronrwolfe.substack.com/p/direct-preference-optimization\nRomero, T. (2024). Direct Preference Optimization Explained In-depth. Blog post. https://www.tylerromero.com/posts/2024-04-dpo/\nBai, Y. et al. (2022). Constitutional AI: Harmlessness from AI Feedback. arXiv:2212.08073. https://arxiv.org/abs/2212.08073\nLee, H. et al. (2024). RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedback. arXiv:2309.00267. https://arxiv.org/abs/2309.00267\nLi, A. et al. (2024). HRLAIF: Improvements in Helpfulness and Harmlessness in Open-domain Reinforcement Learning From AI Feedback. arXiv:2403.08309. https://arxiv.org/abs/2403.08309\nGao, H. et al. (2025). Uncovering the Impact of Chain-of-Thought Reasoning for Direct Preference Optimization. arXiv:2502.11656. https://arxiv.org/abs/2502.11656\nZelikman, E., Wu, Y., Mu, J., \u0026amp; Goodman, N. D. (2022). STaR: Bootstrapping Reasoning With Reasoning. arXiv:2203.14465. https://arxiv.org/abs/2203.14465\nHosseini, A., Yuan, X., Malkin, N., Courville, A., Sordoni, A., \u0026amp; Agarwal, R. (2024). V-STaR: Training Verifiers for Self-Taught Reasoners. arXiv:2402.06457. https://arxiv.org/abs/2402.06457\nPang, R. Y., Yuan, W., Cho, K., He, H., Sukhbaatar, S., \u0026amp; Weston, J. (2024). Iterative Reasoning Preference Optimization. arXiv:2404.19733. https://arxiv.org/abs/2404.19733\nShen, Y. et al. (2024). AIPO: Improving Training Objective for Iterative Preference Optimization. arXiv:2409.08845. https://arxiv.org/abs/2409.08845\nLiu, Z. et al. (2024). Provably Mitigating Overoptimization in RLHF: Your SFT Loss is Implicitly an Adversarial Regularizer. arXiv:2405.16436. https://arxiv.org/abs/2405.16436\nWang, P., Xu, A., Zhou, Y., Xiong, C., \u0026amp; Joty, S. (2024). Direct Judgement Preference Optimization. arXiv:2409.14664. https://arxiv.org/abs/2409.14664\nZhang, L., Hosseini, A., Bansal, H., Kazemi, M., Kumar, A., \u0026amp; Agarwal, R. (2024). Generative Verifiers: Reward Modeling as Next-Token Prediction. arXiv:2408.15240. https://arxiv.org/abs/2408.15240\nMahan, D. et al. (2024). Generative Reward Models. arXiv:2410.12832. https://arxiv.org/abs/2410.12832\nAnkner, Z., Paul, M., Cui, B., Chang, J. D., \u0026amp; Ammanabrolu, P. (2024). Critique-out-Loud Reward Models. arXiv:2408.11791. https://arxiv.org/abs/2408.11791\nYu, Y. et al. (2025). Self-Generated Critiques Boost Reward Modeling for Language Models. arXiv:2411.16646. https://arxiv.org/abs/2411.16646\nWang, Y., Yue, X., \u0026amp; Chen, W. (2025). Critique Fine-Tuning: Learning to Critique is More Effective than Learning to Imitate. arXiv:2501.17703. https://arxiv.org/abs/2501.17703\nMcAleese, N. et al. (2024). LLM Critics Help Catch LLM Bugs. arXiv:2407.00215. https://arxiv.org/abs/2407.00215\nXie, Z. et al. (2025). Teaching Language Models to Critique via Reinforcement Learning. arXiv:2502.03492. https://arxiv.org/abs/2502.03492\nQwen Team. (2024). Qwen2.5 Technical Report. arXiv:2412.15115. https://arxiv.org/abs/2412.15115\nCobbe, K. et al. (2021). Training Verifiers to Solve Math Word Problems. arXiv:2110.14168. https://arxiv.org/abs/2110.14168\nHendrycks, D. et al. (2021). Measuring Mathematical Problem Solving With the MATH Dataset. arXiv:2103.03874. https://arxiv.org/abs/2103.03874\nWe really mean unique equivalence class of rewards and its optimal policy; see Lemma 1 and Lemma 2 in Rafailov et al., 2024.\u0026#160;\u0026#x21a9;\u0026#xfe0e;","href":"https://flowingpurplecrane.com/posts/cdpo/","kind":"page","lang":"en","lastmod":"2026-03-14T15:42:57Z","objectID":"5a2577b50efca1b9d0ead1ae7694aba1","publishDate":"2026-03-14T15:42:57Z","section":"posts","tags":[],"title":"Critique DPO","type":"posts"},{"content":"I recently learnt a drinking game where you had 2 cards (red/black) for round, and people use these as answers to questions. The goal of the game is to locate who the minority answers came from, and with that came some interesting questions.\nGame Rules $n$ players each hold a red card and a black card. A question is posed each round (e.g., yes/no, would-you-rather, etc.). Each player secretly plays one card to reflect their answer. All played cards are collected and shuffled — the total counts of each red and black are revealed, but not who played what. This reveals a minority group of size $k$ and a majority of size $n − k$ The group is given $g$ guesses to identify members of the minority. The group\u0026rsquo;s score is $C$, the number of correct guesses. This is the only random variable in this question; everything else is given. Then I had a question: what\u0026rsquo;s a good number for $g$ so that the game is fun? 1 So not too easy $g \\approx n$ where we\u0026rsquo;ll definitely guess who the minorities are, but not too hard $g=k$ 2 that we\u0026rsquo;ll never find the minorities. 3\nModelling Our Game Before we attempt to answer that question, let\u0026rsquo;s model our game first. Specifically, a core question is what is the probability of getting at least $c$ correct out of $g$ guesses? Or how do we compute\n$$P(C \\geq c)$$Under random guessing, you are choosing $g$ players uniformly at random from a population of $n$. $k$ are minorities in this population. This is sampling without replacement, which gives rise to the hypergeometric distribution.\nHypergeometric Distribution The total number of ways to choose $g$ players from $n$ is $\\binom{n}{g}$. For exactly $c$ of those to be minorities, we need to independently choose $c$ from the $k$ minorities and $g - c$ from the $n - k$ majority. So the probability mass function for $C$ 4 follows a hypergeometric distribution where:\n$$P(C = c) = \\frac{\\binom{k}{c}\\binom{n-k}{g-c}}{\\binom{n}{g}}$$ Term Meaning $\\binom{k}{c}$ Ways to correctly pick $c$ minorities from the $k$ available $\\binom{n-k}{g-c}$ Ways to incorrectly pick $g - c$ non-minorities from the $n - k$ majority $\\binom{n}{g}$ Total ways to pick $g$ players from $n$ Survival Function We can then calculate the probability of getting at least $c$ guesses correct by summing the PMF from $c$ upward:\n$$P(C \\geq c) = \\sum_{i=c}^{\\min(g,\\,k)} \\frac{\\binom{k}{i}\\binom{n-k}{g-i}}{\\binom{n}{g}}$$Expected Value $$E[C] = \\frac{gk}{n}$$ This is intuitive: each guess has a $\\frac{k}{n}$ chance of landing on a minority, and you make $g$ guesses.\nVariance $$\\text{Var}(C) = g \\cdot \\frac{k}{n} \\cdot \\frac{n-k}{n} \\cdot \\frac{n-g}{n-1}$$ Note that when $g = n$ (you guess everyone), the variance collapses to 0 — you deterministically identify all minorities.\nChoosing $g$: What Makes the Game Fun? Now we can return to the original question: how to choose $g$ that makes for a fun game? A natural \u0026ldquo;fun\u0026rdquo; thing to have is finding all $k$ minorities.\nThe Falling Factorial Simplification The probability of a perfect guess under random guessing is $P(C = k)$. Setting $c = k$ in the PMF gives\n$$P(C = k) = \\frac{\\binom{k}{k}\\binom{n-k}{g-k}}{\\binom{n}{g}} = \\frac{\\binom{n-k}{g-k}}{\\binom{n}{g}} = \\frac{\\frac{(n-k)!}{(g-k)!\\,(n-g)!}}{\\frac{n!}{g!\\,(n-g)!}} = \\frac{(n-k)! \\cdot g!}{(g-k)! \\cdot n!}$$Now notice that $\\frac{g!}{(g-k)!} = g(g-1)\\cdots(g-k+1)$ and $\\frac{n!}{(n-k)!} = n(n-1)\\cdots(n-k+1)$. These are falling factorials, written $g^{\\underline{k}}$ and $n^{\\underline{k}}$ respectively. So:\n$$P(C = k) = \\frac{g^{\\underline{k}}}{n^{\\underline{k}}} = \\prod_{i=0}^{k-1} \\frac{g - i}{n - i}$$Each factor $\\frac{g-i}{n-i}$ has a nice interpretation: it\u0026rsquo;s the probability that the $(i+1)$-th minority is among your remaining guesses, given that the previous $i$ minorities were already found.\nDeriving the Heuristic The game is fun when guessing everyone is achievable but unlikely — say $\\approx 50\\%$ for a casual drinking game (below 20% feels difficult; above 70% feels trivial). Let $f$ be probability of guessing everyone correctly (a.k.a our fun probability say $f = 0.5$). The number of guesses $g$ is the main difficulty knob, and since $k$ changes every round, $g$ should depend on both $n$ and $k$. So we want to find $g$ such that:\n$$\\prod_{i=0}^{k-1} \\frac{g - i}{n - i} = f$$This is a degree-$k$ polynomial in $g$, so there\u0026rsquo;s no general closed form.5 But the product has $k$ factors that decrease from $\\frac{g}{n}$ (at $i=0$) down to $\\frac{g-k+1}{n-k+1}$ (at $i=k-1$). If we approximate every factor by the largest one $\\frac{g}{n}$, the product becomes $\\left(\\frac{g}{n}\\right)^k \\approx f$, which gives:\n$$\\boxed{g \\approx \\text{round}\\!\\left(n \\cdot f^{1/k}\\right)}$$This overestimates each factor (since later terms are smaller than $\\frac{g}{n}$), but the error is small and — crucially — you can actually compute this after a few drinks. More accurate alternatives exist6 but aren\u0026rsquo;t worth the mental overhead.\nVerification Tested over 676 $(n, k, f)$ triples: $n \\in [5, 15]$, $k \\in [1, \\lfloor n/2 \\rfloor]$, $f \\in \\{20\\%, 25\\%, \\ldots, 80\\%\\}$.\nThe heuristic $g = \\text{round}(n \\cdot f^{1/k})$ is always within $\\pm 1$ of the true optimal across the entire parameter space — it never misses by 2 or more.\nExact match rate by target probability $f$:\n$f$ 20% 30% 40% 50% 60% 70% 80% Exact match 56% 75% 77% 83% 94% 96% 92% The approximation improves as $f$ increases, since a larger $g$ means each $\\frac{g-i}{n-i}$ is closer to $\\frac{g}{n}$.\nExact match rate by minority size $k$:\n$k$ 1 2 3 4 5 6 7 Exact match 96% 84% 79% 76% 76% 73% 73% Performance degrades as $k$ grows — when $k$ is large relative to $n$, the offset terms $i$ in $\\frac{g-i}{n-i}$ are no longer negligible compared to $g$ and $n$. But even at $k = 7$ (the maximum for $n = 15$), the heuristic is always within $\\pm 1$.\nConclusion To play a fun game where around $f=50\\%$ of the time we guess everyone correctly, at each round pick $g$ such that $g \\approx \\text{round}\\!\\left(n \\cdot f^{1/k}\\right)$.\nCrazy, I know. Especially when drunk - did I mention this is a drinking game? =D\nFor a fun game, $g$ really should be greater than $k$, so it is actually possible to guess all the minorities. But we don\u0026rsquo;t want this to be too big (close to $n$), if not we\u0026rsquo;ll definitely guess all the minorities for sure and the game won\u0026rsquo;t be fun.\u0026#160;\u0026#x21a9;\u0026#xfe0e;\nThe reason why I say $g = k$ (you get exactly as many guesses as there are minorities) makes a hard game is because think about the scenario where $n$ is large and $k = 1$. Then you only have one guess to find the minority out of a huge number of people, which frankly is quite impossible! The probability of all correct guesses is $P(C = k) = \\frac{1}{\\binom{n}{k}}$. This drops off really rapidly. For $n = 10, k = 3$: $P = \\frac{1}{120} \\approx 0.83\\%$.\u0026#160;\u0026#x21a9;\u0026#xfe0e;\nNote that every round $k$ changes, so $g$ should really depend on both $n$ and $k$ for the game to be fun. So $g$ should change every round.\u0026#160;\u0026#x21a9;\u0026#xfe0e;\nThe valid range of $c$ is $c \\in \\{\\max(0,\\; g - n + k),\\;\\dots,\\;\\min(g,\\;k)\\}$. The upper bound $\\min(g, k)$ is clear: you can\u0026rsquo;t guess more minorities than exist ($k$), nor more than you have guesses ($g$). The lower bound $\\max(0, g - n + k)$ is less obvious — it says that if you make enough guesses ($g \u003e n - k$), you\u0026rsquo;re forced to hit some minorities because there aren\u0026rsquo;t enough majority members to fill all your guesses.\u0026#160;\u0026#x21a9;\u0026#xfe0e;\nFor $k \\geq 5$ there\u0026rsquo;s no general closed form by the Abel–Ruffini theorem, but the approximation works well for all $k$.\u0026#160;\u0026#x21a9;\u0026#xfe0e;\nUsing the middle factor $\\frac{g-(k-1)/2}{n-(k-1)/2}$ instead of the first gives $g \\approx \\frac{k-1}{2} + (n - \\frac{k-1}{2})\\,f^{1/k}$, which achieves 96.4% exact match vs the naive\u0026rsquo;s 82.1%. A geometric mean of endpoints approach gives a quadratic $g = \\frac{(k-1) + \\sqrt{(k-1)^2 + 4f^{2/k}\\,n(n-k+1)}}{2}$, at 94.5% exact. Both are always within $\\pm 1$, but neither is worth the calculation involved in some stupid drinking game\u0026hellip;\u0026#160;\u0026#x21a9;\u0026#xfe0e;","href":"https://flowingpurplecrane.com/posts/minority/","kind":"page","lang":"en","lastmod":"2026-02-14T23:50:56Z","objectID":"01cd57cb9ac7ad49f529708e136a6c63","publishDate":"2026-02-14T23:50:56Z","section":"posts","tags":[],"title":"The Minority Guessing Game","type":"posts"},{"content":"I\u0026rsquo;ve wondered about entropy for a long time. From thermodynamics and the arrow of time, to cross-entropy loss in ML cost functions, or KL divergence in stopping intelligence from going batshit crazy in VAE/PPO, entropy keeps cropping up. Maybe the universe is playing some kinda trick on us here. Since I\u0026rsquo;m taking 3F7 Information Theory, I thought it\u0026rsquo;s probably a good time to understand what these ideas really mean.\nDisclaimer: I do not claim this work to be my own; this is merely a synthesis of intuition and math across multiple resources for myself and anyone interested. In particular, we borrow the beautiful figures and analogies from Chris Olah\u0026rsquo;s amazing post1 on Information Theory.\nContents 1 Entropy and Self-Information 1.1 Optimal Encoding 1.2 Source Coding Theorem 2 Cross-Entropy 3 KL Divergence 4 Mutual Information 5 Language Modelling 5.1 Cross-Entropy, Maximum Likelihood Estimation, Multi-class Classification 5.2 Auto-regressive Language Modelling 5.3 Information Theory of LLMs 6 Conclusion References 1 Entropy and Self-Information We introduce entropy as the fundamental limit to compressing information.\n1.1 Optimal Encoding Encoding\nSuppose my friend Jerry is at the London Zoo, while I\u0026rsquo;m (Zach) at the Singapore Zoo. There\u0026rsquo;s only 4 kinds of animals in both zoos: \u0026ldquo;dog\u0026rdquo;, \u0026ldquo;cat\u0026rdquo;, \u0026ldquo;fish\u0026rdquo; and \u0026ldquo;bird\u0026rdquo;. Today, he\u0026rsquo;s telling me which animals he saw at the London Zoo - but for whatever reason he can only communicate in binary (0s and 1s).\nJerry likes to talk in binary To communicate, Jerry and I have a secret code mapping animals to sequences of bits. We use a simple mapping, which uses 2 bits to encode 4 animals/symbols:\nA mapping between animal symbols and binary codewords Technically, the animals are symbols $x_i$, with their collective called an alphabet $\\{x_1 \\dots x_i \\dots x_n\\}$. Their respective binary representations are called \u0026ldquo;codewords\u0026rdquo; and the original string message is \u0026ldquo;encoded\u0026rdquo; into its binary representation.\nSome terminology Variable Encoding\nAll animals are equal, but some animals are more equal than others.\nThere are different numbers for each animal type in the London Zoo. We can plot a diagram showing the probability distribution of the animals and the length of our codeword representations:\nProbability distribution of animals in the London Zoo Vertical axis shows the probability of each animal $p(x_i)$, while the horizontal axis shows the length of the corresponding codeword $L(x_i)$ The vertical axis shows the probability $p(x_i)$ of each animal seen by Jerry, while the horizontal axis shows the length of the corresponding codeword $L(x_i)$ in our secret code mapping. Notice the total area summed over all codewords $\\sum_{i} p(x_i) L(x_i)$ is in fact the average length $\\mathbb{E}[L(X)]$ of codewords in our mapping!\nInformation is expensive - we\u0026rsquo;d like to minimize this average length $\\mathbb{E}[L(X)]$. Let\u0026rsquo;s exploit this uneven-ness in the probability distribution, and assign shorter codewords to more frequent animals:\nOur new codeword mapping with shorter codewords for more frequent animals (like dogs) Visualizing $p(x_i)$ and $L(x_i)$ for our new codeword mapping The average length of codewords (total area above) went down from 2 bits to 1.75 bits! How much further can we reduce this? Turns out there\u0026rsquo;s a fundamental limit - we call this the entropy of the probability distribution. To understand why such a limit exists and how to compute it, we need to understand the tradeoffs between making some codewords short and others long.\nTradeoffs in the Space of Codewords, Kraft\u0026rsquo;s Inequality\nTo understand this tradeoff, we visualize all possible codewords first on a table and then on a tree:\nAll possible codewords as a table All possible codewords of length $L_{max}=3$, with the number of leaves on the tree as $2^{L_{max}}$ We need our code to be uniquely decodable, which means that given an encoded string there is only one way to decode it back to our animal symbols. (e.g. an alphabet $\\{0, 1, 01\\}$ as codewords is not uniquely decodable because 001 could mean $[0,0,1]$ or $[0,01]$) One way to achieve this property is to create a prefix-free code, which means no codeword can be the prefix of another codeword. In terms of trees, this means every codeword must be a leaf node. By enforcing this prefix-tree condition, we have to make tradeoffs whenever we use a codeword. Consider using $01$ as a codeword:\nSacrificing $010$ and $011$ Pruning a tree A quarter of codewords $\\{010, 011\\}$ (assuming we only used 3bit codewords) would be gone from the space of codewords. We see that a shorter codeword requires you to sacrifice more of the space of possible codewords. In fact we can quantify this tradeoff as Kraft\u0026rsquo;s inequality:\nConsider a full binary tree of depth $L_{max}$, which has $2^{L_{max}}$ leaves. Whenever we use a codeword $x_i$ at depth $L(x_i)$, we eliminate all its descendant leaves - that\u0026rsquo;s $2^{L_{max} - L(x_i)}$ leaves becoming unavailable. For a prefix-free code with codewords at depths $L_1, L_2, \\ldots, L_n$, the total number of unusable leaves is:\n$$\\sum_{i=1}^{n} 2^{L_{max} - L(x_i)}$$This sum cannot exceed the total number of leaves $2^{L_{max}}$:\n$$\\sum_{i=1}^{n} 2^{L_{max} - L(x_i)} \\leq 2^{L_{max}}$$Dividing both sides by $2^{L_{max}}$:\n$$\\sum_{i=1}^{n} 2^{-L(x_i)} \\leq 1$$This is Kraft\u0026rsquo;s inequality. It\u0026rsquo;s both a necessary condition (any prefix-free code must satisfy it) and a sufficient condition (if a set of lengths satisfies it, we can construct a prefix-free code with those lengths). This inequality formalizes the tradeoff in the space of codewords - shorter codewords are \u0026ldquo;expensive\u0026rdquo; as they eliminate more possibilities. Assuming a total space of $1$, each codeword $x_i$ of length $L(x_i)$ costs $2^{-L(x_i)}$ of the total space of codewords. We can think of the total space as a maximum fixed \u0026ldquo;budget\u0026rdquo; of $1$, and each codeword costing $2^{-L(x_i)}$ of this budget.\nOptimal Encodings\nWe\u0026rsquo;ve showed the cost of codewords decreases exponentially with its length. We can visualize this 2 with the cost on the vertical axis and the length on the horizontal axis of an exponential curve:\nVisualizing the cost of a codeword against its length. Both the height and right shaded area are numerically equal. Note that since cost decays exponentially, the height and the right shaded area are numerically equal when using natural exponentials2. We can also visualize the contribution $p(x_i) L(x_i)$ of this codeword to the average codeword length:\n$$\\mathbb{E}[L(X)]=\\sum_{i} p(x_i) L(x_i)$$ Contribution $p(x_i) L(x_i)$ of codeword $x_i$ to the average codeword length $\\mathbb{E}[L(X)]$ Visualizing both the cost and length contribution we get:\nFor codeword $x_i$, contribution to average codeword length is the left area, while cost of codeword space is the right area Short codewords reduce the average message length but are expensive, while long codewords increase the average message length but are cheap. The above diagram makes the tradeoff clear: short codewords reduce the average message length but are expensive, while long codewords increase the average message length but are cheap. So what\u0026rsquo;s the optimal allocation of the codeword space budget to each symbol $x_i$, so that the average codeword length is minimized? We propose that if a symbol (or event) $x_i$ appears with probability $p(x_i)$, then the optimal allocation is to allocate $p(x_i)$ of the codeword space budget to this symbol. Here\u0026rsquo;s a handwavy proof:3\n$$ \\text{Cost}(x_i)=2^{-L(x_i)}=p(x_i), \\forall i $$$$ \\frac{d\\big(\\text{Length Contribution}(x_i)\\big)}{dL(x_i)} = \\frac{d\\big(p(x_i) L(x_i)\\big)}{dL(x_i)} = p(x_i) $$$$ \\frac{d\\big(\\text{Cost}(x_i)\\big)}{dL(x_i)} = \\frac{d\\big(2^{-L(x_i)}\\big)}{dL(x_i)} = -2^{-L(x_i)} \\ln(2) $$At the optimal allocation where $2^{-L(x_i)} = p(x_i)$,\n$$\\frac{|d(\\text{Length Contribution})/dL|}{|d(\\text{Cost})/dL|} = \\frac{\\text{Marginal benefit from shorter codeword}}{\\text{Marginal cost in codeword space}} = \\frac{p(x_i)}{2^{-L(x_i)}\\ln(2)} = \\frac{1}{\\ln(2)}$$which is constant regardless of $p(x_i)$. This means at the optimal budget, it\u0026rsquo;s equally worthwhile to invest in making any codeword shorter as all codewords have the same marginal benefit/cost ratio. Note that the actual value of this ratio doesn\u0026rsquo;t matter (the top and bottom quantities don\u0026rsquo;t have the same units so we cannot compare them meaningfully in terms of absolute values) - what matters is the relative ratios between the different symbols.\nIf $2^{-L(x_i)} \\ne p(x_i)$, then this would mean the ratio for each ${x_i}$ would in fact be dependent on both $L(x_i)$ and $p(x_i)$, and will result in uneven - hence suboptimal allocation of budgets. Any uneven-ness of this ratio means it will make more sense to spend more budget in one codeword than other codewords.\nCalculating Entropy\nSince the cost of an encoded message of length $L(x_i)$ is $2^{-L(x_i)}$, then $L(x_i)=\\log_2\\left(\\frac{1}{\\text{Cost}}\\right)$. Since for optimum allocation of budgets (to minimize average lengths), we allocated $p(x_i)$ of the budget to $L(x_i)$, then\n$$L(x_i)=\\log_2\\left(\\frac{1}{p(x_i)}\\right)$$The average message length under optimal allocation is\n$$\\mathbb{E}[L(X)] = \\sum_{i} p(x_i) L(x_i) = \\sum_{i} p(x_i) \\log_2\\left(\\frac{1}{p(x_i)}\\right) $$Kraft\u0026rsquo;s inequality created a budget constraint on how many short codewords you can have. The optimal strategy is to spend this budget proportionally to symbol probabilities. Any other allocation either violates the constraint (making the code invalid, like not prefix-free) or wastes budget (making the average length longer). Hence for a probability distribution of symbols, the minimum average length across all the encoded symbols is:\n$$\\min_{\\text{all encodings}} \\mathbb{E}[L(X)] = \\sum_{i} p(x_i) \\log_2\\left(\\frac{1}{p(x_i)}\\right)$$Note that\n$$H(X) = \\sum_{i} p(x_i) \\log_2\\left(\\frac{1}{p(x_i)}\\right)$$hence\n$$H(X) = \\min_{\\text{all encodings}} \\mathbb{E}[L(X)]$$Entropy emerges because it\u0026rsquo;s the only allocation that balances the budget constraint with minimizing average length! The more concentrated the probability, the easier it is to exploit this by crafting short encodings. The more spread out the probability, the less I am able to exploit this, and I have to make all my encodings longer on average. So a distribution with higher uncertainty requires more bits to communicate information.\n1.2 Source Coding Theorem We\u0026rsquo;ve shown the intuition that the optimal codeword allocation follows $2^{-L(x_i)} = p(x_i)$, yielding entropy. Here\u0026rsquo;s the actual proof for this - Shannon\u0026rsquo;s source coding theorem. We\u0026rsquo;ve seen that prefix-free codes obeys the budget constraint $\\sum_{i} 2^{-L(x_i)} \\leq 1$, and McMillan4 proved the same constraint holds for all uniquely decodable codes (of which prefix-free codes are just a subset). Consider a general (possibly non-optimal) coding scheme giving length $L(x_i)$ for symbol $x_i$:\n$$H(X) - \\mathbb{E}[L(X)] = \\sum_{i} p(x_i) \\log_2\\left(\\frac{1}{p(x_i)}\\right) - \\sum_{i} p(x_i) L(x_i)$$$$= \\sum_{i} p(x_i) \\log_2\\left(\\frac{2^{-L(x_i)}}{p(x_i)}\\right)$$$$= \\frac{1}{\\ln 2} \\sum_{i} p(x_i) \\ln\\left(\\frac{2^{-L(x_i)}}{p(x_i)}\\right)$$Note for any $x \u003e 0$, we have $\\ln(x) \\leq x - 1$ (with equality only when $x = 1$). This gives us:\n$$H(X) - \\mathbb{E}[L(X)] \\leq \\frac{1}{\\ln 2} \\sum_{i} p(x_i) \\left(\\frac{2^{-L(x_i)}}{p(x_i)} - 1\\right)$$$$\\frac{1}{\\ln 2} \\left(\\sum_{i} 2^{-L(x_i)} - 1\\right) \\leq 0$$where we\u0026rsquo;ve used McMillan\u0026rsquo;s constraint $\\sum_i 2^{-L(x_i)} \\leq 1$. Therefore\n$$H(X) \\leq \\mathbb{E}[L(X)]$$no encoding can beat entropy.\nShannon\u0026rsquo;s Source Coding Theorem\nFor any uniquely decodable code encoding symbols (where $L(X)$ gives the length of the codeword for symbol $X$) from a distribution with entropy $H(X)$:\n$$H(X) \\leq \\mathbb{E}[L(X)]$$This limit explains why:\nRandom data doesn\u0026rsquo;t compress — it has maximum entropy because all symbols are equally likely, we can\u0026rsquo;t exploit shorter messages for more common events English text compresses well — it has entropy $\\sim 1.5$ bits per letter (much less than uniform distribution $\\log_2(26) \\approx 4.7$) because letter probabilities are wildly uneven 2 Cross-Entropy Turns out the distribution of animals is different in the London Zoo $P$ and Singapore Zoo $Q$:\nAnimal distributions differ: London Zoo has more dogs, Singapore Zoo has more cats What if I design my code mapping to be locally optimal for the Singapore Zoo, but Jerry is actually sending me data from the London Zoo overseas? That is, I optimize codeword lengths $L(x_i) = \\log_2\\left(\\frac{1}{q(x_i)}\\right)$ for distribution $Q$ (Singapore), but the actual data comes from distribution $P$ (London). The expected codeword length becomes:\n$$H_P(Q) = \\sum_i p(x_i) \\log_2\\left(\\frac{1}{q(x_i)}\\right)$$We call this the cross-entropy from $P$ to $Q$. It represents the average length of encoded messages for data actually drawn from distribution $P$, using an optimal code designed for distribution $Q$. Since we have used the unoptimized code mapping on distribution $P$, we have to pay a cost through an increase in the average length of encoded messages $H_P(Q) \\geq H(P)$:\n$$H_P(Q) = H(P) + D_{KL}(P \\| Q)$$ Cross Entropy measures the average length of encoded messages drawn from $P$ using the encoding of $Q$, which is longer than the encoding optimize for $P$ $H(P)$: The irreducible minimum—the entropy of the true underlying distribution $D_{KL}(P \\| Q)$: The penalty for mismatch—how much extra bits we waste by optimizing for the wrong distribution This mismatch means we waste bits on unexpected events (where $p(x_i) \u003e q(x_i)$) while potentially using longer-than-necessary codes for common events (where $p(x_i) \u003c q(x_i)$), hence giving a length penalty. When $Q = P$ (our assumption is perfect), the second term vanishes and cross-entropy equals entropy—we\u0026rsquo;re encoding optimally. When they differ, $D_{KL}(P \\| Q) \u003e 0$ quantifies exactly the penalty. In this sense, cross-entropy gives us a measure of how different 2 probability distributions are. The more different 2 distributions are, the bigger the cross-entropy $H_P(Q)$.\nAsymmetry\nCross-entropy is fundamentally asymmetric $H_P{(Q)} \\ne H_Q{(P)}$:\nCross-entropy computed in various ways $H_P{(Q)}$ is large because we pay a very large penalty using a long codeword (that\u0026rsquo;s rare in $Q$ but common in $P$) for the purple symbol. On the other hand, this extra cost for using unnecessarily longer codewords when encoding real data from $Q$ is not quite as big.\n3 KL Divergence Now that we understand cross-entropy, we can isolate the pure cost of mismatch. The Kullback-Leibler (KL) divergence is exactly this: cross-entropy minus the intrinsic entropy.\n$$D_{KL}(P \\| Q) = H_P(Q) - H(P) = \\sum_i p(x_i) \\log\\left(\\frac{p(x_i)}{q(x_i)}\\right)$$This is the extra bits we pay because we optimized for the wrong distribution $Q$ instead of the the distribution where samples came from $P$. Note the the following properties:\nNon-negativity: $D_{KL}(P \\| Q) \\geq 0 \\text{, with equality iff } P = Q$ Not really a metric 5 6 Asymmetry $D_{KL}(P \\| Q) \\neq D_{KL}(Q \\| P)$ 7 Forward KL: $D_{KL}(P \\| Q)$ (what we usually minimize) is mean-seeking. If $P$ is bimodal, and we optimize $Q$ to reduce KL, then $Q$ diffuses to cover both modes Reverse KL: $D_{KL}(Q \\| P)$ is mode-seeking. To minimize KL, $Q$ collapses onto a single mode of $P$ rather than spreading across both 4 Mutual Information Mutual information measures how much one variable reduces the uncertainty about another:\n$$I(X; Y) = D_{KL}(P(X,Y) \\| P(X)P(Y))$$This compares the joint distribution $P(X,Y)$ against what we\u0026rsquo;d expect if $X$ and $Y$ were independent: $P(X)P(Y)$. When variables are independent, the joint factorizes and the KL divergence is zero—they share no information. When they\u0026rsquo;re dependent, $I(X; Y) \u003e 0$ quantifies the dependence. Using the decomposition $H_P(Q) = H(P) + D_{KL}(P \\| Q)$, we can also expand:\n$$I(X; Y) = H(X) - H(X|Y)$$This form says: \u0026ldquo;mutual information is the reduction in uncertainty about $X$ when we learn $Y$.\u0026rdquo; Since $H(X|Y)$ is the conditional entropy (uncertainty about $X$ after knowing $Y$), the difference $H(X) - H(X|Y)$ measures exactly how much information $Y$ gave us about $X$.\nBy the chain rule of entropy $H(X,Y) = H(X) + H(Y|X) = H(Y) + H(X|Y)$ we can derive:\n$$I(X; Y) = H(Y) - H(Y|X)$$Combining both:\n$$I(X; Y) = H(X) + H(Y) - H(X,Y)$$Note the following properties:\nSymmetry: $I(X; Y) = I(Y; X)$ knowing $Y$ tells you as much about $X$ as knowing $X$ tells you about $Y$ Non-negativity: $I(X; Y) \\geq 0 $ Due to non-negativity of $D_{KL}$ With equality iff $X$ and $Y$ are independent Self-Information: $I(X; X) = H(X)$ This is why entropy is sometimes called \u0026ldquo;self-information\u0026rdquo;—it\u0026rsquo;s just mutual information of a variable with itself. Note that $P(X,X)=P(X)$. Here\u0026rsquo;s the full picture:\nDiagram of entropy relationships. H(X) and H(Y) are marginal entropies, H(X,Y) is joint entropy, I(X;Y) is mutual information shown as overlap When $X$ and $Y$ are independent, there\u0026rsquo;s no overlap—$I(X;Y) = 0$. When they\u0026rsquo;re perfectly correlated, they share all their information—$I(X;Y) = H(X) = H(Y)$.\n5 Language Modelling 5.1 Cross-Entropy, Maximum Likelihood Estimation, Multi-class Classification The cross entropy is often used as an objective/loss function in multi-class classification, when training machine learning (ML) or large language models (LLM). This objective is essentially minimizing the distance between our model\u0026rsquo;s distribution and the ground truth distribution:\n$P$ = true labels (the ground truth data distribution) $Q$ = predicted probabilities (our model\u0026rsquo;s beliefs) Minimizing cross-entropy is equivalent to maximum likelihood estimation (MLE)! Given a sample dataset $\\{x_1, \\ldots, x_n\\}$ (sample datapoints not the alphabet here) drawn from true distribution $P$, define the likelihood of data under our model with parameters $\\theta$:\n$$\\mathcal{L}(\\theta) = \\prod_{i=1}^{n} q_\\theta(x_i)$$We seek\n$$\\hat{\\theta} = \\arg\\max_\\theta \\mathcal{L}(\\theta) = \\arg\\max_\\theta \\prod_{i=1}^{n} q_\\theta(x_i) $$Convert the product to a sum via logarithms (since logarithms are strictly increasing and do not affect argmax), then normalize by sample count:\n$$\\hat{\\theta} = \\arg\\max_\\theta \\frac{1}{n}\\sum_{i=1}^{n} \\log q_\\theta(x_i)$$By the Law of Large Numbers, this sample average converges to an expected value as $n \\to \\infty$. Specifically, the expectation is with respect to the true underlying distribution $P$:\n$$\\frac{1}{n}\\sum_{i=1}^{n} \\log q_\\theta(x_i) \\to \\mathbb{E}_{x \\sim P}\\left[\\log q_\\theta(x)\\right] = \\sum_{i} p(x_i) \\log q_\\theta(x_i)$$Negate and minimize:\n$$\\hat{\\theta} = \\arg\\min_\\theta \\left\\{ -\\sum_{i} p(x_i) \\log q_\\theta(x_i) \\right\\} = \\arg\\min_\\theta H_P(Q_\\theta)$$MLE is equivalent to minimizing cross-entropy!\nNote that minimizing cross-entropy is also equivalent to minimizing the KL divergence $D_{KL}(P \\| Q)$. Since $H(P)$ is the entropy of the true distribution, it\u0026rsquo;s independent of the model parameters $\\theta$.\n$$\\arg\\min_\\theta H_P(Q_\\theta) = \\arg\\min_\\theta [D_{KL}(P \\| Q_\\theta) + H(P)] = \\arg\\min_\\theta D_{KL}(P \\| Q_\\theta)$$MLE in Practice\nIn practice, we don\u0026rsquo;t have access to the true distribution $P$. Instead, we can approximate this by using a batch of sample data. Consider a classification batch of $n$ samples, where each sample $i$ has a true label $y_i$ (one-hot: probability 1 for the true class, 0 elsewhere) and the model predicts $q_\\theta(y=k)$ for each class $k$. The batch loss is:\n$${L}(\\theta) = -\\frac{1}{n} \\sum_{i=1}^{n} \\log q_\\theta(y_i)$$Note in practice we only sum the model log probabilities for the class corresponding to the true class for each sample, despite having log probs for all classes. Now regroup by class with $n_k$ samples in each class $k$, so we can rewrite the sum as:\n$$ {L}(\\theta) = -\\frac{1}{n} \\sum_{k=1}^{K} n_k \\log q_\\theta(y=k) = -\\sum_{k=1}^{K} \\frac{n_k}{n} \\log q_\\theta(y=k)$$Note that $\\hat{p}(y=k) = \\frac{n_k}{n}$ is the empirical frequency of class $k$ in the batch\n$$ {L}(\\theta) = -\\sum_{k=1}^{K} \\hat{p}(y=k) \\log q_\\theta(y=k) = H_{\\hat{P}}(Q_\\theta)$$This is exactly the cross-entropy formula! One-hot labels, when averaged over a batch, naturally produce an empirical distribution. With sufficient data, $\\hat{p}(y=k) \\to p(y=k)$ (the true class distribution), so batch averaging realizes the theoretical MLE objective.\n5.2 Auto-regressive Language Modelling So we\u0026rsquo;ve seen how cross-entropy applies to multi-class classification. We now extend this to LLMs that generate long sequences of text. Consider a sequence of tokens $X=(x_1, x_2, \\dots, x_T)$. The objective of LLMs is to model the joint probability distribution of this sequence $p(x_1, x_2, \\dots, x_T)$. We can use the chain rule of probability to factor this joint distribution into a product of conditional probabilities:\n$$p(x_1,\\ldots,x_T) = p(x_1) \\cdot p(x_2 | x_1) \\cdot p(x_3 | x_1, x_2) \\cdots = \\prod_{t=1}^T p(x_t|x_1,\\ldots,x_{t-1})$$An autoregressive LLM is just a sequence of simple classification tasks! At each timestep $t$,\nIt takes the context $x_{\\lt t}$ (e.g., \u0026ldquo;The quick brown fox\u0026hellip;\u0026rdquo;). It predicts a probability distribution $Q_\\theta(\\cdot | x_{\\lt t})$ over the entire vocabulary (e.g., 50,000 possible next tokens). Each token is a class. The \u0026ldquo;true label\u0026rdquo; $P_t$ is simply the one-hot vector for the token that actually comes next in the training data (e.g., \u0026ldquo;jumps\u0026rdquo;). The total loss for one sequence ${L}_X(\\theta)$ is just the sum of the cross-entropy losses across all timesteps:\n$${L}_X(\\theta) = \\sum_{t=1}^T -\\log q_\\theta(x_t | x_{\\lt t}) $$This sum of losses is equivalent to the negative log-likelihood of the entire sequence $X$:\n$${L}_X(\\theta) = -\\sum_{t=1}^T \\log q_\\theta(x_t | x_{\\lt t}) = -\\log \\left( \\prod_{t=1}^T q_\\theta(x_t | x_{\\lt t}) \\right) = -\\log q_\\theta(X)$$LLMs are just doing MLE on the joint distribution of the entire sequence.\n5.3 Information Theory of LLMs Prediction is essentially compression.\nGeneralization\nInformation theory can help us analyze generalization in LLMs, by measuring a model\u0026rsquo;s capacity in bits [Morris, 2025]. Morris splits learned information into \u0026ldquo;unintended memorization\u0026rdquo; (information specific to dataset) and \u0026ldquo;generalization\u0026rdquo; (underlying distribution). They then compute the total capacity of LLMs, where models in the GPT family have an approximate capacity of 3.6 bits-per-parameter. When a model is trained on data whose information content exceeds its capacity, it must make a tradeoff between memorizing individual datapoints, or begin extracting general patterns that compress the data more efficiently. This transition point—where unintended memorization gives way to generalization—occurs precisely when the dataset\u0026rsquo;s entropy exceeds the model\u0026rsquo;s capacity. This explains phenomena like \u0026ldquo;grokking\u0026rdquo; and double descent.\nInformation in Chain-Of-Thought\nWhen an LLM uses Chain-of-Thought (CoT) to solve a problem, it generates intermediate steps. But are these steps actually useful? Researchers quantify [Ton, 2025] the usefulness of each reasoning step ($X_t$) by measuring its \u0026ldquo;information gain\u0026rdquo;—the conditional mutual information it shares with the final correct answer ($Y$), given all the previous steps ($X_{\\lt t}$):\n$$G_t = I(Y; X_t | X_{ \\lt t})$$This metric basically answers: \u0026ldquo;How much new and relevant information did this step provide about the final answer, given everything the model has already said?\u0026rdquo;\n6 Conclusion Whether we\u0026rsquo;re designing optimal codes for data compression, training models to predict the next token, or evaluating multi-step reasoning, we return to the same fundamental question: How much information is really here? As we\u0026rsquo;ve seen throughout this post, entropy—in all its forms—is often the answer.\nReferences Entropy, Cross-Entropy, KL Divergence\nZhu, J. Information Theory and Machine Learning. University of Wisconsin-Madison, CS 769 Lecture Notes.\nSarang, N. (2024). Entropy, Cross-Entropy, KL Divergence, and Mutual Information.\nJames Explains. (2024). Information Theory - Entropy, Cross Entropy, KL Divergence [Video]. YouTube.\nBendersky, E. (2025). Cross-Entropy and KL Divergence. Eli Bendersky\u0026rsquo;s Website.\nMaulik, A. Entropy. Indian Institute of Technology Guwahati, CS Lecture Slides.\nOlah, C. (2015). Visual Information Theory.\nGoodfellow, I., Bengio, Y., \u0026amp; Courville, A. (2016). Deep Learning (Chapter 3: Probability and Information Theory). MIT Press.\nTan, Z., Li, C., \u0026amp; Huang, W. (2024). The Information of Large Language Model Geometry.\nTon, J.F., Taufiq, M.F., \u0026amp; Liu, Y. (2025). Understanding Chain-of-Thought in LLMs through Information Theory. ICML 2025\nMorris, J. X., Sitawarin, C., Guo, C., Kokhlikyan, N., Suh, G. E., Rush, A. M., Chaudhuri, K., \u0026amp; Mahloujifar, S. (2025). How much do language models memorize?.\nSource Coding Theorem\nWikipedia. Kraft–McMillan Inequality.\nWikipedia Shannon's Source Coding Theorem.\nBernstein, M. Shannon's Source Coding Theorem. Matthew Bernstein\u0026rsquo;s Blog.\nSingh, A. (2015). Information Theory. Carnegie Mellon University, 10-704 Lecture Notes.\nVenkataramanan, R. (2025). 3F7: Information Theory and Coding - Handouts 2 \u0026 3. University of Cambridge, Department of Engineering.\nOlah, C. (2015). Visual Information Theory. Chris Olah does an awesome job explaining the intuition behind entropy and its related concepts\u0026#160;\u0026#x21a9;\u0026#xfe0e;\nTechnically the equal area math only holds if we work with natural logs instead of base 2 logs as $\\int_{x'}^{\\infty} 2^{-x} \\, dx = \\frac{2^{-x'}}{\\ln 2}$ We are working with base 2 logs instead of natural logs, but we omit the constant scaling factor in our calculations if not they will look rather messy.\u0026#160;\u0026#x21a9;\u0026#xfe0e;\u0026#160;\u0026#x21a9;\u0026#xfe0e;\nI chose not to use Olah\u0026rsquo;s visual proof exactly here because his proof suggests that we need the benefit/cost ratio to be 1. He suggests any deviation in the allocation of budget will change the ratio to not be 1, which is not exactly the right argument. The important thing is that all symbols have the same benefit/cost ratio, so the allocation of budget is optimum.\u0026#160;\u0026#x21a9;\u0026#xfe0e;\nSee Kraft–McMillan Inequality (1956).\u0026#160;\u0026#x21a9;\u0026#xfe0e;\nA true metric must satisfy symmetry: $d(A, B) = d(B, A)$ and the triangle inequality: $d(A, C) \\leq d(A, B) + d(B, C)$. KL divergence doesn\u0026rsquo;t satisfy either property, so it is not really a proper metric for measuring distances between distributions.\u0026#160;\u0026#x21a9;\u0026#xfe0e;\nSome researchers define the Jensen-Shannon divergence as a symmetric alternative: $\\text{JS}(P, Q) = \\frac{1}{2}D_{KL}(P \\| M) + \\frac{1}{2}D_{KL}(Q \\| M)$ where $M = \\frac{P + Q}{2}$. This is a true metric (and $\\sqrt{\\text{JS}}$ is a metric), but it obscures the directed nature of the problem. For machine learning, forward KL is almost always the right choice.\u0026#160;\u0026#x21a9;\u0026#xfe0e;\nSee Nima Sarang's post on Information Theory.\u0026#160;\u0026#x21a9;\u0026#xfe0e;","href":"https://flowingpurplecrane.com/posts/info/","kind":"page","lang":"en","lastmod":"2025-10-25T16:34:06.888Z","objectID":"2c51252e94219472b01971aa710e7c10","publishDate":"2025-10-25T16:34:06.888Z","section":"posts","tags":[],"title":"On Information","type":"posts"},{"content":"We analyze 1 the theoretical properties of both the Discrete Fourier Transform (DFT) and the optimized Fast Fourier Transform (FFT), and estimate their algorithmic complexity.\nAll code available on Github. Slides available here.\nDiscrete Fourier Transform Theory The DFT converts a finite-length time-domain signal into its frequency-domain representation. For an input signal $x[n]$ of length $N$, where $0 \\le n \\le N-1$, the DFT is defined as:\n$$ X[k] = \\sum_{n=0}^{N-1} x[n]\\, W_N^{nk}, \\quad 0 \\le k \\le N-1 \\label{eq:DFT} $$where $W_N=e^{-j\\frac{2\\pi}{N}}$ is the $N$-th principal root of unity, and $X[k]$ is component at frequency $kf_s/N$ with sampling frequency $f_s$. We can rewrite $X[k]$ as an inner product:\n$$ \\begin{align} X[k] \u0026= \\left[ 1 \\quad e^{-j\\frac{2\\pi k}{N}} \\quad \\ldots \\quad e^{-j\\frac{2\\pi k}{N}(N-1)} \\right] \\begin{bmatrix} x[0] \\\\ x[1] \\\\ \\vdots \\\\ x[N-1] \\end{bmatrix} \\\\ \u0026= \\left[ 1 \\quad W_N^k \\quad \\ldots \\quad W_N^{(N-1)k} \\right] \\begin{bmatrix} x[0] \\\\ x[1] \\\\ \\vdots \\\\ x[N-1] \\end{bmatrix} \\end{align} $$By varying $k$ from $0$ to $N-1$, we can collate all outputs ${X[k]}$ into a vector $\\mathbf{X}$, collate all inputs $x[n]$ into a vector $\\mathbf{x}$, to get:\n$$ \\mathbf{X}=\\mathbf{W}\\mathbf{x}, \\quad \\mathbf{W} = \\begin{bmatrix} 1 \u0026 1 \u0026 1 \u0026 \\cdots \u0026 1 \\\\ 1 \u0026 W_N \u0026 W_N^2 \u0026 \\cdots \u0026 W_N^{N-1} \\\\ 1 \u0026 W_N^2 \u0026 W_N^4 \u0026 \\cdots \u0026 W_N^{2(N-1)} \\\\ \\vdots \u0026 \\vdots \u0026 \\vdots \u0026 \\ddots \u0026 \\vdots \\\\ 1 \u0026 W_N^{N-1} \u0026 W_N^{2(N-1)} \u0026 \\cdots \u0026 W_N^{(N-1)(N-1)} \\end{bmatrix} $$where $\\mathbf{W}$ is the DFT matrix.\nComplexity Pseudocode for DFT Direct implementation in MATLAB:\nfunction X = dft_loop(x) % Compute the Discrete Fourier Transform (DFT) of input vector x using naive loops % x : input signal (vector) % X : DFT of x % Convert input to a column vector for consistency x = x(:); N = length(x); % Number of samples in the signal % Pre-allocate the output vector for efficiency X = zeros(N, 1); % Loop over each frequency bin k (from 0 to N-1) for k = 0:N-1 sum_val = 0; % Loop over each time sample n (from 0 to N-1) for n = 0:N-1 % Compute and accumulate the contribution for the k-th frequency component sum_val = sum_val + x(n+1) * exp(-1j * 2 * pi * k * n / N); end % MATLAB uses one-based indexing, so assign to X(k+1) X(k+1) = sum_val; end end MATLAB code (vectorized):\nfunction X = dft_vectorized(x) % Compute the Discrete Fourier Transform (DFT) of input vector x using vectorized operations % x : input signal (vector) % X : DFT of x % Convert input to a column vector for consistency x = x(:); N = length(x); % Number of samples in the signal % Create index vectors: n as a row vector and k as a column vector n = 0:N-1; % Time indices (row vector) k = n\u0026#39;; % Frequency indices (column vector) % Construct the DFT matrix using the formula: exp(-1j*2*pi*k*n/N) W = exp(-1j * 2 * pi * k * n / N); % Multiply the DFT matrix with the signal to obtain the transform X = W * x; end Direct computation of $\\mathbf{X}$ requires $(N-1)^2$ complex multiplications and $N(N-1)$ complex additions. Since multiplication is more compute-intensive than addition, the asymptotic complexity is dominated by multiplication, giving $O(N^2)$.\nFast Fourier Transform Preliminaries We layout some properties of $W_N$ we\u0026rsquo;ll be using later\nProperty 1:\n$$ W_N^2 = W_{N/2} $$Proof:\n$$ W_N^2 = e^{-j\\frac{2\\pi}{N}\\cdot 2} = e^{-j\\frac{2\\pi}{N/2}} = W_{N/2} $$More generally, $W_N^{2nk} = W_{N/2}^{nk}$.\nProperty 2:\n$$ W_N^{k+\\frac{N}{2}} = -W_N^k $$Proof:\n$$ W_N^{k+\\frac{N}{2}} = e^{-j\\frac{2\\pi}{N}\\left(k+\\frac{N}{2}\\right)} = e^{-j\\frac{2\\pi}{N}k} \\cdot e^{-j\\pi} = -W_N^k $$Theory The radix-2 FFT algorithms work by dividing the DFT into 2 DFTs of length $N/2$ each, and iterating. We introduce the simplest variant, called the Decimation-In-Time (DIT) algorithm.\nConsider a $N$-point signal $x[n]$ of even length, indexed from $0$ to $N-1$. The derivation of the DIT radix-2 FFT begins by splitting the $x[n]$ into two parts \u0026mdash; one part for the even-indexed values $x[2n]$ and one part for the odd-indexed values $x[2n + 1]$. Define two $N/2$-point signals $x_{even}[n]$ and $x_{odd}[n]$ as $$x_{even}[n] = x[2n], \\quad x_{odd}[n] = x[2n + 1], \\quad 0 \\leq n \\leq N/2-1$$The DFT can be written as\n$$ \\begin{aligned} X[k] \u0026= \\sum_{n=0 \\atop n \\text{ even}}^{N-1} x[n] W_N^{nk} + \\sum_{n=0 \\atop n \\text{ odd}}^{N-1} x[n] W_N^{nk}\\\\ \u0026= \\sum_{n=0}^{N/2-1} x[2n] W_N^{2nk} + \\sum_{n=0}^{N/2-1} x[2n + 1] W_N^{(2n+1)k}\\\\ \u0026= \\sum_{n=0}^{N/2-1} x_{even}[n] W_N^{2nk} + \\sum_{n=0}^{N/2-1} x_{odd}[n] W_N^{(2n+1)k} \\\\ \u0026= \\sum_{n=0}^{N/2-1} x_{even}[n] W_N^{2nk} + W_N^{k} \\cdot \\sum_{n=0}^{N/2-1} x_{odd}[n] W_N^{2nk}\\\\ \\end{aligned} $$Noting that $W_N^{2}=W_{N/2}$ or more generally $W_N^{2nk}=W_{N/2}^{nk}$ , then\n$$ \\begin{aligned} X[k]\u0026= \\sum_{n=0}^{N/2-1} x_{even}[n] W_{N/2}^{nk} + W_N^{k} \\cdot \\sum_{n=0}^{N/2-1} x_{odd}[n] W_{N/2}^{nk} \\end{aligned} $$Recognizing that the $\\frac{N}{2}$-point DFT of $x_{even}[n]$ and $x_{odd}[n]$ are given by:\n$$X_{even}[k] = \\text{DFT}_{\\frac{N}{2}}\\{x_{even}[n]\\} = \\sum_{n=0}^{N/2-1} x_{even}[n] W_{N/2}^{nk}$$$$X_{odd}[k] = \\text{DFT}_{\\frac{N}{2}}\\{x_{odd}[n]\\} = \\sum_{n=0}^{N/2-1} x_{odd}[n] W_{N/2}^{nk}$$we then obtain our core recursive equation:\n$$ X[k] = X_{even}[k] + W_N^{k} \\cdot X_{odd}[k] \\label{eq:fft_int} $$Since $x_{even}[n]$ and $x_{odd}[n]$ are $N/2$-point signals with $0 \\leq n \\leq N/2-1$, their DFT are also only $N/2$-point signals. However, we require $0 \\le k \\le N-1$. We resolve this by noting their DFT coefficients are periodic with a period of $\\frac{N}{2}$:\n$$X_{even}[k] = X_{even}\\left[k + \\frac{N}{2}\\right], \\quad X_{odd}[k] = X_{odd}\\left[k + \\frac{N}{2}\\right]$$This gives us\n$$ X[k] = \\begin{cases} X_{even}[k] + W_N^{k} \\cdot X_{odd}[k], \u0026 \\text{for } k = 0,1,\\ldots,\\frac{N}{2}-1 \\\\ X_{even}\\left[k-\\frac{N}{2}\\right] + W_N^{k} \\cdot X_{odd}\\left[k-\\frac{N}{2}\\right], \u0026 \\text{for } k = \\frac{N}{2},\\frac{N}{2}+1,\\ldots,N-1 \\end{cases} $$Noting $W_N^{k +\\frac{N}{2}}=-W_{N}^{k}$, we write\n$$ X[k] = \\begin{cases} X_{even}[k] + W_N^{k} \\cdot X_{odd}[k], \u0026 \\text{for } k = 0,1,\\ldots,\\frac{N}{2}-1 \\\\ X_{even}\\left[k-\\frac{N}{2}\\right] - W_N^{k-\\frac{N}{2}} \\cdot X_{odd}\\left[k-\\frac{N}{2}\\right], \u0026 \\text{for } k = \\frac{N}{2},\\frac{N}{2}+1,\\ldots,N-1 \\end{cases} $$finally giving us\n$$ \\begin{aligned} X[k] \u0026= X_{even}[k] + W_N^{k} \\cdot X_{odd}[k] \\quad \\text{for } 0 \\leq k \\leq \\frac{N}{2} - 1 \\\\ X[k + N/2] \u0026= X_{even}[k] - W_N^{k} \\cdot X_{odd}[k] \\quad \\text{for } 0 \\leq k \\leq \\frac{N}{2} - 1 \\label{eq:fft_final} \\end{aligned} $$The multipliers $W_N^k$ are known as twiddle factors. The first computation with $+W_N^k$ give us the first half of the full DFT vector $\\mathbf{X}$, while the second computation with $-W_N^k$ give us the second half of $\\mathbf{X}$.\nComplexity Pseudocode for FFT MATLAB code for FFT:\nfunction X = fft_vectorized(x) % computes the radix-2 FFT recursively using vectorized operations. % If the length of x is not a power of 2, it is zero-padded to the next power of 2, added at the end x = x(:); % Ensure x is a column vector N = length(x); % If N is not a power of 2, zero-pad x to the next power of 2 M = 2^nextpow2(N); % nextpow2 returns the exponent so that 2^exponent \u0026gt;= N if M ~= N %add zeros to the end x = [x; zeros(M - N, 1)]; N = M; % Update N to the new length end % Base case: if the input length is 1, return x if N == 1 X = x; return; end % Recursively compute FFT for even and odd indices X_even = fft_vectorized(x(1:2:end)); %select all the even indices X_odd = fft_vectorized(x(2:2:end)); % selects all the odd indices % Compute twiddle factors (complex exponentials) in a vectorized manner % row array factor = exp(-1j * 2 * pi * (0:N/2-1).\u0026#39; / N); % Combine the FFTs of the even and odd parts using the butterfly operation X = [X_even + factor .* X_odd; X_even - factor .* X_odd]; end We can follow the above procedure to split an $N$-point DFT into two $\\frac{N}{2}$-point DFT, giving us the above algorithim. Let $A_c(N)$ and $M_c(N)$ denote respectively the number of complex additions and multiplications for computing the DFT of an $N$-point complex sequence $x[n]$. Let $N$ be a power of 2, $N = 2^k$. Then, we have $$A_c(N) = 2 A_c(N/2) + N \\quad , \\quad M_c(N) = 2 M_c(N/2) + \\frac{N}{2} - 1$$as $N$ complex additions (addition of even and odd terms) and $\\frac{N}{2} - 1$ complex multiplications ($W_N^{k} \\cdot X_{odd}[k]$) are required to put the two $N/2$-point DFTs together. Note that a 2-point DFT is simply a sum and difference\n$$X[0] = x[0] + x[1] , \\quad X[1] = x[0] - x[1]$$Hence, the starting conditions are $A_c(2) = 2$ and $M_c(2) = 0$. Solving the recursive equation yields $$A_c(N) = N \\log_2 N \\quad , \\quad M_c(N) = \\frac{N}{2} \\log_2 N - N + 1$$Our overall complexity is $O(N\\log N)$.\nExperiments Complexity Execution time versus signal length $N$ for various implementations of DFT and FFT Algorithm O(N) O(N²) O(N³) O(log N) O(N log N) my-DFT 0.9208 0.9999 0.9864 0.4167 0.9417 my-FFT 0.9830 0.9657 0.9161 0.5634 0.9894 MATLAB-FFT 0.8491 0.7141 0.6526 0.8537 0.8287 To verify the theoretical complexity, we measured the execution time for computing the DFT over a range of signal lengths. For each signal length $N$, $N$ samples were taken from a 5Hz sine wave, and the DFT/FFT computation time recorded. We compare our implementation of DFT my-DFT, our implementation of FFT as my-FFT, and the MATLAB FFT implementation as MATLAB-FFT. You can view the measured execution times above on a log-log plot. The FFT results clearly exhibit an $O(N\\log N)$ scaling, while the DFT scales as $O(N^2)$. These experimental results confirm the significant computational advantage of using the FFT for large-scale problems. We also note the much more efficient implementation of MATLAB FFT, which uses the FFTW 2 package. The table above also shows the how well different models fit to data. Both FFT implementations fit well to $O(N)$ and $O(N\\log N)$ models.\nNumerical Error Reconstruction error versus signal length $N$ for various implementations of DFT and FFT We evaluate numerical accuracy of above algorithims by measuring reconstruction error of the reconstructed signal $\\text{RMSE} = \\sqrt{\\frac{1}{N}\\sum_{i=1}^{N}|x_i - \\hat{x}_i|^2}$ where $x_i$ is the original signal and $\\hat{x}_i$ is the reconstructed signal (using the MATLAB Inverse DFT function) Despite all errors falling within the $10^{-12}$ range, indicating high overall accuracy, the custom DFT implementation exhibits exponentially growing error with increasing sequence length. In contrast, both FFT implementations maintain consistently minimal error across all tested sequence lengths. The superior numerical stability of FFT algorithms is possibly due to its lower operation count (lower algorithmic complexity). Floating-point rounding errors accumulate more significantly in DFT.\nConclusion We study the theoretical motivations behind FFT. We verify that the computational complexity of the direct DFT implementation scales as $O(N^2)$, whereas our FFT implementation achieved the expected $O(N \\log N)$ complexity, offering a significant improvement in efficiency.\nReferences Golub, G. H., \u0026amp; Van Loan, C. F. (1996). Matrix computations (3rd ed.). Johns Hopkins University Press.\nStrang, G. (2007). Computational Science and Engineering. Wellesley-Cambridge Press. https://epubs.siam.org/doi/abs/10.1137/1.9780961408817\nRamalingam, C.S. Introduction to the Fast-Fourier Transform (FFT) Algorithm. https://www.ee.iitm.ac.in/~csr/teaching/pg_dsp/lecnotes/fft.pdf\nFast Fourier Transform (FFT). NYU Engineering. https://eeweb.engineering.nyu.edu/iselesni/EL713/zoom/fft\nThis post was created for educational purposes, so much of the analysis is taken directly from the sources quoted in the references\u0026#160;\u0026#x21a9;\u0026#xfe0e;\nThis package is highly optimized to each machine and is written in low level C\u0026#160;\u0026#x21a9;\u0026#xfe0e;","href":"https://flowingpurplecrane.com/posts/dft-fft/","kind":"page","lang":"en","lastmod":"2025-03-28T17:05:08Z","objectID":"edf0899851b8b46beb0b136c155139c4","publishDate":"2025-03-28T17:05:08Z","section":"posts","tags":[],"title":"The Discrete Fourier Transform and Fast Fourier Transform","type":"posts"},{"content":"In this post, I want to explore the key RL architecture and software optimizations DeepSeek made to create Deepseek-R1. I\u0026rsquo;ll provide a brief introduction to the RL setup and explain how GRPO makes traditional algorithims much more efficient. At the end, I\u0026rsquo;ll provide a brief overview of how DeepSeek-R1 was trained. Talk given at Cambridge AI Safety Hub Members Meeting in February, slides available here.\nPreliminaries Transformers Illustration of autoregressive model of transformers, taken from the ARENA curriculum Modern transformers are autoregressive models. They predict the next token (essetially a word) given all the tokens before it:\n$$p(x_1,\\ldots,x_N) = \\prod_{n=1}^N p(x_n|x_1,\\ldots,x_{n-1})$$ Transformer architecture overview, taken from the ARENA curriculum At a high level, input natural language are broken up into tokens, and converted to a numerical form. The output is then a probability distribution over the next possible token (from logits), and the next token is sampled from this distribution.\nReinforcement Learning We provide a brief overview of the RL setup.\nAn agent in state $s$ issues an action $a$ to the environment, and the environment replies with state, reward pairs $(s',r)$. We can then define a trajectory $\\tau$ as a sequence of states and actions: $s_0, a_0, r_1, a_1, r_2, s_2, a_2, r_3...$\nThe agent chooses actions using a policy $\\pi$ which can either be a deterministic function $a=\\pi(s)$ from states to actions, or more generally actions are sampled $a \\sim \\pi(\\cdot|s)$\nThe return at time $t$ is the sum of rewards obtained after time $t$ until the time $T$ when the terminal state is reached, discounted by how far into the future:\n$$G_t = \\sum_{k=0}^{\\infty} \\gamma^k r_{t+k+1} = r_{t+1} + \\gamma r_{t+2} + \\gamma^2 r_{t+3} ...$$ The state value function $V_\\pi(s)$ is the expected return, if you start in state $s$ and always act according to policy $\\pi$:\n$$V_{\\pi}(s) = \\mathbb{E}_{\\pi}\\left[G_t| s_t = s\\right]$$ We deal with these core quantities:\nThe action value function is the expected return if you start in state $s$, take an arbitrary action $a$ (which may not have come from the policy), and then afterwards forever act according to policy $\\pi$: $$Q_\\pi(s, a) = \\mathbb{E}_{\\pi} \\left[G_t| s_t = s, a_t = a\\right]$$ The advantage function describes how much better it is to take a specific action $a$ in state $s$, over randomly selecting an action according to $\\pi(\\cdot|s)$, assuming you act according to $\\pi$ forever after:\n$$A_\\pi(s, a) = Q_\\pi(s, a) - V_\\pi(s)$$ Trust Region Policy Optimization (TRPO)\nLet $\\pi_\\theta$ denote a policy with parameters $\\theta$. We aim to maximize the expected return $J$ of policy $\\pi_\\theta$, where we take expectation over all possible trajectories $\\tau$ generated by following policy $\\pi_\\theta$\n$$J(\\pi_\\theta) = \\underset{\\tau \\sim \\pi_\\theta}{\\mathbb{E}}[G(\\tau)]$$To achieve the optimum $\\pi_\\theta$ that maximizes J, we can iteratively update $\\pi_{\\theta_{k}}$ according to:\n$$\\theta_{k+1} = \\arg \\max_\\theta \\mathcal{L}(\\theta_k, \\theta) = \\arg \\max_\\theta \\underset{s,a\\sim\\pi_{\\theta_k}}{\\mathbb{E}} \\left[L(s, a, \\theta_k, \\theta)\\right], \\quad \\text{s.t. } \\bar{D}_{KL}(\\theta\\|\\theta_k) \\leq \\delta$$where $\\mathcal{L}(\\theta_k, \\theta)$ is the surrogate advantage, a measure of how policy $\\pi_\\theta$ performs relative to the old policy $\\pi_{\\theta_k}$ using data from the old policy:\n$$L(s, a, \\theta_k, \\theta) = \\frac{\\pi_\\theta(a|s)}{\\pi_{\\theta_k}(a|s)} A_{\\pi_{\\theta_k}}(s,a)$$TRPO updates policies by taking the largest step possible to improve performance, while satisfying a special constraint on how close the new and old policies are allowed to be.\nProximal Policy Optimization (PPO)\nProximal Policy Optimization updates policies via:\n$$\\theta_{k+1} = \\arg \\max_\\theta \\mathcal{L}(\\theta_k, \\theta) = \\arg \\max_\\theta \\underset{s,a\\sim\\pi_{\\theta_k}}{\\mathbb{E}} \\left[L(s, a, \\theta_k, \\theta)\\right]$$typically taking multiple steps of SGD to maximize the objective. Here $L$ is given by:\n$$L(s, a, \\theta_k, \\theta) = \\min\\Bigg(\\frac{\\pi_\\theta(a|s)}{\\pi_{\\theta_k}(a|s)}A_{\\pi_{\\theta_k}}(s,a), \\text{clip}\\left(\\frac{\\pi_\\theta(a|s)}{\\pi_{\\theta_k}(a|s)}, 1-\\epsilon, 1+\\epsilon\\right)A_{\\pi_{\\theta_k}}(s,a)\\Bigg)$$PPO-Clip doesn\u0026rsquo;t have a KL-divergence term in the objective and doesn\u0026rsquo;t have a constraint at all. Instead relies on specialized clipping in the objective function to remove incentives for the new policy to get far from the old policy.\nGeneralized Advantage Estimation (GAE) is used in PPO to estimate the advantage function:\n$$A_t = \\sum_{l=0}^{\\infty} (\\gamma \\lambda)^l \\delta_{t+l} \\\\ = \\delta_t + (\\gamma \\lambda) \\delta_{t+1} + (\\gamma \\lambda)^2 \\delta_{t+2} + \\dots$$where:\n$\\delta_t = r_t + \\gamma V(s_{t+1}) - V(s_t)$ (TD error)\nthe reward $r_t$ is usually given by a trained (neural) reward model $\\gamma$ is the discount factor\n$\\lambda$ is the GAE parameter (tradeoff between bias and variance)\n$V(s)$ is the value function estimate, from value network\nOften a value network $V_\\psi$ needs to be trained alongside the policy model $\\pi_\\theta$ to estimate the value, often as big as the policy model itself General RL Training Setup Objective: $$J(\\pi_\\theta) = \\mathbb{E}_{\\tau \\sim \\pi_\\theta}\\left[\\sum_{t=0}^{T} \\gamma^t r_t\\right]$$ where $\\tau = \\{s_0, a_0, r_1, s_1, a_1, \\dots, s_T\\}$ is a trajectory. For LLMs, actions $a_t$ here are tokens!\nPolicy Gradient: $$\\nabla_\\theta J(\\pi_\\theta) = \\mathbb{E}_{s,a \\sim \\pi_\\theta}\\left[\\nabla_\\theta \\log \\pi_\\theta(a|s) \\, Q^{\\pi_\\theta}(s,a)\\right]$$ or equivalently, using the advantage function $A^{\\pi_\\theta}(s,a)$: $$\\nabla_\\theta J(\\pi_\\theta) = \\mathbb{E}_{s,a \\sim \\pi_\\theta}\\left[\\nabla_\\theta \\log \\pi_\\theta(a|s) \\, A^{\\pi_\\theta}(s,a)\\right]$$Trajectory Collection: LLM interacts with the environment (reward model) to sample trajectories. For each timestep: $$s_t \\xrightarrow{\\pi_\\theta} a_t \\rightarrow r_{t+1},\\, s_{t+1}$$Return and Advantage Calculation: $$G_t = \\sum_{k=0}^{\\infty} \\gamma^k r_{t+k+1}, \\quad A(s_t,a_t) = Q^{\\pi_\\theta}(s_t,a_t) - V^{\\pi_\\theta}(s_t)$$ Generalized Advantage Estimation (GAE) are used to compute $A(s_t,a_t)$ practically\nPolicy Update: $$\\theta_{k+1} = \\theta_k + \\alpha\\, \\nabla_\\theta J(\\pi_\\theta)$$Using the computed advantages, update the policy parameters and value network by maximizing a surrogate objective (e.g., the PPO objective) to favor actions with higher expected returns. Repeat until agent\u0026rsquo;s performance converges.\nR1-Zero Group Relative Policy Optimization (GRPO) DeepSeek uses GRPO (created in-house) which foregoes the value network and estimates the advantages directly from group rewards instead.\nFor each question $q$, GRPO samples a group of outputs $\\{o_i\\}_{i=1}^G$ from the old policy $\\pi_{\\theta_{\\text{old}}}$ and then optimizes the policy model $\\pi_\\theta$ by maximizing:\n$$\\begin{aligned} \\mathcal{J}_{\\text{GRPO}}(\\theta) \u0026= \\mathbb{E}[q \\sim P(Q), \\{o_i\\}_{i=1}^G \\sim \\pi_{\\theta_{\\text{old}}}(Q|q)] \\\\ \u0026\\quad \\cdot \\frac{1}{G}\\sum_{i=1}^G\\left(\\min\\left(\\frac{\\pi_\\theta(o_i|q)}{\\pi_{\\theta_{\\text{old}}}(o_i|q)}A_i, \\text{clip}\\left(\\frac{\\pi_\\theta(o_i|q)}{\\pi_{\\theta_{\\text{old}}}(o_i|q)}, 1-\\epsilon, 1+\\epsilon\\right)A_i\\right) - \\beta\\mathbb{D}_{KL}(\\pi_\\theta\\|\\pi_{\\text{ref}})\\right) \\end{aligned}$$$$\\mathbb{D}_{KL}(\\pi_\\theta\\|\\pi_{\\text{ref}}) = \\frac{\\pi_{\\text{ref}}(o_i|q)}{\\pi_\\theta(o_i|q)} - \\log\\frac{\\pi_{\\text{ref}}(o_i|q)}{\\pi_\\theta(o_i|q)} - 1$$where $\\epsilon$ and $\\beta$ are hyper-parameters, and $A_i$ is the advantage 1\nGRPO Advantages\nAdvantages $A_i$ are computed using group rewards $\\{r_1,r_2,\\ldots,r_G\\}$:\n$$A_i = \\frac{r_i - \\text{mean}(\\{r_1,r_2,\\ldots,r_G\\})}{\\text{std}(\\{r_1,r_2,\\ldots,r_G\\})} $$As the value network is typically another model of comparable size as the policy model, it brings a substantial memory and computational burden. GRPO eliminates value network completely, saving enourmous amounts of compute!\nReward Modelling\nR1-Zero uses a rule-based reward system consisting of:\nAccuracy rewards: The accuracy reward model evaluates whether the response is correct. For math problems with deterministic results, can easily check if right or wrong. Provider higher reward if right. For LeetCode problems, a compiler can be used to generate feedback based on predefined test cases Format rewards: Enforces the model to put its thinking process between \u0026lt;think\u0026gt; and \u0026lt;/think\u0026gt; tags. Usually for LLMs, a neural reward model (neural network used to estimate rewards) is used to provide rewards, as human preferences or requirements are difficult to define in a rules-based way. However, R1 does not use any neural reward model:\nNeural reward model may suffer from reward hacking\nRetraining the reward model needs additional training resources and it complicates the whole training pipeline\nPerformance\nThere\u0026rsquo;s no supervised fine tuning involved at all in R1-Zero, which is extremely impressive considering its performance. Majority voting bring performance on AIME benchmark from 71% to 86%.\nR1-Zero naturally acquires the ability to solve increasingly complex reasoning tasks by leveraging extended test-time computation (increased Chain-Of-Thought length). Behaviours like backtracking and reflection naturally emerge, which is absolutely incredible\u0026hellip;\nProblems\nR1-Zero exhibits poor readability of its chain of thoughts, where language mixing often occurs. I suspect this is probably some form of reward over-optimization, as it is more efficient to reason in its own language and achieve higher reward.\nWe know different languages have different information density. This can be thought of as the \u0026ldquo;efficiency\u0026rdquo; of language, given a constraint on its length. Suppose an LLM had a length constraint (the context window), and was incentivised to achieve a goal. Won\u0026rsquo;t you expect the LLM to naturally use more information dense languages? Or even more extreme - if it were intelligent enough - create its own language that is information dense enough to achieve its goals?\nI worry about this aspect quite abit, about future intelligent systems reasoning in its own language uninterpretable to humans. If you\u0026rsquo;ve ever seen Villeneuve\u0026rsquo;s incredible film Arrival, you would have seen the heptapods\u0026rsquo; non-linear language which isn\u0026rsquo;t formed by sequentially combining words linearly. If these systems start communicating in their own language and are given goals by humans, you can see how this could cause all sorts of various problems.\nR1 Training Pipeline Phase 1: Cold Start Construct and collect small amounts (thousands) of long CoT data to fine-tune V3 Using few-shot prompting on V3 with long CoT examples Direct prompting on V3 for detailed answers with reflection Gathering DeepSeek-R1-Zero outputs with human post-processing Key advantages: Readability: R1-Zero has unreadable responses. We define output format as |special_token|\u0026lt;reasoning_process\u0026gt;|special_token|\u0026lt;summary\u0026gt; so reasoning process (CoT) and summary of reasoning provided Potential: Carefully designed patterns with human priors show better performance vs R1-Zero Phase 2: Reasoning Reinforcement Learning After cold start fine-tuning, apply large-scale RL training similar to DeepSeek-R1-Zero:\nFocus on reasoning-intensive tasks: Coding, mathematics, science, logic reasoning Well-defined problems with clear solutions Language Consistency: CoT exhibits language mixing in multi-language prompts Introduce language consistency reward, measured as proportion of target language words in CoT Final reward function combines reasoning task accuracy, language consistency reward, and formatting reward to train until convergence.\nPhase 3: Rejection Sampling and Supervised Fine-Tuning After RL convergence, collect SFT data from checkpoint for further training. Data includes:\nReasoning Data (600k samples)\nRejection sampling from RL checkpoint Expanded dataset includes: Rule-based rewards Generative rewards via DeepSeek-V3 as LLM-Judge Quality filters: Remove mixed languages Filter long paragraphs Remove chaotic code blocks Non-Reasoning Data (200k samples)\nTypes: Writing Factual QA Self-cognition Translation Reuse SFT data for DeepSeek-V3 Final fine-tuning on DeepSeek-V3-Base: 2 epochs on combined 800k samples\nPhase 4: Diverse Reinforcement Learning Phase Secondary RL stage to align with human preferences while improving, helpfulnes, harmlessness, and reasoning:\nTraining Approach\nReasoning data: Same approach as DeepSeek-R1-Zero Rule-based rewards on Math, code, logical reasoning General data: Reward model as in DeepSeek V3 Use preference pairs Helpful Honest Harmless (HHH) Criteria\nHelpfulness: Focus on final summary in summary tag is useful and relevance Preserves reasoning process in the reasoning tag Harmlessness: Evaluate full response for potential biases or hamful content R1 Engineering Unlocks The DeepSeek team also did some insane software and hardware optimizations:\nMulti-Head Latent Attention (for efficient inference) Low-rank join compression for K,Q,V matrices to reduce key-value cache memory usage DeepSeekMoE with Auxiliary-Loss-Free Load Balancing (efficient training) 671B parameters, but only which 37B are activated for each token Multi-Token Prediction Objective Conclusion I think the core reason behind R1\u0026rsquo;s success was its sheer efficiency: getting rid of the value network via simple advantage estimation, using a simple reward model instead of a neural reward model, and all the hardware/software optimizations. The multi-phase training approach, while quite hacky, preserves the reasoning ability of R1-Zero while ensuring readability. All in all I\u0026rsquo;m very impressed by the DeepSeek team, and I hope you learnt something from this post!\nReferences Shao, Z., Wang, P., Zhu, Q., Xu, R., Song, J., Bi, X., Zhang, H., Zhang, M., Li, Y. K., Wu, Y., \u0026amp; Guo, D. (2024). DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models. arXiv:2402.03300 [cs.CL]. https://arxiv.org/abs/2402.03300\nDeepSeek-AI, Guo, D., Yang, D., Zhang, H., Song, J., Zhang, R., Xu, R., Zhu, Q., Ma, S., Wang, P., et al. (2025). DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning. arXiv:2501.12948 [cs.CL]. https://arxiv.org/abs/2501.12948\nDeepSeek-AI, Liu, A., Feng, B., Xue, B., Wang, B., Wu, B., Lu, C., Zhao, C., Deng, C., Zhang, C., et al. (2025). DeepSeek-V3 Technical Report. arXiv:2412.19437 [cs.CL]. https://arxiv.org/abs/2412.19437\nARENA curriculum. Transformer from Scratch. https://arena-chapter1-transformer-interp.streamlit.app/[1.1]_Transformer_from_Scratch\nInstead of adding KL penalty in the reward like in PPO, GRPO regularizes by directly adding the KL divergence between the trained policy and the reference policy to the loss, avoiding complicating the calculation of advantage $A$\u0026#160;\u0026#x21a9;\u0026#xfe0e;","href":"https://flowingpurplecrane.com/posts/r1/","kind":"page","lang":"en","lastmod":"2025-02-17T17:05:08Z","objectID":"aea9ae4ea09924bdadd709902f9af108","publishDate":"2025-02-17T17:05:08Z","section":"posts","tags":[],"title":"Breaking Down DeepSeek R1","type":"posts"},{"content":"We explain how Monte Carlo Tree Search works and introduce its pseudocode. We borrow notes and figures from int8\u0026rsquo;s website and the MCTS Wikipedia page.\nGame Representation A game tree is a tree data structure, where every node represents a distinct state of the game. Given a state $s$, we take an action $a$, and the environment transitions to the new state $s'$ with the transition probability $p_a(s'|s)$. If the transition probablity is $1$ (such as in simple deterministic games like Tic Tac Toe), then each node in the game tree represents a state.\nWe use \u0026ldquo;action\u0026rdquo; and \u0026ldquo;move\u0026rdquo; interchangeably in the context of games\nThe root of the game tree represents the initial state of the game Any transition from one state to another consists a move The game ends at a terminal node Monte Carlo Tree Search Taken from CadiaPlayer Our game tree contains various connected game states, with each node in the tree containing statistics used to compute the value of a particular state. During \u0026ldquo;training\u0026rdquo;, MCTS traverses down the game tree from the root node $R$ until a leaf node $L$ is reached, then expands the game tree by adding a child node $C$.\nAfter expanding the game tree, we simulate the game from the child node until game termination at $T$ - a rollout. The results of the rollout is then used to update the game tree, so that we can choose better nodes to do rollouts from next time.\nThis way we make the game tree contain better estimates of the values of each states, which allows us to make better actions when we actually play the game. This occurs through the 4 stages: selection, expansion, simulation, and backpropagation.\nSelection The bold circles contain the nodes selected using the tree policy UCT. The color of the node represent whose turn it is to move next. The numbers in the nodes represent the statistics of the node. For example, a black circle with $7/10$ means black to move next, $7$ wins played from this state, total of $10$ visits at this state. Hence if there are no draws, white would have won $3$ times from this node.\nThe first phase, selection, works by starting at the root $R$ (initial state of game) and traversing down the game tree using the tree policy until a leaf node $L$. The tree policy used here is Upper Confidence Trees (UCT), which is UCB1 applied to trees:\n$$ a = \\arg\\max_{a \\in A(s)} \\left\\{ Q(s,a) + C_p\\sqrt{\\frac{\\ln N(s)}{N(s,a)}} \\right\\} $$where $N(s)$ is the number of times a node has been visited, $N(s,a)$ is the number of times $a$ has been selected from this node, and $C_p$ is a exploration constant. Increasing $C_p$ will encourage more exploration, while decreasing it encourages a more greedy approach, but $\\sqrt2$ is often chosen.\nIf the transition probability is 1 - taking an action will deterministically move the state to another - then UCT simplifies to\n$$ UCT(v) = \\arg\\max_{i} \\left\\{ \\frac{w(v_i)}{N(v_i)} + C_p \\sqrt{\\frac{\\ln N(v) }{N(v_i)}} \\right\\} $$for a parent node $v$, child node $v_i$, and number of wins $w$ from this child node $v_i$. UCT It is a sum of two components – the first component of our function $ \\frac{w(v_i)}{N(v_i)} $, also called exploitation component, can be read as a win rate. The term on the right is the exploration component - encouraging child nodes that have little visits. During selection, we want to allow some exploration, hence $C_p\u003e0$ here.\nA leaf node $L$ here is a node in the game tree, where not all actions have been explored. In other words, it still has unexplored game states (if one action transitions to a state deterministically). Once a leaf node has no more unexplored actions, it is fully expanded.\nExpansion $3/3$ still has unexplored actions, hence we add a child node to it. This action then becomes \u0026ldquo;explored\u0026rdquo;\nThe second phase expansion occurs when we\u0026rsquo;ve selected a leaf node $L$ to expand on. An unexplored node is randomly chosen, and is added to the game tree as a child node.\nSimulation Once the new child is added to the tree, we do one rollout from the $0/0$ until game termination\nWe do one simulation (rollout) from the new child node $C$ until game termination, where moves are chosen according to the rollout policy. This rollout policy is often completely random.\nBackpropagation We can see only black nodes have their win count updated.\nAfter the simulation reaches an end, all of the nodes taken in this path are updated. All of the nodes number of visits are incremented by one each, and the colour that matches the winner will have its win count incremented by 1. Hence useful statistics to keep are the number of visits and win count for that player.\nIn this repo, we store the win counts of both players in each node. A node also stores the board state, which contains which player it is to move next\nPseudoCode def select(node): while node is not terminal: if node is not fully expanded (has unexplored actions): return node else: node = UCT(node) return node def expand(node): assert unexplored_actions\u0026gt;0 action = unexplored_actions.pop() add this action as a child (to the game tree) return child def simulate(node): while state is not terminal choose an action (rollout policy) move the state return the result of game def backpropagate(node,result): update this node if parent exists: backpropagate(parent,result) def UCT(node,c): choices = (child.value/child.visits)+ c*np.sqrt((np.log(parent.visits)/child.visits)) return node.children[argmax(choices)] def train(root): leaf=select(root) if leaf has unexplored actions: child=expand(leaf) reward=simulate(child) backpropagate(child,reward) else: reward=simulate(leaf) backpropagate(leaf, reward)","href":"https://flowingpurplecrane.com/posts/mcts/","kind":"page","lang":"en","lastmod":"2024-12-07T17:05:08Z","objectID":"4866ff99ed4a5a615c61a11335a5d2e8","publishDate":"2024-12-07T17:05:08Z","section":"posts","tags":[],"title":"Breaking Down Monte Carlo Tree Search","type":"posts"}]