Posts

I chanced upon a pretty interesting idea proposed by Peter Hayes (now at Anthropic) in 2025, to more efficiently use feedback data in the standard RLHF pipeline to speed up the whole thing. Basically, given external feedback/critique/verification of generated answers, can we incorporate this feedback in an NLL term in addition to the standard DPO loss; trained on the base model? Most previous efforts either used this feedback to train a better verifier, or used other additional NLL terms in the base model training.

I recently learnt a drinking game where you had 2 cards (red/black) for round, and people use these as answers to questions. The goal of the game is to locate who the minority answers came from, and with that came some interesting questions. Game Rules $n$ players each hold a red card and a black card. A question is posed each round (e.g., yes/no, would-you-rather, etc.). Each player secretly plays one card to reflect their answer.

I’ve wondered about entropy for a long time. From thermodynamics and the arrow of time, to cross-entropy loss in ML cost functions, or KL divergence in stopping intelligence from going batshit crazy in VAE/PPO, entropy keeps cropping up. Maybe the universe is playing some kinda trick on us here. Since I’m taking 3F7 Information Theory, I thought it’s probably a good time to understand what these ideas really mean. Disclaimer: I do not claim this work to be my own; this is merely a synthesis of intuition and math across multiple resources for myself and anyone interested.

We analyze 1 the theoretical properties of both the Discrete Fourier Transform (DFT) and the optimized Fast Fourier Transform (FFT), and estimate their algorithmic complexity. All code available on Github. Slides available here. Discrete Fourier Transform Theory The DFT converts a finite-length time-domain signal into its frequency-domain representation. For an input signal $x[n]$ of length $N$, where $0 \le n \le N-1$, the DFT is defined as: $$ X[k] = \sum_{n=0}^{N-1} x[n]\, W_N^{nk}, \quad 0 \le k \le N-1 \label{eq:DFT} $$where $W_N=e^{-j\frac{2\pi}{N}}$ is the $N$-th principal root of unity, and $X[k]$ is component at frequency $kf_s/N$ with sampling frequency $f_s$.

In this post, I want to explore the key RL architecture and software optimizations DeepSeek made to create Deepseek-R1. I’ll provide a brief introduction to the RL setup and explain how GRPO makes traditional algorithims much more efficient. At the end, I’ll provide a brief overview of how DeepSeek-R1 was trained. Talk given at Cambridge AI Safety Hub Members Meeting in February, slides available here. Preliminaries Transformers Illustration of autoregressive model of transformers, taken from the ARENA curriculum Modern transformers are autoregressive models.

We explain how Monte Carlo Tree Search works and introduce its pseudocode. We borrow notes and figures from int8’s website and the MCTS Wikipedia page. Game Representation A game tree is a tree data structure, where every node represents a distinct state of the game. Given a state $s$, we take an action $a$, and the environment transitions to the new state $s'$ with the transition probability $p_a(s'|s)$. If the transition probablity is $1$ (such as in simple deterministic games like Tic Tac Toe), then each node in the game tree represents a state.