Posts on Zihe Liu

Critique DPO

zl559@cam.ac.uk (Zihe Liu) — Sat, 14 Mar 2026 15:42:57 +0000

I chanced upon a pretty interesting idea proposed by Peter Hayes (now at Anthropic) in 2025, to more efficiently use feedback data in the standard RLHF pipeline to speed up the whole thing. Basically, given external feedback/critique/verification of generated answers, can we incorporate this feedback in an NLL term in addition to the standard DPO loss; trained on the base model? Most previous efforts either used this feedback to train a better verifier, or used other additional NLL terms in the base model training. I spent couple of months reading around the literature, setting up the HPC infra, and running some experiments. Unfortunately experiments (at least on smaller models) resulted in negative findings - the model was extremely prone to reward hacking…

The Minority Guessing Game

zl559@cam.ac.uk (Zihe Liu) — Sat, 14 Feb 2026 23:50:56 +0000

I recently learnt a drinking game where you had 2 cards (red/black) for round, and people use these as answers to questions. The goal of the game is to locate who the minority answers came from, and with that came some interesting questions.

Game Rules

$n$ players each hold a red card and a black card.
A question is posed each round (e.g., yes/no, would-you-rather, etc.). Each player secretly plays one card to reflect their answer.
All played cards are collected and shuffled — the total counts of each red and black are revealed, but not who played what.
This reveals a minority group of size $k$ and a majority of size $n − k$
The group is given $g$ guesses to identify members of the minority. The group’s score is $C$, the number of correct guesses. This is the only random variable in this question; everything else is given.

Then I had a question: what’s a good number for $g$ so that the game is fun? ¹ So not too easy $g \approx n$ where we’ll definitely guess who the minorities are, but not too hard $g=k$ ² that we’ll never find the minorities. ³

On Information

zl559@cam.ac.uk (Zihe Liu) — Sat, 25 Oct 2025 16:34:06 +0000

I’ve wondered about entropy for a long time. From thermodynamics and the arrow of time, to cross-entropy loss in ML cost functions, or KL divergence in stopping intelligence from going batshit crazy in VAE/PPO, entropy keeps cropping up. Maybe the universe is playing some kinda trick on us here. Since I’m taking 3F7 Information Theory, I thought it’s probably a good time to understand what these ideas really mean.

Disclaimer: I do not claim this work to be my own; this is merely a synthesis of intuition and math across multiple resources for myself and anyone interested. In particular, we borrow the beautiful figures and analogies from Chris Olah’s amazing post¹ on Information Theory.

The Discrete Fourier Transform and Fast Fourier Transform

zl559@cam.ac.uk (Zihe Liu) — Fri, 28 Mar 2025 17:05:08 +0000

We analyze ¹ the theoretical properties of both the Discrete Fourier Transform (DFT) and the optimized Fast Fourier Transform (FFT), and estimate their algorithmic complexity.

All code available on Github. Slides available here.

Discrete Fourier Transform

Theory

The DFT converts a finite-length time-domain signal into its frequency-domain representation. For an input signal $x[n]$ of length $N$, where $0 \le n \le N-1$, the DFT is defined as:

$$ X[k] = \sum_{n=0}^{N-1} x[n]\, W_N^{nk}, \quad 0 \le k \le N-1 \label{eq:DFT} $$

where $W_N=e^{-j\frac{2\pi}{N}}$ is the $N$-th principal root of unity, and $X[k]$ is component at frequency $kf_s/N$ with sampling frequency $f_s$. We can rewrite $X[k]$ as an inner product:

Breaking Down DeepSeek R1

zl559@cam.ac.uk (Zihe Liu) — Mon, 17 Feb 2025 17:05:08 +0000

In this post, I want to explore the key RL architecture and software optimizations DeepSeek made to create Deepseek-R1. I’ll provide a brief introduction to the RL setup and explain how GRPO makes traditional algorithims much more efficient. At the end, I’ll provide a brief overview of how DeepSeek-R1 was trained. Talk given at Cambridge AI Safety Hub Members Meeting in February, slides available here.

Preliminaries

Transformers

Illustration of autoregressive model of transformers, taken from the ARENA curriculum

Modern transformers are autoregressive models. They predict the next token (essetially a word) given all the tokens before it:

Breaking Down Monte Carlo Tree Search

zl559@cam.ac.uk (Zihe Liu) — Sat, 07 Dec 2024 17:05:08 +0000

We explain how Monte Carlo Tree Search works and introduce its pseudocode. We borrow notes and figures from int8’s website and the MCTS Wikipedia page.

Game Representation

A game tree is a tree data structure, where every node represents a distinct state of the game. Given a state $s$, we take an action $a$, and the environment transitions to the new state $s'$ with the transition probability $p_a(s'|s)$. If the transition probablity is $1$ (such as in simple deterministic games like Tic Tac Toe), then each node in the game tree represents a state.