Home on Zihe Liu

Critique DPO

zl559@cam.ac.uk (Zihe Liu) — Sat, 14 Mar 2026 15:42:57 +0000

I chanced upon a pretty interesting idea proposed by Peter Hayes (now at Anthropic) in 2025, to more efficiently use feedback data in the standard RLHF pipeline to speed up the whole thing. Basically, given external feedback/critique/verification of generated answers, can we incorporate this feedback in an NLL term in addition to the standard DPO loss; trained on the base model? Most previous efforts either used this feedback to train a better verifier, or used other additional NLL terms in the base model training. I spent couple of months reading around the literature, setting up the HPC infra, and running some experiments. Unfortunately experiments (at least on smaller models) resulted in negative findings - the model was extremely prone to reward hacking…

The Minority Guessing Game

zl559@cam.ac.uk (Zihe Liu) — Sat, 14 Feb 2026 23:50:56 +0000

I recently learnt a drinking game where you had 2 cards (red/black) for round, and people use these as answers to questions. The goal of the game is to locate who the minority answers came from, and with that came some interesting questions.

Game Rules

$n$ players each hold a red card and a black card.
A question is posed each round (e.g., yes/no, would-you-rather, etc.). Each player secretly plays one card to reflect their answer.
All played cards are collected and shuffled — the total counts of each red and black are revealed, but not who played what.
This reveals a minority group of size $k$ and a majority of size $n − k$
The group is given $g$ guesses to identify members of the minority. The group’s score is $C$, the number of correct guesses. This is the only random variable in this question; everything else is given.

Then I had a question: what’s a good number for $g$ so that the game is fun? ¹ So not too easy $g \approx n$ where we’ll definitely guess who the minorities are, but not too hard $g=k$ ² that we’ll never find the minorities. ³

Transition

zl559@cam.ac.uk (Zihe Liu) — Wed, 31 Dec 2025 00:05:50 +0000

He could not see the ground. No horizon visible other than that of metal against sky, grey against brown. There was scarcely any motion to be seen - save for a few pleasure-crafts lazed against the sky - but all the busy traffic of billions of men were going on, he knew, beneath the metal skin of the world.

Modified from Foundation by Isaac Asimov

I’m on a plane right now with this particular book in hand. To be honest I didn’t finish it because it got too boring and confusing, but I tried :p Actually, the flight destination is back to my hometown 郑州 in China, and I told none of my relatives I was coming. So 姥姥，老爷 and 金刚 will be in for a big surprise. I’ve never surprised someone like this before, so I’m quite excited to see how they’ll react. Hopefully 姥姥 doesn’t get too shocked.

A Box

zl559@cam.ac.uk (Zihe Liu) — Wed, 19 Nov 2025 20:25:51 +0000

Dear Pat,

You came upon me carving some kind of little figure out of wood, and you said, “Why don’t you make something for me?”
I asked you what you wanted, and you said, “A box.”
“What for?”
“To put things in.”
“What things?”
“Whatever you have,” you said.
Well, here’s your box. Nearly everything I have is in it, and it is not full. Pain and excitement are in it, and feeling good or bad and evil thoughts and good thoughts—the pleasure of design and some despair and the indescribable joy of creation.
And on top of these are all the gratitude and love I have for you.
And still the box is not full.

John

Golden

zl559@cam.ac.uk (Zihe Liu) — Fri, 31 Oct 2025 04:05:50 +0000

golden

Kpop demon hunters.

A good boy.

Golden. Retriever. Another night… with the Singaporeans yipeeeeee :p

Fasho

zl559@cam.ac.uk (Zihe Liu) — Thu, 30 Oct 2025 03:20:06 +0000

five bucks
Bezos?? >:( come back plsssss
sorry my granny
squad is on the way
skrrt

Cortis

Cortis.

I may be slightly tipsy while writing this. I got kicked out for dancing too hard to this song. In my defence, I actually learnt the choreography and was on beat. I love my friends though. I recently realised maybe my family and friends really mean the world to me. Its kinda my last (?) year. I’ll miss yall.

On Information

zl559@cam.ac.uk (Zihe Liu) — Sat, 25 Oct 2025 16:34:06 +0000

I’ve wondered about entropy for a long time. From thermodynamics and the arrow of time, to cross-entropy loss in ML cost functions, or KL divergence in stopping intelligence from going batshit crazy in VAE/PPO, entropy keeps cropping up. Maybe the universe is playing some kinda trick on us here. Since I’m taking 3F7 Information Theory, I thought it’s probably a good time to understand what these ideas really mean.

Disclaimer: I do not claim this work to be my own; this is merely a synthesis of intuition and math across multiple resources for myself and anyone interested. In particular, we borrow the beautiful figures and analogies from Chris Olah’s amazing post¹ on Information Theory.

如燕

zl559@cam.ac.uk (Zihe Liu) — Thu, 16 Oct 2025 03:51:34 +0000

A swallow, generated by GPT.

思念如燕它飞舞舌尖
若是真爱配尝几分苦甜
意念婆娑时间里推磨
追随到何处才结果
燕如针线在青空缝编
几幅女红将以泪缀点
誓言斑驳情雾只是经过
风雨中且让我盈步婀娜

Olivia Ong/王俪婷

Recently I did a pretty difficult thing. I put in alot of effort into it, and it paid off. Sure, it came with a couple of trade-offs, but it was well worth it in comparison. I tried really hard to get better at this one other thing (dance), but fell short of my own goals. So I’m a bit at a loss of how to advance on this front, at least for this year. You could say I’m feeling slightly empty, but some people would probably say I’m just being entitled, and I should be grateful for what I already have (rightly so).

Grateful

zl559@cam.ac.uk (Zihe Liu) — Tue, 22 Jul 2025 23:44:25 +0000

The Fallen Angel by Alexandre Cabanel (Wikipedia)

Excerpt from Paradise Lost, by John Milton

What though the field be lost?
All is not lost; the unconquerable Will,
And study of revenge, immortal hate,
And courage never to submit or yield:
And what is else not to be overcome?

I didn’t read Paradise Lost, but I found this painting of Lucifer that was inspired by Paradise Lost somewhat soul-stirring. We see Lucifer’s intense gaze as he looks (at the viewer? slightly off centre? maybe is he angry at us?) ahead, a mixture of angst, betrayal, remorse, anger, and other emotions up to viewer interpretation. I love how the darkness of the side of his face and his right eyebrow contrasts with that blinding gaze, accentuated by small white highlights. I could almost feel what he must have felt getting cast out from heaven.

Always The End

zl559@cam.ac.uk (Zihe Liu) — Sat, 03 May 2025 23:32:42 +0000

Composed and written by Jhung

It’s always the end when I see things different
Always the end when I see you
OK last take, last take
2,3
Then you come in like this
La-la-la, la-da-da
Please!
Don’t you forget about me, no
Cause I can’t let this go
Don’t forget to let me know

Nowadays, a significant portion of my time and energy is channelled to dancing. One thought occurs fairly often to me recently, as I start to see frontier AI models get increasingly intertwined within the lives of the people around me. Perhaps my dance is the very last thing that I can truly call my own. Whatever piece of good quality work I produce nowdays - from supervision answers, code, hell even writing some of these posts - are informed by AI in some way. Especially since my background is in everything AI. But dance is different. I cannot foresee AI generating choreographies anytime soon (certainly not executing them). When I dance, I really do feel the rhythm within me, and getting all the moves on the right beats feels like riding a wave. How I use my body to express the sounds I hear are unique to me. I’m still not sure yet how I feel about this, but I genuinely think my dance is the last thing that’ll be taken away by AI.

The Dog

zl559@cam.ac.uk (Zihe Liu) — Thu, 03 Apr 2025 15:00:10 +0000

I like a dog at my feet when I read,
Whatever his size or whatever his breed.
A dog now and then that will nuzzle my hand
As though I were the greatest of men in the land,
And trying to tell me it’s pleasant to be
On such intimate terms with a fellow like me.

I like a dog at my side when I eat,
I like to give him a bit of my meat;
And though mother objects and insists it is bad
To let dogs in the dining room, still I am glad
To behold him stretched out on the floor by my chair.
It’s cheering to see such a faithful friend there.

A dog leads a curious life at the best.
By the wag of his tail is his pleasure expressed.
He pays a high tribute to man when he stays
True to his friend to the end of his days.
And I wonder sometimes if it happens to be
That dogs pay no heed of the faults which men see.

Should I prove a failure; should I stoop to wrong;
Be weak at a time when I should have been strong;
Should I lose my money, the gossips would sneer
And fill with my blundering many an ear,
But still, as I opened my door, I should see
My dog wag his tail with a welcome for me.

Edgar A. Guest

A Dream Is a Wish Your Heart Makes

zl559@cam.ac.uk (Zihe Liu) — Sat, 29 Mar 2025 23:56:58 +0200

Written and composed by Mack David, Al Hoffman and Jerry Livingston

A dream is a wish your heart makes
When you’re fast asleep
In dreams, you will lose your heartaches
Whatever you wish for, you keep

Have faith in your dreams and maybe someday
Rainbow will come shining through
No matter how your heart is grieving
If you keep on believing
Then the dream that you wish will come true

The original version of this was written for the Walt Disney film Cinderella (1950). When I first listened to the rendition by British artist Puma Blue, I was completely captured by its darker tones and soft vocals. I really did feel like I was in a dream.

The Discrete Fourier Transform and Fast Fourier Transform

zl559@cam.ac.uk (Zihe Liu) — Fri, 28 Mar 2025 17:05:08 +0000

We analyze ¹ the theoretical properties of both the Discrete Fourier Transform (DFT) and the optimized Fast Fourier Transform (FFT), and estimate their algorithmic complexity.

All code available on Github. Slides available here.

Discrete Fourier Transform

Theory

The DFT converts a finite-length time-domain signal into its frequency-domain representation. For an input signal $x[n]$ of length $N$, where $0 \le n \le N-1$, the DFT is defined as:

$$ X[k] = \sum_{n=0}^{N-1} x[n]\, W_N^{nk}, \quad 0 \le k \le N-1 \label{eq:DFT} $$

where $W_N=e^{-j\frac{2\pi}{N}}$ is the $N$-th principal root of unity, and $X[k]$ is component at frequency $kf_s/N$ with sampling frequency $f_s$. We can rewrite $X[k]$ as an inner product:

Unconditional

zl559@cam.ac.uk (Zihe Liu) — Thu, 27 Mar 2025 23:56:58 +0200

Composed and written by Jaehyun

I just, I just bought my baby Prada
Got another for hеr, straight from the store
Fly her ‘round the world so she can hang by the shore
That’s my baby, that’s my sugar
I don’t need no honey on the side
That’s unconditiona-nal
For her, I’d break the law, just take me straight to the court
Buy a billion roses and I’d still get her more
That’s my baby, that’s my sugar
I don’t need no honey on the side
That’s unconditiona-nal

Alright I’ll be honest, the lyrics aren’t the most refined. But the song is an absolute banger. Some songs make you want to get up and dance, and this song is one of them.

Breaking Down DeepSeek R1

zl559@cam.ac.uk (Zihe Liu) — Mon, 17 Feb 2025 17:05:08 +0000

In this post, I want to explore the key RL architecture and software optimizations DeepSeek made to create Deepseek-R1. I’ll provide a brief introduction to the RL setup and explain how GRPO makes traditional algorithims much more efficient. At the end, I’ll provide a brief overview of how DeepSeek-R1 was trained. Talk given at Cambridge AI Safety Hub Members Meeting in February, slides available here.

Preliminaries

Transformers

Illustration of autoregressive model of transformers, taken from the ARENA curriculum

Modern transformers are autoregressive models. They predict the next token (essetially a word) given all the tokens before it:

Breaking Down Monte Carlo Tree Search

zl559@cam.ac.uk (Zihe Liu) — Sat, 07 Dec 2024 17:05:08 +0000

We explain how Monte Carlo Tree Search works and introduce its pseudocode. We borrow notes and figures from int8’s website and the MCTS Wikipedia page.

Game Representation

A game tree is a tree data structure, where every node represents a distinct state of the game. Given a state $s$, we take an action $a$, and the environment transitions to the new state $s'$ with the transition probability $p_a(s'|s)$. If the transition probablity is $1$ (such as in simple deterministic games like Tic Tac Toe), then each node in the game tree represents a state.

About

zl559@cam.ac.uk (Zihe Liu) — Mon, 01 Jan 0001 00:00:00 +0000

Contact

zl559@cam.ac.uk (Zihe Liu) — Mon, 01 Jan 0001 00:00:00 +0000

This is an example of a custom shortcode that you can put right into your content. You will need to add a form action to the the shortcode to make it work. Check out Formspark for a simple, free form service.

Ed theme is not strictly tied to any forms backend. You can easily use any backend like Formspark, Formspree, FabForm and so on just by changing the formAction parameter in the site configuration file.

Credits

zl559@cam.ac.uk (Zihe Liu) — Mon, 01 Jan 0001 00:00:00 +0000

Serghei Iakovlev | Software Engineer

Serghei Iakovlev is a software engineer with 20 years of experience in IT who enjoys finding elegant solutions to non-trivial problems. Serghei focused on the goal and driven by visual sophistication and laser-sharp precision, he takes pride in every finished project, and never stop learning along the way. Serghei work as the Head Of Development with 20 teams of 90+ engineers to develop airSlate — better nocode workflow automation platform.

Documentation

zl559@cam.ac.uk (Zihe Liu) — Mon, 01 Jan 0001 00:00:00 +0000

Prerequisites

This documentation was built with beginners in mind, but has the necessary information for more seasoned producers.

Projects

zl559@cam.ac.uk (Zihe Liu) — Mon, 01 Jan 0001 00:00:00 +0000

Fun stuff I’ve worked on.

GB3 RISC-V CPU | GitHub | CPU Notes | Report

Optimized a tiny RISC-V CPU with a 2-way set-associative instruction cache, cache-hit lookaheads to shorten critical paths, and increasing the clock frequency. Won 1st place at GB3 Project Competition 2026

SF2 Image Compression | Report | Results

Built JPEG-based image compression scheme with Discrete Cosine Transform, quantization, Huffman encoding, and DC intra prediction coding. Won 1st place at SF2 Project Competition 2026

Search

zl559@cam.ac.uk (Zihe Liu) — Mon, 01 Jan 0001 00:00:00 +0000

Found results:

Thank you

zl559@cam.ac.uk (Zihe Liu) — Mon, 01 Jan 0001 00:00:00 +0000

Your message will be sent within a short time. We’ll answer as soon as possible.

← Back to Home.

Home on Zihe Liu

Critique DPO

The Minority Guessing Game

Game Rules

Transition

A Box

Golden

Fasho

On Information

如燕

Grateful

Always The End

The Dog

A Dream Is a Wish Your Heart Makes

The Discrete Fourier Transform and Fast Fourier Transform

Discrete Fourier Transform

Theory

Unconditional

Breaking Down DeepSeek R1

Preliminaries

Transformers

Breaking Down Monte Carlo Tree Search

Game Representation

About

Contact

Credits

Serghei Iakovlev | Software Engineer

Documentation

Contents

Prerequisites

Projects

GB3 RISC-V CPU | GitHub | CPU Notes | Report

SF2 Image Compression | Report | Results

Search

Thank you