Learning to summarize from human feedback
So basically the issue with SFT is it's not good for learning fidelity. So it teaches the exact, what the exact copying the tokens, like the answer. But, for example, there should be a word X, and Y is also equally probable, but it will say penalize Y. Right. You know, you have did it wrong. But what RLHF does is, it makes you learn that there can be multiple outputs doing the same thing. But it makes you learn the right thing. And okay, you need to learn the underlying semantics, not the exact word. Makes sense?
Models are incentivized to place probability mass on all human demonstrations, including those that are low-quality.
Suppose your dataset contains three summaries for similar posts:
Summary A: Excellent Summary B: Good Summary C: Mediocre
SFT doesn't know A is better than C.
It simply learns
Increase P(A) Increase P(B) Increase P(C)
because all demonstrations are treated as correct.
But what we actually want is
P(A) > P(B) > P(C)
SFT has no way of expressing this preference.
RLHF fixes this because humans can say
A > B A > C B > C
and the model learns to prefer better outputs.
Distributional shift during sampling can degrade performance.
During training:
Human: Token1 → Token2 → Token3 → Token4
The model always conditions on correct previous tokens. ......... ........
Also, there was this beam search thing. So what beam search, what happened during beam search was, so at inference time, you would not pick, like, a single token, you would pick, like, some token, like, for example, two tokens, and then branch them, like, generate for each token, and you keep doing it, so you'll get a lot of sequences, and you calculate the cumulative priority of the sequence and which one is higher, because we don't want to maximize the early input's priority, but the total sequence priority, and you will choose the maximum one. The issue is, it would start, it would, okay, so highest sequence priority does not mean highest right answer. It can just start repeating tokens, because, for example, if, you know, if I get teacher, teacher, if teacher is the highest priority token, it will just teacher, teacher, teacher, teacher, teacher, teacher, teacher. It can do like that. So that's why I was not aware of beam search anyways, because this paper only introduced me to that.
So SFT did not generalize and RL generalize. Like they trained on like reddit summaries but news articles summaries also improved by RL
Reward hacking from over-optimization
The authors test what happens when they optimize harder and harder.
Initially:
higher reward-model score⇒better summaries according to humans
But after too much optimization:
higher reward-model score ⇒better summaries
Eventually, the policy discovers weaknesses in the reward model and gets high predicted reward while producing worse text. This is reward hacking or reward-model overoptimization.
The most surprising thing in this paper is that how do they... So, you might be thinking that they take the Reddit summaries, take the pre-trained model, the summary generated, and take the summary generated by the supervised fine-tuned model, and they train the reward model by doing the DPO on that or something like that. But that's not how they do it, man. So first they do it, they train a reward model with this. Then they do PPO. Then they train another reward model, in which they include the PPOs, within the PPO answer also. And then they keep doing this. So the reward models keep improving because it gets the improved PPO summaries. Obviously, they send it to humans for the baseline. That's one thing. Also, they do one more cool thing, which is like they do a top of N. So, basically, in the supervised fine-tuned model, they generate N summaries with this. And then with reward model, they grade the N summaries and pick the top N. So, add this to those notes. This is pretty insightful.
So basically they do a very clever thing. So they wanted to test that, okay, so reward model, if we keep doing PPO, then the reward model, so PPO will try to reward hack the reward model if we do it for long enough, okay. It will find a reward hack, rewarding. Okay. So OpenAI wanted to test that, okay, so at how much it will. So the reward of PPO was the reward from reward model minus beta KL divergence from the SFT policy, right. So they keep increasing this beta policy. And they keep seeing, okay, see like at which does the model digress, kind of, right. They compare to a human baseline. I think how do they evaluate it is simple, like, if the, how do they evaluate it? That's a good question. Wait, let me ask and tell you.
So basically they give it to the humans and ask them which summary is better, the original Reddit one or the one that model generated, and they just attest the, like the final score is as a percentage of times, percentage of, like, answers generated by humans are chosen over, sorry, model PPO model is chosen over humans.
Okay, so what, how do they exactly train the reward model is. So for example, for two pairs y_i and y_j, where i is a correct, like, a summary, like a better summary, what we do is we make them go through the model, we generate the rewards, then we calculate the difference in reward, and we take the sigmoid operator. If the difference is positive, we'll get probability more than 0.5. Like, we look at it as a probability, and if it's less, then we'll get less than 0.5. Now we'll have, so we want to maximize this probability of, right, because we know these are the real pairs. So basically, the loss becomes minus log sigma, sorry, a sigmoid R_j, R_i minus R_j. Also, they do one more thing that they calculate, so they, a lot of human baselines, like the human summaries, they, after the reward model is trained, they make them go through the reward model and take an average of it. So they consider this as a baseline. They want this average to be zero. So they, from every reward, they subtract this average in future, like, just to have a baseline. You can add these two things.