We tackle the fundamental challenge in this regime: how much off-policyness can we tolerate for asynchronous training to speed up learning but maintain performance?
The first thing that happened while reading this paper was was I read this line and I immediately thought that, oh so that means the inf inference is generally slower than a training. But that's not true.
While the trainer is learning, the generators don't stop. They keep producing data.
The catch is that after a while:
Generator is still using policy v10 Trainer has already updated to v11
This is the issue.
Reinforce Leave One Out
First, I was surprised that there exists an algorithm called Reinforce Leave One Out. So, it is same idea as GRPU. Exactly same idea. They just don't divide by, like, they don't make it mean and do the standard deviation stuff. Other than that, other than that, same idea. They, I think they average them and derive the baselines. Yeah. I don't know why people don't take it seriously.
DPO
Second thing, what was the second thing? So, in DPO, there are like two things. So, how DPO works is, so the first is offline DPO. Offline DPO doesn't need any reward model. So, just they take Y plus and Y minus. We have them. And we calculate a pi of Y plus given X and a pi of Y plus minus X. Like we can calculate it, right? Just by multiplying the probabilities of the tokens. Then we take the log on both the sides and we subtract them and take the sigmoid of them and we have to maximize this loss, right? This is the simple objective of the policy. This is the same policy the reward model is trained on.
Online DPO
In the online DPO, we train like this, but we, so we do train like this, but we are not limited to the dataset we have. We also make a reward model. We take more data. We use a policy to generate two answers and then judge them by the reward model. Then because we have them, then we calculate, we have the probabilities, so we try to minimize, sorry, maximize the difference between the loss, whatever you want to say.