use token ids in vllm

[AlexPiche]

Use the token_ids produce by vLLM

• Simplify the Trainable class, e.g. no need to remove leading white space, etc.
• Process the log probs and record the prodsuced token ids
• Remove all the token verification from rl_orchestrator.py
• Modify finetune/data.py to overwrite the huggingface generated input_ids and labels when these are provided in entry.

Replications

DeepSeek RFT

Before

After

Gemma2 2b RFT

train_mean_reward does not improve as much as before, but test_mean_reward behaves similarly.

Before

After

[rizar]

God stuff! LGTM! I'd try to simplify the code in finetune.data before the merge though. I feel like the way is written is a bit of a trap that can give us trouble later on.