rgcKs$ddlm}|di|¤Ž}||_|S)Nr)ÚSamplingParamsre)ÚvllmrhÚ_set_kwargs)ÚkwargsrhÚsampling_paramsrererfÚvLLMSamplingParams3srmcsšeZdZUdZedddid�Zeeed<edddid�Z ee
This class includes only the parameters that are specific to Online DPO training. For a full list of training
arguments, please refer to the [`~transformers.TrainingArguments`] documentation. Note that default values in this
class may differ from those in [`~transformers.TrainingArguments`].
Using [`~transformers.HfArgumentParser`] we can turn this class into
[argparse](https://docs.python.org/3/library/argparse#module-argparse) arguments that can be specified on the
command line.
Parameters:
reward_model_path (`str` or `None`, *optional*, defaults to `None`):
Path to the reward model. Either `judge` or `reward_model_path` must be set, but not both.
judge (`str` or `None`, *optional*, defaults to `None`):
Name of the judge to use. Either `judge` or `reward_model_path` must be set, but not both.
max_new_tokens (`int`, *optional*, defaults to `64`):
Maximum number of tokens to generate per completion.
max_length (`int`, *optional*, defaults to `256`):
Maximum total length of the sequence (prompt + completion) used to compute log probabilities. If the
sequence exceeds this limit, the leftmost tokens will be truncated to preserve as much of the completion as
possible.
temperature (`float`, *optional*, defaults to `0.9`):
Temperature for sampling. The higher the temperature, the more random the completions.
missing_eos_penalty (`float` or `None`, *optional*, defaults to `None`):
Penalty applied to the score when the model fails to generate an EOS token. This is useful to encourage to
generate completions shorter than the maximum length (`max_new_tokens`). The penalty must be a positive
value.
beta (`float` or `list[float]`, *optional*, defaults to `0.1`):
Parameter controlling the deviation from the reference model. Higher β means less deviation from the
reference model. For the IPO loss (`loss_type="ipo"`), β is the regularization parameter denoted by τ in
the [paper](https://huggingface.co/papers/2310.12036). If a list of floats is provided then the β is
selected for each new epoch and the last β is used for the rest of the epochs.
loss_type (`str`, *optional*, defaults to `"sigmoid"`):
Type of loss to use. Possible values are:
- `"sigmoid"`: sigmoid loss from the original [DPO](https://huggingface.co/papers/2305.18290) paper.
- `"ipo"`: IPO loss from the [IPO](https://huggingface.co/papers/2310.12036) paper.
dataset_num_proc (`int` or `None`, *optional*, defaults to `None`):
Number of processes to use for processing the dataset.
disable_dropout (`bool`, *optional*, defaults to `True`):
Whether to disable dropout in the model and reference model.
use_vllm (`bool`, *optional*, defaults to `False`):
Whether to use vLLM for generating completions. Requires vLLM to be installed (`pip install vllm`).
vllm_model_impl (`str`, *optional*, defaults to `"vllm"`):
Model implementation to use for vLLM. Must be one of `"transformers"` or `"vllm"`. `"transformers"`: Use
the `transformers` backend for model implementation. `"vllm"`: Use the `vllm` library for model
implementation.
gpu_memory_utilization (`float`, *optional*, defaults to `0.55`):
The vLLM memory utilization. The default value is 0.55.
ds3_gather_for_generation (`bool`, *optional*, defaults to `True`):
This setting applies to DeepSpeed ZeRO-3. If enabled, the policy model weights are gathered for generation,
improving generation speed. However, disabling this option allows training models that exceed the VRAM
capacity of a single GPU, albeit at the cost of slower generation.
model_init_kwargs (`dict[str, Any]` or `None`, *optional*, defaults to `None`):
Keyword arguments to pass to `AutoModelForCausalLM.from_pretrained` when instantiating the model from a
string.
NÚhelpzvLLM SamplingParams)ÚdefaultÚmetadataÚvllm_sampling_paramsrIz8Chunk size to reduce memory usage. -1 is most efficient.Úunsloth_num_chunksz'Maximum sequence length to truncate to.Úmax_seq_lengthFÚnorJéréúç-Cëâ6
dS)NgH¯¼šò×z>z Unsloth: Your learning rate of `zi` is too small and less than 1e-7! Consider increasing it, otherwise gradient updates will be close to 0!rOza` is way too larger > 1! Consider decreasing it to 1e-1, otherwise gradient updates will explode!rƒr„Úunsloth_training_checkpointsrur)Ú cpu_countrJrvzUUnsloth: Please set a positive non-zero temperature since your results will be wrong.é
zgUnsloth: Please set a positive non-zero temperature less than 10, since sampling will be quite erratic.Ú
t|j/d ƒ�rm|j/0|j1¡|j2|_3|j4�r�|jdu�r‡t5|j|j6|j7|j8ƒ|_|jdu�r›t5|j|j6|j7|j8ƒ|_dSdS|jdu�r¬|j9|j:j;¡|_|jdu�r½|j9|j:j;¡|_dSdS)"NÚvllm_enginerFTzš`model` and `ref_model` cannot be the same object. If you want `ref_model` to be the same as `model`, either omit the `ref_model` argument or pass `None`.z€Both `reward_model` and `judge` are provided. Please choose provide only one of them. Ignoring `judge` and using `reward_model`.z2Either `reward_model` or `judge` must be provided.z@`missing_eos_penalty` is not supported when `judge` is provided.z`args` must be provided.z$`processing_class` must be provided.Útorch_dtyper‡zŽInvalid `torch_dtype` passed to `OnlineDPOConfig`. Expected either 'auto' or a string representing a `torch.dtype` (e.g., 'float32'), but got Ú.z¦You passed `model_init_kwargs` to the `OnlineDPOConfig`, but your model is already instantiated. This argument can only be used when the `model` argument is a string.zfPEFT is not available and passed `peft_config`. Please install PEFT with `pip install peft` to use it.)Úpad_token_id)úobjective/klúobjective/entropyúobjective/non_score_rewardúrewards/chosenúrewards/rejectedúrewards/accuraciesúrewards/marginsúlogps/chosenúlogps/rejectedúval/contain_eos_tokenÚbetaúobjective/rlhf_rewardúobjective/scores_marginúobjective/scoresrrvé2r})ÚnÚ
prompt_idsÚprompt_maskÚcompletion_maskre)r»r¸r[r«rJrfÚ_generate_vllmñs.$"þz'_UnslothOnlineDPOTrainer._generate_vllmcsòˆjj}ˆjj}dd„|Dƒ}‡fdd„|Dƒ}‡fdd„|Dƒ}ˆ|¡}ˆ|¡}|ddd¡}|ddd¡}t|ˆjˆjj d ��}|j
title = {{Direct Language Model Alignment from Online AI Feedback}},
author = {Shangmin Guo and Biao Zhang and Tianlin Liu and Tianqi Liu and Misha Khalman and Felipe Llinares and Alexandre Ram{'{e}} and Thomas Mesnard and Yao Zhao and Bilal Piot and Johan Ferret and Mathieu Blondel},
year = 2024,
eprint = {arXiv:2402.04792}
}z
Online DPOz7Direct Language Model Alignment from Online AI Feedbackz
