2025-08-28 17:57:59 +00:00
o
2025-08-28 22:41:56 +00:00
2025-08-28 17:57:59 +00:00
Z
m
Z
m
Z
m!Z!m"Z"m#Z#m$Z$m%Z%m&Z&m'Z'm(Z(m)Z)m*Z*m+Z+m
Z
m$Z$m%Z%m*Z*m=Z=mCZCm
lWmBZB
d
d
d �
2025.8.9
2025.8.10
4.55.4
0.21.0
__UNSLOTH_VERSIONING__
é
functional)
AutoTokenizerÚ
DPOTrainerÚ
DataLoaderÚ
PeftConfigÚ PeftModelÚ
flush_leftÚ
pad_to_lengthÚ
shape_paddingz
trace.enabledz
� � �
|�
rL
Ú
chunked_indexÚ
� �
i�
S
Configuration class for the [`DPOTrainer`].
This class includes only the parameters that are specific to DPO training. For a full list of training arguments,
please refer to the [`~transformers.TrainingArguments`] documentation. Note that default values in this class may
differ from those in [`~transformers.TrainingArguments`].
Using [`~transformers.HfArgumentParser`] we can turn this class into
[argparse](https://docs.python.org/3/library/argparse#module-argparse) arguments that can be specified on the
command line.
Parameters:
> Parameters that control the model and reference model
model_init_kwargs (`dict[str, Any]` or `None`, *optional*, defaults to `None`):
Keyword arguments for `AutoModelForCausalLM.from_pretrained`, used when the `model` argument of the
[`DPOTrainer`] is provided as a string.
ref_model_init_kwargs (`dict[str, Any]` or `None`, *optional*, defaults to `None`):
Keyword arguments for `AutoModelForCausalLM.from_pretrained`, used when the `ref_model` argument of the
[`DPOTrainer`] is provided as a string.
model_adapter_name (`str` or `None`, *optional*, defaults to `None`):
Name of the train target PEFT adapter, when using LoRA with multiple adapters.
ref_adapter_name (`str` or `None`, *optional*, defaults to `None`):
Name of the reference PEFT adapter, when using LoRA with multiple adapters.
force_use_ref_model (`bool`, *optional*, defaults to `False`):
If you provide a PEFT model as the active model and wish to use a different model for the `ref_model`, set
this flag to `True`.
disable_dropout (`bool`, *optional*, defaults to `True`):
Whether to disable dropout in the model and reference model.
use_logits_to_keep (`bool`, *optional*, defaults to `False`):
If `True`, only a specified number of logits are computed in the forward pass. This can be useful for
saving memory and speeding up training by not computing the logits for all tokens, especially in scenarios
when working with very long prompts where labels are ignored (-100).
> Parameters that control the data preprocessing
dataset_num_proc (`int` or `None`, *optional*, defaults to `None`):
Number of processes to use for processing the dataset.
padding_value (`int` or `None`, *optional*, defaults to `None`):
Padding value to use. If `None`, the padding value of the tokenizer is used.
label_pad_token_id (`int`, *optional*, defaults to `-100`):
Padding value to use for labels.
max_prompt_length (`int` or `None`, *optional*, defaults to `512`):
Maximum length of the prompt.
max_completion_length (`int` or `None`, *optional*, defaults to `None`):
Maximum length of the completion.
max_length (`int` or `None`, *optional*, defaults to `1024`):
Maximum length of the full sequence (prompt + completion).
truncation_mode (`str`, *optional*, defaults to `"keep_end"`):
Truncation mode to use when the sequence exceeds `max_length`. Possible values are `"keep_end"` and
`"keep_start"`.
padding_free (`bool`, *optional*, defaults to `False`):
Whether to perform forward passes without padding by flattening all sequences in the batch into a single
continuous sequence. This reduces memory usage by eliminating padding overhead. Currently, this is only
supported with the `flash_attention_2` attention implementation, which can efficiently handle the flattened
batch structure.
precompute_ref_log_probs (`bool`, *optional*, defaults to `False`):
Whether to precompute the log probabilities from the reference model. Setting this to `True` allows
training without needing the reference model during training, which can help reduce GPU memory usage. If
set to `False` (default), the reference model will be used during training to compute log probabilities
on-the-fly.
precompute_ref_batch_size (`int` or `None`, *optional*, defaults to `None`):
Batch size to use when precomputing reference model log probabilities. This can be set higher than the
training batch size to speed up preprocessing. If `None`, defaults to `per_device_train_batch_size` for
training and `per_device_eval_batch_size` for evaluation.
tools (`Optional[list[Union[dict, Callable]]]`, *optional*, defaults to `None`):
List of tools (callable functions) that will be accessible to the model. If the template does not support
function calling, this argument will have no effect.
> Parameters that control the training
loss_type (`str` or `list[str]`, *optional*, defaults to `"sigmoid"`):
Type of loss to use. Possible values are:
- `"sigmoid"`: sigmoid loss from the original [DPO](https://huggingface.co/papers/2305.18290) paper.
- `"hinge"`: hinge loss on the normalized likelihood from the
[SLiC](https://huggingface.co/papers/2305.10425) paper.
- `"ipo"`: IPO loss from the [IPO](https://huggingface.co/papers/2310.12036) paper.
- `"exo_pair"`: pairwise EXO loss from the [EXO](https://huggingface.co/papers/2402.00856) paper.
- `"nca_pair"`: pairwise NCA loss from the [NCA](https://huggingface.co/papers/2402.05369) paper.
- `"robust"`: unbiased estimate of the DPO loss that is robust to preference noise from the [Robust
DPO](https://huggingface.co/papers/2403.00409) paper.
- `"bco_pair"`: pairwise BCO loss from the [BCO](https://huggingface.co/papers/2404.04656) paper.
- `"sppo_hard"`: SPPO loss with hard label from the [SPPO](https://huggingface.co/papers/2405.00675)
paper.
- `"aot"`: AOT loss for paired datasets from the [AOT](https://huggingface.co/papers/2406.05882) paper.
- `"aot_pair"`: AOT loss for unpaired datasets from the [AOT](https://huggingface.co/papers/2406.05882)
paper.
- `"discopop"`: DiscoPOP (a.k.a Log-Ratio Modulated Loss, LRML) loss from the
[DiscoPOP](https://huggingface.co/papers/2406.08414) paper.
- `"apo_zero"`: APO-zero loss from the [APO](https://huggingface.co/papers/2408.06266) paper.
- `"apo_down"`: APO-down loss from the [APO](https://huggingface.co/papers/2408.06266) paper.
- `"sft"`: Negative log-likelihood loss (standard supervised fine-tuning loss).
Multiple loss types can be combined using comma separation (e.g., `["sigmoid", "bco_pair", "sft"]` for
[MPO](https://huggingface.co/papers/2411.10442)). The `loss_weights` parameter can be used to specify
corresponding weights for each loss type.
use_liger_loss (`bool`, *optional*, defaults to `False`):
Whether to use Liger loss.
base_model_attribute_name (`str`, *optional*, defaults to `"model"`):
Name of the attribute in the model that contains the base model. This is used to get the base model from
the model when the model does not have a `get_decoder` method in the case when `use_liger_loss` is `True`.
beta (`float`, *optional*, defaults to `0.1`):
Parameter controlling the deviation from the reference model. Higher β means less deviation from the
reference model. For the IPO loss (`loss_type="ipo"`), β is the regularization parameter denoted by τ in
the [paper](https://huggingface.co/papers/2310.12036).
f_divergence_type (`str`, *optional*, defaults to `FDivergenceType.REVERSE_KL`):
Type of f-divergence regularization function to compute divergence between policy and reference model.
f_alpha_divergence_coef (`float`, *optional*, defaults to `1.0`):
α coefficient in the α-divergence u^-α regularization function for DPO loss.
reference_free (`bool`, *optional*, defaults to `False`):
Whether to ignore the provided reference model and implicitly use a reference model that assigns equal
probability to all responses.
label_smoothing (`float`, *optional*, defaults to `0.0`):
Robust DPO label smoothing parameter from the [cDPO report](https://ericmitchell.ai/cdpo.pdf) and [Robust
DPO](https://huggingface.co/papers/2403.00409) paper that should be between `0.0` and `0.5`.
use_weighting (`bool`, *optional*, defaults to `False`):
Whether to weight the loss as done in the [WPO paper](https://huggingface.co/papers/2406.11827).
rpo_alpha (`float`, *optional*, defaults to `None`):
α parameter from the [RPO paper](https://huggingface.co/papers/2404.19733) (v3), which controls the
weighting of the NLL term in the loss. If `None`, no weighting is applied and the loss is the same as the
DPO loss. The paper recommends `rpo_alpha=1.0`.
ld_alpha (`float` or `None`, *optional*, defaults to `None`):
α parameter from the [LD-DPO paper](https://huggingface.co/papers/2409.06411), which controls the weighting
of the verbose token log-probabilities in responses. If `None`, no weighting is applied to the verbose
part, and the loss is equivalent to the standard DPO loss. The paper recommends setting `ld_alpha` between
`0.0` and `1.0`.
discopop_tau (`float`, *optional*, defaults to `0.05`):
Ï„/temperature parameter from the [DiscoPOP](https://huggingface.co/papers/2406.08414) paper, which controls
the shape of log ratio modulated loss. The paper recommends the default value `discopop_tau=0.05`.
loss_weights (`list[float]` or `None`, *optional*, defaults to `None`):
List of loss weights for multi-loss combinations. Used when combining multiple loss types.
Example: `[0.8, 0.2, 1.0]` for [MPO](https://huggingface.co/papers/2411.10442). If not provided, defaults
to equal weights (`1.0`) for all loss types.
sync_ref_model (`bool`, *optional*, defaults to `False`):
Whether to synchronize the reference model with the active model every `ref_model_sync_steps` steps, using
the `ref_model_mixup_alpha` parameter. This synchronization originates from the
[TR-DPO](https://huggingface.co/papers/2404.09656) paper.
ref_model_mixup_alpha (`float`, *optional*, defaults to `0.6`):
α parameter from the [TR-DPO](https://huggingface.co/papers/2404.09656) paper, which controls the mix
between the current policy and the previous reference policy during updates. The reference policy is
updated according to the equation: `π_ref = α * π_θ + (1 - α) * π_ref_prev`. To use this parameter, you
must set `sync_ref_model=True`.
ref_model_sync_steps (`int`, *optional*, defaults to `512`):
Ï„ parameter from the [TR-DPO](https://huggingface.co/papers/2404.09656) paper, which determines how
frequently the current policy is synchronized with the reference policy. To use this parameter, you must
set `sync_ref_model=True`.
> Parameters that control the logging
generate_during_eval (`bool`, *optional*, defaults to `False`):
Whether to generate and log completions from both the model and the reference model to W&B or Comet during
evaluation.
NÚ
?ç{
adamw_8bitÚ
every_saveÚ › � ‚ › � ‚ ˆ
d
ƒˆ
“
“ � |t“‚ ˆ ‹ � |€“� “� |‚“� |ƒ“� “� “� |� “� “ � �
output_dirÚ
do_predictÚ
eval_strategyÚ
eval_delayÚ
learning_rateÚ
adam_beta1Ú
adam_beta2Ú
max_grad_normÚ
logging_stepsÚ
save_strategyÚ
save_stepsÚ
jit_mode_evalÚ
local_rankÚ
tpu_num_coresÚ
eval_stepsÚ
past_indexÚ
optim_argsÚ adafactorÚ
mp_parametersÚ
torch_compileÚ
eval_on_startÚ
padding_valueÚ
max_lengthÚ
use_weightingÚ rpo_alphaÚ
Ú
OverflowErrorÚ
× � › ˜ ’ ‘ � � � ‹ ˆ � � � � � � � � � �
� � �
� � � � � � � � � � � � � � � � � � � � � �
� � ‚ ˆ ‹ � � � � ‘ ’ – ’ ˜ › � ‹ ˆ ›
__module__Ú
__classcell__rv
�
e
e
e
e
e
e
e
e
e
e
e
e
e
e
e
e
e
d
Z!dkd
d
e
e
d"d#„
e(
e(
e(
e(
e
Z.e/d3d4„
e
e
e
dUdV„
e"
e
Z?dmd`e
e9
Z@‡
e
e
data_collatorÚ
train_datasetÚ
optimizersÚ
ˆ › � ˆ ˆ ˆ ˆ ˆ ˆ
d ˆ
ˆ ˆ ˆ
d ˆ
ˆ ‚ ‚ ˆ
d ˆ ˆ ˆ ˆ
d � ˆ
ˆ ‚ ˆ ˆ ˆ ˆ ˆ ˆ ˆ ˆ ˆ ˆ ˆ ˆ ˆ ˆ ˆ ˆ ˆ � �
ƒ‚ ˆ � ‚ ˆ ˆ ˆ � ˆ
| � ˆ � ˆ ˆ ˆ ˆ ˆ ˆ ˆ ˆ ˆ ˆ ˆ ˆ ˆ ˆ ˆ ˆ ˆ � � ˆ � ˆ ˆ ˆ ˆ ˆ ˆ ˆ ˆ ˆ � ˆ ˆ ˆ ˆ ˆ ˆ ˆ ˆ ˆ � � ˆ � � › � � ‚ � ˆ ˆ ˆ ˆ ˆ ˆ ˆ ˆ ˆ ˆ � � ˆ ˆ ˆ ˆ ˆ
| � ˆ ˆ � ˆ ˆ ˆ � ‚ ˆ � ˆ � ˆ � ‚ ˆ � ˆ � ˆ � ‚ ˆ � ‚ ˆ � ˆ ˆ ˆ ˆ � ˆ ˆ ˆ
ˆ ˆ
d0� ˆ ˆ � � ‚ ˆ ˆ ˆ � ˆ � ¸ ˆ ˆ � You passed model_init_kwargs to the `DPOConfig`, but your model is already instantiated. The `model_init_kwargs` will be ignored.z� You passed ref_model_init_kwargs to the `DPOConfig`, but your ref_model is already instantiated. The `ref_model_init_kwargs` will be ignored.T)‚
use_ref_modelÚ
\ˆ ˆ ˆ
<dictcomp>ÿ
isinstanceÚ
_name_or_pathÚ
ValueErrorr#
model_typer
is_peft_modelr%
aux_loss_coefÚ
_tag_namesÚ
zero_stager@
prepare_modelÚ
model_namer™ �
model_pathrh › � ‚
| ’ ˆ ˆ � � ‘
‚ ‚
ƒ
z#Prepares a model for PEFT training.FNzvPEFT is not installed and you passed a `peft_config` in the trainer's kwargs, please install it to use the PEFT modelsa8 � ˆ
parametersr ’
� d › � › �
|› �
n
d
d
ˆ –
map_kwargsrv
ï × � � �
Tokenize a row of the dataset.
Args:
features (`dict[str, str]`):
Row of the dataset, should contain the keys `"prompt"`, `"chosen"`, and `"rejected"`.
processing_class (`PreTrainedTokenizerBase`):
Processing class used to process the data.
max_prompt_length (`int` or `None`):
Maximum length of the prompt sequence. If `None`, the prompt sequence is not truncated.
max_completion_length (`int` or `None`):
Maximum length of the completion sequences. If `None`, the completion sequences are not truncated.
add_special_tokens (`bool`):
Whether to add special tokens to the sequences. Typically used for encoder-decoder models. If `True`,
the prompt sequence will have a bos token prepended and an eos token appended. In any case, the
completion sequences will have an eos token appended.
Returns:
`dict[str, list[int]]`:
Tokenized sequences with the keys `"prompt_input_ids"`, `"chosen_input_ids"`, and
`"rejected_input_ids".
Example:
```python
>>> from transformers import GPT2Tokenizer
>>> tokenizer = GPT2Tokenizer.from_pretrained("gpt2")
>>> features = {"prompt": "The sky is", "chosen": " blue", "rejected": " green"}
>>> DPOTrainer.tokenize_row(
... features, tokenizer, max_prompt_length=3, max_completion_length=3, add_special_tokens=False
... )
{'prompt_input_ids': [464, 6766, 318], 'chosen_input_ids': [4171, 50256], 'rejected_input_ids': [4077, 50256]}
```
Ú× ×
� �
|
�
|
|
d
|
|
|
< � |
Same as `tokenize_row` but for vision models. Please refer to `tokenize_row` for more information.
Ú ×
rÞ ×
|
| �
|
| �
d�
Returns the training [`~torch.utils.data.DataLoader`].
Subclass of transformers.src.transformers.trainer.get_train_dataloader to precompute `ref_log_probs`.
F©
batch_sizeÚ
collate_fnÚ
pin_memoryÚ
add_columnrH
|
t‚
|�
t
d� � ‘
| �
Returns the evaluation [`~torch.utils.data.DataLoader`].
Subclass of transformers.src.transformers.trainer.get_eval_dataloader to precompute `ref_log_probs`.
Args:
eval_dataset (`torch.utils.data.Dataset`, *optional*):
If provided, will override `self.eval_dataset`. If it is a [`~datasets.Dataset`], columns not accepted
by the `model.forward()` method are automatically removed. It must implement `__len__`.
Nz-Trainer: evaluation requires an eval_dataset.Fró �
� � *
t � F � 4 �
| �
| �
� � � � �
�
�
� �
Concatenate the `chosen` and `rejected` inputs from the batch into a single tensor for both the prompt and
completion sequences.
Args:
batch (`dict[str, Union[list, torch.LongTensor]]`):
A batch of input data. The batch must contain the following keys:
- `"prompt_input_ids"`: Tensor of shape `(batch_size, prompt_length)` representing the prompt input
IDs.
- `"chosen_input_ids"`: Tensor of shape `(batch_size, chosen_length)` representing the chosen
completion input IDs.
- `"rejected_input_ids"`: Tensor of shape `(batch_size, rejected_length)` representing the rejected
completion input IDs.
- `"prompt_pixel_values"` (optional): Tensor for pixel values, if available.
- `"prompt_pixel_attention_mask"` (optional): Tensor for pixel attention masks, if available.
padding_value (`int`):
The padding value to use for the concatenated completion sequences (`chosen_input_ids` and
`rejected_input_ids`).
Returns:
`dict[str, torch.LongTensor]`: A dictionary containing:
- `"prompt_input_ids"`: Concatenated prompt input IDs of shape `(2 * batch_size, prompt_length)`.
- `"completion_input_ids"`: Concatenated chosen and rejected completion input IDs of shape `(2 *
batch_size, max_completion_length)`.
- `"prompt_attention_mask"`: Concatenated prompt attention masks of shape `(2 * batch_size,
prompt_length)`.
- `"completion_attention_mask"`: Concatenated chosen and rejected attention masks of shape `(2 *
batch_size, max_completion_length)`.
- `"pixel_values"` (optional): Concatenated pixel values if `"prompt_pixel_values"` are present.
- `"pixel_attention_mask"` (optional): Concatenated pixel attention masks if
`"prompt_pixel_attention_mask"` are present.
Notes:
The completion input IDs and attention masks are padded to the maximum completion length of the chosen or
rejected sequences.
râ
ÿ
ÿ
|
r;t
v
t
t
| |
|
�
n
|
|
ˆ � � � �
k� � � � � �
k� � � � � � � �
t �
d� � � �
|
� › � ‚
Compute the DPO loss for a batch of policy and reference model log probabilities.
Args:
chosen_logps (`torch.FloatTensor`):
Log probabilities of the model for the chosen responses. Shape: `(batch_size,)`.
rejected_logps (`torch.FloatTensor`):
Log probabilities of the model for the rejected responses. Shape: `(batch_size,)`.
ref_chosen_logps (`torch.FloatTensor`):
Log probabilities of the reference model for the chosen responses. Shape: `(batch_size,)`.
ref_rejected_logps (`torch.FloatTensor`):
Log probabilities of the reference model for the rejected responses. Shape: `(batch_size,)`.
Returns:
A tuple of three tensors: `(losses, chosen_rewards, rejected_rewards)`. The losses tensor contains the DPO
loss for each example in the batch. The `chosen_rewards` and `rejected_rewards` tensors contain the rewards
for the chosen and rejected responses, respectively.
r �
JS_DIVERGENCEr
logsigmoidr7 ´
zeros_liker�
alpha_coefrm
ref_logratiosÚ
losses_chosenÚ
exp_componentÚ
ÿ
�
�
¡
�
|
j ›
d ›
¡
|
�
¡
� � (
�
¡
� � � � � � ˆ � ˆ � � › � ‚ � � � �
¡ �
d �
¡
t!|
d�
¡
|
ƒ � �
¡ � � �
d �
¡
|
� � � � � � � � �
keep_startr�
ref_weightÚ � ›
hidden_statesÚ
base_modelÚ
�
|
<
|
d
|
� �
f�
|
f� � �
d
d�
› � ‚ � � �
t � � �
j � ¸
d
d
| � � � � � B
d� � � � � � ˜ �
d
d � � � � � � � � � � � � � �
d
d
d
| �
Runs the given model on the given batch of inputs, concatenating the chosen and rejected inputs together.
We do this to avoid doing two forward passes, because it's faster for FSDP.
Args:
model:
Model to run the forward pass on.
batch:
Batch of input data.
is_ref_model:
Whether this method is being called for the reference model. If `True`, length desensitization is not
applied.
râ � � �
cross_entropyÚ ˆ
chosen_logitsÚ
chosen_labelsÚ
front_maskÚ rear_maskÚ
rear_logpsÚ split_idxrj
(
ÿ
train_eval)
n
d
|
}
|
d
u
|
|
› � › � › � › � › � › � › � › �
d
u� � › � �
� ˆ
logits/chosenrk ’ ¸
ú
t � � � ¸
rÀ
store_metrics)
t � ] �
¡ � � �
j�
j�
ref_outputN)
policy_outputrË
ˆ
¡ � $ � � �
¡
�
�
|
¡
f ×
\ˆ ‘
<listcomp>l
¸
dataloaderÚ �
| ¡
}
j
| � �
| �
| �
|
S
Overriding built-in evaluation loop to store metrics for each batch. Prediction/evaluation loop, shared by
