2025-08-28 17:57:59 +00:00
o
2025-08-28 22:41:56 +00:00
2025-08-28 17:57:59 +00:00
Z
m
Z
m
Z
m
Z
m
lOm=Z=
d
d
d �
2025.8.9
2025.8.10
4.55.4
0.21.0
__UNSLOTH_VERSIONING__
é
functional)
DataLoaderÚ
KTOTrainerÚ
has_lengthÚ
itemgetterÚ
pad_to_lengthÚ
shape_paddingz
trace.enabledz
� � �
|�
rD
Ú
chunked_indexÚ
� �
i�
S
Configuration class for the [`KTOTrainer`].
This class includes only the parameters that are specific to KTO training. For a full list of training arguments,
please refer to the [`~transformers.TrainingArguments`] documentation. Note that default values in this class may
differ from those in [`~transformers.TrainingArguments`].
Using [`~transformers.HfArgumentParser`] we can turn this class into
[argparse](https://docs.python.org/3/library/argparse#module-argparse) arguments that can be specified on the
command line.
Parameters:
max_length (`int` or `None`, *optional*, defaults to `1024`):
Maximum length of the sequences (prompt + completion) in the batch. This argument is required if you want
to use the default data collator.
max_prompt_length (`int` or `None`, *optional*, defaults to `512`):
Maximum length of the prompt. This argument is required if you want to use the default data collator.
max_completion_length (`int` or `None`, *optional*, defaults to `None`):
Maximum length of the completion. This argument is required if you want to use the default data collator
and your model is an encoder-decoder.
beta (`float`, *optional*, defaults to `0.1`):
Parameter controlling the deviation from the reference model. Higher β means less deviation from the
reference model.
loss_type (`str`, *optional*, defaults to `"kto"`):
Type of loss to use. Possible values are:
- `"kto"`: KTO loss from the [KTO](https://huggingface.co/papers/2402.01306) paper.
- `"apo_zero_unpaired"`: Unpaired variant of APO-zero loss from the
[APO](https://huggingface.co/papers/2408.06266) paper.
desirable_weight (`float`, *optional*, defaults to `1.0`):
Desirable losses are weighed by this factor to counter unequal number of desirable and undesirable paris.
undesirable_weight (`float`, *optional*, defaults to `1.0`):
Undesirable losses are weighed by this factor to counter unequal number of desirable and undesirable pairs.
label_pad_token_id (`int`, *optional*, defaults to `-100`):
Label pad token id. This argument is required if you want to use the default data collator.
padding_value (`int` or `None`, *optional*, defaults to `None`):
Padding value to use. If `None`, the padding value of the tokenizer is used.
truncation_mode (`str`, *optional*, defaults to `"keep_end"`):
Truncation mode to use when the prompt is too long. Possible values are `"keep_end"` or `"keep_start"`.
This argument is required if you want to use the default data collator.
generate_during_eval (`bool`, *optional*, defaults to `False`):
If `True`, generates and logs completions from both the model and the reference model to W&B or Comet
during evaluation.
is_encoder_decoder (`bool` or `None`, *optional*, defaults to `None`):
When using the `model_init` argument (callable) to instantiate the model instead of the `model` argument,
you need to specify if the model returned by the callable is an encoder-decoder model.
precompute_ref_log_probs (`bool`, *optional*, defaults to `False`):
Whether to precompute reference model log probabilities for training and evaluation datasets. This is
useful when training without the reference model to reduce the total GPU memory needed.
model_init_kwargs (`dict[str, Any]` or `None`, *optional*, defaults to `None`):
Keyword arguments to pass to `AutoModelForCausalLM.from_pretrained` when instantiating the model from a
string.
ref_model_init_kwargs (`dict[str, Any]` or `None`, *optional*, defaults to `None`):
Keyword arguments to pass to `AutoModelForCausalLM.from_pretrained` when instantiating the reference model
from a string.
dataset_num_proc: (`int` or `None`, *optional*, defaults to `None`):
Number of processes to use for processing the dataset.
disable_dropout (`bool`, *optional*, defaults to `True`):
Whether to disable dropout in the model and reference model.
use_liger_loss (`bool`, *optional*, defaults to `False`):
Whether to use Liger loss. It requires liger-kernel to be installed.
base_model_attribute_name (`str`, *optional*, defaults to `"model"`):
Name of the attribute in the model that contains the base model. This is used to get the base model from
the model when the model does not have a `get_decoder` method in the case when `use_liger_loss` is `True`.
NÚ
?ç{
adamw_8bitÚ
every_saveÚ › � ‚ › � ‚ ‘
d ˜
ƒ‘
“
“ � |t“‚ ˆ ‹ � |€“� “� |‚“� |ƒ“� “� “� |� “ –
output_dirÚ
do_predictÚ
eval_strategyÚ
eval_delayÚ
learning_rateÚ
adam_beta1Ú
adam_beta2Ú
max_grad_normÚ
logging_stepsÚ
save_strategyÚ
save_stepsÚ
jit_mode_evalÚ
local_rankÚ
tpu_num_coresÚ
eval_stepsÚ
past_indexÚ
optim_argsÚ adafactorÚ
mp_parametersÚ
torch_compileÚ
eval_on_startÚ
max_lengthÚ
padding_valueÚ
Ú
OverflowErrorÚ › �
¸
× � › ˜ ’ ‘ � � � ‹ ˆ � � � � � � � � � �
� � �
� � � � � � � �
– � ‚ ˆ ˆ ‹ ‹ � � ‹ � ‘ ’ –
__module__Ú
__classcell__rn
�
e
e
e
e
e e
e
e
e
e
e
e
e
e
f
e
e
Z
e
e
Z#d
d
f
e e
e e
e e
e e
e e2f
e
e
e e
e e
e
e(
e
Z;d_dSe
e e2f
e2
Z<‡
e
e ‹
train_datasetÚ
data_collatorÚ
model_initÚ callbacksÚ
optimizersÚ
t‚ ‚ ‚ › � ‚
d ‚
} › � ‚
t
ƒ‚ � � �
ƒ
t
< � � � �
n� � � � � ‚ � � ‚ � � � | � � › � ‚ � � � � × � � � � � � � � � � � � � � � q ˆ � ˆ � ˆ � ˆ � � ˆ � ˆ � ˆ � ˆ � ˆ � ˆ � ˆ � � � ‚ ˆ � � ˆ � ˆ � � � ˆ � ˆ ˆ � � � � � › › › › � t,¡ � ˆ ˆ
| |
| � � � ‚ � � � ‚ � � � ‚ �
| � � � � tgdUƒ‚ � ‚ � ‚ � � ‚ � ‚ ‹
batch_sizerh ˆ ‘
<listcomp>/ ‚
use_ref_model)jÚ
ValueErrorÚ
isinstanceÚ
parametersrÿ
is_peft_modelrM
aux_loss_coefÚ
_tag_namesÚ
zero_stager?
prepare_modelrB
num_desirableÚ ´
�
� � * ’
f �
|
|
j �
|
j �
d
Returns the training [`~torch.utils.data.DataLoader`].
Subclass of transformers.src.transformers.trainer.get_train_dataloader to precompute `ref_log_probs`.
F©
collate_fnÚ
pin_memoryÚ –
add_columnrD
|
t‚ ’ ’
t �
| � � � |
| �
Returns the evaluation [`~torch.utils.data.DataLoader`].
Subclass of transformers.src.transformers.trainer.get_eval_dataloader to precompute `ref_log_probs`.
Args:
eval_dataset (`torch.utils.data.Dataset`, *optional*):
If provided, will override `self.eval_dataset`. If it is a [`~datasets.Dataset`], columns not accepted
by the `model.forward()` method are automatically removed. It must implement `__len__`.
Nz-Trainer: evaluation requires an eval_dataset.Fr¿ ˜ – ×
× � ² � P �
�
�
� �
�
�
�
d�
d� – ×
‚
Args:
logits:
Logits of the model (unnormalized). Shape: (batch_size, sequence_length, vocab_size)
labels:
Labels for which to compute the log probabilities. Label tokens with a value of label_pad_token_id are
ignored. Shape: (batch_size, sequence_length)
average_log_prob:
If True, return the average log probability per (non-masked) token. Otherwise, return the sum of the
log probabilities of the (non-masked) tokens.
Returns:
A tensor of shape (batch_size,) containing the average/sum log probabilities of the given labels under the
given logits.
NrS
ˆ ˆ ˆ ˆ ˆ ˆ
� ˆ
ƒ‚
|
|
|
|
|
|
|
| ˆ ‘
chosen_idxÚ
chosen_logitsÚ �
d
d
|
|
n
|
d
d
|
|
|
n
t
|
|
Args:
policy_chosen_logps:
Log probabilities of the policy model for the chosen responses. Shape: (num(chosen) in batch_size,)
policy_rejected_logps:
Log probabilities of the policy model for the rejected responses. Shape: (num(rejected) in batch_size,)
policy_KL_logps: Log probabilities of the policy model for the KL responses. Shape: (batch_size,)
reference_chosen_logps:
Log probabilities of the reference model for the chosen responses. Shape: (num(chosen) in batch_size,)
reference_rejected_logps:
Log probabilities of the reference model for the rejected responses. Shape: (num(rejected) in
batch_size,)
reference_KL_logps: Log probabilities of the reference model for the KL responses. Shape: (batch_size,)
Returns:
A tuple of four tensors: (losses, chosen_rewards, rejected_rewards, KL). The losses tensor contains the KTO
loss for each example in the batch. The chosen_rewards and rejected_rewards tensors contain the rewards for
the chosen and rejected responses, respectively. The KL tensor contains the detached KL divergence estimate
between the policy and reference models.
r –
chosen_lossesÚ
�
| �
z/Compute KL log probabilities for a given batch.Nrâ –
�
t
|
d � |
d
d
nCt
d
d
|
|
j
d�
j �
| �
j
d� � \ �
Compute the KTO loss using the Liger-Kernel's LigerFusedLinearKTOLoss.
Args:
model:
The policy model used for generating log probabilities and outputs. It could be an encoder-decoder
model or a regular language model.
batch: A dictionary containing the input data and labels for the batch.
Returns:
A dictionary containing the following keys:
- "loss": The computed KTO loss for the batch.
- "chosen_logits_sum": Sum of the logits for the chosen responses from the policy model.
- "rejected_logits_sum": Sum of the logits for the rejected responses from the policy model.
- "chosen_logps": Log probabilities of the chosen responses from the policy model.
- "rejected_logps": Log probabilities of the rejected responses from the policy model.
- "chosen_rewards": Rewards for the chosen responses.
- "rejected_rewards": Rewards for the rejected responses.
- "kl": The KL divergence between the policy and reference models (detached).
If auxiliary loss is enabled, the dictionary will also include:
- "aux_loss": The auxiliary loss from the model outputs.
r ˆ
lin_weightÚ
ref_weightÚ –
base_modelÚ
ˆ ˆ ˆ ˆ ˆ
|ˆ
|
|
ˆ ˆ
d
}ˆ ˆ
ˆ
ˆ ˆ ˆ ˆ
} � B ˆ
u �
d
ˆ ˆ ˆ
d
ˆ
} ˆ � ˆ
ˆ ˆ � � ˆ ˆ ˆ
ˆ � ˆ ˆ
<dictcomp>Ç – ›
num_chosenÚ
‚
t �
|�
train_eval) ‚
store_metrics)
t � ` �
d � �
j �
| �
| � ‚
policy_outputrh
ˆ
¡ � " �
|�
„ � �
|
f
Nr‘
\ˆ ‘ ‘ ‚
dataloaderÚ �
| ¡
| �
t
}
d
| �
| � �
Overriding built-in evaluation loop to store metrics for each batch. Prediction/evaluation loop, shared by
`Trainer.evaluate()` and `Trainer.predict()`.
