2025-08-28 17:57:59 +00:00
o
2025-08-28 22:41:56 +00:00
2025-08-28 17:57:59 +00:00
Z
m
Z
m
Z
m
Z
m Z m!Z!m'Z'm7Z7m=Z=mAZAm
lRm@Z@
d
d
d � �
2025.8.9
2025.8.10
4.55.4
0.21.0
__UNSLOTH_VERSIONING__
é
functional)
BCOTrainerÚ
DataLoaderÚ
has_lengthÚ
itemgetterÚ
pad_to_lengthÚ
shape_paddingz
trace.enabledz
� � �
|�
rG
Ú
chunked_indexÚ
� �
i�
S
Configuration class for the [`BCOTrainer`].
This class includes only the parameters that are specific to BCO training. For a full list of training arguments,
please refer to the [`~transformers.TrainingArguments`] documentation. Note that default values in this class may
differ from those in [`~transformers.TrainingArguments`].
Using [`~transformers.HfArgumentParser`] we can turn this class into
[argparse](https://docs.python.org/3/library/argparse#module-argparse) arguments that can be specified on the
command line.
Parameters:
max_length (`int` or `None`, *optional*, defaults to `1024`):
Maximum length of the sequences (prompt + completion) in the batch. This argument is required if you want
to use the default data collator.
max_prompt_length (`int` or `None`, *optional*, defaults to `512`):
Maximum length of the prompt. This argument is required if you want to use the default data collator.
max_completion_length (`int` or `None`, *optional*, defaults to `None`):
Maximum length of the completion. This argument is required if you want to use the default data collator
and your model is an encoder-decoder.
beta (`float`, *optional*, defaults to `0.1`):
Parameter controlling the deviation from the reference model. Higher β means less deviation from the
reference model.
label_pad_token_id (`int`, *optional*, defaults to `-100`):
Label pad token id. This argument is required if you want to use the default data collator.
padding_value (`int` or `None`, *optional*, defaults to `None`):
Padding value to use. If `None`, the padding value of the tokenizer is used.
truncation_mode (`str`, *optional*, defaults to `"keep_end"`):
Truncation mode to use when the prompt is too long. Possible values are `"keep_end"` or `"keep_start"`.
This argument is required if you want to use the default data collator.
disable_dropout (`bool`, *optional*, defaults to `True`):
Whether to disable dropout in the model and reference model.
generate_during_eval (`bool`, *optional*, defaults to `False`):
If `True`, generates and logs completions from both the model and the reference model to W&B or Comet
during evaluation.
is_encoder_decoder (`bool` or `None`, *optional*, defaults to `None`):
When using the `model_init` argument (callable) to instantiate the model instead of the `model` argument,
you need to specify if the model returned by the callable is an encoder-decoder model.
precompute_ref_log_probs (`bool`, *optional*, defaults to `False`):
Whether to precompute reference model log probabilities for training and evaluation datasets. This is
useful when training without the reference model to reduce the total GPU memory needed.
model_init_kwargs (`dict[str, Any]` or `None`, *optional*, defaults to `None`):
Keyword arguments to pass to `AutoModelForCausalLM.from_pretrained` when instantiating the model from a
string.
ref_model_init_kwargs (`dict[str, Any]` or `None`, *optional*, defaults to `None`):
Keyword arguments to pass to `AutoModelForCausalLM.from_pretrained` when instantiating the reference model
from a string.
dataset_num_proc (`int` or `None`, *optional*, defaults to `None`):
Number of processes to use for processing the dataset.
prompt_sample_size (`int`, *optional*, defaults to `1024`):
Number of prompts that are fed to density ratio classifier.
min_density_ratio (`float`, *optional*, defaults to `0.5`):
Minimum value of the density ratio. The estimated density ratio is clamped to this value.
max_density_ratio (`float`, *optional*, defaults to `10.0`):
Maximum value of the density ratio. The estimated density ratio is clamped to this value.
NÚ
?ç{
adamw_8bitÚ
every_saveÚ › � ‚ › � ‚
d –
ƒ
“
“ � |t“‚ ˆ ‹ � |€“� “� |‚“� |ƒ“� “� “� |� “ ’ ‹
output_dirÚ
do_predictÚ
eval_strategyÚ
eval_delayÚ
learning_rateÚ
adam_beta1Ú
adam_beta2Ú
max_grad_normÚ
logging_stepsÚ
save_strategyÚ
save_stepsÚ
jit_mode_evalÚ
local_rankÚ
tpu_num_coresÚ
eval_stepsÚ
past_indexÚ
optim_argsÚ adafactorÚ
mp_parametersÚ
torch_compileÚ
eval_on_startÚ
max_lengthÚ
padding_valueÚ
Ú
OverflowErrorÚ �
× � › ˜ ’ ‘ � � � ‹ ˆ � � � � � � � � � �
� � �
� � � � � �
� ‚ ˆ ‹ � � � � ‘ ’ � – ˜ – ›
__module__Ú
__classcell__rr
�
e
e
e
e
e e
e
e
e
e
e
e
e
e
f
e
e
e
e
Z
e e
e
e
Z.d5e
d
f
e e
e
e
e e
e e
e e
e e9f
e
e
e e
e e
e
e1
e
ZBdudhe
e e9f
e9
ZC‡
e
e �
train_datasetÚ
data_collatorÚ
model_initÚ callbacksÚ
optimizersÚ
t ‚ ‚ ‚ ‚
t › � ‚
d ‚
}
t › � ‚
| ‚ � �
|
ƒ
|
d
ƒ� � � � � �
|
d
ƒ� � � � � � ‚ � � ˆ ‚ � � � � � ‚ � � � � � � � � � � �
|
| � � �
|
|
|
ƒ
| � � � ’ � � � � � � � � � � �
| |
| �
| � � ‚ � � � ‚ � ˜ � � ‚ �
| � � � � � � � � � › › � ‚ � �
ValueErrorÚ
isinstanceÚ
parametersr
is_peft_modelrP
aux_loss_coefrR
_tag_namesÚ
zero_stagerB
prepare_modelr!
zeros_liker
embeddingsÚ
rejected_meanr6 ¸
� �
j �
|
j� �
zÄ
Calculates the probability if the given prompt embedding is from desirable dataset. This function calculates
the probability in the process and ensemble across processes.
)
process_indexÚ
predict_probar«
� �
Replaces processing_class.pad_token_id to embedding_tokenizer.pad_token_id and applies self.embedding_func
©
ÿ � � ¸
chosen_idxÚ �
d
| � 1 � �
|
¡
t
Sample instances from dataset and get prompt embeddings. Used for density ratio classifier training.
)
batch_sizeÚ
collate_fnÚ
pin_memoryÚ ×
þ
z0_UnslothBCOTrainer._get_sample_prompt_embeddingsc
|
j
|� � � › �
|
rAt
checkpointÚ � � * �
f �
|
j �
d
Returns the training [`~torch.utils.data.DataLoader`].
Subclass of transformers.src.transformers.trainer.get_train_dataloader to precompute `ref_log_probs`.
FrÞ
add_columnrG
|
t‚
t �
| �
�
Returns the evaluation [`~torch.utils.data.DataLoader`].
Subclass of transformers.src.transformers.trainer.get_eval_dataloader to precompute `ref_log_probs`.
Args:
eval_dataset (`torch.utils.data.Dataset`, *optional*):
If provided, will override `self.eval_dataset`. If it is a [`~datasets.Dataset`], columns not accepted
by the `model.forward()` method are automatically removed. It must implement `__len__`.
Nz-Trainer: evaluation requires an eval_dataset.FrÞ �
� h � + � � � �
| � ¸
rG
˜ ¸ ‚
Args:
logits: Logits of the model (unnormalized). Shape: (batch_size, sequence_length, vocab_size)
labels:
Labels for which to compute the log probabilities. Label tokens with a value of label_pad_token_id are
ignored. Shape: (batch_size, sequence_length)
average_log_prob:
If True, return the average log probability per (non-masked) token. Otherwise, return the sum of the
log probabilities of the (non-masked) tokens.
Returns:
A tensor of shape (batch_size,) containing the average/sum log probabilities of the given labels under the
given logits.
NrV � ¸
ˆ ˆ ˆ ˆ ˆ
� ˆ
ƒ‚
|
|
f
| ¸ ˆ ‘
<listcomp>¯ �
r4
chosen_logitsÚ �
|
�
| |
t
|
¡
|
�
|�
Args:
policy_chosen_logps:
Log probabilities of the policy model for the chosen responses. Shape: (num(chosen) in batch_size,)
policy_rejected_logps:
Log probabilities of the policy model for the rejected responses. Shape: (num(rejected) in batch_size,)
reference_chosen_logps:
Log probabilities of the reference model for the chosen responses. Shape: (num(chosen) in batch_size,)
reference_rejected_logps:
Log probabilities of the reference model for the rejected responses. Shape: (num(rejected) in
batch_size,)
chosen_embeddings: embeddings of desirable prompts
rejected_embeddings: embeddings of undesirable prompts
Returns:
A tuple of four tensors: (losses, chosen_rewards, rejected_rewards, delta). The losses tensor contains the
BCO loss for each example in the batch. The chosen_rewards and rejected_rewards tensors contain the rewards
for the chosen and rejected responses, respectively. The delta value contains the moving average of all
implicit rewards.
r
logsigmoidr¼
chosen_lossesÚ
chosen_weightÚ
ˆ
dˆ ˆ ˆ ˆ
f
ˆ
f � @ ˆ � ˆ
} � w ˆ ˆ ˆ
} ˆ
ˆ ˆ
| � ˆ
| ˆ ˆ ˆ
|
| � ˆ
|
< ˆ
| ˆ
| � ˆ
| ˆ
| ˆ
| ˆ � ˆ
ˆ ‚
<dictcomp> ˜
num_chosenÚ
‚
t �
|�
train_eval) ˆ
store_metrics) –
t � ` �
d � �
j �
| �
| � ˆ
policy_outputro
ˆ
¡ � $ � �
|�
<
�
�
|
f �
\ˆ ‘ � ˆ ¸
dataloaderÚ �
| ¡
| �
t
}
d
| �
| � �
Overriding built-in evaluation loop to store metrics for each batch. Prediction/evaluation loop, shared by
`Trainer.evaluate()` and `Trainer.predict()`.
Works both with or without labels.
) ‚ ‘ ‚ ˆ �
target_labelsÚ
start_timec
› � › � › ›
� › › › � › ›
� › � › � › � › � › � ›
� › › � ‘
Log `logs` on the various objects watching training, including stored metrics.
