owusu/DS-LLM-TEMPLATE-FINETUNING
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
78d7063ce53bfe9aca4641e833dae66e9e46b9a4
DS-LLM-TEMPLATE-FINETUNING/unsloth_compiled_cache/__pycache__/UnslothDPOTrainer.cpython-311.pyc
T

745 lines
120 KiB
Plaintext
Raw Normal View History

updated style mimciking fine tuning
2025-08-13 23:50:20 +00:00
§
3$�hãó´—dZddlmZddlZddlmZddlmZddlmZm Z m
Z
m Z m Z m
Z
mZmZddlmZmZmZmZmZmZmZmZmZmZmZmZmZmZmZmZmZmZm Z m
Z
m!Z!m"Z"m#Z#m$Z$m%Z%m&Z&m'Z'm(Z(m)Z)m*Z*m+Z+m Z m,Z,m-Z-m.Z.m/Z/m0Z0m1Z1m2Z2m3Z3m4Z4m5Z5m6Z6m7Z7m8Z8m9Z9m:Z:m;Z;m<Z<m=Z=m>Z>m?Z?m@Z@mAZAmZmBZBmCZCmDZDmEZEmFZFmGZGmHZHmIZImJZJmKZKmLZLmMZMmNZNmZmOZOmPZPmZm
Z
m$Z$m%Z%m*Z*m=Z=mCZCmZddlCZCddlTddlQm0Z0mRZRdd lSmTZTddlZddlUZVdd
lWmBZBddlmZdd lXmYZYmZZ[d d
d d
d
dœZ\ej]d d e\¬¦«d„¦«Z^e0Gd„de¦«¦«Z_ Gd„de*¦«Z`Gd„de`¦«ZadS)z8
2025.8.4
2025.8.5
4.55.1
0.21.0
__UNSLOTH_VERSIONING__
é)ÚTensorN)Ú
functional)ÚAnyÚListÚOptionalÚTupleÚUnionÚDictÚSetÚCallable)OrÚAutoModelForCausalLMÚ
AutoTokenizerÚBaseImageProcessorr Ú DPOConfigÚ
DPOTrainerÚ DataCollatorÚDataCollatorForPreferenceÚ
DataLoaderÚDatasetÚEvalLoopOutputÚFÚFDivergenceConstantsÚFDivergenceTypeÚFeatureExtractionMixinÚIterableDatasetÚLiteralÚ$MODEL_FOR_VISION_2_SEQ_MAPPING_NAMESrÚ PartialStateÚPathÚ
PeftConfigÚ PeftModelÚPreTrainedModelÚPreTrainedTokenizerBaseÚProcessorMixinÚRunningMomentsÚSyncRefModelCallbackÚTrainerÚTrainerCallbackr ÚautocastÚcap_expÚcontextmanagerÚcreate_reference_modelÚ dataclassÚ defaultdictÚdisable_dropout_in_modelÚ empty_cacheÚ
flush_leftÚ flush_rightÚgenerate_model_cardÚget_comet_experiment_urlÚget_peft_modelÚinspectÚis_comet_availableÚis_liger_kernel_availableÚis_mlflow_availableÚis_peft_availableÚis_wandb_availableÚlog_table_to_comet_experimentÚmaybe_apply_chat_templateÚmaybe_extract_promptÚnnÚ nullcontextÚosÚpadÚ
pad_to_lengthÚpdÚpeft_module_casting_to_bf16Úprepare_deepspeedÚ prepare_fsdpÚprepare_model_for_kbit_trainingÚrandomÚselective_log_softmaxÚshift_tokens_rightÚtextwrapÚtorchÚtqdmÚwarningsrrr!r"r'r:rArM)Ú*)r-Úfield)ÚVersion)r@)ÚDataCollatorForSeq2SeqÚDataCollatorForLanguageModelingTF)Úepilogue_fusionÚ max_autotuneÚ
shape_paddingz
trace.enabledztriton.cudagraphs)ÚdynamicÚ fullgraphÚoptionscó’—tj| d|jd¦«dd¬¦«}tj| d¦«dd¬¦«}g}t ||¦«D]‘\}}| tj¦«}tj|d| d¦«¬¦«  d¦«}tj
|d¬¦«}||z
} |  | ¦«Œ’ tj |¦«}| |jd|jdf¦«}|S)Néÿÿÿÿér)ÚchunksÚdim)r_Úindex©r_é)
rMÚchunkÚreshapeÚshapeÚzipÚtoÚfloat32ÚgatherÚ unsqueezeÚsqueezeÚ logsumexpÚappendÚconcat)
Úlogitsr`Úchunked_logitsÚ
chunked_indexÚall_per_token_logpsÚ chunk_logitsÚ chunk_indexÚselected_logitsÚlogsumexp_valuesÚper_token_logpss
ú]/workspace/Fine-tuning/DS-LLM-TEMPLATE-FINETUNING/unsloth_compiled_cache/UnslothDPOTrainer.pyÚchunked_selective_log_softmaxry"s5€õ”[ §¢°°F´LÀÔ4DÑ!EÔ!EÐPQÐYZÐ[Ñ[Ô[€NÝ”[ §¢¨rÑ!2Ô!2¸QÀaÐHÑHÔH€MØÐå%(¨¸Ñ%GÔ%Gð4ð4Ñ!ˆ �kØ#—’¥u¤}Ñ5Ô5ˆ Ýœ, |¸2À{×G\ÒG\Ð]_ÑG`ÔG`ÐaÑaÔa×iÒiÐjlÑmÔmˆÝ œ?¨<¸rÐBÑBÔBÐØ)Ð,<Ñ<ˆØ×"Ò" ?Ñ3Ô3Ð3Ð3ØÝœ,Ð':Ñ;Ô;ÐØ-×5Ò5°v´|ÀA´ÈÌ ÐUVÌÐ6XÑYÔYÐØ ÐócóÔ‡—eZdZUdZedddi¬¦«Zeeed<edddi¬¦«Z ee
ed < d1ˆfd0„ Z ˆxZ S)2ÚUnslothDPOConfigu6.
Configuration class for the [`DPOTrainer`].
This class includes only the parameters that are specific to DPO training. For a full list of training arguments,
please refer to the [`~transformers.TrainingArguments`] documentation. Note that default values in this class may
differ from those in [`~transformers.TrainingArguments`].
Using [`~transformers.HfArgumentParser`] we can turn this class into
[argparse](https://docs.python.org/3/library/argparse#module-argparse) arguments that can be specified on the
command line.
Parameters:
> Parameters that control the model and reference model
model_init_kwargs (`dict[str, Any]` or `None`, *optional*, defaults to `None`):
Keyword arguments for `AutoModelForCausalLM.from_pretrained`, used when the `model` argument of the
[`DPOTrainer`] is provided as a string.
ref_model_init_kwargs (`dict[str, Any]` or `None`, *optional*, defaults to `None`):
Keyword arguments for `AutoModelForCausalLM.from_pretrained`, used when the `ref_model` argument of the
[`DPOTrainer`] is provided as a string.
model_adapter_name (`str` or `None`, *optional*, defaults to `None`):
Name of the train target PEFT adapter, when using LoRA with multiple adapters.
ref_adapter_name (`str` or `None`, *optional*, defaults to `None`):
Name of the reference PEFT adapter, when using LoRA with multiple adapters.
force_use_ref_model (`bool`, *optional*, defaults to `False`):
If you provide a PEFT model as the active model and wish to use a different model for the `ref_model`, set
this flag to `True`.
disable_dropout (`bool`, *optional*, defaults to `True`):
Whether to disable dropout in the model and reference model.
use_logits_to_keep (`bool`, *optional*, defaults to `False`):
If `True`, only a specified number of logits are computed in the forward pass. This can be useful for
saving memory and speeding up training by not computing the logits for all tokens, especially in scenarios
when working with very long prompts where labels are ignored (-100).
> Parameters that control the data preprocessing
dataset_num_proc (`int` or `None`, *optional*, defaults to `None`):
Number of processes to use for processing the dataset.
padding_value (`int` or `None`, *optional*, defaults to `None`):
Padding value to use. If `None`, the padding value of the tokenizer is used.
label_pad_token_id (`int`, *optional*, defaults to `-100`):
Padding value to use for labels.
max_prompt_length (`int` or `None`, *optional*, defaults to `512`):
Maximum length of the prompt.
max_completion_length (`int` or `None`, *optional*, defaults to `None`):
Maximum length of the completion.
max_length (`int` or `None`, *optional*, defaults to `1024`):
Maximum length of the full sequence (prompt + completion).
truncation_mode (`str`, *optional*, defaults to `"keep_end"`):
Truncation mode to use when the sequence exceeds `max_length`. Possible values are `"keep_end"` and
`"keep_start"`.
padding_free (`bool`, *optional*, defaults to `False`):
Whether to perform forward passes without padding by flattening all sequences in the batch into a single
continuous sequence. This reduces memory usage by eliminating padding overhead. Currently, this is only
supported with the `flash_attention_2` attention implementation, which can efficiently handle the flattened
batch structure.
precompute_ref_log_probs (`bool`, *optional*, defaults to `False`):
Whether to precompute the log probabilities from the reference model. Setting this to `True` allows
training without needing the reference model during training, which can help reduce GPU memory usage. If
set to `False` (default), the reference model will be used during training to compute log probabilities
on-the-fly.
precompute_ref_batch_size (`int` or `None`, *optional*, defaults to `None`):
Batch size to use when precomputing reference model log probabilities. This can be set higher than the
training batch size to speed up preprocessing. If `None`, defaults to `per_device_train_batch_size` for
training and `per_device_eval_batch_size` for evaluation.
tools (`Optional[list[Union[dict, Callable]]]`, *optional*, defaults to `None`):
List of tools (callable functions) that will be accessible to the model. If the template does not support
function calling, this argument will have no effect.
> Parameters that control the training
loss_type (`str` or `list[str]`, *optional*, defaults to `"sigmoid"`):
Type of loss to use. Possible values are:
- `"sigmoid"`: sigmoid loss from the original [DPO](https://huggingface.co/papers/2305.18290) paper.
- `"hinge"`: hinge loss on the normalized likelihood from the
[SLiC](https://huggingface.co/papers/2305.10425) paper.
- `"ipo"`: IPO loss from the [IPO](https://huggingface.co/papers/2310.12036) paper.
- `"exo_pair"`: pairwise EXO loss from the [EXO](https://huggingface.co/papers/2402.00856) paper.
- `"nca_pair"`: pairwise NCA loss from the [NCA](https://huggingface.co/papers/2402.05369) paper.
- `"robust"`: unbiased estimate of the DPO loss that is robust to preference noise from the [Robust
DPO](https://huggingface.co/papers/2403.00409) paper.
- `"bco_pair"`: pairwise BCO loss from the [BCO](https://huggingface.co/papers/2404.04656) paper.
- `"sppo_hard"`: SPPO loss with hard label from the [SPPO](https://huggingface.co/papers/2405.00675)
paper.
- `"aot"`: AOT loss for paired datasets from the [AOT](https://huggingface.co/papers/2406.05882) paper.
- `"aot_pair"`: AOT loss for unpaired datasets from the [AOT](https://huggingface.co/papers/2406.05882)
paper.
- `"discopop"`: DiscoPOP (a.k.a Log-Ratio Modulated Loss, LRML) loss from the
[DiscoPOP](https://huggingface.co/papers/2406.08414) paper.
- `"apo_zero"`: APO-zero loss from the [APO](https://huggingface.co/papers/2408.06266) paper.
- `"apo_down"`: APO-down loss from the [APO](https://huggingface.co/papers/2408.06266) paper.
- `"sft"`: Negative log-likelihood loss (standard supervised fine-tuning loss).
Multiple loss types can be combined using comma separation (e.g., `["sigmoid", "bco_pair", "sft"]` for
[MPO](https://huggingface.co/papers/2411.10442)). The `loss_weights` parameter can be used to specify
corresponding weights for each loss type.
use_liger_loss (`bool`, *optional*, defaults to `False`):
Whether to use Liger loss.
base_model_attribute_name (`str`, *optional*, defaults to `"model"`):
Name of the attribute in the model that contains the base model. This is used to get the base model from
the model when the model does not have a `get_decoder` method in the case when `use_liger_loss` is `True`.
beta (`float`, *optional*, defaults to `0.1`):
Parameter controlling the deviation from the reference model. Higher β means less deviation from the
reference model. For the IPO loss (`loss_type="ipo"`), β is the regularization parameter denoted by τ in
the [paper](https://huggingface.co/papers/2310.12036).
f_divergence_type (`str`, *optional*, defaults to `FDivergenceType.REVERSE_KL`):
Type of f-divergence regularization function to compute divergence between policy and reference model.
f_alpha_divergence_coef (`float`, *optional*, defaults to `1.0`):
α coefficient in the α-divergence u^-α regularization function for DPO loss.
reference_free (`bool`, *optional*, defaults to `False`):
Whether to ignore the provided reference model and implicitly use a reference model that assigns equal
probability to all responses.
label_smoothing (`float`, *optional*, defaults to `0.0`):
Robust DPO label smoothing parameter from the [cDPO report](https://ericmitchell.ai/cdpo.pdf) and [Robust
DPO](https://huggingface.co/papers/2403.00409) paper that should be between `0.0` and `0.5`.
use_weighting (`bool`, *optional*, defaults to `False`):
Whether to weight the loss as done in the [WPO paper](https://huggingface.co/papers/2406.11827).
rpo_alpha (`float`, *optional*, defaults to `None`):
α parameter from the [RPO paper](https://huggingface.co/papers/2404.19733) (v3), which controls the
weighting of the NLL term in the loss. If `None`, no weighting is applied and the loss is the same as the
DPO loss. The paper recommends `rpo_alpha=1.0`.
ld_alpha (`float` or `None`, *optional*, defaults to `None`):
α parameter from the [LD-DPO paper](https://huggingface.co/papers/2409.06411), which controls the weighting
of the verbose token log-probabilities in responses. If `None`, no weighting is applied to the verbose
part, and the loss is equivalent to the standard DPO loss. The paper recommends setting `ld_alpha` between
`0.0` and `1.0`.
discopop_tau (`float`, *optional*, defaults to `0.05`):
Ï„/temperature parameter from the [DiscoPOP](https://huggingface.co/papers/2406.08414) paper, which controls
the shape of log ratio modulated loss. The paper recommends the default value `discopop_tau=0.05`.
loss_weights (`list[float]` or `None`, *optional*, defaults to `None`):
List of loss weights for multi-loss combinations. Used when combining multiple loss types.
Example: `[0.8, 0.2, 1.0]` for [MPO](https://huggingface.co/papers/2411.10442). If not provided, defaults
to equal weights (`1.0`) for all loss types.
sync_ref_model (`bool`, *optional*, defaults to `False`):
Whether to synchronize the reference model with the active model every `ref_model_sync_steps` steps, using
the `ref_model_mixup_alpha` parameter. This synchronization originates from the
[TR-DPO](https://huggingface.co/papers/2404.09656) paper.
ref_model_mixup_alpha (`float`, *optional*, defaults to `0.6`):
α parameter from the [TR-DPO](https://huggingface.co/papers/2404.09656) paper, which controls the mix
between the current policy and the previous reference policy during updates. The reference policy is
updated according to the equation: `π_ref = α * π_θ + (1 - α) * π_ref_prev`. To use this parameter, you
must set `sync_ref_model=True`.
ref_model_sync_steps (`int`, *optional*, defaults to `512`):
Ï„ parameter from the [TR-DPO](https://huggingface.co/papers/2404.09656) paper, which determines how
frequently the current policy is synchronized with the reference policy. To use this parameter, you must
set `sync_ref_model=True`.
> Parameters that control the logging
generate_during_eval (`bool`, *optional*, defaults to `False`):
Whether to generate and log completions from both the model and the reference model to W&B or Comet during
evaluation.
NÚhelpzvLLM SamplingParams)ÚdefaultÚmetadataÚvllm_sampling_paramsr\z8Chunk size to reduce memory usage. -1 is most efficient.Úunsloth_num_chunksFÚnor]éréúç-Cëâ6
?ç{®Gáz„?çÍÌÌÌÌÌì?ç+‡ÙÎ÷ï?ç:Œ0âŽyE>çð?ç@Úlinearçš™™™™™¹?ÚpassiveÚwarningTÚstepsrbéôéO
ÚO1ÚautoÚçÚ
adamw_8bitÚlengthÚ
every_saveÚlastééœÿÿÿééÚkeep_endÚmodelçš™™™™™©?ç333333ã?c¤ óê•—|dkrtd|›d�¦«‚|dkrtd|›d�¦«‚|€|#dkr
|$dkrd}d }#|ˆ€!d
d lm}¥t |¥¦«d zd ¦«}ˆt ¦«jd®id
|“d|“d|“d|“d|“d|“d|“d|“d| “d|
“d| “d| “d|
“d|“d|“d|“d|“d|“d|“d |“d!|“d"|“d#|“d$|“d%|“d&|“d'|“d(|“d)|“d*|“d+|“d,| “d-|!“d.|"“d/|#“d0|$“d1|%“d2|&“d3|'“d4|(“d5|)“d6|*“d7|+“d8|,“d9|-“d:|.“d;|/“d<|0“d=|1“d>|2“d?|3“d@|4“dA|5“dB|6“dC|7“dD|8“dE|9“dF|:“dG|;“dH|<“dI|=“dJ|>“dK|?“dL|@“dM|A“dN|B“dO|C“dP|D“dQ|E“dR|F“dS|G“dT|H“dU|I“dV|J“dW|K“dX|L“dY|M“dZ|N“d[|O“d\|P“d]|Q“d^|R“d_|S“d`|T“da|U“db|V“dc|W“dd|X“de|Y“df|Z“dg|[“dh|\“di|]“dj|^“dk|_“dl|`“dm|a“dn|b“do|c“dp|d“dq|e“dr|f“ds|g“dt|h“du|i“dv|j“dw|k“dx|l“dy|m“dz|n“d{|o“d||p“d}|q“d~|r“d|s“d€|t“d�|u“d‚|v“dƒ|w“d„|x“d…|y“d†|z“d‡|{“dˆ||“d‰|}“dŠ|~“d‹|“dŒ|€“d�|�“dŽ|‚“d�|ƒ“d�|„“d‘|…“d’|†“d“|‡“d”|ˆ“d•|‰“d–|Š“d—|‹“d˜|Œ“d™|�“dš|Ž“d›|�“dœ|�“d�|‘“dž|’“dŸ|““d |”“d¡|•“d¢|–“d£|—“d¤|˜“d¥|™“d¦|š“d§|›“d¨|œ“d©|�“dª|ž“d«|Ÿ“d¬| “d­|¡“|¤¤Ž|¢|_|£|_dS)¯NgH¯¼šò×z>z Unsloth: Your learning rate of `zi` is too small and less than 1e-7! Consider increasing it, otherwise gradient updates will be close to 0!rbza` is way too larger > 1! Consider decreasing it to 1e-1, otherwise gradient updates will explode!r�r‘Úunsloth_training_checkpointsr‚r)Ú cpu_countrƒÚ
output_dirÚoverwrite_output_dirÚdo_trainÚdo_evalÚ
do_predictÚ
eval_strategyÚprediction_loss_onlyÚper_device_train_batch_sizeÚper_device_eval_batch_sizeÚper_gpu_train_batch_sizeÚper_gpu_eval_batch_sizeÚgradient_accumulation_stepsÚeval_accumulation_stepsÚ
eval_delayÚtorch_empty_cache_stepsÚ
learning_rateÚ weight_decayÚ
adam_beta1Ú
adam_beta2Ú adam_epsilonÚ
max_grad_normÚnum_train_epochsÚ max_stepsÚlr_scheduler_typeÚ warmup_ratioÚ warmup_stepsÚ log_levelÚlog_level_replicaÚlog_on_each_nodeÚ logging_dirÚlogging_strategyÚlogging_first_stepÚ
logging_stepsÚlogging_nan_inf_filterÚ
save_strategyÚ
save_stepsÚsave_total_limitÚsave_safetensorsÚsave_on_each_nodeÚsave_only_modelÚ'restore_callback_states_from_checkpointÚno_cudaÚuse_cpuÚuse_mps_deviceÚseedÚ data_seedÚ
jit_mode_evalÚuse_ipexÚbf16Úfp16Úfp16_opt_levelÚhalf_precision_backendÚbf16_full_evalÚfp16_full_evalÚtf32Ú
local_rankÚ ddp_backendÚ
tpu_num_coresÚtpu_metrics_debugÚdebugÚdataloader_drop_lastÚ
eval_stepsÚdataloader_num_workersÚdataloader_prefetch_factorÚ
past_indexÚrun_nameÚ disable_tqdmÚremove_unused_columnsÚ label_namesÚload_best_model_at_endÚmetric_for_best_modelÚgreater_is_betterÚignore_data_skipÚfsdpÚfsdp_min_num_paramsÚ fsdp_configÚ"fsdp_transformer_layer_cls_to_wrapÚaccelerator_configÚ deepspeedÚlabel_smoothing_factorÚoptimÚ
optim_argsÚ adafactorÚgroup_by_lengthÚlength_column_nameÚ report_toÚddp_find_unused_parametersÚddp_bucket_cap_mbÚddp_broadcast_buffersÚdataloader_pin_memoryÚdataloader_persistent_workersÚskip_memory_metricsÚuse_legacy_prediction_loopÚ push_to_hubÚresume_from_checkpointÚ hub_model_idÚ hub_strategyÚ hub_tokenÚhub_private_repoÚhub_always_pushÚ hub_revisionÚgradient_checkpointingÚgradient_checkpointing_kwargsÚinclude_inputs_for_metricsÚeval_do_concat_batchesÚ fp16_backendÚpush_to_hub_model_idÚpush_to_hub_organizationÚpush_to_hub_tokenÚ
mp_parametersÚauto_find_batch_sizeÚfull_determinismÚ torchdynamoÚ ray_scopeÚ ddp_timeoutÚ
torch_compileÚtorch_compile_backendÚtorch_compile_modeÚinclude_tokens_per_secondÚinclude_num_input_tokens_seenÚneftune_noise_alphaÚoptim_target_modulesÚbatch_eval_metricsÚ
eval_on_startÚuse_liger_kernelÚliger_kernel_configÚeval_use_gather_objectÚaverage_tokens_across_devicesÚmodel_init_kwargsÚref_model_init_kwargsÚmodel_adapter_nameÚref_adapter_nameÚforce_use_ref_modelÚdisable_dropoutÚuse_logits_to_keepÚdataset_num_procÚ
padding_valueÚlabel_pad_token_idÚmax_prompt_lengthÚmax_completion_lengthÚ
max_lengthÚtruncation_modeÚ padding_freeÚprecompute_ref_log_probsÚprecompute_ref_batch_sizeÚtoolsÚuse_liger_lossÚbase_model_attribute_nameÚbetaÚf_alpha_divergence_coefÚreference_freeÚlabel_smoothingÚ
use_weightingÚ rpo_alphaÚld_alphaÚ discopop_tauÚ loss_weightsÚsync_ref_modelÚref_model_mixup_alphaÚref_model_sync_stepsÚgenerate_during_eval©) ÚFloatingPointErrorÚ
OverflowErrorÚmultiprocessingr¥ÚminÚsuperÚ__init__r€r�)§Úselfr¦r§r¨r©rªr«r¬r­r®r¯r°r±r²r³r´rµr¶r·r¸r¹rºr»r¼r½r¾r¿rÀrÁrÂrÃrÄrÅrÆrÇrÈrÉrÊrËrÌrÍrÎrÏrÐrÑrÒrÓrÔrÕrÖr×rØrÙrÚrÛrÜrÝrÞrßràrárârãrärårærçrèrérêrërìrírîrïrðrñròrórôrõrör÷rørùrúrûrürýrþrÿrrrrrrrrrr r
r r r
rrrrrrrrrrrrrrrrrrr r!r"r#r$r%r&r'r(r)r*r+r,r-r.r/r0r1r2r3r4r5r6r7r8r9r:r;r<r=r>r?r@rArBrCrDrErFr€r�Úkwargsr¥Ú __class__s§ €rxrMzUnslothDPOConfig.__init__ÚsM ø€ðN ˜4Ò Ð Õ'9ð;VÐ]jð;Vð;Vð;Vñ(Wô(Wð"WØ ˜1Ò Ð ¥Mð3FÐUbð3Fð3Fð3Fñ%Gô%GðGØ Ð  -°7Ò":Ð":¸zÈSÒ?PÐ?PØ7ˆJØ ˆMØ Ð #Ø 1Ð 1Ð 1Ð 1Ð 1Ð 1Ý" 9 9¡;¤;¨q¡=°!Ñ4Ô4Ð à�‰ŒÔða Bða Bða BØ#˜ða Bà#7Ð#7ða Bð �xða Bð�gð a Bð
$˜ð a Bð *˜Mð
a Bð$8Ð#7ða Bð+FÐ*Eða Bð*DÐ)Cða Bð(@Ð'?ða Bð'>Ð&=ða Bð+FÐ*Eða Bð'>Ð&=ða Bð$˜ða Bð'>Ð&=ða Bð *˜Mð!a Bð"(˜<ð#a Bð$$˜ð%a Bð&$˜ð'a Bð((˜<ð)a Bð**˜Mð+a Bð, 0Ð/ð-a Bð."˜ ð/a Bð0!2Ð 1ð1a Bð2(˜<ð3a Bð4(˜<ð5a Bð6"˜ ð7a Bð8!2Ð 1ð9a Bð: 0Ð/ð;a Bð<&˜+ð=a Bð> 0Ð/ð?a Bð@"4Ð!3ðAa BðB*˜MðCa BðD&<Ð%;ðEa BðF*˜MðGa BðH$˜ðIa BðJ 0Ð/ðKa BðL 0Ð/ðMa BðN!2Ð 1ðOa BðP.˜oðQa BðR7^Ð6]ðSa BðT�gðUa BðV�gðWa BðX,˜^ðYa BðZ�4ð[a Bð\"˜ ð]a Bð^*˜Mð_a Bð` �xðaa Bðb�4ðca Bðd�4ðea Bðf,˜^ðga Bðh&<Ð%;ðia Bðj,˜^ðka Bðl,˜^ðma Bðn�4ðoa Bðp$˜ðqa Bðr&˜+ðsa Bðt*˜Mðua Bðv!2Ð 1ðwa Bðx�Eðya Bðz$8Ð#7ð{a Bð|$˜ð}a Bð~&<Ð%;ða Bð@*DÐ)CðAa BðB$˜ðCa BðD �xðEa BðF(˜<ðGa BðH%:Ð$9ðIa BðJ&˜+ðKa BðL&<Ð%;ðMa BðN%:Ð$9ðOa BðP!2Ð 1ðQa BðR 0Ð/ðSa BðT�4ðUa BðV#6Ð"5ðWa BðX&˜+ðYa BðZ2TÐ1Sð[a Bð\"4Ð!3ð]a Bð^"˜ ð_a Bð`&<Ð%;ðaa Bðb�Eðca Bðd$˜ðea Bðf"˜ ðga Bðh.˜oðia Bðj"4Ð!3ðka Bðl"˜ ðma Bðn*DÐ)Cðoa Bðp!2Ð 1ðqa Bðr%:Ð$9ðsa Bðt%:Ð$9ðua Bðv-JÐ,Iðwa Bðx#6Ð"5ðya Bðz*DÐ)Cð{a Bð|&˜+ð}a Bð~&<Ð%;ða Bð@(˜<ðAa BðB(˜<ðCa BðD"˜ ðEa BðF 0Ð/ðGa BðH.˜oðIa BðJ(˜<ðKa BðL&<Ð%;ðMa BðN-JÐ,IðOa BðP*DÐ)CðQa BðR&<Ð%;ðSa BðT(˜<ðUa BðV$8Ð#7ðWa BðX(@Ð'?ðYa BðZ!2Ð 1ð[a Bð\*˜Mð]a Bð^$8Ð#7ð_a Bð` 0Ð/ðaa Bðb&˜+ðca Bðd"˜ ðea Bðf&˜+ðga Bðh*˜Mðia Bðj%:Ð$9ðka Bðl"4Ð!3ðma Bðn)BÐ(Aðoa Bðp-JÐ,Iðqa Bðr#6Ð"5ðsa Bðt$8Ð#7ðua Bðv"4Ð!3ðwa Bðx*˜Mðya Bðz 0Ð/ð{a Bð|#6Ð"5ð}a Bð~&<Ð%;ða Bð@-JÐ,IðAa BðB!2Ð 1ðCa BðD%:Ð$9ðEa BðF"4Ð!3ðGa BðH 0Ð/ðIa BðJ#6Ð"5ðKa BðL.˜oðMa BðN"4Ð!3ðOa BðP 0Ð/ðQa BðR*˜MðSa BðT"4Ð!3ðUa BðV!2Ð 1ðWa BðX%:Ð$9ðYa BðZ$˜ð[a Bð\.˜oð]a Bð^(˜<ð_a Bð`(@Ð'?ðaa Bðb)BÐ(Aðca Bðd�Eðea Bðf,˜^ðga Bðh)BÐ(Aðia Bðj�4ðka Bðl'>Ð&=ðma Bðn,˜^ðoa Bðp.˜oðqa Bðr*˜Mðsa Bðt"˜ ðua Bðv �xðwa Bðx(˜<ðya Bðz(˜<ð{a Bð|,˜^ð}a Bð~%:Ð$9ða Bð@$8Ð#7ðAa BðB$8Ð#7¸&ðCa Bða Bða BðD%9ˆÔ!Ø"4ˆÔÐÐrz)£NNFFFr‚Fr]r]NNrƒrƒrr„r…r†r‡rˆr‰rŠr‹r\rŒr�rrŽr�TNr�FrbFr�r‘NTFFFFFFr’r’FFFFr“r”FFNr\NNFr•FNrNr\NNTNFNNFr•rNNNNr–r—NFFr˜NNNNTFTFFNNr™NNFNFNFTr”NNNr•TFNršr›FNNFFNNFFFNFTNNNNFTFNNrœr�NržrŸFFNNFr r�rŠFr–FNNr¡NFr¢r�FNr\)
Ú__name__Ú
__module__Ú __qualname__Ú__doc__rQr€rrÚ__annotations__r�ÚintrMÚ
__classcell__©rPs@rxr|r|3s¶ø€€€€€€ð\ð\ðz+0¨%ØØÐ1Ð2ð+ñ+ô+И( 3œ-ððñð*/¨ØØÐVÐWð*ñ*ô*И #œððñð Ø#ØØØØØ$Ø&'Ø%&Ø#'Ø"&Ø&'Ø"#ØØ"%ØØØØØØØØØ$ØØØØ%ØØØ"Ø"ØØ!&ØØØØØ!ØØ27ØØØØØØØØØØØ!'ØØØØØØØ!ØØ$ØØ!"Ø%)ØØØØ $ØØ!&Ø $Ø Ø ØØØØ-1Ø!ØØ!$ØØØØØ%ØØ%)Ø Ø $Ø $Ø(-Ø"Ø%*ØØ!%ØØ#ØØØØØ!&Ø(,Ø%*Ø!%ØØ#Ø#'Ø ØØ#Ø ØØØØØ $Ø!Ø$)Ø(-Ø"Ø#Ø"ØØ Ø"Ø!&Ø(,Ø Ø $Ø!ØØ#ØØ"ØØØ!ØØ $ØØ$ØØ#(Ø$(ØØØ$+ØØ"%ØØØØØØØØØ #Ø"Ø$Ø#ØðIS5ðS5ðS5ðS5ðS5ðS5ðS5ðS5ðS5ðS5rzr|cóº ‡—eZdZdZddgZ dQdeeeje fde
ee ejefde
e d e
e d
e
ee
efd e
ee
eeeee
efffd e
eeeeefd
e
eegefde
eedee
ejje
ejjjfde
eeejjeeeffde
eej ej gej fde
dfˆfd„
Z!dRdede de"de fd„Z#de de dede de f
d„Z$de de fd„Z%dee
efd eeeeefde dedee
eff
d„Z&e' dSd eeefd ed!e
e(d"e
e(d#e"deeee(ff d$„¦«Z)e' dSd eeefd ed!e
e(d"e
e(d#e"deeee(ff d%„¦«Z*d&„Z+de,fˆfd'„ Z-dTd e
e
de,fˆfd(„
Z.e/d)„¦«Z0d*eeej1fdeej ej ffd+„Z2e'd*eeeeej1ffd,e(deeej1ffd-„¦«Z3 dUd/ej4d0ej4d1ej4d2ej4d3ed4eeej4fdeej4ej4ej4ffd5„Z5dejd*eeeeej1ffdeeej ffd6„Z6 dRdejd*eeeeej1ffd7e"deeej ffd8„Z7 dVdee ejfd*eeeeej1ffd:e8d;deej eee9fffd<„Z: dWdee ejfd=eeeej effdeej eej eee9ffffd>„Z;d*eeej1fdeeeffd?„Z< dTdee ejfd=eeeej effd@e"dAe
eedeej e
ej e
ej ff
dB„Z=dVdCeee9fd:e8d;ddfdD„Z> dXdFe,dGed@e
e"dAe
eedHedef ˆfdI„
Z?dTdJeee9fdKe
e9ddfˆfdL„
Z@ˆfdM„ZA dYdNe
ede
edOeeeedffdP„ZBˆxZCS)ZÚ_UnslothDPOTrainerr•ÚtrlÚdpoN©NNr Ú ref_modelÚargsÚ
data_collatorÚ
train_datasetÚ eval_datasetÚprocessing_classÚcompute_metricsÚ callbacksÚ
optimizersÚoptimizer_cls_and_kwargsÚpreprocess_logits_for_metricsÚ peft_configr c
óÆ•‡‡‡—t|t¦«r|n |jj}‰€-| d¦«d}t |›d�¦«Š‰€t
j|¦«Š‰j�
‰j‰_nat‰d¦«r‰j
�
‰j
‰_n=t‰d¦«r‰j j
�‰j j
‰_ntd¦«‚t|t¦«s||urtd¦«‚‰j
�)t|t¦«stjd¦«t|t¦«r‰ |‰¦«}‰j�)t|t¦«stjd ¦«t|t¦«r‰ |‰d
¬ ¦«}‰ |||
‰¦«}‰jr9t)¦«s+t+¦«st-¦«std ¦«‚|jj‰_|jjt3j¦«v‰_t9¦«ot|t:¦«‰_‰j‰_‰j ‰_ ‰j!‰_!|r|‰_"n*‰js‰j#rd‰_"ntI|¦«‰_"‰j%r*tM|¦«‰j"�tM‰j"¦«‰j'r_tQ¦«stSd
¦«‚‰j*dkrtd¦«‚tW‰j,‰j-‰j! d¬¦«‰_.d
|j/d<|€ta‰j¬¦«}‰j‰_‰j,‰_,‰j1‰_1‰j2‰_2‰j3‰_3‰j4‰_4‰j#‰_#‰j5‰_5‰j6rC|jj7dkrtjd¦«‰j8dkrtjd¦«‰j6‰_6d‰_9d‰_:‰j-‰_-‰j;‰_;t‰j*tx¦«r‰j*n‰j*g‰_*‰j=‰_=t}|jdd¦«‰_?‰j@‰_@t}|jdd¦«‰_A‰j?r%‰jAdkrtjdt„¦«‰j*D]D}|dvr)‰j;dkrtjd|›d�t„¦«|d krtd!¦«‚ŒEt‡d"„¦«‰_D‰jE‰_EtŒjG‰jHi‰_I‰jJ‰_J‰ K|‰‰d#¦«}|�Pt|t˜¦«r#ˆˆˆfd$„| M¦«D¦«}n‰ K|‰‰d%¦«}t�¦« O|‰|||‰|| |
| | ¬&¦ « d‰_Pt‰jQd'¦«r‰jQ R‰jS¦«t‰d(¦«st©d)¦«‚‰jUr0‰jVjWjXjYd*kr‰j#rtd+¦«‚‰j"€4‰js‰j#std,¦«‚‰jZrtd-¦«‚nt‰jUr t·‰j"‰jV¦«‰_"nM‰j\r t»‰j"‰jV¦«‰_"n&‰jV ^‰j"d
¬.¦«‰_"‰jZrD‰j#rtd/¦«‚‰ _tÁ‰j"‰jV¬0¦«¦«d1‰j*vrtÉjV¦«‰_bdSdS)2Nú/r\z-DPOÚ pad_token_idÚ tokenizera`padding_value` is not specified in `DPOConfig`, and `pad_token_id` is missing in the `processing_class`. Please either set the `padding_value` argument in `DPOConfig`, or set `tokenizer.pad_token` (e.g., `tokenizer.pad_token = tokenizer.eos_token`) before instantiating the trainer.zœ`model` and `ref_model` cannot be the same object. If you want `ref_model` to be the same as `model`, you must mass a copy of it, or `None` if you use peft.z�You passed model_init_kwargs to the `DPOConfig`, but your model is already instantiated. The `model_init_kwargs` will be ignored.z�You passed ref_model_init_kwargs to the `DPOConfig`, but your ref_model is already instantiated. The `ref_model_init_kwargs` will be ignored.T)Úis_refz”`generate_during_eval=True` requires Weights and Biases, MLFlow or Comet to be installed. Please install `wandb`, `mlflow` or `comet-ml` to resolve.z‚You set `use_liger_loss=True` but the liger kernel is not available. Please install liger-kernel first: `pip install liger-kernel`Úsigmoidz{You set `use_liger_loss=True` but the loss type is not `sigmoid`. Please set `loss_type='sigmoid'` to use the liger kernel.F)Ú ignore_indexr:Ú
use_ref_modelÚaverage_log_probÚestimate_tokens)rlÚflash_attention_2açPadding-free training is enabled, but the attention implementation is not set to 'flash_attention_2'. Padding-free training flattens batches into a single sequence, and 'flash_attention_2' is the only known attention mechanism that reliably supports this. Using other implementations may lead to unexpected behavior. To ensure compatibility, set `attn_implementation='flash_attention_2'` in the model configuration, or verify that your attention mechanism can handle flattened sequences.rbzÎYou are using a per_device_train_batch_size of 1 with padding-free training. Using a batch size of 1 anihilate the benefits of padding-free training. Please consider increasing the batch size to at least 2.Úoutput_router_logitsÚrouter_aux_loss_coefr–a-You set `output_router_logits` to `True` in the model config, but `router_aux_loss_coef` is set to `0.0`, meaning the auxiliary loss will not be used. Either set `router_aux_loss_coef` to a value greater than `0.0`, or set `output_router_logits` to `False` if you don't want to use the auxiliary loss.)ÚhingeÚipoÚbco_pairÚ sppo_hardÚnca_pairÚapo_zeroÚapo_downrzYou are using the z™ loss type that does not support label smoothing. The `label_smoothing` parameter will be ignored. Set `label_smoothing` to `0.0` to remove this warning.Úkto_pairzKSupport for kto_pair has been removed in DPOTrainer. Please use KTOTrainer.có*—tt¦«S©N)r.ÚlistrGrzrxú<lambda>z-_UnslothDPOTrainer.__init__.<locals>.<lambda>ðs€µ;½tÑ3DÔ3D€rzÚtrainc óH•—i|]\}}|‰ |‰‰|¦«“ŒSrG)Ú_prepare_dataset)Ú.0ÚkeyÚdatasetr_rcrNs €€€rxú
<dictcomp>z/_UnslothDPOTrainer.__init__.<locals>.<dictcomp>ùsEø€ð ð ð á$˜˜Wð˜×.Ò.¨wÐ8HÈ$ÐPSÑTÔTð ð ð rzÚeval) r r_r`rarbrcrdrerfrgrhÚadd_model_tagsÚ acceleratorzXYour `Trainer` does not have an `accelerator` object. Consider upgrading `transformers`.ézrYou cannot use `precompute_ref_log_probs=True` with Deepspeed ZeRO-3. Please set `precompute_ref_log_probs=False`.z]No reference model and model is not a Peft model. Try setting `precompute_ref_log_probs=True`zYYou currently cannot use `ref_model=None` with TR-DPO method. Please provide `ref_model`.)Úevaluation_modezoYou cannot use `precompute_ref_log_probs=True` with TR-DPO method. Please set `precompute_ref_log_probs=False`.)r^rŒry)cÚ
isinstanceÚstrÚconfigÚ
_name_or_pathÚsplitrrÚfrom_pretrainedr.ÚhasattrrlrmÚ
ValueErrorr&rOÚwarnÚ_create_model_from_pathr'Ú_prepare_peft_modelrFr;r7r9Úis_encoder_decoderÚ
model_typerÚkeysÚis_vision_modelr:r!Ú
is_peft_modelr(r)r<r^r5r,r+r/r8r8Ú ImportErrorÚ loss_typeÚLigerFusedLinearDPOLossr/r:Ú dpo_loss_fnÚwarnings_issuedrr0r1r2r3r,r4Ú_attn_implementationr­Ú _precomputed_train_ref_log_probsÚ_precomputed_eval_ref_log_probsr=r�rBÚgetattrÚaux_loss_enabledr>Ú
aux_loss_coefÚ UserWarningr.Ú_stored_metricsÚf_divergence_typerÚALPHA_DIVERGENCE_COEF_KEYr;Úf_divergence_paramsr-r…ÚdictÚitemsrLrMÚmodel_accepts_loss_kwargsr r‹Ú
_tag_namesÚAttributeErrorÚis_deepspeed_enabledrŒÚstateÚdeepspeed_pluginÚ
zero_stagerCrFÚis_fsdp_enabledrGÚ
prepare_modelÚ add_callbackr&r%Úrunning)rNr r^r_r`rarbrcrdrerfrgrhriÚmodel_idÚ
model_namer rPs` ` ` €rxrMz_UnslothDPOTrainer.__init__5sìøøøø€õ&' u­cÑ2Ô2ÐR�5�5¸¼ Ô8RˆØ ˆ<Ø!Ÿš¨Ñ,Ô,¨RÔ0ˆJÝ 
Ð0Ð0Ð0Ñ1Ô1ˆDð Ð #Ý,Ô<¸XÑFÔFÐ à Ô Ð )Ø!%Ô!3ˆDÔ Ð åÐ'¨Ñ8Ô8ð
Ð=MÔ=ZÐ=fØ%5Ô%B�Ô"Ð"ÝÐ)¨;Ñ7Ô7ð
Ð<LÔ<VÔ<cÐ<oØ%5Ô%?Ô%L�Ô"Ð"å ð#ñôðõ˜%¥Ñ%Ô%ð ¨)°uÐ*<Ð*<ÝðZñôð
ð
Ô !Ð -µjÀÍÑ6LÔ6LÐ -Ý ŒMð;ñ
ô
ð
õ �e�SÑ !Ô !ð >Ø×0Ò0°¸Ñ=Ô=ˆEà Ô %Ð 1½*ÀYÕPSÑ:TÔ:TÐ 1Ý ŒMð?ñ
ô
ð
õ �i¥Ñ %Ô %ð SØ×4Ò4°YÀÈTÐ4ÑRÔRˆIð×(Ò(¨° ¸;ÈÑMÔMˆà Ô $ð Õ.@Ñ.BÔ.Bð ÕFXÑFZÔFZð Õ^qÑ^sÔ^sð ÝðNñôð
ð
#(¤,Ô"AˆÔØ$œ|Ô6Õ:^Ô:cÑ:eÔ:eÐeˆÔÝ.Ñ0Ô0ÐQµZÀÅyÑ5QÔ5QˆÔØ"&Ô"9ˆÔØ $Ô 5ˆÔØ"Ô1ˆÔà ð ;Ø&ˆDŒNˆNØ
Ô
ð ; 4Ô#@ð ;à!ˆDŒNˆNå3°EÑ:Ô:ˆDŒNð Ô ð 9Ý $ UÑ +Ô +Ð +ØŒ~Ð)Ý(¨¬Ñ8Ô8Ð8ð Ô ð Ý,Ñ.Ô.ð
Ý!ðTñôððŒ~ Ò*Ð*Ý ðPñôðõ 7Ø!Ô4Ø”YØ"&Ô"5Ð5Ø!&ð  ñ ô ˆDÔ ð48ˆÔÐ/Ñ0ð Ð Ý5À4ÔCUÐVÑVÔVˆMà$(Ô$=ˆÔ!Ø"&Ô"9ˆÔØ!%Ô!7ˆÔØ%)Ô%?ˆÔ"Øœ/ˆŒØ#Ô3ˆÔØ(,Ô(EˆÔ%Ø"&Ô"9ˆÔà Ô ð ØŒ|Ô0Ð4GÒGÐGÝ”
ðJñôððÔ/°1Ò4Ð4Ý”
ð%ñôðð
!Ô-ˆÔð16ˆÔ-Ø/4ˆÔ,à”IˆŒ Ø#Ô3ˆÔÝ+5°d´nÅdÑ+KÔ+KÐa˜œ˜ÐRVÔR`ÐQaˆŒØ Ô-ˆÔÝ '¨¬ Ð6LÈeÑ TÔ TˆÔØ!Ô/ˆÔÝ$ U¤\Ð3IÈ3ÑOÔOˆÔØ Ô ð  TÔ%7¸3Ò%>Ð%>Ý ŒMðõñ 
ô
ð
ðœð pð pˆIàÐjÐjÐjØÔ(¨1Ò,Ð,å”
ðz¨ðzðzðzåñôðð
˜JÒ&Ð&Ý Ð!nÑoÔoÐoð'õ +Ð+DÐ+DÑEÔEˆÔØ!%Ô!7ˆÔÝ$8Ô$RÐTXÔTpÐ#qˆÔ Ø $Ô 5ˆÔð×-Ò-¨mÐ=MÈtÐU\Ñ]Ô]ˆ
Ø Ð #ݘ,­Ñ-Ô-ð
cð ð ð ð ð ð à(4×(:Ò(:Ñ(<Ô(<ð ñ ô � � ð
$×4Ò4°\ÐCSÐUYÐ[aÑbÔb� å
‰Œ×ÒØØØ'Ø'Ø%Ø-Ø+ØØ!Ø%=Ø*Gð ñ
ô
ð
ð"*/ˆÔ&õ �4”:Ð/Ñ 0Ô 0ð 7Ø ŒJ× %Ò % d¤oÑ 6Ô 6Ð 6å�t˜]Ñ+Ô+ð Ý Øjñôð
ð
Ô $ð ØÔÔ%Ô6ÔAÀQÒFÐFÈ4ÔKhÐFÝ ðIñôðð Œ>Ð !ØÔ&ð
¨$Ô*Gð
Ý ØsñôððÔ"ð
Ý Øoñôðð
ð
Ô(ð
fÝ!2°4´>À4ÔCSÑ!TÔ!T�”�ØÔ%ð
fÝ!-¨d¬n¸dÔ>NÑ!OÔ!O�”�à!%Ô!1×!?Ò!?ÀÄÐ`dÐ!?Ñ!eÔ!e�”à Ô ð lØÔ,ð
Ý ðFñôðð
× Ò Õ2¸T¼^ÐY]ÔYiÐjÑjÔjÑ kÔ kÐ kà ˜œÐ 'Ð 'Ý)¨$Ô*:Ñ;Ô;ˆDŒLˆLˆLð (Ð 'rzFÚ
model_pathrnÚreturncóJ—|s
|jpi}n |jpi}| d¦«}t|tj¦«s|dks|€nCt|t ¦«rtt|¦«}||d<ntd|›d�¦«‚tj
|fi|¤Ž}|S)z0Creates a model from a path or model identifier.Ú torch_dtyper”NzˆInvalid `torch_dtype` passed to `DPOConfig`. Expected either 'auto' or a string representing a `torch.dtype` (e.g., 'float32'), but got ú.) r&r'Úgetr�rMÚdtyper�r§r–r
r”)rNr¾r_rnr&rÁr s rxr˜z*_UnslothDPOTrainer._create_model_from_path?sá€àð AØ $Ô 6Ð <¸"Ð Ð à $Ô :Ð @¸bÐ ð(×+Ò+¨MÑ:Ô:ˆ Ý �k¥5¤;Ñ /Ô /ð °;À&Ò3HÐ3HÈKÐL_Ø Ý
˜ ¥SÑ
)Ô
)ð Ý!¥%¨Ñ5Ô5ˆKØ/:Ð ˜mÑ ,Ð ,åðMØ>IðMðMðMñôð
õ%Ô4°ZÐUÐUÐCTÐUÐUˆØˆ rzcóÒ—d|_t¦«s|�td¦«‚t¦«�r|��t|t¦«r| ¦«}|�|jstd¦«‚t|dd¦«st|dd¦«r`t|d¦«o,dttj t¦«j
¦«v}d|ji}|r
|j|d<t|fi|¤Ž}n| ||¦«}t#||¦«}|jr't|dd¦«rt'|¦«d |_n| ||¦«}|S)
z#Prepares a model for PEFT training.FNzvPEFT is not installed and you passed a `peft_config` in the trainer's kwargs, please install it to use the PEFT modelsa8You passed both a ref_model and a peft_config. For training PEFT adapters with DPO there is no need to pass a reference model. Please pass `ref_model=None` in case you want to train PEFT adapters, or pass a ref_model with `force_use_ref_model=True` in DPOTrainer's init. if you want to use a different ref_model.Úis_loaded_in_8bitÚis_loaded_in_4bitr Úuse_gradient_checkpointingT)Ú_peft_has_been_casted_to_bf16r:r–r�r!Úmerge_and_unloadr*r§r•r�r6Ú signaturerHÚ
parametersr r Ú_prepare_gradient_checkpointingr5rÖrE)rNr r^rir_Ú_support_gc_kwargsÚprepare_model_kwargss rxr™z&_UnslothDPOTrainer._prepare_peft_modelZs€ð .3ˆÔ*å Ñ"Ô"ð) F {Ð'>ÝðIñôð
õÑ
Ô
ñ% F [Ñ%<å˜%¥Ñ+Ô+ð
1Ø×.Ò.Ñ0Ô0�àÐ$¨TÔ-EÐ$Ý ðAñôðõ �uÐ1°5Ñ9Ô9ð
J½WÀUÐL_ÐafÑ=gÔ=gð
JÝ%,ØÐ9ñ&ô&ð&à5½ÝÔ%Õ&EÑFÔFÔQñ:ô:ðð#ð )EÀdÔFaÐ'bÐ$à%ðoØLPÔLnÐ(Ð)HÑIå7¸ÐVÐVÐAUÐVÐV��ð×<Ò<¸UÀDÑIÔI�õ# 5¨+Ñ6Ô6ˆEØŒyð
:�W UÐ,?ÀÑGÔGð
:Ý+¨EÑ2Ô2Ð2à59�Ô2øð×8Ò8¸ÀÑEÔEˆEàˆ rzcó²—|jrOt|d¦«r| ¦«n*d„}| ¦« |¦«|S)z4Prepare the gradienting checkpointing for the model.Úenable_input_require_gradscó0—| d¦«dS)NT)Úrequires_grad_)ÚmoduleÚinputÚoutputs rxÚmake_inputs_require_gradzT_UnslothDPOTrainer._prepare_gradient_checkpointing.<locals>.make_inputs_require_gradšs€Ø×)Ò)¨$Ñ/Ô/Ð/Ð/Ð/rz)r r•rÑÚget_input_embeddingsÚregister_forward_hook)rNr r_r×s rxrÍz2_UnslothDPOTrainer._prepare_gradient_checkpointing�sp€ð
Ô &ð ]å�uÐ:Ñ;Ô;ð
]Ø×0Ò0Ñ2Ô2Ð2Ð2ð0ð0ð0ð×*Ò*Ñ,Ô,×BÒBÐC[Ñ\Ô\Ð\àˆ rzrˆÚ dataset_namec
ó@—i}t|t¦«r|j|d<d|d<t¦« ¦«5t|t¦«r d|›d�|d<|jt fi|¤Ž}t|t¦«r d|›d�|d<|jtfd||jd œi|¤Ž}t|t¦«r d
|›d�|d<|j|j s|j
n|j fd d g||j |j
d
dœdœ|¤Ž}ddd¦«n #1swxYwY|S)NÚnum_procé
Úwriter_batch_sizezExtracting prompt in z datasetÚdesczApplying chat template to Ú fn_kwargs)rmr7z Tokenizing ÚchosenÚrejectedF)rcr0r1Úadd_special_tokens)Úremove_columnsrà)r�rr-rÚmain_process_firstÚmapr>r=r7r�Ú tokenize_rowÚ process_rowr0r1)rNrˆrcr_rÚÚ
map_kwargss rxr…z#_UnslothDPOTrainer._prepare_dataset¡sÓ€ðˆ
Ý �g�wÑ 'Ô 'ð 1Ø%)Ô%:ˆJ�zÑ "Ø.0ˆJÐ*Ñ +å
‰^Œ^×
.Ò
.Ñ
0Ô
0ð ð å˜'¥7Ñ+Ô+ð
TØ%S¸\Ð%SÐ%SÐ%S�
˜6Ñ"Ø!�g”kÕ"6ÐEÐE¸*ÐEÐEˆGõ˜'¥7Ñ+Ô+ð
YØ%XÀ,Ð%XÐ%XÐ%X�
˜6Ñ"Ø!�g”kÝ)ððØCSÐ^bÔ^hÐ5iÐ5iðØmwððˆGõ
˜'¥7Ñ+Ô+ð
JØ%I°<Ð%IÐ%IÐ%I�
˜6Ñ"à!�g”kØ)-Ô)=ÐS�Ô!Ð!À4ÔCSð à (¨*Ð5à(8Ø)-Ô)?Ø-1Ô-Gà*/ð ðð ð ðð ð ˆGð# ð ð ñ ô ð ð ð ð ð ð øøøð ð ð ð ð<ˆsÁCDÄDÄDTÚfeaturesr0r1rãcóf—|}||dd¬¦«d}||dd¬¦«d}||dd¬¦«d}|r$|j� |jg|z}|j� ||jgz}||jgz}||jgz}|� || d…}|�|d|…}|d|…}|||dœS) a©
Tokenize a row of the dataset.
Args:
features (`dict[str, str]`):
Row of the dataset, should contain the keys `"prompt"`, `"chosen"`, and `"rejected"`.
processing_class (`PreTrainedTokenizerBase`):
Processing class used to process the data.
max_prompt_length (`int` or `None`):
Maximum length of the prompt sequence. If `None`, the prompt sequence is not truncated.
max_completion_length (`int` or `None`):
Maximum length of the completion sequences. If `None`, the completion sequences are not truncated.
add_special_tokens (`bool`):
Whether to add special tokens to the sequences. Typically used for encoder-decoder models. If `True`,
the prompt sequence will have a bos token prepended and an eos token appended. In any case, the
completion sequences will have an eos token appended.
Returns:
`dict[str, list[int]]`:
Tokenized sequences with the keys `"prompt_input_ids"`, `"chosen_input_ids"`, and
`"rejected_input_ids".
Example:
```python
>>> from transformers import GPT2Tokenizer
>>> tokenizer = GPT2Tokenizer.from_pretrained("gpt2")
>>> features = {"prompt": "The sky is", "chosen": " blue", "rejected": " green"}
>>> DPOTrainer.tokenize_row(
... features, tokenizer, max_prompt_length=3, max_completion_length=3, add_special_tokens=False
... )
{'prompt_input_ids': [464, 6766, 318], 'chosen_input_ids': [4171, 50256], 'rejected_input_ids': [4077, 50256]}
```
ÚpromptF©rãÚ input_idsrárâN)Úprompt_input_idsÚchosen_input_idsÚrejected_input_ids)Ú bos_token_idÚ eos_token_id) rêrcr0r1rãrmrïrðrñs rxrçz_UnslothDPOTrainer.tokenize_rowÎs.€ðT%ˆ Ø$˜9 X¨hÔ%7ÈEÐRÑRÔRÐS^Ô_ÐØ$˜9 X¨hÔ%7ÈEÐRÑRÔRÐS^Ô_ÐØ&˜Y x°
Ô';ÐPUÐVÑVÔVÐWbÔcÐð ð OØÔ%Ð1Ø$-Ô$:Ð#;Ð>NÑ#NÐ ØÔ%Ð1Ø#3°yÔ7MÐ6NÑ#NÐ Ø+¨yÔ/EÐ.FÑFÐØ/°9Ô3IÐ2JÑJÐð Ð (Ø/Ð1BÐ0BÐ0CÐ0CÔDÐ Ø Ð ,Ø/Ð0FÐ1FÐ0FÔGÐ Ø!3Ð4JÐ5JÐ4JÔ!KÐ ð!1Ø 0Ø"4ð
ð
ð
rzcó—||j}}||d|dd¬¦«}|dd}|dd} ||dd¬ ¦«d}
||d
d¬ ¦«d} |r$|j� |jg|z}|j� ||jgz}|
|jgz}
| |jgz} |� || d …}|�|
d |…}
| d |…} || |
| d œ} d
|vr|d
d| d
<d|vr|dd| d<| S)zt
Same as `tokenize_row` but for vision models. Please refer to `tokenize_row` for more information.
ÚimagesrìF)rõÚtextrãrîrÚ pixel_valuesrárírâN)rïr÷rðrñÚpixel_attention_maskÚ image_sizes)rmròró)
rêrcr0r1rãÚ processorrmÚprocessed_featuresrïr÷rðrñrÖs
rxrèz_UnslothDPOTrainer.process_rows¡€ð 0Ð1AÔ1K�9ˆ Ø&˜Y¨h°xÔ.@ÀxÐPXÔGYÐnsÐtÑtÔtÐà-¨kÔ:¸1Ô=ÐØ)¨.Ô9¸!Ô<ˆ Ø$˜9 X¨hÔ%7ÈEÐRÑRÔRÐS^Ô_ÐØ&˜Y x°
Ô';ÐPUÐVÑVÔVÐWbÔcÐð ð OØÔ%Ð1Ø$-Ô$:Ð#;Ð>NÑ#NÐ ØÔ%Ð1Ø#3°yÔ7MÐ6NÑ#NÐ Ø+¨yÔ/EÐ.FÑFÐØ/°9Ô3IÐ2JÑJÐð Ð (Ø/Ð1BÐ0BÐ0CÐ0CÔDÐ Ø Ð ,Ø/Ð0FÐ1FÐ0FÔGÐ Ø!3Ð4JÐ5JÐ4JÔ!KÐ ð!1Ø(Ø 0Ø"4ð 
ð
ˆð "Ð%7Ð 7Ð 7Ø-?Ð@VÔ-WÐXYÔ-ZˆFÐ)Ñ *Ø Ð.Ð .Ð .Ø$6°}Ô$EÀaÔ$HˆF�=Ñ !àˆ
rzcó*—|j€ gd¢|_dSdS)N)rïrðrñrùÚref_chosen_logpsÚref_rejected_logps)Ú_signature_columns©rNs rxÚ _set_signature_columns_if_neededz3_UnslothDPOTrainer._set_signature_columns_if_neededDs/€ð
Ô "Ð *ð'ð'ð'ˆDÔ #Ð #Ð #ð +Ð *rzcó$•—|j�rè|j�sà|jjp |jj}||j|jj|jjddœ}|j  t|j fi|¤Ž¦«}g}g}t|d¬¦«D]®}| 
|¦«\}}|j ||f¦«\}}| | ¦«¦«| | ¦«¦«t#¦«|j ¦«Œ¯t'j|¦« ¦« ¦«} t'j|¦« ¦« ¦«}
|j  d| ¬¦«|_ |j  d|
¬¦«|_ d|_t1¦« ¦«S) z·
Returns the training [`~torch.utils.data.DataLoader`].
Subclass of transformers.src.transformers.trainer.get_train_dataloader to precompute `ref_log_probs`.
F©Ú
batch_sizeÚ
collate_fnÚ num_workersÚ
pin_memoryÚshufflez!Train dataset reference log probs©Úiterablerßrý©ÚnameÚcolumnrþT)r5r¥r_r6r­r`rärÿrŒÚpreparerrarNÚcompute_ref_log_probsÚgather_for_metricsrmÚcpur0Ú free_memoryrMÚcatÚfloatÚnumpyÚ
add_columnrLÚget_train_dataloader) rNrÚdataloader_paramsÚ data_loaderrýrþÚ padded_batchÚref_chosen_logpÚref_rejected_logpÚall_ref_chosen_logpsÚall_ref_rejected_logpsrPs €rxrz'_UnslothDPOTrainer.get_train_dataloaderSs ø€ð Ô (ñ# 9°Ô1Vñ# 9ØœÔ<ÐeÀÄ Ô@eˆJà(Ø"Ô0Ø#œyÔ?Ø"œiÔ=Ø ð !ð!Ð ðÔ*×2Ò2µ:¸dÔ>PÐ3fÐ3fÐTeÐ3fÐ3fÑgÔgˆKà!Ð Ø!#Ð Ý $¨kÐ@cÐ dÑ dÔ dð
/ð
/� Ø59×5OÒ5OÐP\Ñ5]Ô5]Ñ2�Ð!2Ø59Ô5E×5XÒ5XØ$Ð&7Ð8ñ6ô6Ñ2�Ð!2ð!×'Ò'¨×(;Ò(;Ñ(=Ô(=Ñ>Ô>Ð>Ø"×)Ò)Ð*;×*?Ò*?Ñ*AÔ*AÑBÔBÐBõ‘
”
�
ØÔ ×,Ò,Ñ.Ô.Ð.Ð.å#(¤9Ð-=Ñ#>Ô#>×#DÒ#DÑ#FÔ#F×#LÒ#LÑ#NÔ#NÐ Ý%*¤YÐ/AÑ%BÔ%B×%HÒ%HÑ%JÔ%J×%PÒ%PÑ%RÔ%RÐ "à!%Ô!3×!>Ò!>ÐDVÐ_sÐ!>Ñ!tÔ!tˆDÔ Ø!%Ô!3×!>Ò!>Ø)Ð2Hð"?ñ"ô"ˆDÔ ð59ˆDÔ 1å‰wŒw×+Ò+Ñ-Ô-Ð-rzcó
•—|€|j€td¦«‚|�|n|j}|j�r¶|j�s®|jjp |jj}||j|jj|jj ddœ}|j
  t|fi|¤Ž¦«}g}g}t|d¬¦«D]‡}| |¦«\}} |j
 || f¦«\}} | | ¦«¦«| |  ¦«¦«Œˆt%j|¦« ¦« ¦«}
t%j|¦« ¦« ¦«} | d|
¬¦«}| d | ¬¦«}|j�||_d
|_t/¦« |¬ ¦«S) aé
Returns the evaluation [`~torch.utils.data.DataLoader`].
Subclass of transformers.src.transformers.trainer.get_eval_dataloader to precompute `ref_log_probs`.
Args:
eval_dataset (`torch.utils.data.Dataset`, *optional*):
If provided, will override `self.eval_dataset`. If it is a [`~datasets.Dataset`], columns not accepted
by the `model.forward()` method are automatically removed. It must implement `__len__`.
Nz-Trainer: evaluation requires an eval_dataset.Frz Eval dataset reference log probsr rýr rþT)rb)rbr–r5r¦r_r6r®r`rärÿrŒrrrNrrrmrrMrrrrrLÚget_eval_dataloader)
rNrbrrrrýrþrrrrrrPs
€rxr z&_UnslothDPOTrainer.get_eval_dataloader�sø€ð Ð  DÔ$5Ð$=ÝÐLÑMÔMÐ MØ'3Ð'?�|�|ÀTÔEVˆ à Ô (ñ 8°Ô1Uñ 8ØœÔ<ÐdÀÄ Ô@dˆJà(Ø"Ô0Ø#œyÔ?Ø"œiÔ=Ø ð !ð!Ð ðÔ*×2Ò2µ:¸lÐ3`Ð3`ÐN_Ð3`Ð3`ÑaÔaˆKà!Ð Ø!#Ð Ý $¨kÐ@bÐ cÑ cÔ cð
Cð
C� Ø59×5OÒ5OÐP\Ñ5]Ô5]Ñ2�Ð!2Ø59Ô5E×5XÒ5XØ$Ð&7Ð8ñ6ô6Ñ2�Ð!2ð!×'Ò'¨×(;Ò(;Ñ(=Ô(=Ñ>Ô>Ð>Ø"×)Ò)Ð*;×*?Ò*?Ñ*AÔ*AÑBÔBÐBÐBå#(¤9Ð-=Ñ#>Ô#>×#DÒ#DÑ#FÔ#F×#LÒ#LÑ#NÔ#NÐ Ý%*¤YÐ/AÑ%BÔ%B×%HÒ%HÑ%JÔ%J×%PÒ%PÑ%RÔ%RÐ "à'×2Ò2Ð8JÐSgÐ2ÑhÔhˆLØ'×2Ò2Ð8LÐUkÐ2ÑlÔlˆLðÔ Ð,Ø$0�Ô!Ø37ˆDÔ 0å‰wŒw×*Ò*¸ Ð*ÑEÔEÐErzc#ózK—|jr8|js1|j |j¦« ¦«n
t
¦«5|jr|j |j¦«dV—|jr!|j |jpd¦«ddd¦«dS#1swxYwYdS)zWContext manager for handling null reference model (that is, peft adapter manipulation).Nr~) ržr)rŒÚ unwrap_modelr Údisable_adapterr@Ú set_adapterr(rs rxÚnull_ref_contextz#_UnslothDPOTrainer.null_ref_context´sèè€ð
Ô!ð
Ø*.Ô*?ð
ˆDÔ × )Ò )¨$¬*Ñ 5Ô 5× EÒ EÑ GÔ GÐ Gå‘”ð Mð Mð
Ô$ð
>Ø”
×&Ò& tÔ'<Ñ=Ô=Ð=Ø ˆEˆEˆEØÔ$ð
MØ”
×&Ò& tÔ'>Ð'KÀ)ÑLÔLÐLð Mð Mð Mñ Mô Mð Mð Mð Mð Mð Mð Mð Møøøð Mð Mð Mð Mð Mð MsÁAB0Â0B4Â7B4Úbatchcóî—|jrt|jjj¦«n
t ¦«}t
j¦«5|5|j€J|  ¦«5| 
|j |d¬¦«}ddd¦«n #1swxYwYn| 
|j|d¬¦«}ddd¦«n #1swxYwYddd¦«n #1swxYwY|d|dfS)zfComputes log probabilities of the reference model for a single padded batch of a DPO specific dataset.NT)Ú is_ref_modelÚ chosen_logpsÚrejected_logps) rÉr)rŒÚdeviceÚtyper@rMÚno_gradr^r%Úconcatenated_forwardr )rNr&Úcompte_ref_context_managerÚref_model_outputs rxrz(_UnslothDPOTrainer.compute_ref_log_probsÂsé€ð7;Ô6XÐ k�H�TÔ%Ô,Ô1Ñ 2Ô 2Ð 2Õ^iÑ^kÔ^kð #õŒ]‰_Œ_ð gð gÐ8ð gð gØŒ~Ð%Ø×*Ò*Ñ,Ô,ðgðgØ'+×'@Ò'@ÀÄÈUÐaeÐ'@Ñ'fÔ'fÐ$ðgðgðgñgôgðgðgðgðgðgðgøøøðgðgðgðgøð$(×#<Ò#<¸T¼^ÈUÐaeÐ#<Ñ#fÔ#fÐ ð  gð gð gñ gô gð gð gð gð gð gð gøøøð gð gð gð gð gð gð gñ gô gð gð gð gð gð gð gøøøð gð gð gð gð   Ô/Ð1AÐBRÔ1SÐSÐSsZÁCÁ
CÁ&BÂ CÂB ÂCÂB Â!CÂ9 CÃC Ã CÃ C Ã
CÃC Ã#C r.có —i}tj|d|dgd¬¦«|d<tj|d|dgd¬¦«|d<d|vr'tj|d|dgd¬¦«|d<d|vr'tj|d|dgd¬¦«|d<d|vr'tj|d|dgd¬¦«|d<t|djd |d
jd ¦«}tjt |d||¬ ¦«t |d
||¬ ¦«f¦«|d <tjt |d
|d¬ ¦«t |d|d¬ ¦«f¦«|d<|S)aÕ
Concatenate the `chosen` and `rejected` inputs from the batch into a single tensor for both the prompt and
completion sequences.
Args:
batch (`dict[str, Union[list, torch.LongTensor]]`):
A batch of input data. The batch must contain the following keys:
- `"prompt_input_ids"`: Tensor of shape `(batch_size, prompt_length)` representing the prompt input
IDs.
- `"chosen_input_ids"`: Tensor of shape `(batch_size, chosen_length)` representing the chosen
completion input IDs.
- `"rejected_input_ids"`: Tensor of shape `(batch_size, rejected_length)` representing the rejected
completion input IDs.
- `"prompt_pixel_values"` (optional): Tensor for pixel values, if available.
- `"prompt_pixel_attention_mask"` (optional): Tensor for pixel attention masks, if available.
padding_value (`int`):
The padding value to use for the concatenated completion sequences (`chosen_input_ids` and
`rejected_input_ids`).
Returns:
`dict[str, torch.LongTensor]`: A dictionary containing:
- `"prompt_input_ids"`: Concatenated prompt input IDs of shape `(2 * batch_size, prompt_length)`.
- `"completion_input_ids"`: Concatenated chosen and rejected completion input IDs of shape `(2 *
batch_size, max_completion_length)`.
- `"prompt_attention_mask"`: Concatenated prompt attention masks of shape `(2 * batch_size,
prompt_length)`.
- `"completion_attention_mask"`: Concatenated chosen and rejected attention masks of shape `(2 *
batch_size, max_completion_length)`.
- `"pixel_values"` (optional): Concatenated pixel values if `"prompt_pixel_values"` are present.
- `"pixel_attention_mask"` (optional): Concatenated pixel attention masks if
`"prompt_pixel_attention_mask"` are present.
Notes:
The completion input IDs and attention masks are padded to the maximum completion length of the chosen or
rejected sequences.
rïrraÚprompt_attention_maskr÷rørùrðrbrñ)Ú pad_valueÚcompletion_input_idsÚchosen_attention_maskÚrejected_attention_maskÚcompletion_attention_mask)rMrÚmaxrerC)r&r.rÖr1s rxÚconcatenated_inputsz&_UnslothDPOTrainer.concatenated_inputsÏsí€ðVˆõ&+¤Y°Ð6HÔ0IÈ5ÐQcÔKdÐ/eÐklÐ%mÑ%mÔ%mˆÐ!Ñ"Ý*/¬)Ø
Ð*Ô
+¨UÐ3JÔ-KÐ LÐRSð+
ñ+
ô+
ˆÐ&Ñ'ð ˜UÐ "Ð "Ý%*¤Y°°nÔ0EÀuÈ^ÔG\Ð/]ÐcdÐ%eÑ%eÔ%eˆF�>Ñ "à ! UÐ *Ð *Ý-2¬YØÐ-Ô.°Ð6LÔ0MÐNÐTUð.ñ.ô.ˆFÐ)Ñ *ð ˜EÐ !Ð !Ý$)¤I¨u°]Ô/CÀUÈ=ÔEYÐ.ZÐ`aÐ$bÑ$bÔ$bˆF�=Ñ !õ!$ EÐ*<Ô$=Ô$CÀAÔ$FÈÐNbÔHcÔHiÐjkÔHlÑ mÔ mÐÝ).¬å˜eÐ$6Ô7Ð9NÐZgÐhÑhÔhݘeÐ$8Ô9Ð;PÐ\iÐjÑjÔjð
ñ*
ô*
ˆÐ%Ñ&õ /4¬iå˜eÐ$;Ô<Ð>SÐ_`ÐaÑaÔaݘeÐ$=Ô>Ð@UÐabÐcÑcÔcð
ñ/
ô/
ˆÐ*Ñ+ðˆ
rzror)r*rýrþr Ú model_outputcóL—|jj}| |¦«|j | |¦«zz
}| |¦«|j | |¦«zz
} |jt
jjkrutj }
|j
r7tj |j
vr$t|j
tj ¦«}
t| |
z¦«t||
z¦«z
|
z } n½||z
} |jr#tjdg| j| j¬¦«}
n||z
}
|  |jj¦«} |
 |jj¦«}
| |
z
} |jt
jjkr,| t%j|¦«t%j| ¦«z
z} |dkrPt%j|j| z¦« d|jz
zt%j|j | z¦«|jzz
}�n€|dkr^t%j|j| z¦« d|jz
zt%j|j | z¦«|jzzdd|jzz
z }�n|dkrÆddl}|jdkrd |_|j| z ¦«t%j|j| z¦«| d|jz
¦«z
z|j | z ¦«t%j|j | z¦«| |j¦«z
zz}�nP|d
kr!tjd|j| zz
¦«}�n)|d kr| dd|jzz z
dz}�n|d krÂ||z
}||z
} |j|z}|j| z}tj||fd¦« ¦« ¦«}|j |¦«|jj}t%j|j|z|z
¦« t%j|j| z|z
¦«z
}�nF|d
kr-||z
}||z
}|d|jz z
dz|d|jz zdzz}�n|dkrc||z
|jz}||z
|jz}t%j|¦« dt%j| ¦«zz
dt%j| ¦«zz
}�nª|dkr‘||z
}||z
} tj |d¬¦«\}}tj | d¬¦«\}}||z
}t%j|j|z¦« d|jz
zt%j|j |z¦«|jzz
}�n|dkr‘||z
} ||z
}
tj | d¬¦«\}}tj |
d¬¦«\}}||z
}t%j|j|z¦« d|jz
zt%j|j |z¦«|jzz
}�n||dkrBdt%j|j|z¦«z
}t%j|j| z¦«}||z}�n4|dkrDt%j|j|z¦«}dt%j|j|| z
z¦«z
}||z}nê|dkrs||z
} ||z
}
| |
z
} | |jz} tj| |j!j"z ¦«}t%j| ¦« }tj#| ¦«}|d|z
z||zz}nq|dkrS|d} |j$d}!|  %|!¦«}tj&|¦«}tj&|¦«}ntOd|j(›d�¦«‚|j| |¦«| |¦«z
 ¦«z}|j| |¦«| |¦«z
 ¦«z}|||fS)a
Compute the DPO loss for a batch of policy and reference model log probabilities.
Args:
chosen_logps (`torch.FloatTensor`):
Log probabilities of the model for the chosen responses. Shape: `(batch_size,)`.
rejected_logps (`torch.FloatTensor`):
Log probabilities of the model for the rejected responses. Shape: `(batch_size,)`.
ref_chosen_logps (`torch.FloatTensor`):
Log probabilities of the reference model for the chosen responses. Shape: `(batch_size,)`.
ref_rejected_logps (`torch.FloatTensor`):
Log probabilities of the reference model for the rejected responses. Shape: `(batch_size,)`.
Returns:
A tuple of three tensors: `(losses, chosen_rewards, rejected_rewards)`. The losses tensor contains the DPO
loss for each example in the batch. The `chosen_rewards` and `rejected_rewards` tensors contain the rewards
for the chosen and rejected responses, respectively.
r)rÄr+rorbÚrobustrƒÚexo_pairNgü©ñÒMbP?rwrxryrzgà?r{Úaot_pairraÚaotr|r}ÚdiscopopÚsftÚnll_losszUnknown loss type: z§. Should be one of ['sigmoid', 'hinge', 'ipo', 'exo_pair', 'nca_pair', 'robust', 'bco_pair', 'sppo_hard', 'aot', 'aot_pair', 'discopop', 'apo_zero', 'apo_down', 'sft']))rŒr+rgr<r¬rÚALPHA_DIVERGENCEÚvaluerÚALPHA_DIVERGENCE_COEF_DEFAULTr®r­rr*rMÚtensorrÄÚ
JS_DIVERGENCErÚsoftplusÚ
logsigmoidr:r=ÚmathroÚlogÚrelurÚmeanÚdetachr»ÚupdateÚsortr_rAÚexpreÚexpandÚ
zeros_liker–r )"rNr)r*rýrþr r:r+Úchosen_logratiosÚrejected_logratiosÚ
alpha_coefroÚ logratiosÚ
ref_logratiosÚlossesrJÚchosen_rewardsÚrejected_rewardsÚrewardsÚdeltaÚaÚbÚchosen_logratios_sortedÚ_Úrejected_logratios_sortedÚlogratios_sortedÚref_logratios_sortedÚ
losses_chosenÚlosses_rejectedÚlog_ratio_modulationÚlogistic_componentÚ
exp_componentÚsft_lossrs" rxÚdpo_lossz_UnslothDPOTrainer.dpo_losssÛ€ð6Ô!Ô(ˆð(Ÿ?š?¨6Ñ2Ô2¸$Ô:MÐ6MÐQa×QdÒQdÐekÑQlÔQlÑ5lÑlÐØ+×.Ò.¨vÑ6Ô6¸dÔ>QÐ:QÐUg×UjÒUjÐkqÑUrÔUrÑ9rÑrÐà Ô !¥_Ô%EÔ%KÒ KÐ Kõ.ÔKˆJØÔ'ð
mÕ,@Ô,ZÐ^bÔ^vÐ,vÐ,vÝ" 4Ô#;Õ<PÔ<jÔ#kÑlÔl�
ÝÐ0°J°;Ñ>Ñ?Ô?Å'ÐJZÐ^hÐ]hÑJhÑBiÔBiÑiÐmwÑwˆFˆFà$ ~Ñ5ˆIØÔ"ð
FÝ %¤ ¨a¨S¸ ¼ÐPYÔP`Ð aÑ aÔ a�
�
à 0Ð3EÑ E�
à!Ÿ š  TÔ%5Ô%<Ñ=Ô=ˆIØ)×,Ò,¨TÔ-=Ô-DÑEÔEˆMØ Ñ.ˆFàÔ%­Ô)FÔ)LÒLÐLð�!œ*Ð%5Ñ6Ô6½¼ÐDVÑ9WÔ9WÑWÑW�ð
˜ Ò !Ð !唘dœi¨&Ñ0Ñ1Ô1Ð1°Q¸Ô9MÑ5MÑNÝ”, ¤ ˜z¨FÑ2Ñ3Ô3°dÔ6JÑJñKð
‰Fð
˜(Ò
"Ð
"唘dœi¨&Ñ0Ñ1Ô1Ð1°Q¸Ô9MÑ5MÑNÝ”, ¤ ˜z¨FÑ2Ñ3Ô3°dÔ6JÑJñKà�Q˜Ô-Ñ-Ñ-ñ/ˆF‰Fð
˜*Ò
$Ð
$à ˆKˆKˆKàÔ# qÒ(Ð(Ø'+�Ô$Ø”i &Ñ(×1Ò1Ñ3Ô3Ý” ˜TœY¨Ñ/Ñ0Ô0°4·8²8¸AÀÔ@TÑ<TÑ3UÔ3UÑUñà”)�˜fÑ$×-Ò-Ñ/Ô/µ1´<ÀÄÀ
ÈVÑ@SÑ3TÔ3TÐW[×W_ÒW_Ð`dÔ`tÑWuÔWuÑ3uÑvñwˆF‰Fð˜'Ò
!Ð
!Ý”Z  D¤I°Ñ$6Ñ 6Ñ7Ô7ˆF‰Fà
˜%Ò
Ð
à˜q A¨¬ ¡MÑ2Ñ2°qÑ8ˆF‰Fà
˜*Ò
$Ð
$Ø+Ð.>Ñ>Ð Ø!/Ð2DÑ!DÐ Ø!œYÐ)9Ñ9ˆNØ#œyÐ+=Ñ=Ð Ý”i Ð1AÐ BÀAÑFÔF×KÒKÑMÔM×TÒTÑVÔVˆGØ ŒL× Ò  Ñ (Ô (Ð (Ø”LÔ%ˆEÝ”l D¤IÐ0@Ñ$@ÀEÑ#IÑJÔJÐJÍQÌ\Ø”)Ð0Ñ0°5Ñ8Ð9ñNôNñˆF‰Fð˜+Ò
%Ð
%ð
Ð/Ñ/ˆAØÐ!3Ñ3ˆAؘ# ¤ ™/Ñ)¨aÑ/°1°s¸T¼Y±Ñ3FÈ1Ñ2LÑLˆF‰Fà
˜*Ò
$Ð
$Ø*Ð-=Ñ=ÀÄÑJˆNØ .Ð1CÑ CÀtÄyÑPР唘nÑ-Ô-Ð-Ø�œ  n _Ñ5Ô5Ñ5ñ6à�œ Ð&6Ð%6Ñ7Ô7Ñ7ñ8ð
‰Fð ˜*Ò
$Ð
$Ø+Ð.>Ñ>Ð Ø!/Ð2DÑ!DÐ Ý).¬Ð4DÈ!Ð)LÑ)LÔ)LÑ &Ð # QÝ+0¬:Ð6HÈaÐ+PÑ+PÔ+PÑ (Ð % qØ+Ð.GÑGˆE唘dœi¨%Ñ/Ñ0Ô0Ð0°A¸Ô8LÑ4LÑMÝ”, ¤ ˜z¨EÑ1Ñ2Ô2°TÔ5IÑIñJð
‰Fð
˜%Ò
Ð
Ø$ ~Ñ5ˆIØ,Ð/AÑAˆMÝ"'¤*¨Y¸AÐ">Ñ">Ô">Ñ Ð ˜aÝ&+¤j°ÀAÐ&FÑ&FÔ&FÑ #Ð  !Ø$Ð';Ñ;ˆE唘dœi¨%Ñ/Ñ0Ô0Ð0°A¸Ô8LÑ4LÑMÝ”, ¤ ˜z¨EÑ1Ñ2Ô2°TÔ5IÑIñJð
‰Fð
˜*Ò
$Ð
$𥤠¨$¬)Ð6FÑ*FÑ GÔ GÑGˆMÝœi¨¬ Ð4FÑ(FÑGÔGˆOØ" _Ñ4ˆF‰Fà
˜*Ò
$Ð
$õœI d¤iÐ2BÑ&BÑCÔCˆMØ¥!¤)¨D¬IÐ9IÐL^Ñ9^Ñ,_Ñ"`Ô"`Ñ`ˆOØ" _Ñ4ˆFˆFà
˜*Ò
$Ð
$ð% ~Ñ5ˆIØ,Ð/AÑAˆMØ Ñ.ˆFؘdœiÑ'ˆFå#(¤=°¸$¼)Ô:PÑ1PÑ#QÔ#QÐ Ý"#¤,¨vÑ"6Ô"6Ð!6Ð Ý!œI v gÑ.Ô.ˆMà'¨1Ð/CÑ+CÑDÀ}ÐWkÑGkÑkˆFˆFà
˜%Ò
Ð
ð$ JÔ/ˆHà%Ô+¨AÔ.ˆJØ—_’_ ZÑ0Ô0ˆFå"Ô-¨lÑ;Ô;ˆNÝ$Ô/°Ñ?Ô?Ð Ð õð% d¤nð%ð%ð%ñôð
𠜠l§o¢o°fÑ&=Ô&=Ð@P×@SÒ@SÐTZÑ@[Ô@[Ñ&[×%cÒ%cÑ%eÔ%eÑeˆØœ9¨×(9Ò(9¸&Ñ(AÔ(AÐDV×DYÒDYÐZ`ÑDaÔDaÑ(a×'iÒ'iÑ'kÔ'kÑkÐà�~Ð'7Ð7Ð7rzc óÚ—|j |¦«}| ||j¬¦«}i}|jrd|d<d|vr |d|d<d|vr |d|d<d|vr |d|d<|d}|d}|j�rÓ| ¦«|d |dd¬
¦«}t|d |jj ¦«} | 
¦«| |d|j |dd ¬
¦«}
|
j } d} |j s�|j
�†|j |j
¦«}
|
 ¦«|d |dd¬
¦«}|
 
¦«| |d|j |dd ¬
¦«}|j } n™|j s’| ¦«5| ¦«|d |dd¬
¦«}| 
¦«| |d|j |dd ¬
¦«}|j } ddd¦«n #1swxYwY|d }| ¦«}�nNt!j|d |d fd¬¦«}t!j|d|dfd¬¦«}t!jt!j|¦«|fd¬¦«}|j��|j| d¦«krä|jdkrOt-|||¦«\}}}|dd…d|j…f}|dd…d|j…f}|dd…d|j…f}nŸ|jdkrgt/|||¦«\}}}|dd…|j d…f}|dd…|j d…f}|dd…|j d…f}t-|||¦«\}}}n-t1d|j›d�¦«‚t-|||¦«\}}}|jrX| d¬¦«d ¦«}|jd|z
 ¦«dz}||d<d|d<|jr£|| ¦« d¦«}|| ¦« d¦«}|  d¦«| ¦« d¦«dz
}||d<n||d<tC|d¦«r| 
¦«}ntE||j#j$|¦«}||fdd i|¤Ž}|j dd…dd…f} d} |j s…|j
�~|j |j
¦«}
tC|
d¦«r|
 
¦«}ntE|
|j#j$|
¦«}||fdd i|¤Ž}|j dd…dd…f} n‘|j sŠtC|d¦«r| 
¦«}ntE||j#j$|¦«}| ¦«5||fdd i|¤Ž}|j dd…dd…f} ddd¦«n #1swxYwYt!j%|dk||j&¦«}|dd…dd…f}| '¦«}d}d}|j s›|j
�4|j |j
¦«}
|
 '¦«}n@| ¦«5| '¦«}ddd¦«n #1swxYwY|j(}tC|d¦«r|j)nd}| *|j(| |tC|d¦«r|j)nd|j s| nd|j s|nd|j s|nd¬¦«} | \}!^}"}#}$}%}&}'|!|"|#|$|%|&|'d|'ddœ}(|jr
|j+|(d <|(S)!N©r.Trur÷rørùr2r7rï)Úattention_maskÚ return_dictr4F)rîrnÚencoder_hidden_statesÚencoder_attention_maskÚ use_cacherbraÚ
keep_startrŸúUnknown truncation mode: 'ú/'. Should be one of ['keep_end', 'keep_start'].©Úas_tupleÚlogits_to_keepÚoutput_hidden_statesrÚ position_idsrnÚ get_decoderrrr\Úbias)r|Ú ref_inputÚ
ref_weightÚref_bias)Úlossr)r*Úmean_chosen_logitsÚmean_rejected_logitsrBrZr[Úaux_loss),rŒr"r9r.r¨ršÚ get_encoderrKr‘Údecoder_start_token_idr{Úlast_hidden_stater<r^r%ÚboolrMrrSr2Úsizer3r1r2r–r,ÚnonzerorKreÚitemr4rjÚcumsumr•r§r_r9Úwherer/Úget_output_embeddingsÚweightr|r¢rƒ))rNr r&Úunwrapped_modelÚconcatenated_batchÚ model_kwargsr2r7Úencoder_outputsÚdecoder_input_idsÚdecoder_outputsÚ
hidden_statesÚref_hidden_statesÚunwrapped_ref_modelÚref_encoder_outputsÚref_decoder_outputsÚlabelsÚ loss_maskrîrnÚfirst_compute_indexrxrzÚ
base_modelÚoutputsÚref_base_modelÚ ref_outputsÚmasked_input_idsÚlm_headr~rÚ ref_lm_headÚ loss_outputr€r)r*Úchosen_logits_meanÚrejected_logits_meanrBÚ aux_outputsrÖs) rxÚ_compute_loss_ligerz&_UnslothDPOTrainer._compute_loss_ligeräsö
€ðÔ*×7Ò7¸Ñ>Ô>ˆØ!×5Ò5°eÈ4ÔK]Ð5Ñ^Ô^Ðàˆ Ø Ô ð 8Ø37ˆLÐ/Ñ 0ð Ð/Ð /Ð /Ø+=¸nÔ+MˆL˜Ñ (Ø !Ð%7Ð 7Ð 7Ø3EÐF\Ô3]ˆLÐ/Ñ 0Ø Ð.Ð .Ð .Ø*<¸]Ô*KˆL˜Ñ 'à 2Ð3JÔ KÐØ$6Ð7RÔ$SÐ!à Ô "ñb -à;˜o×9Ò9Ñ;Ô;Ø"Ð#5Ô6Ø1Ð2IÔJØ ðñôˆOõ !3Ø"Ð#9Ô:ØÔ&Ô=ñ!ô!Ð ð
<˜o×9Ò9Ñ;Ô;Ø+Ø1Ð2MÔNØ&5Ô&GØ'9Ð:QÔ'RØð ñôˆOð,Ô=ˆMà $Ð ØÔ&ð
N¨4¬>Ð+EØ&*Ô&6×&CÒ&CÀDÄNÑ&SÔ&SÐ#Ø&GÐ&9×&EÒ&EÑ&GÔ&GØ&Ð'9Ô:Ø#5Ð6MÔ#NØ $ð'ñ'ô'Ð#ð
'HÐ&9×&EÒ&EÑ&GÔ&GØ/Ø#5Ð6QÔ#RØ*=Ô*OØ+=Ð>UÔ+VØ#ð 'ñ'ô'Ð#ð%8Ô$IÐ!Ð!ØÔ(ð
NØ×*Ò*Ñ,Ô,ð
Nð
NØ*G¨/×*EÒ*EÑ*GÔ*GØ*Ð+=Ô>Ø'9Ð:QÔ'RØ$(ð+ñ+ô+Ð'ð
+H¨/×*EÒ*EÑ*GÔ*GØ"3Ø'9Ð:UÔ'VØ.AÔ.SØ/AÐBYÔ/ZØ"'ð +ñ+ô+Ð'ð)<Ô(MÐ%ð
Nð
Nð
Nñ
Nô
Nð
Nð
Nð
Nð
Reference in New Issue Copy Permalink