Files
DS-LLM-TEMPLATE-FINETUNING/unsloth_compiled_cache/__pycache__/UnslothCPOTrainer.cpython-311.pyc
T

418 lines
77 KiB
Plaintext
Raw Normal View History

2025-08-13 23:50:20 +00:00
§
3$hÚ/ãóhdZddlmZddlZddlmZddlmZddlmZm Z m
Z
m Z m Z m
Z
mZmZddlmZmZmZmZmZmZmZmZmZmZmZmZmZmZm
Z
mZmZmZmZm Z m!Z!m"Z"m#Z#m Z m$Z$m%Z%m&Z&m'Z'm(Z(m)Z)m*Z*m+Z+m,Z,m-Z-m.Z.m/Z/m0Z0m1Z1m2Z2mZm3Z3m4Z4m5Z5m6Z6m7Z7m8Z8m9Z9m:Z:m;Z;m<Z<mZm=Z=mZm
Z
mZmZm"Z"m-Z-m5Z5mZddl5Z5ddlTddl>m?Z?m@Z@dd lAmBZBddlZddlCZ3dd
lDm4Z4ddlmZdd lEmFZFmGZHd d
d d
d
dœZIejJd d eI¬¦«d¦«ZKe?Gdde¦«¦«ZL Gdde"¦«ZMGddeM¦«ZNdS)z8
2025.8.4
2025.8.5
4.55.1
0.21.0
__UNSLOTH_VERSIONING__
é)ÚTensorN)Ú
functional)ÚAnyÚListÚOptionalÚTupleÚUnionÚDictÚSetÚCallable)<rÚAutoModelForCausalLMÚBaseImageProcessorÚ CPOConfigÚ
CPOTrainerr ÚDPODataCollatorWithPaddingÚ DataCollatorÚ
DataLoaderÚDatasetÚEvalLoopOutputÚFeatureExtractionMixinÚLiteralrÚ PartialStateÚPathÚ PeftModelÚPreTrainedModelÚPreTrainedTokenizerBaseÚProcessorMixinÚTrainerÚTrainerCallbackr Úadd_bos_token_if_neededÚadd_eos_token_if_neededÚautocastÚ defaultdictÚdisable_dropout_in_modelÚgenerate_model_cardÚget_comet_experiment_urlÚinspectÚis_comet_availableÚis_peft_availableÚis_torch_fx_proxyÚis_wandb_availableÚlog_table_to_comet_experimentÚmaybe_apply_chat_templateÚmaybe_extract_promptÚnnÚnpÚ nullcontextÚosÚ
pad_to_lengthÚpdÚpeft_module_casting_to_bf16Úprepare_model_for_kbit_trainingÚrandomÚselective_log_softmaxÚtextwrapÚtorchÚwarningsrrrrrr*r3r;)Ú*)Ú dataclassÚfield)ÚVersion)r2)ÚDataCollatorForSeq2SeqÚDataCollatorForLanguageModelingTF)Úepilogue_fusionÚ max_autotuneÚ
shape_paddingz
trace.enabledztriton.cudagraphs)ÚdynamicÚ fullgraphÚoptionscó’tj| d|jd¦«dd¬¦«}tj| d¦«dd¬¦«}g}t ||¦«D]\}}| tj¦«}tj|d| d¦«¬¦«  d¦«}tj
|d¬¦«}||z
} |  | ¦«Œ’ tj |¦«}| |jd|jdf¦«}|S)Néÿÿÿÿér)ÚchunksÚdim)rMÚindex©rMé)
r;ÚchunkÚreshapeÚshapeÚzipÚtoÚfloat32ÚgatherÚ unsqueezeÚsqueezeÚ logsumexpÚappendÚconcat)
ÚlogitsrNÚchunked_logitsÚ
chunked_indexÚall_per_token_logpsÚ chunk_logitsÚ chunk_indexÚselected_logitsÚlogsumexp_valuesÚper_token_logpss
ú]/workspace/Fine-tuning/DS-LLM-TEMPLATE-FINETUNING/unsloth_compiled_cache/UnslothCPOTrainer.pyÚchunked_selective_log_softmaxrg"s5õ”[ §¢°°F´LÀÔ4DÑ!EÔ!EÐPQÐYZÐ[€NÝ”[ §¢¨rÑ!2Ô!2¸QÀaÐH€MØÐå%(¨¸Ñ%GÔ%Gð #—¥u¤}Ñ Ýœ, |¸2À{×G\ÒG\Ð]_ÑG`ÔG`Ða×iÐjlÑmˆÝ œ?¨<¸Ø)Ð,<Ñ<ˆØ×" Ýœ,Ð':ÑØ-×5°v´|ÀA´ÈÌ ÐUVÌÐ6XÑØ Ðócó²eZdZUdZedddi¬¦«Zeeed<edddi¬¦«Z ee
ed < d0ˆfd/„ Z ˆxZ S)1ÚUnslothCPOConfiguy
Configuration class for the [`CPOTrainer`].
This class includes only the parameters that are specific to CPO training. For a full list of training arguments,
please refer to the [`~transformers.TrainingArguments`] documentation. Note that default values in this class may
differ from those in [`~transformers.TrainingArguments`].
Using [`~transformers.HfArgumentParser`] we can turn this class into
[argparse](https://docs.python.org/3/library/argparse#module-argparse) arguments that can be specified on the
command line.
Parameters:
max_length (`int` or `None`, *optional*, defaults to `1024`):
Maximum length of the sequences (prompt + completion) in the batch. This argument is required if you want
to use the default data collator.
max_prompt_length (`int` or `None`, *optional*, defaults to `512`):
Maximum length of the prompt. This argument is required if you want to use the default data collator.
max_completion_length (`int` or `None`, *optional*, defaults to `None`):
Maximum length of the completion. This argument is required if you want to use the default data collator
and your model is an encoder-decoder.
beta (`float`, *optional*, defaults to `0.1`):
Parameter controlling the deviation from the reference model. Higher β means less deviation from the
reference model. For the IPO loss (`loss_type="ipo"`), β is the regularization parameter denoted by τ in
the [paper](https://huggingface.co/papers/2310.12036).
label_smoothing (`float`, *optional*, defaults to `0.0`):
Label smoothing factor. This argument is required if you want to use the default data collator.
loss_type (`str`, *optional*, defaults to `"sigmoid"`):
Type of loss to use. Possible values are:
- `"sigmoid"`: sigmoid loss from the original [DPO](https://huggingface.co/papers/2305.18290) paper.
- `"hinge"`: hinge loss on the normalized likelihood from the
[SLiC](https://huggingface.co/papers/2305.10425) paper.
- `"ipo"`: IPO loss from the [IPO](https://huggingface.co/papers/2310.12036) paper.
- `"simpo"`: SimPO loss from the [SimPO](https://huggingface.co/papers/2405.14734) paper.
disable_dropout (`bool`, *optional*, defaults to `True`):
Whether to disable dropout in the model.
cpo_alpha (`float`, *optional*, defaults to `1.0`):
Weight of the BC regularizer in CPO training.
simpo_gamma (`float`, *optional*, defaults to `0.5`):
Target reward margin for the SimPO loss, used only when the `loss_type="simpo"`.
label_pad_token_id (`int`, *optional*, defaults to `-100`):
Label pad token id. This argument is required if you want to use the default data collator.
padding_value (`int` or `None`, *optional*, defaults to `None`):
Padding value to use. If `None`, the padding value of the tokenizer is used.
truncation_mode (`str`,*optional*, defaults to `"keep_end"`):
Truncation mode to use when the prompt is too long. Possible values are `"keep_end"` or `"keep_start"`.
This argument is required if you want to use the default data collator.
generate_during_eval (`bool`, *optional*, defaults to `False`):
If `True`, generates and logs completions from the model to W&B or Comet during evaluation.
is_encoder_decoder (`bool` or `None`, *optional*, defaults to `None`):
When using the `model_init` argument (callable) to instantiate the model instead of the `model` argument,
you need to specify if the model returned by the callable is an encoder-decoder model.
model_init_kwargs (`dict[str, Any]` or `None`, *optional*, defaults to `None`):
Keyword arguments to pass to `AutoModelForCausalLM.from_pretrained` when instantiating the model from a
string.
dataset_num_proc (`int` or `None`, *optional*, defaults to `None`):
Number of processes to use for processing the dataset.
helpzvLLM SamplingParams)ÚdefaultÚmetadataÚvllm_sampling_paramsrJz8Chunk size to reduce memory usage. -1 is most efficient.Úunsloth_num_chunksFÚnorKéréúç-Cëâ6
?ç{®Gáz„?çÍÌÌÌÌÌì?ç+‡ÙÎ÷ï?ç:Œ0âŽyE>çð?çlinearçš™™™™™¹?ÚpassiveÚwarningTÚstepsrPéôéO
ÚO1ÚautoÚçÚ
adamw_8bitÚlengthÚ
every_saveÚlastéééÚsigmoidçà?éœÿÿÿÚkeep_endc“ ó„|dkrtd|d¦«|dkrtd|d¦«||#dkr
|$dkrd}d }#|€!d
d lm}”t |”¦«d zd ¦«}t ¦«jdid
|d|d|d|d|d|d|d|d| “d|
d| d| d|
d|d|d|d|d|d|d |d!|d"|d#|d$|d%|d&|d'|d(|d)|d*|d+|d,| “d-|!“d.|"“d/|#“d0|$“d1|%“d2|&“d3|'“d4|(“d5|)“d6|*“d7|+“d8|,“d9|-“d:|.“d;|/“d<|0“d=|1“d>|2“d?|3“d@|4“dA|5“dB|6“dC|7“dD|8“dE|9“dF|:“dG|;“dH|<“dI|=“dJ|>“dK|?“dL|@“dM|A“dN|B“dO|C“dP|D“dQ|E“dR|F“dS|G“dT|H“dU|I“dV|J“dW|K“dX|L“dY|M“dZ|N“d[|O“d\|P“d]|Q“d^|R“d_|S“d`|T“da|U“db|V“dc|W“dd|X“de|Y“df|Z“dg|[“dh|\“di|]“dj|^“dk|_“dl|`“dm|a“dn|b“do|c“dp|d“dq|e“dr|f“ds|g“dt|h“du|i“dv|j“dw|k“dx|l“dy|m“dz|n“d{|o“d||p“d}|q“d~|r“d|s“d€|t“d|u“d|v“dƒ|w“d„|x“d…|y“d†|z“d‡|{“dˆ||“d‰|}“dŠ|~“d|dŒ|€“d|dŽ|‚“d|ƒ“d|„“d‘|…“d’|†“d“|‡“d”|ˆ“d•|‰“d–|Š“d—|‹“d˜|Œ“d™|dš|Ž“d›|dœ||“¤Ž||_||_dS)žNgH¯¼šò×z>z Unsloth: Your learning rate of `zi` is too small and less than 1e-7! Consider increasing it, otherwise gradient updates will be close to 0!rPza` is way too larger > 1! Consider decreasing it to 1e-1, otherwise gradient updates will explode!r~rÚunsloth_training_checkpointsrpr)Ú cpu_countrqÚ
output_dirÚoverwrite_output_dirÚdo_trainÚdo_evalÚ
do_predictÚ
eval_strategyÚprediction_loss_onlyÚper_device_train_batch_sizeÚper_device_eval_batch_sizeÚper_gpu_train_batch_sizeÚper_gpu_eval_batch_sizeÚgradient_accumulation_stepsÚeval_accumulation_stepsÚ
eval_delayÚtorch_empty_cache_stepsÚ
learning_rateÚ weight_decayÚ
adam_beta1Ú
adam_beta2Ú adam_epsilonÚ
max_grad_normÚnum_train_epochsÚ max_stepsÚlr_scheduler_typeÚ warmup_ratioÚ warmup_stepsÚ log_levelÚlog_level_replicaÚlog_on_each_nodeÚ logging_dirÚlogging_strategyÚlogging_first_stepÚ
logging_stepsÚlogging_nan_inf_filterÚ
save_strategyÚ
save_stepsÚsave_total_limitÚsave_safetensorsÚsave_on_each_nodeÚsave_only_modelÚ'restore_callback_states_from_checkpointÚno_cudaÚuse_cpuÚuse_mps_deviceÚseedÚ data_seedÚ
jit_mode_evalÚuse_ipexÚbf16Úfp16Úfp16_opt_levelÚhalf_precision_backendÚbf16_full_evalÚfp16_full_evalÚtf32Ú
local_rankÚ ddp_backendÚ
tpu_num_coresÚtpu_metrics_debugÚdebugÚdataloader_drop_lastÚ
eval_stepsÚdataloader_num_workersÚdataloader_prefetch_factorÚ
past_indexÚrun_nameÚ disable_tqdmÚremove_unused_columnsÚ label_namesÚload_best_model_at_endÚmetric_for_best_modelÚgreater_is_betterÚignore_data_skipÚfsdpÚfsdp_min_num_paramsÚ fsdp_configÚ"fsdp_transformer_layer_cls_to_wrapÚaccelerator_configÚ deepspeedÚlabel_smoothing_factorÚoptimÚ
optim_argsÚ adafactorÚgroup_by_lengthÚlength_column_nameÚ report_toÚddp_find_unused_parametersÚddp_bucket_cap_mbÚddp_broadcast_buffersÚdataloader_pin_memoryÚdataloader_persistent_workersÚskip_memory_metricsÚuse_legacy_prediction_loopÚ push_to_hubÚresume_from_checkpointÚ hub_model_idÚ hub_strategyÚ hub_tokenÚhub_private_repoÚhub_always_pushÚ hub_revisionÚgradient_checkpointingÚgradient_checkpointing_kwargsÚinclude_inputs_for_metricsÚeval_do_concat_batchesÚ fp16_backendÚpush_to_hub_model_idÚpush_to_hub_organizationÚpush_to_hub_tokenÚ
mp_parametersÚauto_find_batch_sizeÚfull_determinismÚ torchdynamoÚ ray_scopeÚ ddp_timeoutÚ
torch_compileÚtorch_compile_backendÚtorch_compile_modeÚinclude_tokens_per_secondÚinclude_num_input_tokens_seenÚneftune_noise_alphaÚoptim_target_modulesÚbatch_eval_metricsÚ
eval_on_startÚuse_liger_kernelÚliger_kernel_configÚeval_use_gather_objectÚaverage_tokens_across_devicesÚ
max_lengthÚmax_prompt_lengthÚmax_completion_lengthÚbetaÚlabel_smoothingÚ loss_typeÚdisable_dropoutÚ cpo_alphaÚ simpo_gammaÚlabel_pad_token_idÚ
padding_valueÚtruncation_modeÚgenerate_during_evalÚis_encoder_decoderÚmodel_init_kwargsÚdataset_num_proc©) ÚFloatingPointErrorÚ
OverflowErrorÚmultiprocessingrÚminÚsuperÚ__init__rnro)–Úselfr“r”r•rr—r™rrr r­r¿rÿrrrrrrrrrr r
r r r
rrrrrrrrrrrrrrrrrrr r!r"rnroÚkwargsrÚ __class__s €rfr)zUnslothCPOConfig.__init__zs ø€ðl ˜ Ð Õ'9ð;VÐ]jð;Vð;Vð;Vñ(Wô(Wð"WØ ˜ Ð ¥Mð3FÐUbð3Fð3Fð3Fñ%Gô%GðGØ Ð  -°7Ò":Ð":¸zÈSÒ?PÐ?PØ7ˆ ˆ Ð " 9 9¡;¤;¨q¡=°!Ñ àŒÔðPPP#˜ðP :à#7Ð#7ðP Pgð P
$˜ð P *˜
P$8Ð#7ðP+FÐ*EðP*DÐ)CðP(@Ð'?ðP'>Ð&=ðP+FÐ*EðP'>Ð&=ðP$˜ðP'>Ð&=ðP*˜Mð!P :ð"(˜<ð#P :ð$$˜ð%P :ð&$˜ð'P :ð((˜<ð)P :ð**˜Mð+P :ð,/ð-P :ð."˜ ð/P :ð0!2Ð 1ð1P :ð2(˜<ð3P :ð4(˜<ð5P :ð6"˜ ð7P :ð8!2Ð 1ð9P :ð:/ð;P :ð<&˜+ð=P :ð>/ð?P :ð@"4Ð!3ðAP :ðB*˜MðCP :ðD&<Ð%;ðEP :ðF*˜MðGP :ðH$˜ðIP :ðJ/ðKP :ðL/ðMP :ðN!2Ð 1ðOP :ðP.˜oðQP :ðR7^Ð6]ðSP :ðTgðUP :ðVgðWP :ðX,˜^ðYP :ðZ4ð[P :ð\"˜ ð]P :ð^*˜Mð_P :ð` xðaP :ðb4ðcP :ðd4ðeP :ðf,˜^ðgP :ðh&<Ð%;ðiP :ðj,˜^ðkP :ðl,˜^ðmP :ðn4ðoP :ðp$˜ðqP :ðr&˜+ðsP :ðt*˜MðuP :ðv!2Ð 1ðwP :ðxEðyP :ðz$8Ð#7ð{P :ð|$˜ð}P :ð~&<Ð%;ðP :ð@*DÐ)CðAP :ðB$˜ðCP :ðD xðEP :ðF(˜<ðGP :ðH%:Ð$9ðIP :ðJ&˜+ðKP :ðL&<Ð%;ðMP :ðN%:Ð$9ðOP :ðP!2Ð 1ðQP :ðR/ðSP :ðT4ðUP :ðV#6Ð"5ðWP :ðX&˜+ðYP :ðZ2TÐ1Sð[P :ð\"4Ð!3ð]P :ð^"˜ ð_P :ð`&<Ð%;ðaP :ðbEðcP :ðd$˜ðeP :ðf"˜ ðgP :ðh.˜oðiP :ðj"4Ð!3ðkP :ðl"˜ ðmP :ðn*DÐ)CðoP :ðp!2Ð 1ðqP :ðr%:Ð$9ðsP :ðt%:Ð$9ðuP :ðv-JÐ,IðwP :ðx#6Ð"5ðyP :ðz*DÐ)Cð{P :ð|&˜+ð}P :ð~&<Ð%;ðP :ð@(˜<ðAP :ðB(˜<ðCP :ðD"˜ ðEP :ðF/ðGP :ðH.˜oðIP :ðJ(˜<ðKP :ðL&<Ð%;ðMP :ðN-JÐ,IðOP :ðP*DÐ)CðQP :ðR&<Ð%;ðSP :ðT(˜<ðUP :ðV$8Ð#7ðWP :ðX(@Ð'?ðYP :ðZ!2Ð 1ð[P :ð\*˜Mð]P :ð^$8Ð#7ð_P :ð`/ðaP :ðb&˜+ðcP :ðd"˜ ðeP :ðf&˜+ðgP :ðh*˜MðiP :ðj%:Ð$9ðkP :ðl"4Ð!3ðmP :ðn)BÐ(AðoP :ðp-JÐ,IðqP :ðr#6Ð"5ðsP :ðt$8Ð#7ðuP :ðv"4Ð!3ðwP :ðx*˜MðyP :ðz/ð{P :ð|#6Ð"5ð}P :ð~&<Ð%;ðP :ð@-JÐ,IðAP :ðB$˜ðCP :ðD!2Ð 1ðEP :ðF%:Ð$9ðGP :ðH4ðIP :ðJ.˜oðKP :ðL"˜ ðMP :ðN.˜oðOP :ðP"˜ ðQP :ðR&˜+ðSP :ðT"4Ð!3ðUP :ðV*˜MðWP :ðX.˜oðYP :ðZ$8Ð#7ð[P :ð\"4Ð!3ð]P :ð^!2Ð 1ð_P :ð`/°&ðaPPP :ðb%9ˆÔ!Ø"4ˆÔÐÐrh)NNFFFrpFrKrKNNrqrqrrrrsrtrurvrwrxryrJrzr{rr|r}TNr~FrPFr~rNTFFFFFFr€r€FFFFrrFFNrJNNFrƒFNrNrJNNTNFNNFrƒrNNNNr„r…NFFr†NNNNTFTFFNNr‡NNFNFNFTrNNNrƒTFNrˆr‰FNNFFNNFFFNFTrŠrNr{r„TrxrNrFNNNNrJ)
Ú__name__Ú
__module__Ú __qualname__Ú__doc__r?rnrrÚ__annotations__roÚintr)Ú
__classcell__©r,s@rfrjrj3sø€ð<ð<ðz+0¨%ØØÐ+ñ+ô+И( 3œ-ððñð*/¨ØØÐ*ñ*ô*И #œððñð ØØØØØ$Ø&'Ø%&Ø#'Ø"&Ø&'Ø"#ØØ"%ØØØØØØØØØØØØØØØ!&ØØØØØØ27ØØØØØØØØØØØ!'ØØØØØØØØØ!"Ø%)ØØØØ $ØØ!&Ø $Ø Ø ØØØØ-1ØØ!$ØØØØØØ%)Ø Ø $Ø $Ø(-Ø"Ø%*ØØ!%ØØØØØØ!&Ø(,Ø%*Ø!%ØØ#Ø#'Ø ØØ ØØØØØ $Ø!Ø$)Ø(-ØØ Ø"Ø!&Ø(,ØØØ $ØØØØØØØØ ØØðgqqqqqqqqqq5rhrjcó¢eZdZdZddgZ dAdeeeej e
fdee dee d ee
d
eee
ee
e
ffd eeeeeefd eegefd
eeedeejjejjjfdeeejejgejfdeedeeegeffˆfd
ZdZdBdeeeej fdefdZ e! dCdee
eeej"ffde#de$de$deej%dee
ej"ff d¦«Z&dej'd ej'deej'ej'ej'ffd!„Z(e! dDd"ej'd#ej"d$e#de$de#dej'f d%„¦«Z)dej dee
eeej"ffdeej'ej'ej'ej'ffd&„Z* dEdee
eeej"ffd(e+d)fd*„Z, dFdeeej fd+ee
eeje-ffdeejeejee
ejffffd,„Z.dee
ej"fde
fd-„Z/ dBdeeej fd+ee
eeje-ffd.e#d/eee
fd0„Z0dEd1ee
e1fd(e+d)ddfd2„Z2 dGd4e3d5e
d.ee#d/eee
d6e
def ˆfd7„
Z4dBd8ee
e1fd9ee1ddfˆfd:„
Z5d;„Z6ˆfd<„Z7 dHd=ee
d>ee
d?ee
ee
dffd@„Z8ˆxZ9S)IÚ_UnslothCPOTrainerrƒÚtrlÚcpoN©NNÚmodelÚargsÚ
data_collatorÚ
train_datasetÚ eval_datasetÚprocessing_classÚ
model_initÚ callbacksÚ
optimizersÚpreprocess_logits_for_metricsÚ peft_configÚcompute_metricsc
ó¬|ji}
nªt|t¦«std¦«|j}
|
 d¦«}|ht|t¦«r|dkrt t |¦«}|dkr-t|t j¦«std|d¦«||
d<t|t¦«rtj |fi|
¤Ž}d|_
t¦«s| td¦«t¦«r5| 2t|t¦«r| 
¦«}t |dd¦«st |d d¦«r`t|d
¦«o,d
tt!jt$¦«j¦«v}d |ji}|r
|j|d
<t%|fi|¤Ž}nV|jrOt|d ¦«r| ¦«n*d
}| ¦« |¦«|}|jr't |d d¦«rt5|¦«d|_
nV|jrOt|d ¦«r| ¦«n*d}| ¦« |¦«|jr+t9¦«st;¦«std¦«||jj|_n"|jtd¦«|j|_|jr"|jj |_ |jj!|_!|td¦«|j"tGj$dtJ¦«d}n|j"}|j&tGj$dtJ¦«d}n|j&}||kstd|d|d¦«|j'€$|jrtGj$dtJ¦«d}n|j'}|€QtQ|j!|j)|j¬¦«}|j*r!d|_*tGj$dtJ¦«d|_+nd|_+|j,rt[|¦«||_"|j|_|j)|_)|j.|j.n|j!|_.||_&|j/|_/||_'||_0|j1dvr.|j2dkr#tGj$d|j1d tJ¦«|j1d!krtd"¦«|j3|_3|j2|_2|j1|_1|j4|_4t |jd#d¦«|_5t |jd$d%¦«|_6|j5r%|j6d%krtGj$d&tJ¦«|j1d'kr |j7|_7tqd(„¦«|_9d|j:d)<tw¦« <¦«5| =t||j?¬*¦«}| =t€d+|i|j?¬,¦«}|E| =t||j?¬*¦«}| =t€d+|i|j?¬,¦«}| =|jA|j?¬*¦«}|!| =|jA|j?¬*¦«}ddd¦«n #1swxYwYt…¦« C|||||||| || |
¬-¦ « d|_Dt|jEd.¦«r|jE F|jG¦«t|d/¦«std0¦«dS)1NzRYou passed model_kwargs to the CPOTrainer. But your model is already instantiated.Ú torch_dtyperznInvalid `torch_dtype` passed to the CPOConfig. Expected a string with either `torch.dtype` or 'auto', but got ú.FzvPEFT is not installed and you passed a `peft_config` in the trainer's kwargs, please install it to use the PEFT modelsÚis_loaded_in_8bitÚis_loaded_in_4bitrùÚuse_gradient_checkpointingÚenable_input_require_gradscó0| d¦«dS©NT©Úrequires_grad_©ÚmoduleÚinputÚoutputs rfÚmake_inputs_require_gradz=_UnslothCPOTrainer.__init__.<locals>.make_inputs_require_gradøsØ×-¨dÑ3rhTcó0| d¦«dSrNrOrQs rfrUz=_UnslothCPOTrainer.__init__.<locals>.make_inputs_require_grad
sØ×)¨$Ñ/rhz`generate_during_eval=True` requires Weights and Biases or Comet to be installed. Please install `wandb` or `comet-ml` to resolve.zMWhen no model is provided, you need to pass the parameter is_encoder_decoder.z=processing_class must be specified to tokenize a CPO dataset.z`max_length` is not set in the CPOConfig's init it will default to `512` by default, but you should do it yourself in the future.rzˆ`max_prompt_length` is not set in the CPOConfig's init it will default to `128` by default, but you should do it yourself in the future.é€zmax_prompt_length (z+) should be strictly less than max_length (z).z¼When using an encoder decoder architecture, you should set `max_completion_length` in the CPOConfig's init it will default to `128` by default, but you should do it yourself in the future.)Ú pad_token_idrr z²When using DPODataCollatorWithPadding, you should set `remove_unused_columns=False` in your TrainingArguments we have set it for you, but you should do it yourself in the future.)ÚhingeÚiporzYou are using the z™ loss type that does not support label smoothing. The `label_smoothing` parameter will be ignored. Set `label_smoothing` to `0.0` to remove this warning.Úkto_pairzKSupport for kto_pair has been removed in CPOTrainer. Please use KTOTrainer.Úoutput_router_logitsÚrouter_aux_loss_coefr„a-You set `output_router_logits` to `True` in the model config, but `router_aux_loss_coef` is set to `0.0`, meaning the auxiliary loss will not be used. Either set `router_aux_loss_coef` to a value greater than `0.0`, or set `output_router_logits` to `False` if you don't want to use the auxiliary loss.Úsimpocó*tt¦«S©N)r$Úlistr#rhrfú<lambda>z-_UnslothCPOTrainer.__init__.<locals>.<lambda>ƒsµ;½tÑ3DÔ3D€rhÚestimate_tokens)Únum_procÚ tokenizer)Ú fn_kwargsrd) r:r;r<r=r>r?r@rErArBrCÚadd_model_tagsÚ acceleratorzXYour `Trainer` does not have an `accelerator` object. Consider upgrading `transformers`.)Ir!Ú
isinstanceÚstrÚ
ValueErrorÚgetÚgetattrr;Údtyper
Úfrom_pretrainedÚ_peft_has_been_casted_to_bf16r*rÚmerge_and_unloadÚhasattrrar(Ú signaturer7Ú
parametersrørLÚget_input_embeddingsÚregister_forward_hookrÃr6rr,r)Úconfigr Údecoder_start_token_idrXrr<ÚwarnÚ UserWarningrrrrÚuse_dpo_data_collatorrr%rrr?rrrrÚaux_loss_enabledÚ
aux_loss_coefrr$Ú_stored_metricsÚwarnings_issuedrÚmain_process_firstÚmapr/r"r.Ú tokenize_rowr(r)Úmodel_accepts_loss_kwargsr:rgÚ
_tag_namesÚAttributeError)r*r:r;r<r=r>r?r@rArBrCrDrEr!rGÚ_support_gc_kwargsÚprepare_model_kwargsrUrrrr,s €rfr)z_UnslothCPOTrainer.__init__³ø€ð" Ô )Ø "РРݘE¥3Ñ
Ð rà $Ô 6Ð Ø
Ñ>ˆÐ˜k­>°KÀ6Ò4IÐ4IÝ")­Ñ"=Ô"= ¸ÌÑ1UÔ1UÐXðJUðXðXðXñôðð4?Ð! -Ñ e  UÝ8¸ÐTÐBSÐTˆ.3ˆÔ Ñ4 ] {Ð'>ÝðIñôð
õÑ
Ô
ñ0 ] [Ñ%<å˜Ñ
×0å1°5Ñ
a½WÀUÐL_ÐafÑ=gÔ=gð
aÝ%,ØÐ&ô&ð&àÝÔ%Õ&EÑ:ô:ðð )EÀdÔFaÐ'bÐoØLPÔLnÐ(Ð)HÑ7¸ÐVÐAUÐVØÔ
aå˜5Ð">ÑaØ××FÐG_шEØŒyð
:W UÐ,?ÀÑ
+¨EÑ2à59Ô2øð
Ô
]å
]Ø××BÐC[Ñ Ô  Õ.@Ñ.BÔ.Bð ÕFXÑFZÔFZð ÝðDñôð
ð
Ð Ø&+¤lÔ&Eˆ
Ô
Ð mà&*Ô&=ˆ Ô  :Ø*/¬,Ô*Mˆ 'Ø %¤ Ô 9ˆ à Ð Ð Œ?Ð ŒMðeåñ
ô
ð
ð
ˆJˆJàœˆJØ Ô ŒMðeåñ
ô
ð
ð
!$Ð Ð à $Ô 6Ð à  ØrÐ&7ÐrÐdnÐôð
ð Ô -°$Ô2IÐ ŒMðeåñ
ô
ð
ð
%(Ð !à$(Ô$>Ð Ð Ý:Ø#'Ô#:Ø#'Ô#:ðñôˆMð Ô
Ø-2Ô
ð\åñôðð *.ˆ &à).ˆ  Ô ð $  $ˆŒØ$(Ô$=ˆÔ!Ø"&Ô"9ˆÔØ37Ô3EÐ3Q˜/ÐWgÔWtˆÔØ!2ˆÔØÔØ%:ˆÔ"Ø 0ˆÔà Œ>Ð -°$Ô2FÈÒ2JÐ2JÝ ŒMðv T¤^ðvðvðvåñ
ô
ð
ð
Œ>˜ Ð ”IˆŒ ØÔØœˆŒØœˆŒÝ '¨¬ Ð6LÈeÑ TÔ TˆÔÝ$ U¤\Ð3IÈ3ÑÔØ Ô ð  TÔ%7¸3Ò%>Ð%>Ý ŒMðõñ 
ô
ð
ð Œ>˜WÒ /ˆ å*Ð+DÐ+DÑÔð48ˆÔЉ^Œ^×
 cð cà)×-Õ.BÈTÔMbÐcˆ)×)°kÐCSÐ5TÐ_cÔ_tðôˆÐ+×/Õ0DÈtÔOdÐe Ø+×*Ð,<Ð ô  ð-¨dÔ.?È$ÔJ_Ð`ˆÐ+×Ô0AÈDÔLaÐb ð# cð cð cñ cô cð cð cð cð cð cð cøøøð cð cð cð cõ& Œ×ÒØØØØ!Ø*Gð ñ
ô
ð
ð"*/ˆÔ 4”:Ð  ŒJ× % d¤oÑ t˜]Ñ Ý Øôð
ð ð s×&C[Û[Û
[cóö| ||zd¬¦«}| |d¬¦«d}|dt|¦«d}|dt|¦«d}tj||g¦«}tj|d¦«}t|¦«t|¦«krt d¦«t|¦«} ||dd| …kr| dz} |dd| …}|dd| …}
t|¦«t|
¦«krt d¦«|d| d}|d| d}t
||
||¬ ¦«S)
a
Llama tokenizer does satisfy `enc(a + b) = enc(a) + enc(b)`. It does ensure `enc(a + b) = enc(a) + enc(a +
b)[len(enc(a)):]`. Reference:
https://github.com/EleutherAI/lm-evaluation-harness/pull/531#issuecomment-1595586257
Úadd_special_tokensÚ input_idsNÚattention_maskzBPrompt input ids and answer input ids should have the same length.rPz@Prompt input ids and attention mask should have the same length.)Úprompt_input_idsÚprompt_attention_maskr)r?Úlenr1Ú concatenateÚarrayrkÚdict) r*ÚpromptÚanswerÚfull_tokenizedrÚanswer_input_idsÚanswer_attention_maskÚfull_concat_input_idsÚfull_input_idsÚresponse_token_ids_start_idxrŽs rfÚbuild_tokenized_answerz)_UnslothCPOTrainer.build_tokenized_answer¿ð×.¨v¸©ÐSXÐØ×ÈEÐRÐS^Ôà)¨+Ô6µsÐ;KÑ7LÔ7LÐ7NÐ7NÔØ .Ð/?Ô @ÅÐEUÑAVÔAVÐAXÐAXÔ YÐõ!#¤Ð0@ÐBRÐ/SÑ TÔ TÐõœ Ô"=Ñ>ˆå ˆ Ô ¥#Ð&;Ñ"<Ô"<Ò Ð (+Ð+;Ñ'<Ô'<Ð ˜~¨kÔ:Ð;XÐ<XÐ;XÔ (¨AÑ )¨+Ô6Ð7TÐ8TÐ7TÔØ .Ð/?Ô @ÐA^ÐB^ÐA^Ô _Ðå ÐÑ Ô ¥CÐ(=Ñ$>Ô$>Ò Ð )¨+Ô6Ð7SÐ7TÐ7TÔØ .Ð/?Ô @ÐA]ÐA^ÐA^Ô _ÐåØ-Ø"7Ø
ñ
ô
ð
rhÚreturnc óœ
i}|d}|d}|d}|js-t|t¦«stdt |¦«¦«| |d¬¦«}d| ¦«D¦«}t|t¦«stdt |¦«¦«| ||¦«Št|t¦«std t |¦«¦«| ||¦«Št|d
¦«}td
¦«} td
¦«}
t| |
¦«}| ¦«D]\} } | d ||| <Œtd td
d
¦«D¦«¦«}
t| |
z
¦«}|
d
ks|d
krtd¦«t|jj||| ‰|
¦«\}ŠŠt|jj¦«\ŠŠt#td¦«td¦«¦«}|fD]}t|d
¦«|z|jkrj|jdkrdD]} || d |j|| <ŒŒL|jdkrdD]} || |j d || <ŒŒvtd|j¦«ŒŽ‰fD]H}t|d
¦«|z|jkr%dD]"} || d |j|jz
|| <Œ#ŒIˆfddD¦«}ˆfddD¦«}|dd d |d<|jgtd
¦«z|dd td
¦«<|dd d |d<|jgtd
¦«z|dd td
¦«<|||dœ ¦«D]/\} }| ¦«D]\}}|dkrŒ ||| |<ŒŒ0nú| |d|jd¬¦«Š| |d|jd¬¦«Š| |d|jd¬¦«}d|d<d|d<|d|d
<|d|d<|rt/|d ¦«rb| t3j|d¦«¬!¦«|d"<| t3j|d¦«¬!¦«|d#<|S)$a.Tokenize a single row from a CPO specific dataset.
At this stage, we don't convert to PyTorch tensors yet; we just handle the truncation in case the prompt +
chosen or prompt + rejected responses is/are too long. First we truncate the prompt; if we're still too long,
we truncate the chosen/rejected.
We also create the labels for the chosen/rejected responses, which are of length equal to the sum of the length
of the prompt and the chosen/rejected response, with label_pad_token_id for the prompt tokens.
r“ÚchosenÚrejectedz prompt should be an str but got Fr‰có i|] \}}d||Œ S©Úprompt_r#)Ú.0Úvs rfú
<dictcomp>z3_UnslothCPOTrainer.tokenize_row.<locals>.<dictcomp>s$ÐP±$°!°Q˜] q˜]˜]¨AÐPrhz chosen should be an str but got z"rejected should be an str but got rNcó g|] \}}||kŒ Sr#r#)Úbs rfú
<listcomp>z3_UnslothCPOTrainer.tokenize_row.<locals>.<listcomp> s Ðp™D˜A˜qaÐprhrPzdChosen and rejected prompt_input_ids might only differ on the last token due to tokenizer merge ops.rÚ
keep_start)rrzUnknown truncation mode: )rcó:i|]}|d||zŒSr#)Ú
chosen_tokenss €rfz3_UnslothCPOTrainer.tokenize_row.<locals>.<dictcomp>Ns=ø€ð&ð&ð&ØGH= ¨1  Ô/°-ÀÔ2BÑ&ð&ð&rhcó:i|]}|d||zŒSr#)Úrejected_tokenss €rfz3_UnslothCPOTrainer.tokenize_row.<locals>.<dictcomp>Qs=ø€ð(ð(ð(ØKL? =¨Q = 1°OÀAÔ4FÑ(ð(ð(rhÚlabels)Úchosen_Ú rejected_rƒÚtoken_type_idsT)Ú
truncationrÚ
chosen_labelsÚrejected_labelsrŒÚ%prepare_decoder_input_ids_from_labels)Úrejected_decoder_input_idsÚchosen_decoder_input_ids)r rirjrkÚtyper?Úitemsrrr'ÚsumrTÚabsr!Ú bos_token_idr"Ú eos_token_idÚmaxrrrrrrrr;Útensor)r*Úfeaturer:Úbatchr“Ú
prompt_tokensÚprompt_len_input_idsÚchosen_prompt_len_input_idsÚrejected_prompt_len_input_idsr¤Únum_diff_tokensÚ num_diff_lenÚlonger_response_lengthÚ
answer_tokensÚchosen_sequence_tokensÚrejected_sequence_tokensÚtoksÚtype_keyÚtokensr­s @@rfrz_UnslothCPOTrainer.tokenize_rowðøø€ðˆØ˜ÔØ˜Ô"ˆØ˜&ˆàÔ~ õ ˜f¥cÑ
TÝ Ð!RÅDÈÁLÄLÐ!RÐ!RÑ ×1°&ÈUÐSˆP¸-×:MÒ:MÑ:OÔ:OÐPˆ˜f¥cÑ
TÝ Ð!RÅDÈÁLÄLÐ!RÐ!RÑ ×ÀÑGˆMå˜h­Ñ
XÝ Ð!VÅdÈ8ÁnÄnÐ!VÐ!VÑ"×9¸&À(ÑKˆ$' }Ð5GÔ'HÑ#IÔ#IÐ å*-¨mÐ<NÔ.OÑ*PÔ*PÐ 'Ý,/°Ð@RÔ0SÑ,TÔ,TÐ )Ý#&Ð'BÐDaÑ#bÔ#bÐ à%×

<1Ø#$Ð%:Ð&:Ð%:Ô#;
˜ Ð õp¥C¨
Ð6HÔ(IÈ?Ð[mÔKnÑ$oÔ$oÐôˆOõÐ:Ð=ZÑ[ˆ Ò" l°QÒ&6Ð&6Ý ðôðõ =TØÔØØñ=ô=Ñ 9ˆM˜=¨/õ.EØÔ2°MÀ?ñ.ô.Ñ *ˆM˜&)­¨]¸;Ô-GÑ)HÔ)HÍ#ÈoÐ^iÔNjÑJkÔJkÑ%lÔ%lÐ #0°À-Ð!Pð
]ð
]
Ý}Ð%7Ô9Ð<RÑRÐUYÔUdÒÔ+¨|Ò;Ø!NðZðZ˜AØ/<¸QÔ/?Ð@XÀ$ÔBXÐ@XÔ/Y˜M¨!ÑZàÔÒ;Ø!Nð[ð[˜AØ/<¸QÔ/?ÀÔAWÐ@WÐ@YÐ@YÔ/Z˜M¨!Ñ[õ)Ð)[ÀTÔEYÐ)[Ð)[Ñeð#0°Ð!Að
hð
h
Ý}Ð%7Ô9Ð<RÑRÐUYÔUdÒhðh˜Ø+8¸Ô+;Ð<f¸d¼oÐPTÔPfÑ>fÐ<fÔ+g˜
 (øð&ð&ð&ð&ØLkð&ñ&ô&Ð (ð(ð(ð(ØPoð(ñ(ô(Ð 0FÀkÔ/RÐSTÐSTÐSTÔ/UÐ "  ÔZåMÐ"4ÔZ "  ,Ð-U­s°=ÐASÔ3TÑ/UÔ/UÐ-UÑ 2JÈ+Ô1VÐWXÐWXÐWXÔ1YÐ $  Ô^åOÐ$6Ô^ $  .Ð/Yµ°_ÐEWÔ5XÑ1YÔ1YÐ/YÑ
ð÷Še‰gŒgð 

5
)-¯
ª
© ¬ ð$H˜fØÐ#3Ò Ø.4E˜QÐ* Ð 
!× 4°DÔ4NÐcgðôˆ T°dÔ6PÐeiðôˆ 4°DÔ4JÐ_cðôˆ&3°;Ô%?ˆE "Ø'6°{Ô'Cˆ $Ø(5°kÔ(Bˆ %Ø-:Ð;KÔ-Lˆ Ð ¥W¨UÐ4[Ñ%\Ô%\Ð Ø6;×6aÒ6aÝ œ<¨Ð.?Ô(@Ñ7bñ7ô7Ð5:×4_Ò4_Ý œ<¨¨oÔ(>Ñ5`ñ5ô5Ј rhFrŽrr rrÚdevicec ó”i}|r3t|djd|djd¦«}n2t|djd|djd¦«}|D] }| d¦«r‰t||tj¦«rid|vs|r|}n/| d¦«r|}n| d ¦«rd
}| dd ¦«} t||||¬ ¦«|| <Œ¡|D]Ð}| d
¦«r¹t||tj¦«r™d|vs|r|}n/| d¦«r|}n| d ¦«rd
}| d
d ¦«} t j || t||||¬ ¦«fd
¬¦« 
|¬¦«|| <ŒÑ|rf|d  dd¦« 
|¬¦«|d<|d  dd¦« 
|¬¦«|d<|S)Concatenate the chosen and rejected inputs into a single tensor.
Args:
batch:
A batch of data. Must contain the keys 'chosen_input_ids' and 'rejected_input_ids', which are tensors
of shape (batch_size, sequence_length).
is_encoder_decoder:
Whether the model is an encoder-decoder model.
label_pad_token_id:
The label pad token id.
padding_value:
The padding value to use for the concatenated inputs_ids.
device:
The device for the concatenated inputs.
Returns:
A dictionary containing the concatenated inputs under the key 'concatenated_input_ids'.
rPÚchosen_input_idsÚrejected_input_idsržÚ
_input_idsÚ_attention_maskrÚ concatenated)Ú pad_valuerŸrO©rrqÚconcatenated_input_idsrŽÚconcatenated_attention_mask) rSÚ
startswithrir;rÚendswithÚreplacer4ÚcatrUÚrepeat)
r rrÚconcatenated_batchrÚconcatenated_keys
rfÚconcatenated_inputsz&_UnslothCPOTrainer.concatenated_inputss˜ð4 Ðà ð gݘU 9¸<¸eÐDUÔ>VÔ>\Ð]^Ô>_Ñ`ˆJˆ˜UÐ#5Ô<¸ÐG[ÔA\ÔAbÐcdÔAeÑfˆð pð pˆ|Š|˜
p­*°U¸1´X½u¼|Ñ*LÔ*Lð
pؘq==Ð$6=Ø 2I—ZZ  Ñ"Ø -I—ZZÐ 1Ñ"Ø !IØ#$§9¢9¨X°~Ñ#FÔ#FÐ Ý7DÀUÈ1ÄXÈzÐenÐ7oÑ7oÔ7oÐ"Ð#3Ñ4øØð $ˆ|Š|˜
$­J°u¸Q´ÄÑ,NÔ,Nð
˜q==Ð$6=Ø 2I—ZZ  Ñ"Ø -I—ZZÐ 1Ñ"Ø !IØ#$§9¢9¨Z¸Ñ#HÔ#HÐ Ý7<´*Ð+;Ô% e¨A¤h°
ÀiÐðð 8ñ8ô8÷ "˜F
#Ð#3Ñ4øð ð Ø;@ÐASÔ;T×;[Ò;[Ð\]Ð_`Ñ;aÔ;a×;dÒ;dÐlrÐ;dÑ;sÔ;sÐ Ð Ð5°a¸Ñ;×>ÀfÐ
Ð !rhÚpolicy_chosen_logpsÚpolicy_rejected_logpscóˆ||z
 |jj¦«}|jdkrc|j|jz }||z
}t
j|j|z¦« d|jz
zt
j|j |z¦«|jzz
}n¼|jdkrOt
j|j|z¦« d|jz
zt
j|j |z¦«|jzz
}nb|jdkr tj
d|j|zz
¦«}n7|jdkr|dd|jzz z
dz}ntd|jd¦«|j| |jj¦«  ¦«z}|j| |jj¦«  ¦«z}|||fS) aµCompute the CPO loss for a batch of policy and reference model log probabilities.
Args:
policy_chosen_logps:
Log probabilities of the policy model for the chosen responses. Shape: (batch_size,)
policy_rejected_logps:
Log probabilities of the policy model for the rejected responses. Shape: (batch_size,)
Returns:
A tuple of three tensors: (losses, chosen_rewards, rejected_rewards). The losses tensor contains the CPO
loss for each example in the batch. The chosen_rewards and rejected_rewards tensors contain the rewards for
the chosen and rejected responses, respectively.
r^rPrYrZrqzUnknown loss type: z7. Should be one of ['sigmoid', 'hinge', 'ipo', 'simpo'])
rUrhrrrrÚ
logsigmoidrr;ÚrelurkÚdetach)r*r]Úgamma_logratiosÚlossesÚchosen_rewardsÚrejected_rewardss rfÚcpo_lossz_UnslothCPOTrainer.cpo_lossÅð$&Ð(=Ñ=×AÀ$ÔBRÔBYÑZˆð Œ>˜ ´Ñ:ˆOؘoÑ-ˆFõ˜dœi¨&Ñ1°Q¸Ô9MÑ5MÑ”, ¤ ˜z¨FÑ3°dÔ6JÑKð
ˆŒ^˜
˜dœi¨&Ñ1°Q¸Ô9MÑ5MÑ”, ¤ ˜z¨FÑ3°dÔ6JÑKð
ˆŒ^˜
”Z  D¤I°Ñ$6Ñ 6Ñ7ˆFˆ
Œ^˜
˜q A¨¬ ¡MÑ2°qÑ8ˆFˆFåØm d¤nÐôð
ðœÐ&9×&<Ò&<¸TÔ=MÔ=TÑ&UÔ&U×%]Ò%]Ñ%_Ô%_Ñ_ˆØœ9Ð(=×(@Ò(@ÀÔAQÔAXÑ(YÔ(Y×'aÒ'aÑ'cÔ'cÑà~Ð'7Ð7rhr]Úaverage_log_probcó„|jdd|jkrtd¦«|s1|ddddf ¦«}|ddddddf}||k}d|||k<t||¦«}|r.||z d¦«| d¦«z S||z d¦«S)Compute the log probabilities of the given labels under the given logits.
Args:
logits: Logits of the model (unnormalized). Shape: (batch_size, sequence_length, vocab_size)
labels:
Labels for which to compute the log probabilities. Label tokens with a value of label_pad_token_id are
ignored. Shape: (batch_size, sequence_length)
average_log_prob:
If True, return the average log probability per (non-masked) token. Otherwise, return the sum of the
log probabilities of the (non-masked) tokens.
label_pad_token_id: The label pad token id.
is_encoder_decoder: Whether the model is an encoder-decoder model.
Returns:
A tensor of shape (batch_size,) containing the average/sum log probabilities of the given labels under the
given logits.
NrJzKLogits (batch and sequence length dim) and labels must have the same shape.rPr)rSrkÚcloner9)r]rr Ú loss_maskres rfÚget_batch_logpsz"_UnslothCPOTrainer.get_batch_logpsúð2 Œ<˜˜˜Ô  ¤ Ò Ð ˜A˜A˜A˜q˜r˜r˜E”]×*ˆFؘA˜A˜A˜s ˜s A A A˜&ˆÐ0ˆ ð01ˆˆvиÑà ð# iÑ/×4°RÑ8¸9¿=º=ÈÑ;LÔ;LÑ # /×4°RÑ 8rhcó
 |jjjjj¬¦«}|djd}jrd |d¦«ini}jrd|d<||df|d d
d œ|¤Ž}|j }ˆfd }|d 
¦«} ‰j dkr2tj
d
¦« jj¦«}
n||d|| d|¦«}
 ||djdvjj¬¦«} | d|} | |d}
|d|}||d}jr
| |
|||
|jfS| |
|||
fS)zÆRun the given model on the given batch of inputs, concatenating the chosen and rejected inputs together.
We do this to avoid doing two forward passes, because it's faster for FSDP.
)r rrrÚdecoder_input_idsÚconcatenated_labelsTr\F)Ú use_cachecórjs?|dddddf ¦«}|dddf ¦«}tj¦«}| d|jd¦«}| d¦«}| |j¦«}|||¦«}|S)N.rJrP)r Ú
contiguousr0ÚCrossEntropyLossÚviewrSrU)r]Úloss_fctÚlossr*s €rfÚcross_entropy_losszC_UnslothCPOTrainer.concatenated_forward.<locals>.cross_entropy_lossHø€ØÔ
  S b S¨!¨!¨!  Ô,×9Ø  Q R R œ×5åÔ,ˆ—[[  V¤\°"Ô%5Ñ6ˆ—[‘_”_ˆFà—Y’Y˜vœ}Ñ-ˆFØ8˜F +ˆˆKrhr„N)rZr^)r r)r rrrhrSÚ _shift_rightr|r]rr;rUrÚaux_loss)r*r:Ú