unsloth_compiled_cache/__pycache__/UnslothCPOTrainer.cpython-310.pyc

o
õ×°h=3ã@sdZddlmZddlZddlmZddlmZddlmZm	Z	m
Z
mZmZm
Z
mZmZddlmZmZmZmZmZmZmZmZmZmZmZmZmZmZm
Z
mZmZmZmZm Z m!Z!m"Z"m#Z#mZm$Z$m%Z%m&Z&m'Z'm(Z(m)Z)m*Z*m+Z+m,Z,m-Z-m.Z.m/Z/m0Z0m1Z1m2Z2mZm3Z3m4Z4m5Z5m6Z6m7Z7m8Z8m9Z9m:Z:m;Z;m<Z<mZm=Z=mZm
Z
mZmZm"Z"m-Z-m5Z5mZddl5Z5ddlTddl>m?Z?m@Z@dd	lAmBZBddlZddlCZ3dd
lDm4Z4ddlmZddlEmFZFmGZHdd
dd
d
dœZIejJddeId�dd„ƒZKe?Gdd„deƒƒZL	Gdd„de"ƒZMGdd„deMƒZNdS)z9
2025.8.9
2025.8.10
4.55.4
0.21.0
__UNSLOTH_VERSIONING__
é)ÚTensorN)Ú
functional)ÚAnyÚListÚOptionalÚTupleÚUnionÚDictÚSetÚCallable)<rÚAutoModelForCausalLMÚBaseImageProcessorÚ	CPOConfigÚ
CPOTrainerrÚDPODataCollatorWithPaddingÚDataCollatorÚ
DataLoaderÚDatasetÚEvalLoopOutputÚFÚFeatureExtractionMixinÚLiteralrÚPartialStateÚPathÚ	PeftModelÚPreTrainedModelÚPreTrainedTokenizerBaseÚProcessorMixinÚTrainerÚTrainerCallbackrÚadd_bos_token_if_neededÚadd_eos_token_if_neededÚautocastÚdefaultdictÚdisable_dropout_in_modelÚgenerate_model_cardÚget_comet_experiment_urlÚinspectÚis_comet_availableÚis_peft_availableÚis_torch_fx_proxyÚis_wandb_availableÚlog_table_to_comet_experimentÚmaybe_apply_chat_templateÚmaybe_extract_promptÚnnÚnpÚnullcontextÚosÚ
pad_to_lengthÚpdÚpeft_module_casting_to_bf16Úprepare_model_for_kbit_trainingÚrandomÚselective_log_softmaxÚtextwrapÚtorchÚwarningsrrrrrr)r2r:)Ú*)Ú	dataclassÚfield)ÚVersion)r1)ÚDataCollatorForSeq2SeqÚDataCollatorForLanguageModelingTF)Úepilogue_fusionÚmax_autotuneÚ
shape_paddingz
trace.enabledztriton.cudagraphs)ÚdynamicÚ	fullgraphÚoptionsc
Cs¾tj| d|jd¡ddd�}tj| d¡ddd�}g}t||ƒD](\}}| tj¡}tj|d| d¡d� 	d¡}tj
|dd�}||}	| |	¡q!	t |¡}| |jd|jdf¡}|S)Néÿÿÿÿér)ÚchunksÚdim)rKÚindex©rKé)
r:ÚchunkÚreshapeÚshapeÚzipÚtoÚfloat32ÚgatherÚ	unsqueezeÚsqueezeÚ	logsumexpÚappendÚconcat)
ÚlogitsrLÚchunked_logitsÚ
chunked_indexÚall_per_token_logpsÚchunk_logitsÚchunk_indexÚselected_logitsÚlogsumexp_valuesÚper_token_logps©rdúQ/workspace/DS-LLM-TEMPLATE-FINETUNING/unsloth_compiled_cache/UnslothCPOTrainer.pyÚchunked_selective_log_softmax"s
rfcsžeZdZUdZedddid�Zeeed<edddid�Z	ee
ed	<eddd
id�Zee
ed<						
																														 									!	!					"	#								$														$						%	&				'												(									#				$				)	*														+	,			%	-			.	/		0							d3‡fd1d2„	Z‡Z
S)4ÚUnslothCPOConfiguy
    
    Configuration class for the [`CPOTrainer`].

    This class includes only the parameters that are specific to CPO training. For a full list of training arguments,
    please refer to the [`~transformers.TrainingArguments`] documentation. Note that default values in this class may
    differ from those in [`~transformers.TrainingArguments`].

    Using [`~transformers.HfArgumentParser`] we can turn this class into
    [argparse](https://docs.python.org/3/library/argparse#module-argparse) arguments that can be specified on the
    command line.

    Parameters:
        max_length (`int` or `None`, *optional*, defaults to `1024`):
            Maximum length of the sequences (prompt + completion) in the batch. This argument is required if you want
            to use the default data collator.
        max_prompt_length (`int` or `None`, *optional*, defaults to `512`):
            Maximum length of the prompt. This argument is required if you want to use the default data collator.
        max_completion_length (`int` or `None`, *optional*, defaults to `None`):
            Maximum length of the completion. This argument is required if you want to use the default data collator
            and your model is an encoder-decoder.
        beta (`float`, *optional*, defaults to `0.1`):
            Parameter controlling the deviation from the reference model. Higher Î² means less deviation from the
            reference model. For the IPO loss (`loss_type="ipo"`), Î² is the regularization parameter denoted by Ï„ in
            the [paper](https://huggingface.co/papers/2310.12036).
        label_smoothing (`float`, *optional*, defaults to `0.0`):
            Label smoothing factor. This argument is required if you want to use the default data collator.
        loss_type (`str`, *optional*, defaults to `"sigmoid"`):
            Type of loss to use. Possible values are:

                - `"sigmoid"`: sigmoid loss from the original [DPO](https://huggingface.co/papers/2305.18290) paper.
                - `"hinge"`: hinge loss on the normalized likelihood from the
                  [SLiC](https://huggingface.co/papers/2305.10425) paper.
                - `"ipo"`: IPO loss from the [IPO](https://huggingface.co/papers/2310.12036) paper.
                - `"simpo"`: SimPO loss from the [SimPO](https://huggingface.co/papers/2405.14734) paper.

        disable_dropout (`bool`, *optional*, defaults to `True`):
            Whether to disable dropout in the model.
        cpo_alpha (`float`, *optional*, defaults to `1.0`):
            Weight of the BC regularizer in CPO training.
        simpo_gamma (`float`, *optional*, defaults to `0.5`):
            Target reward margin for the SimPO loss, used only when the `loss_type="simpo"`.
        label_pad_token_id (`int`, *optional*, defaults to `-100`):
            Label pad token id. This argument is required if you want to use the default data collator.
        padding_value (`int` or `None`, *optional*, defaults to `None`):
            Padding value to use. If `None`, the padding value of the tokenizer is used.
        truncation_mode (`str`,*optional*,  defaults to `"keep_end"`):
            Truncation mode to use when the prompt is too long. Possible values are `"keep_end"` or `"keep_start"`.
            This argument is required if you want to use the default data collator.
        generate_during_eval (`bool`, *optional*, defaults to `False`):
            If `True`, generates and logs completions from the model to W&B or Comet during evaluation.
        is_encoder_decoder (`bool` or `None`, *optional*, defaults to `None`):
            When using the `model_init` argument (callable) to instantiate the model instead of the `model` argument,
            you need to specify if the model returned by the callable is an encoder-decoder model.
        model_init_kwargs (`dict[str, Any]` or `None`, *optional*, defaults to `None`):
            Keyword arguments to pass to `AutoModelForCausalLM.from_pretrained` when instantiating the model from a
            string.
        dataset_num_proc (`int` or `None`, *optional*, defaults to `None`):
            Number of processes to use for processing the dataset.
    
    NÚhelpzvLLM SamplingParams)ÚdefaultÚmetadataÚvllm_sampling_paramsrHz8Chunk size to reduce memory usage. -1 is most efficient.Úunsloth_num_chunksz'Maximum sequence length to truncate to.Úmax_seq_lengthFÚnorIéréúç-Cëâ6
?ç{®Gáz„?çÍÌÌÌÌÌì?ç+‡ÙÎ÷ï?ç:Œ0âŽyE>çð?ç@Úlinearçš™™™™™¹?ÚpassiveÚwarningTÚstepsrNéôéO
ÚO1ÚautoÚçÚ
adamw_8bitÚlengthÚ
every_saveÚlastéééÚsigmoidçà?éœÿÿÿÚkeep_endc”–sü|dkrtd|›d�ƒ‚|dkrtd|›d�ƒ‚|dur(|#dkr(|$dkr(d}d	}#|�dur:d
dlm}•t|•ƒdd
ƒ}�tƒjdžid|“d|“d|“d|“d|“d|“d|“d|“d|	“d|
“d|“d|“d|
“d|“d|“d|“d|“d|“d |“d!|“d"|“d#|“d$|“d%|“d&|“d'|“d(|“d)|“d*|“d+|“d,|“d-| “d.|!“d/|"“d0|#“d1|$“d2|%“d3|&“d4|'“d5|(“d6|)“d7|*“d8|+“d9|,“d:|-“d;|.“d<|/“d=|0“d>|1“d?|2“d@|3“dA|4“dB|5“dC|6“dD|7“dE|8“dF|9“dG|:“dH|;“dI|<“dJ|=“dK|>“dL|?“dM|@“dN|A“dO|B“dP|C“dQ|D“dR|E“dS|F“dT|G“dU|H“dV|I“dW|J“dX|K“dY|L“dZ|M“d[|N“d\|O“d]|P“d^|Q“d_|R“d`|S“da|T“db|U“dc|V“dd|W“de|X“df|Y“dg|Z“dh|[“di|\“dj|]“dk|^“dl|_“dm|`“dn|a“do|b“dp|c“dq|d“dr|e“ds|f“dt|g“du|h“dv|i“dw|j“dx|k“dy|l“dz|m“d{|n“d||o“d}|p“d~|q“d|r“d€|s“d�|t“d‚|u“dƒ|v“d„|w“d…|x“d†|y“d‡|z“dˆ|{“d‰||“dŠ|}“d‹|~“dŒ|“d�|€“dŽ|�“d�|‚“d�|ƒ“d‘|„“d’|…“d“|†“d”|‡“d•|ˆ“d–|‰“d—|Š“d˜|‹“d™|Œ“dš|�“d›|Ž“dœ|�“d�|�“|”¤Ž|‘|_|’|_|“|_	dS)ŸNgH¯¼šò×z>z Unsloth: Your learning rate of `zi` is too small and less than 1e-7! Consider increasing it, otherwise gradient updates will be close to 0!rNza` is way too larger > 1! Consider decreasing it to 1e-1, otherwise gradient updates will explode!r|r}Úunsloth_training_checkpointsrnr)Ú	cpu_countrIroÚ
output_dirÚoverwrite_output_dirÚdo_trainÚdo_evalÚ
do_predictÚ
eval_strategyÚprediction_loss_onlyÚper_device_train_batch_sizeÚper_device_eval_batch_sizeÚper_gpu_train_batch_sizeÚper_gpu_eval_batch_sizeÚgradient_accumulation_stepsÚeval_accumulation_stepsÚ
eval_delayÚtorch_empty_cache_stepsÚ
learning_rateÚweight_decayÚ
adam_beta1Ú
adam_beta2Úadam_epsilonÚ
max_grad_normÚnum_train_epochsÚ	max_stepsÚlr_scheduler_typeÚwarmup_ratioÚwarmup_stepsÚ	log_levelÚlog_level_replicaÚlog_on_each_nodeÚlogging_dirÚlogging_strategyÚlogging_first_stepÚ
logging_stepsÚlogging_nan_inf_filterÚ
save_strategyÚ
save_stepsÚsave_total_limitÚsave_safetensorsÚsave_on_each_nodeÚsave_only_modelÚ'restore_callback_states_from_checkpointÚno_cudaÚuse_cpuÚuse_mps_deviceÚseedÚ	data_seedÚ
jit_mode_evalÚuse_ipexÚbf16Úfp16Úfp16_opt_levelÚhalf_precision_backendÚbf16_full_evalÚfp16_full_evalÚtf32Ú
local_rankÚddp_backendÚ
tpu_num_coresÚtpu_metrics_debugÚdebugÚdataloader_drop_lastÚ
eval_stepsÚdataloader_num_workersÚdataloader_prefetch_factorÚ
past_indexÚrun_nameÚdisable_tqdmÚremove_unused_columnsÚlabel_namesÚload_best_model_at_endÚmetric_for_best_modelÚgreater_is_betterÚignore_data_skipÚfsdpÚfsdp_min_num_paramsÚfsdp_configÚ"fsdp_transformer_layer_cls_to_wrapÚaccelerator_configÚ	deepspeedÚlabel_smoothing_factorÚoptimÚ
optim_argsÚ	adafactorÚgroup_by_lengthÚlength_column_nameÚ	report_toÚddp_find_unused_parametersÚddp_bucket_cap_mbÚddp_broadcast_buffersÚdataloader_pin_memoryÚdataloader_persistent_workersÚskip_memory_metricsÚuse_legacy_prediction_loopÚpush_to_hubÚresume_from_checkpointÚhub_model_idÚhub_strategyÚ	hub_tokenÚhub_private_repoÚhub_always_pushÚhub_revisionÚgradient_checkpointingÚgradient_checkpointing_kwargsÚinclude_inputs_for_metricsÚeval_do_concat_batchesÚfp16_backendÚpush_to_hub_model_idÚpush_to_hub_organizationÚpush_to_hub_tokenÚ
mp_parametersÚauto_find_batch_sizeÚfull_determinismÚtorchdynamoÚ	ray_scopeÚddp_timeoutÚ
torch_compileÚtorch_compile_backendÚtorch_compile_modeÚinclude_tokens_per_secondÚinclude_num_input_tokens_seenÚneftune_noise_alphaÚoptim_target_modulesÚbatch_eval_metricsÚ
eval_on_startÚuse_liger_kernelÚliger_kernel_configÚeval_use_gather_objectÚaverage_tokens_across_devicesÚ
max_lengthÚmax_prompt_lengthÚmax_completion_lengthÚbetaÚlabel_smoothingÚ	loss_typeÚdisable_dropoutÚ	cpo_alphaÚsimpo_gammaÚlabel_pad_token_idÚ
padding_valueÚtruncation_modeÚgenerate_during_evalÚis_encoder_decoderÚmodel_init_kwargsÚdataset_num_procrd)
ÚFloatingPointErrorÚ
OverflowErrorÚmultiprocessingr�ÚmaxÚsuperÚ__init__rkrlrm)–Úselfr�r‘r’r“r”r•r–r—r˜r™ršr›rœr�ržrŸr r¡r¢r£r¤r¥r¦r§r¨r©rªr«r¬rr®r¯r°r±r²r³r´rµr¶r·r¸r¹rºr»r¼r½r¾r¿rÀrÁrÂrÃrÄrÅrÆrÇrÈrÉrÊrËrÌrÍrÎrÏrÐrÑrÒrÓrÔrÕrÖr×rØrÙrÚrÛrÜrÝrÞrßràrárârãrärårærçrèrérêrërìrírîrïrðrñròrórôrõrör÷rørùrúrûrürýrþrÿrrrrrrrrrr	r
rrr
rrrrrrrrrrrrrrrrrrrkrlrmÚkwargsr�©Ú	__class__rdrer%~s ÿþýüûúùø	÷
öõô
óòñðïîíìëêéèçæåäãâá à!ß"Þ#Ý$Ü%Û&Ú'Ù(Ø)×*Ö+Õ,Ô-Ó.Ò/Ñ0Ð1Ï2Î3Í4Ì5Ë6Ê7É8È9Ç:Æ;Å<Ä=Ã>Â?Á@ÀA¿B¾C½D¼E»FºG¹H¸I·J¶KµL´M³N²O±P°Q¯R®ST¬U«VªW©X¨Y§Z¦[¥\¤]£^¢_¡` aŸbžc�dœe›fšg™h˜i—j–k•l”m“n’o‘p�q�rŽs�tŒu‹vŠw‰xˆy‡z†{…|„}ƒ~‚��ÿ�þ�ý�ü�û�ú�ù�ø	�÷
�ö�õ�ô
�ó�ò�ñ�ð�ï
zUnslothCPOConfig.__init__)“NNFFFrnFrIrINNrororrprqrrrsrtrurvrwrHrxryrrzr{TNr|FrNFr|r}NTFFFFFFr~r~FFFFrr€FFNrHNNFr�FNrNrHNNTNFNNFr�rNNNNr‚rƒNFFr„NNNNTFTFFNNr…NNFNFNFTr€NNNr�TFNr†r‡FNNFFNNFFFNFTrˆr‰Nryr‚rŠTrvr‹rŒNr�FNNNNrHN)Ú__name__Ú
__module__Ú__qualname__Ú__doc__r>rkrrÚ__annotations__rlÚintrmr%Ú
__classcell__rdrdr(rerg3sF
=þþþ�ërgcsÈeZdZdZddgZ												dRdeeeej	e
fdeedeed	ee
d
eee
ee
e
ffdeeeeeefdeegefd
eeedeejjejjjfdeeejejgejfdeedeeegeff‡fdd„
Zdd„ZdSdeeeej	fdefdd„Z e!				dTdee
eeej"ffde#de$de$d eej%dee
ej"ffd!d"„ƒZ&d#ej'd$ej'deej'ej'ej'ffd%d&„Z(e!			dUd'ej'd(ej"d)e#de$de#dej'fd*d+„ƒZ)dej	dee
eeej"ffdeej'ej'ej'ej'ffd,d-„Z*	.dVdee
eeej"ffd/e+d0fd1d2„Z,		dWdeeej	fd3ee
eeje-ffdeejeejee
ejffffd4d5„Z.dee
ej"fde
fd6d7„Z/	dSdeeej	fd3ee
eeje-ffd8e#d9eee
fd:d;„Z0dVd<ee
e1fd/e+d0ddfd=d>„Z2			?dXd@e3dAe
d8ee#d9eee
dBe
def‡fdCdD„
Z4dSdEee
e1fdFee1ddf‡fdGdH„
Z5dIdJ„Z6‡fdKdL„Z7			dYdMee
dNee
dOee
ee
dffdPdQ„Z8‡Z9S)ZÚ_UnslothCPOTrainerr�ÚtrlÚcpoN©NNÚmodelÚargsÚ
data_collatorÚ
train_datasetÚeval_datasetÚprocessing_classÚ
model_initÚ	callbacksÚ
optimizersÚpreprocess_logits_for_metricsÚpeft_configÚcompute_metricsc

sF|jduri}
n9t|tƒstdƒ‚|j}
|
 d¡}|durAt|tƒr+|dkr+tt|ƒ}|dkr=t|tjƒs=td|›d�ƒ‚||
d<t|tƒrOtj	|fi|
¤Ž}d|_
tƒs]|dur]tdƒ‚tƒrÈ|durÈt|tƒrm| 
¡}t|ddƒsyt|d	dƒr�t|d
ƒo‡d
tt t¡jƒv}d|ji}|r”|j|d
<t|fi|¤Ž}n|jrµt|dƒrª| ¡nd
d„}| ¡ |¡|}|jrÇt|d	dƒrÇt|ƒd|_
n|jràt|dƒrÕ| ¡ndd„}| ¡ |¡|jrítƒsítƒsítdƒ‚|dur÷|jj|_n|jdu�rtdƒ‚|j|_|j�r|jj |_ |jj!|_!|du�rtdƒ‚|j"du�r+t# $dt%¡d}n|j"}|j&du�r=t# $dt%¡d}n|j&}||k�sPtd|›d|›d�ƒ‚|j'du�rc|j�rct# $dt%¡d}n|j'}|du�r†t(|j!|j)|jd�}|j*�r‚d|_*t# $dt%¡d|_+nd|_+|j,�r‘t-|ƒ||_"|j|_|j)|_)|j.du�r¥|j.n|j!|_.||_&|j/|_/||_'||_0|j1dv�rÍ|j2dk�rÍt# $d |j1›d!�t%¡|j1d"k�r×td#ƒ‚|j3|_3|j2|_2|j1|_1|j4|_4t|jd$dƒ|_5t|jd%d&ƒ|_6|j5�r|j6d&k�rt# $d't%¡|j1d(k�r|j7|_7t8d)d*„ƒ|_9d|j:d+<t;ƒ <¡�J|j=t>|j?d,�}|j=t@d-|i|j?d.�}|du�rN|j=t>|j?d,�}|j=t@d-|i|j?d.�}|j=|jA|j?d,�}|du�re|j=|jA|j?d,�}Wdƒn	1�spwYtBƒjC||||||||||	|
d/�d|_Dt|jEd0ƒ�r—|jE F|jG¡t|d1ƒ�s¡tHd2ƒ‚dS)3NzRYou passed model_kwargs to the CPOTrainer. But your model is already instantiated.Útorch_dtyper€znInvalid `torch_dtype` passed to the CPOConfig. Expected a string with either `torch.dtype` or 'auto', but got Ú.FzvPEFT is not installed and you passed a `peft_config` in the trainer's kwargs, please install it to use the PEFT modelsÚis_loaded_in_8bitÚis_loaded_in_4bitröÚuse_gradient_checkpointingÚenable_input_require_gradscSó| d¡dS©NT©Úrequires_grad_©ÚmoduleÚinputÚoutputrdrdreÚmake_inputs_require_gradþóz=_UnslothCPOTrainer.__init__.<locals>.make_inputs_require_gradTcSrGrHrIrKrdrdrerOrPz‚`generate_during_eval=True` requires Weights and Biases or Comet to be installed. Please install `wandb` or `comet-ml` to resolve.zMWhen no model is provided, you need to pass the parameter is_encoder_decoder.z=processing_class must be specified to tokenize a CPO dataset.z�`max_length` is not set in the CPOConfig's init it will default to `512` by default, but you should do it yourself in the future.r‰zˆ`max_prompt_length` is not set in the CPOConfig's init it will default to `128` by default, but you should do it yourself in the future.é€zmax_prompt_length (z+) should be strictly less than max_length (z).z¼When using an encoder decoder architecture, you should set `max_completion_length` in the CPOConfig's init it will default to `128` by default, but you should do it yourself in the future.)Úpad_token_idrrz²When using DPODataCollatorWithPadding, you should set `remove_unused_columns=False` in your TrainingArguments we have set it for you, but you should do it yourself in the future.)ÚhingeÚiporzYou are using the z™ loss type that does not support label smoothing. The `label_smoothing` parameter will be ignored. Set `label_smoothing` to `0.0` to remove this warning.Úkto_pairzKSupport for kto_pair has been removed in CPOTrainer. Please use KTOTrainer.Úoutput_router_logitsÚrouter_aux_loss_coefr‚a-You set `output_router_logits` to `True` in the model config, but `router_aux_loss_coef` is set to `0.0`, meaning the auxiliary loss will not be used. Either set `router_aux_loss_coef` to a value greater than `0.0`, or set `output_router_logits` to `False` if you don't want to use the auxiliary loss.ÚsimpocSsttƒS©N)r#ÚlistrdrdrdreÚ<lambda>‰sz-_UnslothCPOTrainer.__init__.<locals>.<lambda>Úestimate_tokens)Únum_procÚ	tokenizer)Ú	fn_kwargsr])r5r6r7r8r9r:r;r@r<r=r>Úadd_model_tagsÚacceleratorzXYour `Trainer` does not have an `accelerator` object. Consider upgrading `transformers`.)IrÚ
isinstanceÚstrÚ
ValueErrorÚgetÚgetattrr:ÚdtyperÚfrom_pretrainedÚ_peft_has_been_casted_to_bf16r)rÚmerge_and_unloadÚhasattrrZr'Ú	signaturer6Ú
parametersrõrörFÚget_input_embeddingsÚregister_forward_hookrÀr5rr+r(ÚconfigrÚdecoder_start_token_idrRrr;ÚwarnÚUserWarningrrrrrÓÚuse_dpo_data_collatorrr$rrr:rrrrÚaux_loss_enabledÚ
aux_loss_coefrr#Ú_stored_metricsÚwarnings_issuedrÚmain_process_firstÚmapr.rr-Útokenize_rowr$r%Úmodel_accepts_loss_kwargsr5r`Ú
_tag_namesÚAttributeError)r&r5r6r7r8r9r:r;r<r=r>r?r@rrAÚ_support_gc_kwargsÚprepare_model_kwargsrOrrrr(rdrer%¹s\


ÿ
ÿ
ÿ
ÿþ


€

ÿ


ýý
ÿý
ýýýû
	ÿ
ý
€ïõÿÿz_UnslothCPOTrainer.__init__cCs|j||dd�}|j|dd�d}|dt|ƒd…}|dt|ƒd…}t ||g¡}t |d¡}t|ƒt|ƒkr@tdƒ‚t|ƒ}	||dd|	…krR|	d8}	|dd|	…}|dd|	…}
t|ƒt|
ƒkrntdƒ‚|d|	d…}|d|	d…}t||
||d	�S)
a
        Llama tokenizer does satisfy `enc(a + b) = enc(a) + enc(b)`. It does ensure `enc(a + b) = enc(a) + enc(a +
        b)[len(enc(a)):]`. Reference:
            https://github.com/EleutherAI/lm-evaluation-harness/pull/531#issuecomment-1595586257
        F©Úadd_special_tokensÚ	input_idsNÚattention_maskzBPrompt input ids and answer input ids should have the same length.rNz@Prompt input ids and attention mask should have the same length.)Úprompt_input_idsÚprompt_attention_maskrƒr„)r:Úlenr0ÚconcatenateÚarrayrdÚdict)r&ÚpromptÚanswerÚfull_tokenizedr…Úanswer_input_idsÚanswer_attention_maskÚfull_concat_input_idsÚfull_input_idsÚresponse_token_ids_start_idxr†rdrdreÚbuild_tokenized_answerÅs.üz)_UnslothCPOTrainer.build_tokenized_answerÚreturncsi}|d}|d}|d}|j�s±t|tƒs tdt|ƒ›�ƒ‚|j|dd�}dd„| ¡Dƒ}t|tƒs>td	t|ƒ›�ƒ‚| ||¡‰t|tƒsRtd
t|ƒ›�ƒ‚| ||¡‰t|dƒ}tˆdƒ}	tˆdƒ}
t	|	|
ƒ}| ¡D]\}}|d|…||<qst
d
d„tˆdˆdƒDƒƒ}
t|	|
ƒ}|
dksž|dkr¢tdƒ‚t
|jj|||	ˆ|
ˆƒ\}‰‰t|jjˆˆƒ\‰‰ttˆdƒtˆdƒƒ}ˆˆ|fD]D}t|dƒ||jk�r|jdkrñdD]
}||d|j…||<qâqÍ|jdk�r	dD]}|||jd…||<qùqÍtd|j›�ƒ‚qÍˆˆfD]#}t|dƒ||jk�r8dD]}||d|j|j…||<�q&�q‡fdd„dDƒ}‡fdd„dDƒ}|ddd…|d<|jgtˆdƒ|ddtˆdƒ…<|ddd…|d<|jgtˆdƒ|ddtˆdƒ…<|||dœ ¡D]\}}| ¡D]\}}|dk�r£�q˜|||›|›�<�q˜�q�|S|j|d|jdd�‰|j|d|jdd�‰|j|d|jdd�}ˆd|d<ˆd|d<|d|d<|d |d!<|du�rt|d"ƒ�r|jt |d¡d#�|d$<|jt |d¡d#�|d%<|S)&a.Tokenize a single row from a CPO specific dataset.

        At this stage, we don't convert to PyTorch tensors yet; we just handle the truncation in case the prompt +
        chosen or prompt + rejected responses is/are too long. First we truncate the prompt; if we're still too long,
        we truncate the chosen/rejected.

        We also create the labels for the chosen/rejected responses, which are of length equal to the sum of the length
        of the prompt and the chosen/rejected response, with label_pad_token_id for the prompt tokens.
        r‹ÚchosenÚrejectedz prompt should be an str but got Fr�cSsi|]
\}}d|›�|“qS©Úprompt_rd©Ú.0ÚkÚvrdrdreÚ
<dictcomp>óz3_UnslothCPOTrainer.tokenize_row.<locals>.<dictcomp>z chosen should be an str but got z"rejected should be an str but got r…NcSsg|]\}}||k‘qSrdrd)ršÚaÚbrdrdreÚ
<listcomp>&sz3_UnslothCPOTrainer.tokenize_row.<locals>.<listcomp>rNzdChosen and rejected prompt_input_ids might only differ on the last token due to tokenizer merge ops.rƒÚ
keep_start)r…r†r�zUnknown truncation mode: )rƒr„có$i|]}|ˆd|›�ˆ|“qSr—rd©ršr›)Ú
chosen_tokensrdrer�Tóÿcr£r—rdr¤)Úrejected_tokensrdrer�Wr¦Úlabels)Úchosen_Ú	rejected_r�Útoken_type_idsT)Ú
truncationrr‚Ú
chosen_labelsÚrejected_labelsr„r†Ú%prepare_decoder_input_ids_from_labels)r¨Úrejected_decoder_input_idsÚchosen_decoder_input_ids)rrbrcrdÚtyper:Úitemsr“r‡ÚminÚsumrRÚabsr Úbos_token_idr!Úeos_token_idr#rrrrrrkr¯r:Útensor)r&Úfeaturer5Úbatchr‹r•r–Ú
prompt_tokensÚprompt_len_input_idsÚchosen_prompt_len_input_idsÚrejected_prompt_len_input_idsr›rœÚnum_diff_tokensÚnum_diff_lenÚlonger_response_lengthÚ
answer_tokensÚchosen_sequence_tokensÚrejected_sequence_tokensÚtoksÚtype_keyÚtokensrd)r¥r§rer{ösØ


ÿÿ
ù
ÿ
ÿÿø €
ÿ
ÿÿ
þÿ
þý
ü
ýé
ÿ
ÿ
ÿ
ÿ
ÿz_UnslothCPOTrainer.tokenize_rowFrŒrr»rrrÚdevicec
CsŒi}|rt|djd|djdƒ}nt|djd|djdƒ}|D]8}| d¡r]t||tjƒr]d|vs:|r=|}n| d¡rE|}n| d	¡rLd
}| dd¡}	t||||d�||	<q%|D]E}| d
¡r¥t||tjƒr¥d|vsu|rx|}n| d¡r€|}n| d	¡r‡d
}| d
d¡}	tj	||	t||||d�fd
d�j
|d�||	<q`|rÄ|d dd¡j
|d�|d<|d dd¡j
|d�|d<|S)açConcatenate the chosen and rejected inputs into a single tensor.

        Args:
            batch:
                A batch of data. Must contain the keys 'chosen_input_ids' and 'rejected_input_ids', which are tensors
                of shape (batch_size, sequence_length).
            is_encoder_decoder:
                Whether the model is an encoder-decoder model.
            label_pad_token_id:
                The label pad token id.
            padding_value:
                The padding value to use for the concatenated inputs_ids.
            device:
                The device for the concatenated inputs.

        Returns:
            A dictionary containing the concatenated inputs under the key 'concatenated_input_ids'.
        rrNr®Úchosen_input_idsÚrejected_input_idsr•r¨Ú
_input_idsÚ_attention_maskrÚconcatenated)Ú	pad_valuer–rM©rÉr…roÚconcatenated_input_idsr†Úconcatenated_attention_mask)r#rQÚ
startswithrbr:rÚendswithÚreplacer3ÚcatrSÚrepeat)
r»rrrrÉÚconcatenated_batchrr›rÏÚconcatenated_keyrdrdreÚconcatenated_inputs‡sL 

€

þû
ú€ÿz&_UnslothCPOTrainer.concatenated_inputsÚpolicy_chosen_logpsÚpolicy_rejected_logpscCs4|| |jj¡}|jdkr3|j|j}||}t |j|¡d|jt |j|¡|j}nJ|jdkrSt |j|¡d|jt |j|¡|j}n*|jdkrct	 
d|j|¡}n|jdkrt|dd|jd}n	td|j›d�ƒ‚|j| |jj¡ ¡}|j| |jj¡ ¡}|||fS)	aµCompute the CPO loss for a batch of policy and reference model log probabilities.

        Args:
            policy_chosen_logps:
                Log probabilities of the policy model for the chosen responses. Shape: (batch_size,)
            policy_rejected_logps:
                Log probabilities of the policy model for the rejected responses. Shape: (batch_size,)

        Returns:
            A tuple of three tensors: (losses, chosen_rewards, rejected_rewards). The losses tensor contains the CPO
            loss for each example in the batch. The chosen_rewards and rejected_rewards tensors contain the rewards for
            the chosen and rejected responses, respectively.
        rXrNrŠrSrTrozUnknown loss type: z7. Should be one of ['sigmoid', 'hinge', 'ipo', 'simpo'])
rSrarÉrrrrÚ
logsigmoidrr:ÚrelurdÚdetach)r&rÛrÜr[Úgamma_logratiosÚlossesÚchosen_rewardsÚrejected_rewardsrdrdreÚcpo_lossËs.
ÿÿ
ÿÿ

ÿ
z_UnslothCPOTrainer.cpo_lossr[r¨Úaverage_log_probcCsš|jdd…|jkrtdƒ‚|s)|dd…dd…f ¡}|dd…dd…dd…f}||k}d|||k<t||ƒ}|rF|| d¡| d¡S|| d¡S)aŽCompute the log probabilities of the given labels under the given logits.

        Args:
            logits: Logits of the model (unnormalized). Shape: (batch_size, sequence_length, vocab_size)
            labels:
                Labels for which to compute the log probabilities. Label tokens with a value of label_pad_token_id are
                ignored. Shape: (batch_size, sequence_length)
            average_log_prob:
                If True, return the average log probability per (non-masked) token. Otherwise, return the sum of the
                log probabilities of the (non-masked) tokens.
            label_pad_token_id: The label pad token id.
            is_encoder_decoder: Whether the model is an encoder-decoder model.

        Returns:
            A tensor of shape (batch_size,) containing the average/sum log probabilities of the given labels under the
            given logits.
        NrHzKLogits (batch and sequence length dim) and labels must have the same shape.rNr)rQrdÚcloner8rµ)r[r¨rårrÚ	loss_maskrcrdrdreÚget_batch_logpss
z"_UnslothCPOTrainer.get_batch_logpscsDˆj|ˆjˆjˆjˆjjd�}|djd}ˆjr"dˆ |d¡ini}ˆjr+d|d<||df|d	d
dœ|¤Ž}|j	}‡fdd
„}|d 
¡}	ˆjdkrYt 
d¡ ˆjj¡}
n
||d|…|	d|…ƒ}
ˆj||dˆjdvˆjˆjd�}|d|…}||d…}
|d|…}||d…}ˆjr›||
|||
|jfS||
|||
fS)zÆRun the given model on the given batch of inputs, concatenating the chosen and rejected inputs together.

        We do this to avoid doing two forward passes, because it's faster for FSDP.
        )rrrrÉrrÚdecoder_input_idsÚconcatenated_labelsTrVrÑrÒF)r„Ú	use_cachecsrˆjs|ddd…dd…f ¡}|ddd…f ¡}t ¡}| d|jd¡}| d¡}| |j¡}|||ƒ}|S)N.rHrN)rÚ
contiguousr/ÚCrossEntropyLossÚviewrQrSrÉ)r[r¨Úloss_fctÚloss©r&rdreÚcross_entropy_lossNs

zC_UnslothCPOTrainer.concatenated_forward.<locals>.cross_entropy_lossr‚N)rTrX)rårr)rÚrrrrarÉrQÚ_shift_rightrur[rærr:r¹rSrèrÚaux_loss)r&r5r»rØÚ
len_chosenÚmodel_kwargsÚoutputsÚ
all_logitsròr¨Únll_lossÚ	all_logpsÚchosen_logpsÚrejected_logpsÚ
chosen_logitsÚrejected_logitsrdrñreÚconcatenated_forward+sXûýÿûÿýü
ûz'_UnslothCPOTrainer.concatenated_forwardÚtrainÚ
train_eval)rÚevalcCs¾i}| ||¡}|dd…\}}}}	}
|jr|d}| ||¡\}}
}| ¡|j|
}|
|k ¡}|dkr8dnd}|j |
¡ ¡ ¡||›d�<|j |¡ ¡ ¡||›d�<|j |¡ ¡ ¡||›d�<|j |
|¡ ¡ ¡||›d	�<|j |¡ 	¡ ¡ ¡||›d
�<|j |¡ 	¡ ¡ ¡||›d�<|j |	 	¡ ¡¡ ¡ ¡||›d�<|j | 	¡ ¡¡ ¡ ¡||›d
�<|j |
¡ 	¡ ¡ ¡||›d�<|jrÛ||j
|7}||fS)zWCompute the CPO loss and other metrics for the given batch of inputs for train or test.NérÚeval_r�zrewards/chosenzrewards/rejectedzrewards/accuracieszrewards/marginszlogps/rejectedzlogps/chosenzlogits/rejectedz
logits/chosenrù)rÿruräÚmeanrÚfloatraÚgather_for_metricsÚitemrßrv)r&r5r»rÚmetricsÚforward_outputrÛrÜÚpolicy_chosen_logitsÚpolicy_rejected_logitsÚpolicy_nll_lossrôrárârãrðÚreward_accuraciesÚprefixrdrdreÚget_batch_loss_metricsvsF
ú
þÿÿÿÿÿ"z)_UnslothCPOTrainer.get_batch_loss_metricsÚinputscCsp|jr
t|jjjƒntƒ}|�|j||dd�\}}Wdƒn1s$wY|j|dd�|r6||fS|S)Nr©r)rir"rarÉr²r1rÚ
store_metrics)r&r5rÚreturn_outputsÚnum_items_in_batchÚcompute_loss_context_managerrðr	rdrdreÚcompute_loss¬sÿÿz_UnslothCPOTrainer.compute_losscCs†|jr
t|jjjƒntƒ}|�|j|d|d|jd|jj	d�}Wdƒn1s+wYt
||j|jj	ƒ}|jj|dd�}|S)zRGenerate samples from the model and reference model for the given batch of inputs.r…r†T)rƒr„rÚ	do_samplerRN)Úskip_special_tokens)rir"rarÉr²r1Úgeneraterr:rRr3Úbatch_decode)r&r5r»Úgenerate_context_managerÚ
policy_outputÚpolicy_output_decodedrdrdreÚgenerate_from_modelÁsÿûÿ	z&_UnslothCPOTrainer.generate_from_modelr–Úignore_keysc	s ˆdurt|dƒrt|jdgƒ‰ng‰|jrt|jjjƒntƒ}t	 
¡�$|�|j||dd�\}}Wdƒn1s<wYWdƒn1sKwY|j|dd�|r`| 
¡ddfS|d|ddœ}‡fdd	„| ¡Dƒ}	t	j|	|jjd
�}	t	j|	jd|jjd
�}
| 
¡|	|
fS)NrpÚkeys_to_ignore_at_inferencerrúeval_logits/chosenúeval_logits/rejected)r"r#csg|]
\}}|ˆvr|‘qSrdrdr™©r rdrer¡÷ržz6_UnslothCPOTrainer.prediction_step.<locals>.<listcomp>rÐr)rkrfrprir"rarÉr²r1r:Úno_gradrrrßr³r¹ÚzerosrQ)r&r5rr–r Úprediction_context_managerrðr	Úlogits_dictr[r¨rdr$reÚprediction_stepØs*
ÿÿ€þz"_UnslothCPOTrainer.prediction_stepr	cCs*| ¡D]\}}|j|| |¡qdSrY)r³rwrY)r&r	rÚkeyÚvaluerdrdrerýsÿz _UnslothCPOTrainer.store_metricsrÚ
dataloaderÚdescriptionÚmetric_key_prefixc
sÌ|jrZt|jƒ}tjt|ƒ|jjd�}|j |¡}| 	|¡}	| 
|	¡}	| |j|	¡}
t
jddgdd„t|	d|
ƒDƒd�}d|jjvrNt d	tj|d
�i¡d|jjvrZtd|d
�tƒ |||||¡}|S)zÞ
        Overriding built-in evaluation loop to store metrics for each batch. Prediction/evaluation loop, shared by
        `Trainer.evaluate()` and `Trainer.predict()`.

        Works both with or without labels.
        )r›ÚPromptÚPolicycSs$g|]\}}||t|ƒd…g‘qSrY)r‡)ršr‹Úpolrdrdrer¡r¦z6_UnslothCPOTrainer.evaluation_loop.<locals>.<listcomp>r‹)ÚcolumnsÚdataÚwandbÚgame_log)r3Úcomet_mlzgame_log.csv)ÚnameÚtable)rr‡Údatasetr7ÚsampleÚranger6Úeval_batch_sizeÚselectr7Ú_prepare_inputsrr5r4Ú	DataFramerRrår4ÚlogÚTabler,r$Úevaluation_loop)
r&r,r-r–r r.Únum_samplesÚrandom_indicesÚrandom_batch_datasetÚrandom_batchrr8Úinitial_outputr(rdrerBs0


ÿþþ
ÿz"_UnslothCPOTrainer.evaluation_loopÚlogsÚ
start_timecsTd|vrdnd}|j| ¡D]\}}t |¡ ¡ ¡||<q|j|=tƒ ||¡S)a1
        Log `logs` on the various objects watching training, including stored metrics.

        Args:
            logs (`dict[str, float]`):
                The values to log.
            start_time (`float` or `None`, *optional*, defaults to `None`):
                Start time of the training.
        rðrr)rwr³r:r¹rrr$r@)r&rHrIrr*r	r(rdrer@3s
z_UnslothCPOTrainer.logcCs´|jdur	tdƒ‚t|ƒr+t |jdd…d|j¡}tj||ddd…fgdd�}n| |j¡}|ddd…f ¡|ddd…f<|j|d<|j	durOtdƒ‚| 
|d	k|j	¡|S)
Nz]model.config.decoder_start_token_id has to be defined. It is usually set to the pad_token_id.rH)rN.rMrN).rz,model.config.pad_token_id has to be defined.rŒ)rqrdr*r:ÚfullrQrÖÚ	new_zerosrærRÚmasked_fill_)r&rƒÚshifted_input_idsrdrdreróEs
ÿ  

z_UnslothCPOTrainer._shift_rightcsL|jjdurt|jjƒj}n	|jj d¡d}|j|d�tƒ ||¡dS)Nú/rH)Ú
model_name)	r6rïrr�r7ÚsplitÚcreate_model_cardr$Ú_save_checkpoint)r&r5ÚtrialrOr(rdrerR]s
z#_UnslothCPOTrainer._save_checkpointrOÚdataset_nameÚtagsc
Csê| ¡sdSt|jjdƒrtj |jjj¡s|jjj}nd}|dur&tƒ}n
t	|t
ƒr/|h}nt|ƒ}t|jjdƒr?| d¡| |j
¡t d¡}t|||j||tƒr]tjdur]tjjndtƒd|ddd	�}| tj |jjd
¡¡dS)aî
        Creates a draft of a model card using the information available to the `Trainer`.

        Args:
            model_name (`str` or `None`, *optional*, defaults to `None`):