unsloth_compiled_cache/__pycache__/UnslothORPOTrainer.cpython-310.pyc

o
<—°h.,ã@sdZddlmZddlZddlmZddlmZddlmZm	Z	m
Z
mZmZm
Z
mZmZddlmZmZmZmZmZmZmZmZmZmZmZmZmZmZm
Z
mZmZmZmZm Z m!Z!m"Z"m#Z#mZm$Z$m%Z%m&Z&m'Z'm(Z(m)Z)m*Z*m+Z+m,Z,m-Z-m.Z.m/Z/m0Z0m1Z1m2Z2m3Z3mZm4Z4m5Z5m6Z6m7Z7m8Z8m9Z9m:Z:m;Z;m<Z<m=Z=mZm>Z>mZm
Z
mZmZm"Z"m-Z-m6Z6mZddl6Z6ddlTddl?m@Z@mAZAdd	lBmCZCddlZddlDZ4dd
lEm5Z5ddlmZddlFmGZGmHZIdd
dd
d
dœZJejKddeJd�dd„ƒZLe@Gdd„deƒƒZM	Gdd„de"ƒZNGdd„deNƒZOdS)z9
2025.8.9
2025.8.10
4.55.4
0.21.0
__UNSLOTH_VERSIONING__
é)ÚTensorN)Ú
functional)ÚAnyÚListÚOptionalÚTupleÚUnionÚDictÚSetÚCallable)=rÚAutoModelForCausalLMÚBaseImageProcessorrÚDPODataCollatorWithPaddingÚDataCollatorÚ
DataLoaderÚDatasetÚEvalLoopOutputÚFÚFeatureExtractionMixinÚLiteralÚ
ORPOConfigÚORPOTrainerrÚPartialStateÚPathÚ	PeftModelÚPreTrainedModelÚPreTrainedTokenizerBaseÚProcessorMixinÚTrainerÚTrainerCallbackrÚadd_bos_token_if_neededÚadd_eos_token_if_neededÚautocastÚdefaultdictÚdisable_dropout_in_modelÚgenerate_model_cardÚget_comet_experiment_urlÚinspectÚis_comet_availableÚis_peft_availableÚis_torch_fx_proxyÚis_torch_xla_availableÚis_wandb_availableÚlog_table_to_comet_experimentÚmaybe_apply_chat_templateÚmaybe_extract_promptÚnnÚnpÚnullcontextÚosÚ
pad_to_lengthÚpdÚpeft_module_casting_to_bf16Úprepare_model_for_kbit_trainingÚrandomÚselective_log_softmaxÚtextwrapÚtorchÚwarningsrrrrrr)r3r;)Ú*)Ú	dataclassÚfield)ÚVersion)r2)ÚDataCollatorForSeq2SeqÚDataCollatorForLanguageModelingTF)Úepilogue_fusionÚmax_autotuneÚ
shape_paddingz
trace.enabledztriton.cudagraphs)ÚdynamicÚ	fullgraphÚoptionsc
Cs¾tj| d|jd¡ddd�}tj| d¡ddd�}g}t||ƒD](\}}| tj¡}tj|d| d¡d� 	d¡}tj
|dd�}||}	| |	¡q!	t |¡}| |jd|jdf¡}|S)Néÿÿÿÿér)ÚchunksÚdim)rLÚindex©rLé)
r;ÚchunkÚreshapeÚshapeÚzipÚtoÚfloat32ÚgatherÚ	unsqueezeÚsqueezeÚ	logsumexpÚappendÚconcat)
ÚlogitsrMÚchunked_logitsÚ
chunked_indexÚall_per_token_logpsÚchunk_logitsÚchunk_indexÚselected_logitsÚlogsumexp_valuesÚper_token_logps©reúR/workspace/DS-LLM-TEMPLATE-FINETUNING/unsloth_compiled_cache/UnslothORPOTrainer.pyÚchunked_selective_log_softmax"s
rgcs–eZdZUdZedddid�Zeeed<edddid�Z	ee
ed	<eddd
id�Zee
ed<						
																														 									!	!					"	#								$														$						%	&				'												(									#				$				)	*														+	,				-		.							d1‡fd/d0„	Z‡Z
S)2ÚUnslothORPOConfiguB
    
    Configuration class for the [`ORPOTrainer`].

    This class includes only the parameters that are specific to ORPO training. For a full list of training arguments,
    please refer to the [`~transformers.TrainingArguments`] documentation. Note that default values in this class may
    differ from those in [`~transformers.TrainingArguments`].

    Using [`~transformers.HfArgumentParser`] we can turn this class into
    [argparse](https://docs.python.org/3/library/argparse#module-argparse) arguments that can be specified on the
    command line.

    Parameters:
        max_length (`int` or `None`, *optional*, defaults to `1024`):
            Maximum length of the sequences (prompt + completion) in the batch. This argument is required if you want
            to use the default data collator.
        max_prompt_length (`int` or `None`, *optional*, defaults to `512`):
            Maximum length of the prompt. This argument is required if you want to use the default data collator.
        max_completion_length (`int` or `None`, *optional*, defaults to `None`):
            Maximum length of the completion. This argument is required if you want to use the default data collator
            and your model is an encoder-decoder.
        beta (`float`, *optional*, defaults to `0.1`):
            Parameter controlling the relative ratio loss weight in the ORPO loss. In the
            [paper](https://huggingface.co/papers/2403.07691), it is denoted by Î». In the
            [code](https://github.com/xfactlab/orpo), it is denoted by `alpha`.
        disable_dropout (`bool`, *optional*, defaults to `True`):
            Whether to disable dropout in the model.
        label_pad_token_id (`int`, *optional*, defaults to `-100`):
            Label pad token id. This argument is required if you want to use the default data collator.
        padding_value (`int` or `None`, *optional*, defaults to `None`):
            Padding value to use. If `None`, the padding value of the tokenizer is used.
        truncation_mode (`str`, *optional*, defaults to `"keep_end"`):
            Truncation mode to use when the prompt is too long. Possible values are `"keep_end"` or `"keep_start"`.
            This argument is required if you want to use the default data collator.
        generate_during_eval (`bool`, *optional*, defaults to `False`):
            If `True`, generates and logs completions from the model to W&B or Comet during evaluation.
        is_encoder_decoder (`bool` or `None`, *optional*, defaults to `None`):
            When using the `model_init` argument (callable) to instantiate the model instead of the `model` argument,
            you need to specify if the model returned by the callable is an encoder-decoder model.
        model_init_kwargs (`dict[str, Any]` or `None`, *optional*, defaults to `None`):
            Keyword arguments to pass to `AutoModelForCausalLM.from_pretrained` when instantiating the model from a
            string.
        dataset_num_proc (`int` or `None`, *optional*, defaults to `None`):
            Number of processes to use for processing the dataset.
    
    NÚhelpzvLLM SamplingParams)ÚdefaultÚmetadataÚvllm_sampling_paramsrIz8Chunk size to reduce memory usage. -1 is most efficient.Úunsloth_num_chunksz'Maximum sequence length to truncate to.Úmax_seq_lengthFÚnorJéréúç-Cëâ6
?ç{®Gáz„?çÍÌÌÌÌÌì?ç+‡ÙÎ÷ï?ç:Œ0âŽyE>çð?ç@Úlinearçš™™™™™¹?ÚpassiveÚwarningTÚstepsrOéôéO
ÚO1ÚautoÚçÚ
adamw_8bitÚlengthÚ
every_saveÚlastééééœÿÿÿÚkeep_endc�’sä|dkrtd|›d�ƒ‚|dkrtd|›d�ƒ‚|dur(|#dkr(|$dkr(d}d	}#|Œdur:d
dlm}‘t|‘ƒdd
ƒ}Œtƒjdšid|“d|“d|“d|“d|“d|“d|“d|“d|	“d|
“d|“d|“d|
“d|“d|“d|“d|“d|“d |“d!|“d"|“d#|“d$|“d%|“d&|“d'|“d(|“d)|“d*|“d+|“d,|“d-| “d.|!“d/|"“d0|#“d1|$“d2|%“d3|&“d4|'“d5|(“d6|)“d7|*“d8|+“d9|,“d:|-“d;|.“d<|/“d=|0“d>|1“d?|2“d@|3“dA|4“dB|5“dC|6“dD|7“dE|8“dF|9“dG|:“dH|;“dI|<“dJ|=“dK|>“dL|?“dM|@“dN|A“dO|B“dP|C“dQ|D“dR|E“dS|F“dT|G“dU|H“dV|I“dW|J“dX|K“dY|L“dZ|M“d[|N“d\|O“d]|P“d^|Q“d_|R“d`|S“da|T“db|U“dc|V“dd|W“de|X“df|Y“dg|Z“dh|[“di|\“dj|]“dk|^“dl|_“dm|`“dn|a“do|b“dp|c“dq|d“dr|e“ds|f“dt|g“du|h“dv|i“dw|j“dx|k“dy|l“dz|m“d{|n“d||o“d}|p“d~|q“d|r“d€|s“d�|t“d‚|u“dƒ|v“d„|w“d…|x“d†|y“d‡|z“dˆ|{“d‰||“dŠ|}“d‹|~“dŒ|“d�|€“dŽ|�“d�|‚“d�|ƒ“d‘|„“d’|…“d“|†“d”|‡“d•|ˆ“d–|‰“d—|Š“d˜|‹“d™|Œ“|�¤Ž|�|_|Ž|_|�|_	dS)›NgH¯¼šò×z>z Unsloth: Your learning rate of `zi` is too small and less than 1e-7! Consider increasing it, otherwise gradient updates will be close to 0!rOza` is way too larger > 1! Consider decreasing it to 1e-1, otherwise gradient updates will explode!r}r~Úunsloth_training_checkpointsror)Ú	cpu_countrJrpÚ
output_dirÚoverwrite_output_dirÚdo_trainÚdo_evalÚ
do_predictÚ
eval_strategyÚprediction_loss_onlyÚper_device_train_batch_sizeÚper_device_eval_batch_sizeÚper_gpu_train_batch_sizeÚper_gpu_eval_batch_sizeÚgradient_accumulation_stepsÚeval_accumulation_stepsÚ
eval_delayÚtorch_empty_cache_stepsÚ
learning_rateÚweight_decayÚ
adam_beta1Ú
adam_beta2Úadam_epsilonÚ
max_grad_normÚnum_train_epochsÚ	max_stepsÚlr_scheduler_typeÚwarmup_ratioÚwarmup_stepsÚ	log_levelÚlog_level_replicaÚlog_on_each_nodeÚlogging_dirÚlogging_strategyÚlogging_first_stepÚ
logging_stepsÚlogging_nan_inf_filterÚ
save_strategyÚ
save_stepsÚsave_total_limitÚsave_safetensorsÚsave_on_each_nodeÚsave_only_modelÚ'restore_callback_states_from_checkpointÚno_cudaÚuse_cpuÚuse_mps_deviceÚseedÚ	data_seedÚ
jit_mode_evalÚuse_ipexÚbf16Úfp16Úfp16_opt_levelÚhalf_precision_backendÚbf16_full_evalÚfp16_full_evalÚtf32Ú
local_rankÚddp_backendÚ
tpu_num_coresÚtpu_metrics_debugÚdebugÚdataloader_drop_lastÚ
eval_stepsÚdataloader_num_workersÚdataloader_prefetch_factorÚ
past_indexÚrun_nameÚdisable_tqdmÚremove_unused_columnsÚlabel_namesÚload_best_model_at_endÚmetric_for_best_modelÚgreater_is_betterÚignore_data_skipÚfsdpÚfsdp_min_num_paramsÚfsdp_configÚ"fsdp_transformer_layer_cls_to_wrapÚaccelerator_configÚ	deepspeedÚlabel_smoothing_factorÚoptimÚ
optim_argsÚ	adafactorÚgroup_by_lengthÚlength_column_nameÚ	report_toÚddp_find_unused_parametersÚddp_bucket_cap_mbÚddp_broadcast_buffersÚdataloader_pin_memoryÚdataloader_persistent_workersÚskip_memory_metricsÚuse_legacy_prediction_loopÚpush_to_hubÚresume_from_checkpointÚhub_model_idÚhub_strategyÚ	hub_tokenÚhub_private_repoÚhub_always_pushÚhub_revisionÚgradient_checkpointingÚgradient_checkpointing_kwargsÚinclude_inputs_for_metricsÚeval_do_concat_batchesÚfp16_backendÚpush_to_hub_model_idÚpush_to_hub_organizationÚpush_to_hub_tokenÚ
mp_parametersÚauto_find_batch_sizeÚfull_determinismÚtorchdynamoÚ	ray_scopeÚddp_timeoutÚ
torch_compileÚtorch_compile_backendÚtorch_compile_modeÚinclude_tokens_per_secondÚinclude_num_input_tokens_seenÚneftune_noise_alphaÚoptim_target_modulesÚbatch_eval_metricsÚ
eval_on_startÚuse_liger_kernelÚliger_kernel_configÚeval_use_gather_objectÚaverage_tokens_across_devicesÚ
max_lengthÚmax_prompt_lengthÚmax_completion_lengthÚbetaÚdisable_dropoutÚlabel_pad_token_idÚ
padding_valueÚtruncation_modeÚgenerate_during_evalÚis_encoder_decoderÚmodel_init_kwargsÚdataset_num_procre)
ÚFloatingPointErrorÚ
OverflowErrorÚmultiprocessingrŽÚmaxÚsuperÚ__init__rlrmrn)’Úselfr�r�r‘r’r“r”r•r–r—r˜r™ršr›rœr�ržrŸr r¡r¢r£r¤r¥r¦r§r¨r©rªr«r¬rr®r¯r°r±r²r³r´rµr¶r·r¸r¹rºr»r¼r½r¾r¿rÀrÁrÂrÃrÄrÅrÆrÇrÈrÉrÊrËrÌrÍrÎrÏrÐrÑrÒrÓrÔrÕrÖr×rØrÙrÚrÛrÜrÝrÞrßràrárârãrärårærçrèrérêrërìrírîrïrðrñròrórôrõrör÷rørùrúrûrürýrþrÿrrrrrrrrrr	r
rrr
rrrrrrrrrrrrrrlrmrnÚkwargsrŽ©Ú	__class__rerfr os€ÿþýüûúùø	÷
öõô
óòñðïîíìëêéèçæåäãâá à!ß"Þ#Ý$Ü%Û&Ú'Ù(Ø)×*Ö+Õ,Ô-Ó.Ò/Ñ0Ð1Ï2Î3Í4Ì5Ë6Ê7É8È9Ç:Æ;Å<Ä=Ã>Â?Á@ÀA¿B¾C½D¼E»FºG¹H¸I·J¶KµL´M³N²O±P°Q¯R®ST¬U«VªW©X¨Y§Z¦[¥\¤]£^¢_¡` aŸbžc�dœe›fšg™h˜i—j–k•l”m“n’o‘p�q�rŽs�tŒu‹vŠw‰xˆy‡z†{…|„}ƒ~‚��ÿ�þ�ý�ü�û�ú�ù�ø	�÷
�ö�õ�ô
�ó
zUnslothORPOConfig.__init__)�NNFFFroFrJrJNNrprprrqrrrsrtrurvrwrxrIryrzrr{r|TNr}FrOFr}r~NTFFFFFFrrFFFFr€r�FFNrINNFr‚FNrNrINNTNFNNFr‚rNNNNrƒr„NFFr…NNNNTFTFFNNr†NNFNFNFTr�NNNr‚TFNr‡rˆFNNFFNNFFFNFTr‰rŠNrzTr‹NrŒFNNNNrIN)Ú__name__Ú
__module__Ú__qualname__Ú__doc__r?rlrrÚ__annotations__rmÚintrnr Ú
__classcell__rerer#rfrh3s>
.þþþ�ïrhcsÐeZdZdZddgZ												dRdeeeej	e
fdeedeed	ee
d
eee
ee
e
ffdeeeeeefdeegefd
eeedeejjejjjfdeeejejgejfdeedeeegeff‡fdd„
Zdd„ZdSdeeeej	fdefdd„Z e!				dTdee
eeej"ffde#de$de$d eej%dee
ej"ffd!d"„ƒZ&d#ej'd$ej'deej'ej'ej'ej'ej'ffd%d&„Z(e!			dUd'ej'd(ej"d)e#de$de#dej'fd*d+„ƒZ)dej	dee
eeej"ffdeej'ej'ej'ej'ffd,d-„Z*	.dVdee
eeej"ffd/e+d0fd1d2„Z,		dWdeeej	fd3ee
eeje-ffdeejeejee
ejffffd4d5„Z.dee
ej"fde
fd6d7„Z/	dSdeeej	fd3ee
eeje-ffd8e#d9eee
fd:d;„Z0dVd<ee
e1fd/e+d0ddfd=d>„Z2			?dXd@e3dAe
d8ee#d9eee
dBe
def‡fdCdD„
Z4dSdEee
e1fdFee1ddf‡fdGdH„
Z5dIdJ„Z6‡fdKdL„Z7			dYdMee
dNee
dOee
ee
dffdPdQ„Z8‡Z9S)ZÚ_UnslothORPOTrainerr‚ÚtrlÚorpoN©NNÚmodelÚargsÚ
data_collatorÚ
train_datasetÚeval_datasetÚprocessing_classÚ
model_initÚ	callbacksÚ
optimizersÚpreprocess_logits_for_metricsÚpeft_configÚcompute_metricsc

s¬|jduri}
n9t|tƒstdƒ‚|j}
|
 d¡}|durAt|tƒr+|dkr+tt|ƒ}|dkr=t|tjƒs=td|›d�ƒ‚||
d<t|tƒrOtj	|fi|
¤Ž}d|_
tƒs]|dur]tdƒ‚tƒrÈ|durÈt|tƒrm| 
¡}t|ddƒsyt|d	dƒr�t|d
ƒo‡d
tt t¡jƒv}d|ji}|r”|j|d
<t|fi|¤Ž}n|jrµt|dƒrª| ¡nd
d„}| ¡ |¡|}|jrÇt|d	dƒrÇt|ƒd|_
n|jràt|dƒrÕ| ¡ndd„}| ¡ |¡|jrítƒsítƒsítdƒ‚|dur÷|jj|_n|jdu�rtdƒ‚|j|_|j�r|jj |_ |jj!|_!|du�rtdƒ‚|j"du�r+t# $dt%¡d}n|j"}|j&du�r=t# $dt%¡d}n|j&}|j'du�rT|j�rTt# $dt%¡d|_'n|j'|_'|du�rxt(|j!|j)|jd�}|j*�rtd|_*t# $dt%¡d|_+nd|_+|j,�rƒt-|ƒ||_"|j|_|j)|_)|j.du�r—|j.n|j!|_.||_&|j/|_/||_0|j1|_1t|jddƒ|_2t|jddƒ|_3|j2�rÉ|j3dk�rÉt# $dt%¡t4dd „ƒ|_5d|j6d!<t7ƒ 8¡�E|j9t:|j;d"�}|j9t<d#|i|j;d$�}|j9|j=|j;d"�}|du�r|j9t:|j;d"�}|j9t<d#|i|j;d$�}|j9|j=|j;d"�}Wdƒn	1�s#wYt>ƒj?||||||||||	|
d%�d|_@t|jAd&ƒ�rJ|jA B|jC¡t|d'ƒ�sTtDd(ƒ‚dS))NzSYou passed model_kwargs to the ORPOTrainer. But your model is already instantiated.Útorch_dtyper�zoInvalid `torch_dtype` passed to the ORPOConfig. Expected a string with either `torch.dtype` or 'auto', but got Ú.FzvPEFT is not installed and you passed a `peft_config` in the trainer's kwargs, please install it to use the PEFT modelsÚis_loaded_in_8bitÚis_loaded_in_4bitrõÚuse_gradient_checkpointingÚenable_input_require_gradscSó| d¡dS©NT©Úrequires_grad_©ÚmoduleÚinputÚoutputrererfÚmake_inputs_require_gradçóz>_UnslothORPOTrainer.__init__.<locals>.make_inputs_require_gradTcSrBrCrDrFrererfrJürKz‚`generate_during_eval=True` requires Weights and Biases or Comet to be installed. Please install `wandb` or `comet-ml` to resolve.zMWhen no model is provided, you need to pass the parameter is_encoder_decoder.z>processing_class must be specified to tokenize a ORPO dataset.z‚`max_length` is not set in the ORPOConfig's init it will default to `512` by default, but you should do it yourself in the future.rŠz‰`max_prompt_length` is not set in the ORPOConfig's init it will default to `128` by default, but you should do it yourself in the future.é€z½When using an encoder decoder architecture, you should set `max_completion_length` in the ORPOConfig's init it will default to `128` by default, but you should do it yourself in the future.)Úpad_token_idrrz²When using DPODataCollatorWithPadding, you should set `remove_unused_columns=False` in your TrainingArguments we have set it for you, but you should do it yourself in the future.Úoutput_router_logitsÚrouter_aux_loss_coefrƒa-You set `output_router_logits` to `True` in the model config, but `router_aux_loss_coef` is set to `0.0`, meaning the auxiliary loss will not be used. Either set `router_aux_loss_coef` to a value greater than `0.0`, or set `output_router_logits` to `False` if you don't want to use the auxiliary loss.cSsttƒS©N)r#ÚlistrerererfÚ<lambda>]sz._UnslothORPOTrainer.__init__.<locals>.<lambda>Úestimate_tokens)Únum_procÚ	tokenizer)Ú	fn_kwargsrT)r0r1r2r3r4r5r6r;r7r8r9Úadd_model_tagsÚacceleratorzXYour `Trainer` does not have an `accelerator` object. Consider upgrading `transformers`.)ErÚ
isinstanceÚstrÚ
ValueErrorÚgetÚgetattrr;ÚdtyperÚfrom_pretrainedÚ_peft_has_been_casted_to_bf16r)rÚmerge_and_unloadÚhasattrrQr'Ú	signaturer7Ú
parametersrôrõrAÚget_input_embeddingsÚregister_forward_hookr¿r6rr,r(ÚconfigrÚdecoder_start_token_idrMrr<ÚwarnÚUserWarningrrrrrÒÚuse_dpo_data_collatorrr$rrr5rÚaux_loss_enabledÚ
aux_loss_coefr#Ú_stored_metricsÚwarnings_issuedrÚmain_process_firstÚmapr/rr.Útokenize_rowrr Úmodel_accepts_loss_kwargsr0rWÚ
_tag_namesÚAttributeError)r!r0r1r2r3r4r5r6r7r8r9r:r;rr<Ú_support_gc_kwargsÚprepare_model_kwargsrJrrr#rerfr ¢s8


ÿ
ÿ
ÿ
ÿþ


€

ÿ


ýýý
ýýû
	ÿ
ý€òõÿÿz_UnslothORPOTrainer.__init__cCs|j||dd�}|j|dd�d}|dt|ƒd…}|dt|ƒd…}t ||g¡}t |d¡}t|ƒt|ƒkr@tdƒ‚t|ƒ}	||dd|	…krR|	d8}	|dd|	…}|dd|	…}
t|ƒt|
ƒkrntdƒ‚|d|	d…}|d|	d…}t||
||d	�S)
a
        Llama tokenizer does satisfy `enc(a + b) = enc(a) + enc(b)`. It does ensure `enc(a + b) = enc(a) + enc(a +
        b)[len(enc(a)):]`. Reference:
            https://github.com/EleutherAI/lm-evaluation-harness/pull/531#issuecomment-1595586257
        F©Úadd_special_tokensÚ	input_idsNÚattention_maskzBPrompt input ids and answer input ids should have the same length.rOz@Prompt input ids and attention mask should have the same length.)Úprompt_input_idsÚprompt_attention_maskrzr{)r5Úlenr1ÚconcatenateÚarrayr[Údict)r!ÚpromptÚanswerÚfull_tokenizedr|Úanswer_input_idsÚanswer_attention_maskÚfull_concat_input_idsÚfull_input_idsÚresponse_token_ids_start_idxr}rererfÚbuild_tokenized_answer–s.üz*_UnslothORPOTrainer.build_tokenized_answerÚreturncs�i}|d}|d}|d}|j�s°t|tƒs tdt|ƒ›�ƒ‚|j|dd�}dd„| ¡Dƒ}t|tƒs>td	t|ƒ›�ƒ‚| ||¡‰t|tƒsRtd
t|ƒ›�ƒ‚| ||¡‰t|dƒ}tˆdƒ}	tˆdƒ}
t	|	|
ƒ}| ¡D]\}}|d|…||<qst
d
d„tˆdˆdƒDƒƒ}
t|	|
ƒ}|
dksž|dkr¢tdƒ‚t
|jj|||	ˆ|
ˆƒ\}‰‰t|jjˆˆƒ\‰‰ttˆdƒtˆdƒƒ}ˆˆ|fD]D}t|dƒ||jk�r|jdkrñdD]
}||d|j…||<qâqÍ|jdk�r	dD]}|||jd…||<qùqÍtd|j›�ƒ‚qÍˆˆfD]#}t|dƒ||jk�r8dD]}||d|j|j…||<�q&�q‡fdd„dDƒ}‡fdd„dDƒ}|ddd…|d<|jgtˆdƒ|ddtˆdƒ…<|ddd…|d<|jgtˆdƒ|ddtˆdƒ…<|||dœ ¡D]\}}| ¡D]\}}|dk�r£�q˜|||›|›�<�q˜�q�n[|j|d|jdd�‰|j|d|jdd�‰|j|d|jdd�}ˆd|d<ˆd|d<|d|d<|d |d!<|du�rt|d"ƒ�r|jt |d¡d#�|d$<|jt |d¡d#�|d%<tƒ�rF|D]4}d|v�s|j�r |j}n| d&¡�r*|j}n| d'¡�r2d(}|||g|jt||ƒ||<�q|S))a/Tokenize a single row from a ORPO specific dataset.

        At this stage, we don't convert to PyTorch tensors yet; we just handle the truncation in case the prompt +
        chosen or prompt + rejected responses is/are too long. First we truncate the prompt; if we're still too long,
        we truncate the chosen/rejected.

        We also create the labels for the chosen/rejected responses, which are of length equal to the sum of the length
        of the prompt and the chosen/rejected response, with label_pad_token_id for the prompt tokens.
        r‚ÚchosenÚrejectedz prompt should be an str but got FrxcSsi|]
\}}d|›�|“qS©Úprompt_re©Ú.0ÚkÚvrererfÚ
<dictcomp>ßóz4_UnslothORPOTrainer.tokenize_row.<locals>.<dictcomp>z chosen should be an str but got z"rejected should be an str but got r|NcSsg|]\}}||k‘qSrere)r‘ÚaÚbrererfÚ
<listcomp>÷sz4_UnslothORPOTrainer.tokenize_row.<locals>.<listcomp>rOzdChosen and rejected prompt_input_ids might only differ on the last token due to tokenizer merge ops.rzÚ
keep_start)r|r}rŒzUnknown truncation mode: )rzr{có$i|]}|ˆd|›�ˆ|“qSrŽre©r‘r’)Ú
chosen_tokensrerfr”%óÿcršrŽrer›)Úrejected_tokensrerfr”(r�Úlabels)Úchosen_Ú	rejected_r‚Útoken_type_idsT)Ú
truncationrryÚ
chosen_labelsÚrejected_labelsr{r}Ú%prepare_decoder_input_ids_from_labels)rŸÚrejected_decoder_input_idsÚchosen_decoder_input_idsÚ
_input_idsÚ_attention_maskr)rrYrZr[Útyper5ÚitemsrŠr~ÚminÚsumrSÚabsr Úbos_token_idr!Úeos_token_idrrrrrrrbr¦r;Útensorr+Úendswithr)r!Úfeaturer0Úbatchr‚rŒr�Ú
prompt_tokensÚprompt_len_input_idsÚchosen_prompt_len_input_idsÚrejected_prompt_len_input_idsr’r“Únum_diff_tokensÚnum_diff_lenÚlonger_response_lengthÚ
answer_tokensÚchosen_sequence_tokensÚrejected_sequence_tokensÚtoksÚtype_keyÚtokensÚ	pad_valuere)rœržrfrrÇsê


ÿÿ
ù
ÿ
ÿÿø €
ÿ
ÿÿ
þÿ
þý
ü
ýû
ÿ
ÿ
ÿ
ÿ
ÿ(z _UnslothORPOTrainer.tokenize_rowFr‹rrµrrrÚdevicec
CsŒi}|rt|djd|djdƒ}nt|djd|djdƒ}|D]8}| d¡r]t||tjƒr]d|vs:|r=|}n| d¡rE|}n| d	¡rLd
}| dd¡}	t||||d�||	<q%|D]E}| d
¡r¥t||tjƒr¥d|vsu|rx|}n| d¡r€|}n| d	¡r‡d
}| d
d¡}	tj	||	t||||d�fd
d�j
|d�||	<q`|rÄ|d dd¡j
|d�|d<|d dd¡j
|d�|d<|S)açConcatenate the chosen and rejected inputs into a single tensor.

        Args:
            batch:
                A batch of data. Must contain the keys 'chosen_input_ids' and 'rejected_input_ids', which are tensors
                of shape (batch_size, sequence_length).
            is_encoder_decoder:
                Whether the model is an encoder-decoder model.
            label_pad_token_id:
                The label pad token id.
            padding_value:
                The padding value to use for the concatenated inputs_ids.
            device:
                The device for the concatenated inputs.

        Returns:
            A dictionary containing the concatenated inputs under the key 'concatenated_input_ids'.
        r¤rOr¥Úchosen_input_idsÚrejected_input_idsrŒrŸr©rªrÚconcatenated)rÃr�rN©rÄr|rpÚconcatenated_input_idsr}Úconcatenated_attention_mask)rrRÚ
startswithrYr;rr³Úreplacer4ÚcatrTÚrepeat)
rµrrrrÄÚconcatenated_batchrr’rÃÚconcatenated_keyrererfÚconcatenated_inputsbsL 

€

þû
ú€ÿz'_UnslothORPOTrainer.concatenated_inputsÚpolicy_chosen_logpsÚpolicy_rejected_logpscCsŠ||t t |¡¡t t |¡¡}t |¡}|j|}|j| |jj¡ 	¡}|j| |jj¡ 	¡}|||t 
|¡t 
|¡fS)anCompute ORPO's odds ratio (OR) loss for a batch of policy and reference model log probabilities.

        Args:
            policy_chosen_logps:
                Log probabilities of the policy model for the chosen responses. Shape: (batch_size,)
            policy_rejected_logps:
                Log probabilities of the policy model for the rejected responses. Shape: (batch_size,)

        Returns:
            A tuple of three tensors: (losses, chosen_rewards, rejected_rewards). The losses tensor contains the ORPO
            loss for each example in the batch. The chosen_rewards and rejected_rewards tensors contain the rewards for
            the chosen and rejected responses, respectively. The log odds ratio of the chosen responses over the
            rejected responses ratio for logging purposes. The `log(sigmoid(log_odds_chosen))` for logging purposes.
        )r;Úlog1pÚexprÚ
logsigmoidrrTrXrÄÚdetachÚmean)r!rÒrÓÚlog_oddsÚratioÚlossesÚchosen_rewardsÚrejected_rewardsrererfÚodds_ratio_loss¦s"ÿ

z#_UnslothORPOTrainer.odds_ratio_lossr\rŸÚaverage_log_probcCs |jdd…|jkrtdƒ‚|s)|dd…dd…f ¡}|dd…dd…dd…f}||k}t ||kd|¡}t||ƒ}|rI|| d¡| d¡S|| d¡S)aŽCompute the log probabilities of the given labels under the given logits.

        Args:
            logits: Logits of the model (unnormalized). Shape: (batch_size, sequence_length, vocab_size)
            labels:
                Labels for which to compute the log probabilities. Label tokens with a value of label_pad_token_id are
                ignored. Shape: (batch_size, sequence_length)
            average_log_prob:
                If True, return the average log probability per (non-masked) token. Otherwise, return the sum of the
                log probabilities of the (non-masked) tokens.
            label_pad_token_id: The label pad token id.
            is_encoder_decoder: Whether the model is an encoder-decoder model.

        Returns:
            A tensor of shape (batch_size,) containing the average/sum log probabilities of the given labels under the
            given logits.
        NrIzKLogits (batch and sequence length dim) and labels must have the same shape.rOr)rRr[Úcloner;Úwherer9r®)r\rŸrßrrÚ	loss_maskrdrererfÚget_batch_logpsÆs
z#_UnslothORPOTrainer.get_batch_logpscsŠˆj|ˆjˆjˆjˆjjd�}|djd}ˆjr"dˆ |d¡ini}ˆjr+d|d<||df|d	d
dœ|¤Ž}|j	}‡fdd
„}ˆjrM|d 
¡}	n|d 
¡}	|d	}
t |
dk|	ˆj¡}	||d|…|	d|…ƒ}ˆj
||ddˆjˆjd�}|d|…}
||d…}ˆjs¦|d|…dd…dd…f}||d…dd…dd…f}n|d|…}||d…}ˆjr¾|
|||||jfS|
||||fS)zÆRun the given model on the given batch of inputs, concatenating the chosen and rejected inputs together.

        We do this to avoid doing two forward passes, because it's faster for FSDP.
        )rrrrÄr¤rÚdecoder_input_idsÚconcatenated_labelsTrNrÉrÊF)r{Ú	use_cachecsrˆjs|ddd…dd…f ¡}|ddd…f ¡}t ¡}| d|jd¡}| d¡}| |j¡}|||ƒ}|S)N.rIrO)rÚ
contiguousr0ÚCrossEntropyLossÚviewrRrTrÄ)r\rŸÚloss_fctÚloss©r!rerfÚcross_entropy_losss

zD_UnslothORPOTrainer.concatenated_forward.<locals>.cross_entropy_lossrON)rßrrrI)rÑrrrrXrÄrRÚ_shift_rightrlr\ràr;rárãÚaux_loss)r!r0rµrÏÚ
len_chosenÚmodel_kwargsÚoutputsÚ
all_logitsrírŸr{Úchosen_nll_lossÚ	all_logpsÚchosen_logpsÚrejected_logpsÚ
chosen_logitsÚrejected_logitsrerìrfÚconcatenated_forwardñsbûýÿûÿýüûz(_UnslothORPOTrainer.concatenated_forwardÚtrainÚ
train_eval)rûÚevalcCsi}| ||¡}|dd…\}}}}	}
|jr|d}| ||¡\}}
}}}|
| ¡}|
|k ¡}|dkr7dnd}|j |
¡ ¡||›d�<|j |¡ ¡||›d�<|j |¡ ¡||›d�<|j |
|¡ ¡||›d	�<|j |¡ ¡ ¡||›d
�<|j |¡ ¡ ¡||›d�<|j |	 ¡ ¡¡ ¡||›d�<|j | ¡ ¡¡ ¡||›d
�<|j |
¡ ¡ ¡||›d�<|j |¡ ¡ ¡||›d�<|j |¡ ¡ ¡||›d�<tƒrãt	 
¡| ¡D]
\}}| ¡||<qç|jrü||j
|7}||fS)zXCompute the ORPO loss and other metrics for the given batch of inputs for train or test.NérýÚeval_r‚zrewards/chosenzrewards/rejectedzrewards/accuracieszrewards/marginszlogps/rejectedzlogps/chosenzlogits/rejectedz
logits/chosenÚnll_lossÚlog_odds_ratioÚlog_odds_chosen)rúrlrÞrØÚfloatrXÚgather_for_metricsr×r+ÚxmÚ	mark_stepr¬Úitemrm)r!r0rµrüÚmetricsÚforward_outputrÒrÓÚpolicy_chosen_logitsÚpolicy_rejected_logitsÚpolicy_nll_lossrïrÛrÜrÝrrrëÚreward_accuraciesÚprefixr’r“rererfÚget_batch_loss_metricsBs^
úÿÿþ
ÿþ
ÿþz*_UnslothORPOTrainer.get_batch_loss_metricsÚinputscCs~|jr
t|jjjƒntƒ}|�|j||dd�\}}Wdƒn1s$wY| |jj¡}|j	|dd�|r=||fS|S)Nrû©rü)
r`r"rXrÄr«r2rrTr1Ú
store_metrics)r!r0rÚreturn_outputsÚnum_items_in_batchÚcompute_loss_context_managerrërrererfÚcompute_lossysÿÿz _UnslothORPOTrainer.compute_losscCs†|jr
t|jjjƒntƒ}|�|j|d|d|jd|jj	d�}Wdƒn1s+wYt
||j|jj	ƒ}|jj|dd�}|S)zRGenerate samples from the model and reference model for the given batch of inputs.r|r}T)rzr{rÚ	do_samplerMN)Úskip_special_tokens)r`r"rXrÄr«r2Úgeneraterr5rMr4Úbatch_decode)r!r0rµÚgenerate_context_managerÚ
policy_outputÚpolicy_output_decodedrererfÚgenerate_from_model‘sÿûÿ	z'_UnslothORPOTrainer.generate_from_modelr•Úignore_keysc	s0|jst d¡ˆdurt|dƒrt|jdgƒ‰ng‰|jr%t|jj	j
ƒntƒ}t 
¡�$|�|j||dd�\}}Wdƒn1sDwYWdƒn1sSwY|j|dd�|rh| ¡ddfS|d|ddœ}‡fd	d
„| ¡Dƒ}	tj|	|jj	d�}	tj|	jd|jj	d�}
| ¡|	|
fS)
Na!prediction_step is only implemented for DPODataCollatorWithPadding, and you passed a datacollator that is different than DPODataCollatorWithPadding - you might see unexpected behavior. Alternatively, you can implement your own prediction_step method if you are using a custom data collatorrgÚkeys_to_ignore_at_inferencerýrúeval_logits/chosenúeval_logits/rejected)r!r"csg|]
\}}|ˆvr|‘qSrerer�©rrerfr˜Ìr•z7_UnslothORPOTrainer.prediction_step.<locals>.<listcomp>rÈr)rkr<rirbr]rgr`r"rXrÄr«r2r;Úno_gradrrr×r¬r²ÚzerosrR)r!r0rr•rÚprediction_context_managerrërÚlogits_dictr\rŸrer#rfÚprediction_step¨s2ÿ
ÿÿ€þz#_UnslothORPOTrainer.prediction_steprcCs*| ¡D]\}}|j|| |¡qdSrP)r¬rnrZ)r!rrüÚkeyÚvaluerererfrÒsÿz!_UnslothORPOTrainer.store_metricsrýÚ
dataloaderÚdescriptionÚmetric_key_prefixc
sÌ|jrZt|jƒ}tjt|ƒ|jjd�}|j |¡}| 	|¡}	| 
|	¡}	| |j|	¡}
t
jddgdd„t|	d|
ƒDƒd�}d|jjvrNt d	tj|d
�i¡d|jjvrZtd|d
�tƒ |||||¡}|S)zÞ
        Overriding built-in evaluation loop to store metrics for each batch. Prediction/evaluation loop, shared by
        `Trainer.evaluate()` and `Trainer.predict()`.

        Works both with or without labels.
        )r’ÚPromptÚPolicycSs$g|]\}}||t|ƒd…g‘qSrP)r~)r‘r‚Úpolrererfr˜ôr�z7_UnslothORPOTrainer.evaluation_loop.<locals>.<listcomp>r‚)ÚcolumnsÚdataÚwandbÚgame_log)r2Úcomet_mlzgame_log.csv)ÚnameÚtable)rr~Údatasetr8ÚsampleÚranger1Úeval_batch_sizeÚselectr2Ú_prepare_inputsrr0r5Ú	DataFramerSrär3ÚlogÚTabler-rÚevaluation_loop)
r!r+r,r•rr-Únum_samplesÚrandom_indicesÚrandom_batch_datasetÚrandom_batchrr7Úinitial_outputr#rerfrAÖs0


ÿþþ
ÿz#_UnslothORPOTrainer.evaluation_loopÚlogsÚ
start_timecsTd|vrdnd}|j| ¡D]\}}t |¡ ¡ ¡||<q|j|=tƒ ||¡S)a1
        Log `logs` on the various objects watching training, including stored metrics.

        Args:
            logs (`dict[str, float]`):
                The values to log.
            start_time (`float` or `None`, *optional*, defaults to `None`):
                Start time of the training.
        rërûrý)rnr¬r;r²rØrrr?)r!rGrHrür)rr#rerfr?s
z_UnslothORPOTrainer.logcCs´|jdur	tdƒ‚t|ƒr+t |jdd…d|j¡}tj||ddd…fgdd�}n| |j¡}|ddd…f ¡|ddd…f<|j|d<|j	durOtdƒ‚| 
|d	k|j	¡|S)
Nz]model.config.decoder_start_token_id has to be defined. It is usually set to the pad_token_id.rI)rO.rNrO).rz,model.config.pad_token_id has to be defined.r‹)rhr[r*r;ÚfullrRrÍÚ	new_zerosràrMÚmasked_fill_)r!rzÚshifted_input_idsrererfrîs
ÿ  

z _UnslothORPOTrainer._shift_rightcsL|jjdurt|jjƒj}n	|jj d¡d}|j|d�tƒ ||¡dS)Nú/rI)Ú
model_name)	r1rîrr�r6ÚsplitÚcreate_model_cardrÚ_save_checkpoint)r!r0ÚtrialrNr#rerfrQ2s
z$_UnslothORPOTrainer._save_checkpointrNÚdataset_nameÚtagsc
Csê| ¡sdSt|jjdƒrtj |jjj¡s|jjj}nd}|dur&tƒ}n
t	|t
ƒr/|h}nt|ƒ}t|jjdƒr?| d¡| |j
¡t d¡}t|||j||tƒr]tjdur]tjjndtƒd|ddd	�}| tj |jjd
¡¡dS)aî
        Creates a draft of a model card using the information available to the `Trainer`.