470 lines
94 KiB
Plaintext
470 lines
94 KiB
Plaintext
|
§
|
|||
|
|
4$�h߃ ã ó” — d Z ddlmZ ddlZddlmZ ddlmZ ddlmZm Z m
|
|||
|
|
Z
|
|||
|
|
m Z m Z m
|
|||
|
|
Z
|
|||
|
|
mZmZ ddlmZmZmZmZmZmZmZmZmZmZmZmZmZmZm
|
|||
|
|
Z
|
|||
|
|
m Z m Z m Z m Z m Z m!Z!m"Z"m#Z#m$Z$m%Z%m Z m&Z&m'Z'm(Z(m)Z)m*Z*m+Z+m,Z,m-Z-m.Z.m/Z/m0Z0m1Z1m2Z2m3Z3m4Z4m5Z5m6Z6m7Z7m8Z8m9Z9m:Z:m;Z;mZm<Z<m=Z=m>Z>m?Z?m@Z@mAZAmBZBmCZCmDZDmEZEmFZFmZmGZGmHZHmZm
|
|||
|
|
Z
|
|||
|
|
m Z m Z m#Z#m5Z5m>Z>mZ ddl>Z>ddlT ddlImJZJmKZK dd lLmMZM ddlZddlNZ<dd
|
|||
|
|
lOm=Z= ddlmZ dd lPmQZQmRZS d d
|
|||
|
|
d d
|
|||
|
|
d
|
|||
|
|
dœZT ejU d d eT¬¦ « d„ ¦ « ZVeJ G d„ de¦ « ¦ « ZW G d„ de#¦ « ZX G d„ deX¦ « ZYdS )z8
|
|||
|
|
2025.8.4
|
|||
|
|
2025.8.5
|
|||
|
|
4.55.1
|
|||
|
|
0.21.0
|
|||
|
|
__UNSLOTH_VERSIONING__
|
|||
|
|
é )ÚTensorN)Ú
|
|||
|
|
functional)ÚAnyÚListÚOptionalÚTupleÚUnionÚDictÚSetÚCallable)Gr ÚAutoModelForCausalLMÚBaseImageProcessorr ÚDPODataCollatorWithPaddingÚ DataCollatorÚ
|
|||
|
|
DataLoaderÚDatasetÚEvalLoopOutputÚFÚFeatureExtractionMixinÚ KTOConfigÚ
|
|||
|
|
KTOTrainerÚLiteralr Ú PartialStateÚPathÚ PeftModelÚPreTrainedModelÚPreTrainedTokenizerBaseÚProcessorMixinÚSequentialSamplerÚTrainerÚTrainerCallbackÚTrainingArgumentsr Ú_get_kl_datasetÚ_process_tokensÚ _tokenizeÚautocastÚconcatenate_datasetsÚcontextmanagerÚcreate_reference_modelÚ defaultdictÚdisable_dropout_in_modelÚgenerate_model_cardÚget_comet_experiment_urlÚ
|
|||
|
|
has_lengthÚinspectÚis_comet_availableÚis_liger_kernel_availableÚis_peft_availableÚis_wandb_availableÚ
|
|||
|
|
itemgetterÚ log_table_to_comet_experimentÚmaybe_apply_chat_templateÚmaybe_extract_promptÚ maybe_unpair_preference_datasetÚnnÚnpÚ nullcontextÚosÚ
|
|||
|
|
pad_to_lengthÚpdÚpeft_module_casting_to_bf16Úprepare_deepspeedÚ prepare_model_for_kbit_trainingÚrandomÚselective_log_softmaxÚtextwrapÚtorchÚtqdmÚwarningsr r r r r r2 r< rE )Ú*)Ú dataclassÚfield)ÚVersion)r; )ÚDataCollatorForSeq2SeqÚ DataCollatorForLanguageModelingTF)Úepilogue_fusionÚ max_autotuneÚ
|
|||
|
|
shape_paddingz
|
|||
|
|
trace.enabledztriton.cudagraphs)ÚdynamicÚ fullgraphÚoptionsc ó’ — t j | d| j d ¦ « dd¬¦ « }t j | d¦ « dd¬¦ « }g }t ||¦ « D ]‘\ }}| t j ¦ « }t j |d| d¦ « ¬¦ « d¦ « }t j
|
|||
|
|
|d¬¦ « }||z
|
|||
|
|
} | | ¦ « Œ’ t j |¦ « }| | j d | j d f¦ « }|S )Néÿÿÿÿé r )ÚchunksÚdim)rX Úindex)rX é )
|
|||
|
|
rE ÚchunkÚreshapeÚshapeÚzipÚtoÚfloat32ÚgatherÚ unsqueezeÚsqueezeÚ logsumexpÚappendÚconcat)
|
|||
|
|
ÚlogitsrY Úchunked_logitsÚ
|
|||
|
|
chunked_indexÚall_per_token_logpsÚ chunk_logitsÚ chunk_indexÚselected_logitsÚlogsumexp_valuesÚper_token_logpss
|
|||
|
|
ú]/workspace/Fine-tuning/DS-LLM-TEMPLATE-FINETUNING/unsloth_compiled_cache/UnslothKTOTrainer.pyÚ chunked_selective_log_softmaxrq " s5 € õ ”[ § ¢ °°F´LÀÔ4DÑ!EÔ!EÐPQÐYZÐ[Ñ[Ô[€NÝ”[ § ¢ ¨rÑ!2Ô!2¸QÀaÐHÑHÔH€MØ Ðå%(¨ ¸ Ñ%GÔ%Gð 4ð 4Ñ!ˆ �kØ#—’¥u¤}Ñ5Ô5ˆ Ý œ, |¸2À{×G\ÒG\Ð]_ÑG`ÔG`ÐaÑaÔa×iÒiÐjlÑmÔmˆÝ œ?¨<¸rÐBÑBÔBÐØ)Ð,<Ñ<ˆØ×"Ò" ?Ñ3Ô3Ð3Ð3ØÝ œ,Ð':Ñ;Ô;ÐØ-×5Ò5°v´|ÀA´ÈÌ ÐUVÌÐ6XÑYÔYÐØ Ð ó c ó¸ ‡ — e Zd ZU dZ edddi¬¦ « Zee ed< edddi¬¦ « Z ee
|
|||
|
|
ed < d0ˆ fd/„ Z ˆ xZ S )1ÚUnslothKTOConfiguÐ
|
|||
|
|
|
|||
|
|
Configuration class for the [`KTOTrainer`].
|
|||
|
|
|
|||
|
|
This class includes only the parameters that are specific to KTO training. For a full list of training arguments,
|
|||
|
|
please refer to the [`~transformers.TrainingArguments`] documentation. Note that default values in this class may
|
|||
|
|
differ from those in [`~transformers.TrainingArguments`].
|
|||
|
|
|
|||
|
|
Using [`~transformers.HfArgumentParser`] we can turn this class into
|
|||
|
|
[argparse](https://docs.python.org/3/library/argparse#module-argparse) arguments that can be specified on the
|
|||
|
|
command line.
|
|||
|
|
|
|||
|
|
Parameters:
|
|||
|
|
max_length (`int` or `None`, *optional*, defaults to `1024`):
|
|||
|
|
Maximum length of the sequences (prompt + completion) in the batch. This argument is required if you want
|
|||
|
|
to use the default data collator.
|
|||
|
|
max_prompt_length (`int` or `None`, *optional*, defaults to `512`):
|
|||
|
|
Maximum length of the prompt. This argument is required if you want to use the default data collator.
|
|||
|
|
max_completion_length (`int` or `None`, *optional*, defaults to `None`):
|
|||
|
|
Maximum length of the completion. This argument is required if you want to use the default data collator
|
|||
|
|
and your model is an encoder-decoder.
|
|||
|
|
beta (`float`, *optional*, defaults to `0.1`):
|
|||
|
|
Parameter controlling the deviation from the reference model. Higher β means less deviation from the
|
|||
|
|
reference model.
|
|||
|
|
loss_type (`str`, *optional*, defaults to `"kto"`):
|
|||
|
|
Type of loss to use. Possible values are:
|
|||
|
|
|
|||
|
|
- `"kto"`: KTO loss from the [KTO](https://huggingface.co/papers/2402.01306) paper.
|
|||
|
|
- `"apo_zero_unpaired"`: Unpaired variant of APO-zero loss from the
|
|||
|
|
[APO](https://huggingface.co/papers/2408.06266) paper.
|
|||
|
|
|
|||
|
|
desirable_weight (`float`, *optional*, defaults to `1.0`):
|
|||
|
|
Desirable losses are weighed by this factor to counter unequal number of desirable and undesirable paris.
|
|||
|
|
undesirable_weight (`float`, *optional*, defaults to `1.0`):
|
|||
|
|
Undesirable losses are weighed by this factor to counter unequal number of desirable and undesirable pairs.
|
|||
|
|
label_pad_token_id (`int`, *optional*, defaults to `-100`):
|
|||
|
|
Label pad token id. This argument is required if you want to use the default data collator.
|
|||
|
|
padding_value (`int` or `None`, *optional*, defaults to `None`):
|
|||
|
|
Padding value to use. If `None`, the padding value of the tokenizer is used.
|
|||
|
|
truncation_mode (`str`, *optional*, defaults to `"keep_end"`):
|
|||
|
|
Truncation mode to use when the prompt is too long. Possible values are `"keep_end"` or `"keep_start"`.
|
|||
|
|
This argument is required if you want to use the default data collator.
|
|||
|
|
generate_during_eval (`bool`, *optional*, defaults to `False`):
|
|||
|
|
If `True`, generates and logs completions from both the model and the reference model to W&B or Comet
|
|||
|
|
during evaluation.
|
|||
|
|
is_encoder_decoder (`bool` or `None`, *optional*, defaults to `None`):
|
|||
|
|
When using the `model_init` argument (callable) to instantiate the model instead of the `model` argument,
|
|||
|
|
you need to specify if the model returned by the callable is an encoder-decoder model.
|
|||
|
|
precompute_ref_log_probs (`bool`, *optional*, defaults to `False`):
|
|||
|
|
Whether to precompute reference model log probabilities for training and evaluation datasets. This is
|
|||
|
|
useful when training without the reference model to reduce the total GPU memory needed.
|
|||
|
|
model_init_kwargs (`dict[str, Any]` or `None`, *optional*, defaults to `None`):
|
|||
|
|
Keyword arguments to pass to `AutoModelForCausalLM.from_pretrained` when instantiating the model from a
|
|||
|
|
string.
|
|||
|
|
ref_model_init_kwargs (`dict[str, Any]` or `None`, *optional*, defaults to `None`):
|
|||
|
|
Keyword arguments to pass to `AutoModelForCausalLM.from_pretrained` when instantiating the reference model
|
|||
|
|
from a string.
|
|||
|
|
dataset_num_proc: (`int` or `None`, *optional*, defaults to `None`):
|
|||
|
|
Number of processes to use for processing the dataset.
|
|||
|
|
disable_dropout (`bool`, *optional*, defaults to `True`):
|
|||
|
|
Whether to disable dropout in the model and reference model.
|
|||
|
|
use_liger_loss (`bool`, *optional*, defaults to `False`):
|
|||
|
|
Whether to use Liger loss. It requires liger-kernel to be installed.
|
|||
|
|
base_model_attribute_name (`str`, *optional*, defaults to `"model"`):
|
|||
|
|
Name of the attribute in the model that contains the base model. This is used to get the base model from
|
|||
|
|
the model when the model does not have a `get_decoder` method in the case when `use_liger_loss` is `True`.
|
|||
|
|
|
|||
|
|
NÚhelpzvLLM SamplingParams)ÚdefaultÚmetadataÚvllm_sampling_paramsrU z8Chunk size to reduce memory usage. -1 is most efficient.Úunsloth_num_chunksFÚnorV é r éú ç-C ëâ6
|
|||
|
|
?ç{®Gáz„?çÍÌÌÌÌÌì?ç+‡ÙÎ÷ï?ç:Œ0âŽyE>ç ð?ç @Úlinearçš™™™™™¹?ÚpassiveÚwarningTÚstepsrZ éô éO
|
|||
|
|
ÚO1ÚautoÚ ç Ú
|
|||
|
|
adamw_8bitÚlengthÚ
|
|||
|
|
every_saveÚlasté é é ÚktoéœÿÿÿÚkeep_endÚmodelc– ó– •— |dk rt d|› d�¦ « ‚|dk rt d|› d�¦ « ‚|€|#dk r
|
|||
|
|
|$dk rd}d }#|‘€!d
|
|||
|
|
d lm}— t |—¦ « d z d ¦ « }‘ t ¦ « j d i d
|
|||
|
|
|“d|“d|“d|“d|“d|“d|“d|“d| “d|
|
|||
|
|
“d| “d| “d|
|
|||
|
|
“d|“d|“d |“d |“d |“d |“d |“d!|“d"|“d#|“d$|“d%|“d&|“d'|“d(| “d)| “d*| “d+| “d,| “d-|!“d.|"“d/|#“d0|$“d1|%“d2|&“d3|'“d4|(“d5|)“d6|*“d7|+“d8|,“d9|-“d:|.“d;|/“d<|0“d=|1“d>|2“d?|3“d@|4“dA|5“dB|6“dC|7“dD|8“dE|9“dF|:“dG|;“dH|<“dI|=“dJ|>“dK|?“dL|@“dM|A“dN|B“dO|C“dP|D“dQ|E“dR|F“dS|G“dT|H“dU|I“dV|J“dW|K“dX|L“dY|M“dZ|N“d[|O“d\|P“d]|Q“d^|R“d_|S“d`|T“da|U“db|V“dc|W“dd|X“de|Y“df|Z“dg|[“dh|\“di|]“dj|^“dk|_“dl|`“dm|a“dn|b“do|c“dp|d“dq|e“dr|f“ds|g“dt|h“du|i“dv|j“dw|k“dx|l“dy|m“dz|n“d{|o“d||p“d}|q“d~|r“d|s“d€|t“d�|u“d‚|v“dƒ|w“d„|x“d…|y“d†|z“d‡|{“dˆ||“d‰|}“dŠ|~“d‹|“dŒ|€“d�|�“dŽ|‚“d�|ƒ“d�|„“d‘|…“d’|†“d“|‡“d”|ˆ“d•|‰“d–|Š“d—|‹“d˜|Œ“d™|�“dš|Ž“d›|�“dœ|�“d�|‘“dž|’“dŸ|““|–¤Ž |”| _ |•| _ d S )¡NgH¯¼šò×z>z Unsloth: Your learning rate of `zi` is too small and less than 1e-7! Consider increasing it, otherwise gradient updates will be close to 0!rZ za` is way too larger > 1! Consider decreasing it to 1e-1, otherwise gradient updates will explode!rˆ r‰ Ú unsloth_training_checkpointsrz r )Ú cpu_countr{ Ú
|
|||
|
|
output_dirÚoverwrite_output_dirÚdo_trainÚdo_evalÚ
|
|||
|
|
do_predictÚ
|
|||
|
|
eval_strategyÚprediction_loss_onlyÚper_device_train_batch_sizeÚper_device_eval_batch_sizeÚper_gpu_train_batch_sizeÚper_gpu_eval_batch_sizeÚgradient_accumulation_stepsÚeval_accumulation_stepsÚ
|
|||
|
|
eval_delayÚtorch_empty_cache_stepsÚ
|
|||
|
|
learning_rateÚ weight_decayÚ
|
|||
|
|
adam_beta1Ú
|
|||
|
|
adam_beta2Ú adam_epsilonÚ
|
|||
|
|
max_grad_normÚnum_train_epochsÚ max_stepsÚlr_scheduler_typeÚ warmup_ratioÚ warmup_stepsÚ log_levelÚlog_level_replicaÚlog_on_each_nodeÚ logging_dirÚlogging_strategyÚlogging_first_stepÚ
|
|||
|
|
logging_stepsÚlogging_nan_inf_filterÚ
|
|||
|
|
save_strategyÚ
|
|||
|
|
save_stepsÚsave_total_limitÚsave_safetensorsÚsave_on_each_nodeÚsave_only_modelÚ'restore_callback_states_from_checkpointÚno_cudaÚuse_cpuÚuse_mps_deviceÚseedÚ data_seedÚ
|
|||
|
|
jit_mode_evalÚuse_ipexÚbf16Úfp16Úfp16_opt_levelÚhalf_precision_backendÚbf16_full_evalÚfp16_full_evalÚtf32Ú
|
|||
|
|
local_rankÚ ddp_backendÚ
|
|||
|
|
tpu_num_coresÚtpu_metrics_debugÚdebugÚdataloader_drop_lastÚ
|
|||
|
|
eval_stepsÚdataloader_num_workersÚdataloader_prefetch_factorÚ
|
|||
|
|
past_indexÚrun_nameÚ disable_tqdmÚremove_unused_columnsÚ label_namesÚload_best_model_at_endÚmetric_for_best_modelÚgreater_is_betterÚignore_data_skipÚfsdpÚfsdp_min_num_paramsÚ fsdp_configÚ"fsdp_transformer_layer_cls_to_wrapÚaccelerator_configÚ deepspeedÚlabel_smoothing_factorÚoptimÚ
|
|||
|
|
optim_argsÚ adafactorÚgroup_by_lengthÚlength_column_nameÚ report_toÚddp_find_unused_parametersÚddp_bucket_cap_mbÚddp_broadcast_buffersÚdataloader_pin_memoryÚ dataloader_persistent_workersÚskip_memory_metricsÚuse_legacy_prediction_loopÚ push_to_hubÚresume_from_checkpointÚ hub_model_idÚ hub_strategyÚ hub_tokenÚhub_private_repoÚhub_always_pushÚ hub_revisionÚgradient_checkpointingÚ gradient_checkpointing_kwargsÚinclude_inputs_for_metricsÚeval_do_concat_batchesÚ fp16_backendÚpush_to_hub_model_idÚpush_to_hub_organizationÚpush_to_hub_tokenÚ
|
|||
|
|
mp_parametersÚauto_find_batch_sizeÚfull_determinismÚ torchdynamoÚ ray_scopeÚ ddp_timeoutÚ
|
|||
|
|
torch_compileÚtorch_compile_backendÚtorch_compile_modeÚinclude_tokens_per_secondÚ include_num_input_tokens_seenÚneftune_noise_alphaÚoptim_target_modulesÚbatch_eval_metricsÚ
|
|||
|
|
eval_on_startÚuse_liger_kernelÚliger_kernel_configÚeval_use_gather_objectÚ average_tokens_across_devicesÚ
|
|||
|
|
max_lengthÚmax_prompt_lengthÚmax_completion_lengthÚbetaÚ loss_typeÚdesirable_weightÚundesirable_weightÚlabel_pad_token_idÚ
|
|||
|
|
padding_valueÚtruncation_modeÚgenerate_during_evalÚis_encoder_decoderÚdisable_dropoutÚprecompute_ref_log_probsÚmodel_init_kwargsÚref_model_init_kwargsÚdataset_num_procÚuse_liger_lossÚbase_model_attribute_name© ) ÚFloatingPointErrorÚ
|
|||
|
|
OverflowErrorÚmultiprocessingrœ ÚminÚsuperÚ__init__rx ry )™Úselfr� rž rŸ r r¡ r¢ r£ r¤ r¥ r¦ r§ r¨ r© rª r« r¬ r r® r¯ r° r± r² r³ r´ rµ r¶ r· r¸ r¹ rº r» r¼ r½ r¾ r¿ rÀ rÁ r rà rÄ rÅ rÆ rÇ rÈ rÉ rÊ rË rÌ rÍ rÎ rÏ rÐ rÑ rÒ rÓ rÔ rÕ rÖ r× rØ rÙ rÚ rÛ rÜ rÝ rÞ rß rà rá râ rã rä rå ræ rç rè ré rê rë rì rí rî rï rð rñ rò ró rô rõ rö r÷ rø rù rú rû rü rý rþ rÿ r r r r r r r r r r r
|
|||
|
|
r r r
|
|||
|
|
r r r r r r r r r r r r r r r r r r r r! r" r# r$ r% r& r' r( r) r* r+ r, r- r. r/ rx ry Úkwargsrœ Ú __class__s™ €rp r6 zUnslothKTOConfig.__init__� sg
|
|||
|
|
ø€ ðr ˜4Ò Ð Õ'9ð ;VÐ]jð ;Vð ;Vð ;Vñ (Wô (Wð "WØ ˜1Ò Ð ¥Mð 3FÐUbð 3Fð 3Fð 3Fñ %Gô %Gð GØ Ð -°7Ò":Ð":¸zÈSÒ?PÐ?PØ7ˆJØ ˆMØ Ð #Ø 1Ð 1Ð 1Ð 1Ð 1Ð 1Ý " 9 9¡;¤;¨q¡=°!Ñ 4Ô 4Ð à�‰ŒÔð S Lð S Lð S LØ#˜ðS Là#7Ð#7ðS Lð �xðS Lð �gð S Lð
|
|||
|
|
$˜ð S Lð *˜Mð
|
|||
|
|
S Lð $8Ð#7ðS Lð +FÐ*EðS Lð *DÐ)CðS Lð (@Ð'?ðS Lð '>Ð&=ðS Lð +FÐ*EðS Lð '>Ð&=ðS Lð $˜ð S Lð '>Ð&=ð S Lð *˜Mð!S Lð" (˜<ð#S Lð$ $˜ð%S Lð& $˜ð'S Lð( (˜<ð)S Lð* *˜Mð+S Lð, 0Ð /ð-S Lð. "˜ ð/S Lð0 !2Ð 1ð1S Lð2 (˜<ð3S Lð4 (˜<ð5S Lð6 "˜ ð7S Lð8 !2Ð 1ð9S Lð: 0Ð /ð;S Lð< &˜+ð=S Lð> 0Ð /ð?S Lð@ "4Ð!3ðAS LðB *˜MðCS LðD &<Ð%;ðES LðF *˜MðGS LðH $˜ðIS LðJ 0Ð /ðKS LðL 0Ð /ðMS LðN !2Ð 1ðOS LðP .˜oðQS LðR 7^Ð6]ðSS LðT �gðUS LðV �gðWS LðX ,˜^ðYS LðZ �4ð[S Lð\ "˜ ð]S Lð^ *˜Mð_S Lð` �xðaS Lðb �4ðcS Lðd �4ðeS Lðf ,˜^ðgS Lðh &<Ð%;ðiS Lðj ,˜^ðkS Lðl ,˜^ðmS Lðn �4ðoS Lðp $˜ðqS Lðr &˜+ðsS Lðt *˜MðuS Lðv !2Ð 1ðwS Lðx �EðyS Lðz $8Ð#7ð{S Lð| $˜ð}S Lð~ &<Ð%;ðS Lð@ *DÐ)CðAS LðB $˜ðCS LðD �xðES LðF (˜<ðGS LðH %:Ð$9ðIS LðJ &˜+ðKS LðL &<Ð%;ðMS LðN %:Ð$9ðOS LðP !2Ð 1ðQS LðR 0Ð /ðSS LðT �4ðUS LðV #6Ð"5ðWS LðX &˜+ðYS LðZ 2TÐ1Sð[S Lð\ "4Ð!3ð]S Lð^ "˜ ð_S Lð` &<Ð%;ðaS Lðb �EðcS Lðd $˜ðeS Lðf "˜ ðgS Lðh .˜oðiS Lðj "4Ð!3ðkS Lðl "˜ ðmS Lðn *DÐ)CðoS Lðp !2Ð 1ðqS Lðr %:Ð$9ðsS Lðt %:Ð$9ðuS Lðv -JÐ,IðwS Lðx #6Ð"5ðyS Lðz *DÐ)Cð{S Lð| &˜+ð}S Lð~ &<Ð%;ðS Lð@ (˜<ðAS LðB (˜<ðCS LðD "˜ ðES LðF 0Ð /ðGS LðH .˜oðIS LðJ (˜<ðKS LðL &<Ð%;ðMS LðN -JÐ,IðOS LðP *DÐ)CðQS LðR &<Ð%;ðSS LðT (˜<ðUS LðV $8Ð#7ðWS LðX (@Ð'?ðYS LðZ !2Ð 1ð[S Lð\ *˜Mð]S Lð^ $8Ð#7ð_S Lð` 0Ð /ðaS Lðb &˜+ðcS Lðd "˜ ðeS Lðf &˜+ðgS Lðh *˜MðiS Lðj %:Ð$9ðkS Lðl "4Ð!3ðmS Lðn )BÐ(AðoS Lðp -JÐ,IðqS Lðr #6Ð"5ðsS Lðt $8Ð#7ðuS Lðv "4Ð!3ðwS Lðx *˜MðyS Lðz 0Ð /ð{S Lð| #6Ð"5ð}S Lð~ &<Ð%;ðS Lð@ -JÐ,IðAS LðB $˜ðCS LðD !2Ð 1ðES LðF %:Ð$9ðGS LðH �4ðIS LðJ "˜ ðKS LðL 0Ð /ðMS LðN "4Ð!3ðOS LðP "4Ð!3ðQS LðR *˜MðSS LðT .˜oðUS LðV $8Ð#7ðWS LðX "4Ð!3ðYS LðZ .˜oð[S Lð\ (@Ð'?ð]S Lð^ !2Ð 1ð_S Lð` %:Ð$9ðaS Lðb 0Ð /ðcS Lðd ,˜^ðeS Lðf )BÐ(AÀFðgS Lð S Lð S Lðh %9ˆÔ!Ø"4ˆÔ Ð Ð rr )•NNFFFrz FrV rV NNr{ r{ r r| r} r~ r r€ r� r‚ rƒ rU r„ r… r r† r‡ TNrˆ FrZ Frˆ r‰ NTFFFFFFrŠ rŠ FFFFr‹ rŒ FFNrU NNFr� FNr NrU NNTNFNNFr� r NNNNrŽ r� NFFr� NNNNTFTFFNNr‘ NNFNFNFTrŒ NNNr� TFNr’ r“ FNNFFNNFFFNFTr” r• Nr… r– r‚ r‚ r— Nr˜ FNTFNNNFr™ NrU )
|
|||
|
|
Ú__name__Ú
|
|||
|
|
__module__Ú __qualname__Ú__doc__rJ rx r r Ú__annotations__ry Úintr6 Ú
|
|||
|
|
__classcell__©r9 s @rp rt rt 3 sŒ ø€ € € € € € ðCð CðH +0¨%ØØÐ 1Ð2ð+ñ +ô +И( 3œ-ð ð ñ ð */¨ØØÐ VÐWð*ñ *ô *И #œ ð ð ñ ð Ø #ØØØØ Ø $Ø&'Ø%&Ø#'Ø"&Ø&'Ø"#ØØ"%Ø ØØØØ ØØ ØØ $ØØØ Ø %Ø ØØ"Ø "ØØ!&Ø ØØ Ø Ø !Ø Ø27ØØØ ØØØ ØØØØ Ø!'Ø Ø ØØØØ Ø !ØØ $ØØ!"Ø%)ØØØØ $ØØ!&Ø $Ø Ø ØØ ØØ-1Ø !ØØ!$Ø ØØØ Ø %ØØ%)Ø Ø $Ø $Ø(-Ø "Ø%*ØØ!%ØØ#ØØ Ø ØØ!&Ø(,Ø%*Ø!%Ø Ø #Ø#'Ø ØØ #Ø ØØØØ Ø $Ø !Ø$)Ø(-Ø "Ø #Ø "Ø Ø Ø "Ø!&Ø(,ØØ Ø $ØØØ Ø Ø !Ø Ø$Ø $Ø !Ø Ø#(Ø Ø $Ø Ø Ø$+Ø #Ø ðmw5ð w5ð w5ð w5ð w5ð w5ð w5ð w5ð w5ð w5rr rt c óð ‡ — e Zd ZdZddgZ dJdeeej e f de
|
|||
|
|
eeej e f de d e
|
|||
|
|
e d
|
|||
|
|
e
|
|||
|
|
ee e
|
|||
|
|
e e f f d e
|
|||
|
|
eeeeef d e
|
|||
|
|
e d
|
|||
|
|
e
|
|||
|
|
eg ef de
|
|||
|
|
ee deej j ej j j f de
|
|||
|
|
eej ej gej f de
|
|||
|
|
e
|
|||
|
|
de
|
|||
|
|
ee ge
|
|||
|
|
f de
|
|||
|
|
e de
|
|||
|
|
e f ˆ fd„
|
|||
|
|
Z e d„ ¦ « Z de!fˆ fd„ Z"dKd
|
|||
|
|
e
|
|||
|
|
e de!fˆ fd„
|
|||
|
|
Z#de
|
|||
|
|
de
|
|||
|
|
fd„Z$e% dLd ej& d ej' d e(d!e)d"e(dej& f d#„¦ « Z*dej d$e
|
|||
|
|
e eeej' f f deej& ej& ej& ej& f fd%„Z+d&ej& d'ej& d(ej& d)ej& d*ej& d+ej& deej& ej& ej& ej& f fd,„Z,d-„ Z-d.„ Z.d$e
|
|||
|
|
e eeej' f f fd/„Z/ dMdeeej f d0e
|
|||
|
|
e eej e0f f deej eej e
|
|||
|
|
e ej f f f fd1„Z1dNd3e
|
|||
|
|
e e2f d4e3d5 ddfd6„Z4dKd7e
|
|||
|
|
e de
|
|||
|
|
ej5 j6 j7 fd8„Z8d$e
|
|||
|
|
e ej' f dee e f fd9„Z9 dKdeeej f d0e
|
|||
|
|
e eej e0f f d:e(d;e
|
|||
|
|
ee fd<„Z: dOd>e!d?e d:e
|
|||
|
|
e( d;e
|
|||
|
|
ee d@e de f ˆ fdA„
|
|||
|
|
Z;dKdBe
|
|||
|
|
e e2f dCe
|
|||
|
|
e2 ddfˆ fdD„
|
|||
|
|
Z<ˆ fdE„Z= dPdFe
|
|||
|
|
e dGe
|
|||
|
|
e dHee ee df fdI„Z>ˆ xZ?S )QÚ_UnslothKTOTrainerr� Útrlr– N©NNr™ Ú ref_modelÚargsÚ
|
|||
|
|
train_datasetÚ eval_datasetÚprocessing_classÚ
|
|||
|
|
data_collatorÚ
|
|||
|
|
model_initÚ callbacksÚ
|
|||
|
|
optimizersÚ preprocess_logits_for_metricsÚ peft_configÚcompute_metricsÚmodel_adapter_nameÚref_adapter_namec
|
|||
|
|
ó˜ •‡‡— t |¦ « t u rt d¦ « ‚t |t ¦ « s||u rt d¦ « ‚|j €i }nªt |t ¦ « st d¦ « ‚|j }| d¦ « }|�ht |t ¦ « r|dk rt t |¦ « }|dk r-t |t j ¦ « st d|› d�¦ « ‚||d< |j
|
|||
|
|
€i }nªt |t ¦ « st d¦ « ‚|j
|
|||
|
|
}| d¦ « }|�ht |t ¦ « r|dk rt t |¦ « }|dk r-t |t j ¦ « st d|› d�¦ « ‚||d< t |t ¦ « rt j |fi |¤Ž}t |t ¦ « rt j |fi |¤Ž}d | _
|
|||
|
|
t ¦ « s| �t d
|
|||
|
|
¦ « ‚t ¦ « �r5| ��2t |t ¦ « r| ¦ « }t |d d ¦ « st |d d ¦ « r`t# |d
|
|||
|
|
¦ « o,d
|
|||
|
|
t% t' j t* ¦ « j ¦ « v }d|j i}|r
|
|||
|
|
|j |d
|
|||
|
|
< t+ |fi |¤Ž}nV|j rOt# |d¦ « r| ¦ « n*d„ }| ¦ « |¦ « |}|j r't |d d ¦ « rt; |¦ « d| _
|
|||
|
|
nV|j rOt# |d¦ « r| ¦ « n*d„ }| ¦ « |¦ « |j r+t? ¦ « s tA ¦ « st d¦ « ‚|�|j! j" | _" n"|j" €t d¦ « ‚|j" | _" t ¦ « ot |t ¦ « | _# || _$ || _% |r|| _& n*| j# s|j' rd | _& ntQ |¦ « | _& |€t d¦ « ‚|j) € tU j+ dtX ¦ « d}|j) �|j) }|j- € tU j+ dtX ¦ « d}|j- �|j- }d }|j. €#| j" r tU j+ dtX ¦ « d}|j. �| j" r|j. }|€Qt_ |j0 |j1 | j" ¬¦ « }|j2 r!d |_2 tU j+ d tX ¦ « d| _3 nd | _3 |j4 r*tk |¦ « | j& �tk | j& ¦ « |j6 | _6 || _) |j | _ |j1 | _1 |j7 �|j7 n|j0 | _7 || _- |j8 | _8 || _. || _9 |j' | _' d| _: | j6 d v rd | _: d | _; d | _< t{ d „ ¦ « | _> |j? | _? |j@ | _@ |jA | _A t |j! d d ¦ « | _B t |j! d d!¦ « | _C | jB r%| jC d!k rtU j+ d"tX ¦ « d|jD d#< t‹ ¦ « F ¦ « 5 ‰ G t� |jI d$¬%¦ « Št• ‰|jI d&¬'¦ « Š‰ G t– d(|i|jI d)¬*¦ « Š‰�^‰ G t� |jI d+¬%¦ « Št• ‰|jI d,¬'¦ « Š‰ G t– d(|i|jI d-¬*¦ « Š‰ G t˜ dd(| j9 i|jI d.¬/¦ « Šd0| j" | j9 | j) | j8 | j1 | j- | j. d1œ}‰ G tš ||jI d2¬*¦ « Š‰�N‰ G t˜ d(| j9 id|jI d3¬4¦ « Š‰ G tš ||jI d5¬*¦ « Š| j: �r|jN d6k rt d7¦ « ‚‰ G tž d|jN |jI d8¬9¦ « }d:|d;< | G tš ||jI ˆfd<„|jP D ¦ « d=¬>¦ « }t£ ‰|gd6¬?¦ « Š‰�q‰ G tž d|jN |jI d@¬9¦ « }| G tš ||jI ˆfdA„|jP D ¦ « dB¬>¦ « }t£ ‰|gd6¬?¦ « Št¥ t§ ‰dC ¦ « d6¦ « } t¥ t© ‰dC ¦ « | z
|
|||
|
|
d6¦ « } | | k rÍt« | | jA z | z d6z dD¦ « } t« | | jA z | z dEz dD¦ « } t« | | j@ z | z dEz dD¦ « } t« | | j@ z | z d6z dD¦ « }!| | j@ cxk o| k nc }"| | jA cxk o|!k nc }#|"s)|#s'tU j+ dF| › dG| › dH| › dG|!› dI� tX ¦ « d d d ¦ « n # 1 swxY w Y t ¦ « W |||‰‰|||
|
|||
|
|
| |
|
|||
|
|
| ¬J¦ « d | _X t# | jY dK¦ « r | jY Z | j[ ¦ « t# | dL¦ « st¹ dM¦ « ‚| j] r0| j^ j_ j` ja dNk r| j' rt dO¦ « ‚| j& € | j# s| j' st dP¦ « ‚nM| j] r tÅ | j& | j^ ¦ « | _& n&| j^ c | j& d¬Q¦ « | _& | jd je r’tÍ ¦ « stÏ dR¦ « ‚| j6 d v rt dS¦ « ‚| j' rt dT¦ « ‚| j# s| j% �t dU¦ « ‚tÑ | j1 | j? | j& d u¬V¦ « | _i d S d S )WNz1Please use `KTOConfig` instead TrainingArguments.zœ`model` and `ref_model` cannot be the same object. If you want `ref_model` to be the same as `model`, you must mass a copy of it, or `None` if you use peft.zRYou passed model_kwargs to the KTOTrainer. But your model is already instantiated.Ú torch_dtyperŒ znInvalid `torch_dtype` passed to the KTOConfig. Expected a string with either `torch.dtype` or 'auto', but got ú.zZYou passed ref_model_kwargs to the KTOTrainer. But your ref_model is already instantiated.FzŽPEFT is not installed and you passed a `peft_config` in the trainer's kwargs, please install it with `pip install peft` to use the PEFT modelsÚis_loaded_in_8bitÚis_loaded_in_4bitr Úuse_gradient_checkpointingÚenable_input_require_gradsc ó0 — | d¦ « d S ©NT©Úrequires_grad_©ÚmoduleÚinputÚoutputs rp Úmake_inputs_require_gradz=_UnslothKTOTrainer.__init__.<locals>.make_inputs_require_grad' s € Ø ×-Ò-¨dÑ3Ô3Ð3Ð3Ð3rr Tc ó0 — | d¦ « d S r\ r] r_ s rp rc z=_UnslothKTOTrainer.__init__.<locals>.make_inputs_require_grad< s € Ø×)Ò)¨$Ñ/Ô/Ð/Ð/Ð/rr z‚`generate_during_eval=True` requires Weights and Biases or Comet to be installed. Please install `wandb` or `comet-ml` to resolve.zMWhen no model is provided, you need to pass the parameter is_encoder_decoder.zdmax_length or a processing_class must be specified when using the default DPODataCollatorWithPaddingz¬When using DPODataCollatorWithPadding, you should set `max_length` in the KTOTrainer's init it will be set to `512` by default, but you should do it yourself in the future.r• z³When using DPODataCollatorWithPadding, you should set `max_prompt_length` in the KTOTrainer's init it will be set to `128` by default, but you should do it yourself in the future.é€ zÜWhen using DPODataCollatorWithPadding with an encoder decoder architecture, you should set `max_completion_length` in the KTOTrainer's init it will be set to `128` by default, but you should do it yourself in the future.)Ú pad_token_idr$ r( zªWhen using DPODataCollatorWithPadding, you should set `remove_unused_columns=False` in your KTOConfig we have set it for you, but you should do it yourself in the future.)Úapo_zero_unpairedc ó* — t t ¦ « S ©N)r* Úlistr0 rr rp ú<lambda>z-_UnslothKTOTrainer.__init__.<locals>.<lambda> s € µ;½tÑ3DÔ3D€ rr Úoutput_router_logitsÚrouter_aux_loss_coefrŽ a- You set `output_router_logits` to `True` in the model config, but `router_aux_loss_coef` is set to `0.0`, meaning the auxiliary loss will not be used. Either set `router_aux_loss_coef` to a value greater than `0.0`, or set `output_router_logits` to `False` if you don't want to use the auxiliary loss.Úestimate_tokensz$Extracting prompt from train dataset)Únum_procÚdesczUnpairing train dataset)rp Ú tokenizerz'Applying chat template to train dataset)Ú fn_kwargsro rp z#Extracting prompt from eval datasetzUnpairing eval datasetz&Applying chat template to eval datasetzTokenizing train dataset)Úbatchedrr ro rp r�
|
|||
|
|
batch_sizero rp ÚKL_rt c ó& •— g | ]
|
|||
|
|
}|‰j v ¯ |‘ŒS r0 ©Ú column_names)Ú.0ÚcrH s €rp ú
|
|||
|
|
<listcomp>z/_UnslothKTOTrainer.__init__.<locals>.<listcomp>) s( ø€ Ð#pÐ#pÐ#p¨!ÐPQÐUbÔUoÐPoÐPo AÐPoÐPoÐPorr z%Processing tokenized train KL dataset)rr ro Úremove_columnsrp )ÚaxiszExtracting eval KL datasetc ó& •— g | ]
|
|||
|
|
}|‰j v ¯ |‘ŒS r0 rx )rz r{ rI s €rp r| z/_UnslothKTOTrainer.__init__.<locals>.<listcomp>> s( ø€ Ð'rÐ'rÐ'r¨aÐSTÐXdÔXqÐSqÐSq¨ÐSqÐSqÐSqrr z$Processing tokenized eval KL datasetÚlabelr{ gHáz®Gõ?zìYou have different amounts of desirable/positive and undesirable/negative examples but the weights on the desirable and undesirable losses don't seem to be in an ideal range. Based on your data, we recommend EITHER desirable_weight in [z, z ] or undesirable_weight in [zN] (but NOT BOTH). See the documentation on how to optimally set these weights.) r™ rG rK rH rI rJ rL rQ rM rN rO Úadd_model_tagsÚ acceleratorzXYour `Trainer` does not have an `accelerator` object. Consider upgrading `transformers`.é zrYou cannot use `precompute_ref_log_probs=True` with Deepspeed ZeRO-3. Please set `precompute_ref_log_probs=False`.z]No reference model and model is not a Peft model. Try setting `precompute_ref_log_probs=True`)Úevaluation_modez‚You set `use_liger_loss=True` but the liger kernel is not available. Please install liger-kernel first: `pip install liger-kernel`znYou cannot set `loss_type='apo_zero_unpaired'` with liger-kernel.Only KTO loss is supported with liger-kernel.znYou cannot use `precompute_ref_log_probs=True` with liger kernel. Please set `precompute_ref_log_probs=False`.zYYou cannot use `use_liger_loss=True` with Peft models. Please set `use_liger_loss=False`.)Ú ignore_indexr Ú
|
|||
|
|
use_ref_model)jÚtyper" Ú
|
|||
|
|
ValueErrorÚ
|
|||
|
|
isinstanceÚstrr+ ÚgetÚgetattrrE Údtyper, r
|
|||
|
|
Úfrom_pretrainedÚ _peft_has_been_casted_to_bf16r2 r Úmerge_and_unloadÚhasattrrj r/ Ú signaturerA Ú
|
|||
|
|
parametersr r rZ Úget_input_embeddingsÚregister_forward_hookrÍ r? r' r3 r0 Úconfigr( Ú
|
|||
|
|
is_peft_modelrR rS rF r* r) r rG ÚwarnÚ UserWarningr r r rf r$ rà Úuse_dpo_data_collatorr) r+ r! r% r& rJ Ú calculate_KLÚ _precomputed_train_ref_log_probsÚ _precomputed_eval_ref_log_probsr* Ú_stored_metricsr r" r# Úaux_loss_enabledÚ
|
|||
|
|
aux_loss_coefÚwarnings_issuedr Úmain_process_firstÚmapr7 r- r8 r6 r% r$ r¤ r# ry r' ÚmaxÚsumÚlenÚroundr5 r6 Úmodel_accepts_loss_kwargsr™ r� Ú
|
|||
|
|
_tag_namesÚAttributeErrorÚis_deepspeed_enabledr‚ ÚstateÚdeepspeed_pluginÚ
|
|||
|
|
zero_stager@ Ú
|
|||
|
|
prepare_modelrG r. r1 Ú ImportErrorÚLigerFusedLinearKTOLossÚ kto_loss_fn)%r7 r™ rF rG rH rI rJ rK rL rM rN rO rP rQ rR rS r+ rU r, Ú_support_gc_kwargsÚprepare_model_kwargsrc r r r rr Útrain_kl_datasetÚeval_kl_datasetÚ
|
|||
|
|
num_desirableÚnum_undesirableÚdes_weight_lower_boundÚdes_weight_upper_boundÚund_weight_lower_boundÚund_weight_upper_boundÚdes_weight_in_rangeÚund_weight_in_ranger9 s% `` €rp r6 z_UnslothKTOTrainer.__init__À s9 øøø€ õ( �‰:Œ:Õ*Ð *Ð *Ý Ð PÑQÔQÐ Qå˜%¥Ñ%Ô%ð ¨)°uÐ*<Ð*<Ý ðZñô ð
|
|||
|
|
ð
|
|||
|
|
Ô !Ð )Ø "РРݘE¥3Ñ'Ô'ð
|
|||
|
|
?Ý Ð qÑrÔrÐ rà $Ô 6Ð Ø+×/Ò/°
|
|||
|
|
Ñ>Ô>ˆKØÐ&å ˜k3Ñ/Ô/ð >°KÀ6Ò4IÐ4IÝ")%°Ñ"=Ô"=�KØ &Ò(Ð(µ¸KÍÌÑ1UÔ1UÐ(Ý$ð Xð JUð Xð Xð Xñô ð ð 4?Ð! -Ñ0à Ô %Ð -Ø$&Ð !Ð !ݘI¥sÑ+Ô+ð CÝ Ølñô ð
|
|||
|
|
ð %)Ô$>Ð !Ø/×3Ò3°MÑBÔBˆKØÐ&å ˜k3Ñ/Ô/ð >°KÀ6Ò4IÐ4IÝ")%°Ñ"=Ô"=�KØ &Ò(Ð(µ¸KÍÌÑ1UÔ1UÐ(Ý$ð Xð JUð Xð Xð Xñô ð ð 8CÐ% mÑ4å �e�SÑ !Ô !ð UÝ(Ô8¸ÐTÐTÐBSÐTÐTˆEå �i¥Ñ %Ô %ð aÝ,Ô<¸YÐ`Ð`ÐJ_Ð`Ð`ˆIð .3ˆÔ*å Ñ"Ô"ð 4 ] {Ð'>Ý ð añô ð
|
|||
|
|
õ Ñ
|
|||
|
|
Ô
|
|||
|
|
ñ 0 ] [Ñ%<å˜%¥Ñ+Ô+ð
|
|||
|
|
1Ø ×.Ò.Ñ0Ô0�å�uÐ 1°5Ñ9Ô9ð
|
|||
|
|
a½WÀUÐL_ÐafÑ=gÔ=gð
|
|||
|
|
aÝ%,ØÐ9ñ&ô &ð &à5½ÝÔ%Õ&EÑFÔFÔQñ:ô :ð ð #ð )EÀdÔFaÐ'bÐ$à%ð oØLPÔLnÐ(Ð)HÑIå7¸ÐVÐVÐAUÐVÐV��ØÔ,ð
|
|||
|
|
aå˜5Ð">Ñ?Ô?ð aØ×4Ò4Ñ6Ô6Ð6Ð6ð4ð 4ð 4ð ×.Ò.Ñ0Ô0×FÒFÐG_Ñ`Ô`Ð`ð ˆEØŒyð
|
|||
|
|
:�W UÐ,?ÀÑ GÔ Gð
|
|||
|
|
:Ý+¨EÑ2Ô2Ð2à59�Ô2øð
|
|||
|
|
Ô
|
|||
|
|
(ð ]å�uÐ :Ñ;Ô;ð
|
|||
|
|
]Ø×0Ò0Ñ2Ô2Ð2Ð2ð0ð 0ð 0ð ×*Ò*Ñ,Ô,×BÒBÐC[Ñ\Ô\Ð\à Ô $ð Õ.@Ñ.BÔ.Bð ÕFXÑFZÔFZð Ý ðDñô ð
|
|||
|
|
ð
|
|||
|
|
Ð Ø&+¤lÔ&EˆDÔ #Ð #Ø
|
|||
|
|
Ô
|
|||
|
|
$Ð
|
|||
|
|
,Ý Ð lÑmÔmÐ mà&*Ô&=ˆDÔ #å .Ñ 0Ô 0Ð QµZÀÅyÑ5QÔ5QˆÔØ"4ˆÔ Ø 0ˆÔ à ð ;Ø &ˆDŒNˆNØ
|
|||
|
|
Ô
|
|||
|
|
ð ; 4Ô#@ð ;à !ˆDŒNˆNå 3°EÑ :Ô :ˆDŒNà Ð #Ý Øvñô ð
|
|||
|
|
ð Œ?Ð "Ý ŒMðdåñ
|
|||
|
|
ô
|
|||
|
|
ð
|
|||
|
|
ð
|
|||
|
|
ˆJØ Œ?Ð &Ø œ ˆJà Ô !Ð )Ý ŒMðdåñ
|
|||
|
|
ô
|
|||
|
|
ð
|
|||
|
|
ð
|
|||
|
|
!$Ð Ø Ô !Ð -Ø $Ô 6Ð à $Ð Ø Ô %Ð -°$Ô2IÐ -Ý ŒMðdåñ
|
|||
|
|
ô
|
|||
|
|
ð
|
|||
|
|
ð
|
|||
|
|
%(Ð !Ø Ô %Ð 1°dÔ6MÐ 1Ø$(Ô$>Ð !à Ð Ý 6Ø -Ô :Ø#'Ô#:Ø#'Ô#:ð ñ ô ˆMð Ô)ð
|
|||
|
|
Ø-2�Ô*å”
|
|||
|
|
ð\å ñô ð ð *.ˆDÔ &Ð &à).ˆDÔ &ð Ô ð 9Ý $ UÑ +Ô +Ð +ØŒ~Ð)Ý(¨¬ Ñ8Ô8Ð8à œ ˆŒØ$ˆŒØ$(Ô$=ˆÔ!Ø"&Ô"9ˆÔ Ø37Ô3EÐ3Q˜TÔ /Ð /ÐWgÔWtˆÔØ!2ˆÔ Ø #Ô 3ˆÔ Ø%:ˆÔ"Ø 0ˆÔ Ø(,Ô(EˆÔ%ð !ˆÔØ Œ>Ð 2Ð 2Ð 2Ø %ˆDÔ ð 16ˆÔ-Ø/4ˆÔ,õ +Ð+DÐ+DÑ EÔ EˆÔ ð ”IˆŒ Ø $Ô 5ˆÔ Ø"&Ô"9ˆÔ Ý '¨¬ Ð6LÈeÑ TÔ TˆÔ Ý $ U¤\Ð3IÈ3Ñ OÔ OˆÔØ Ô ð TÔ%7¸3Ò%>Ð%>Ý ŒMðõ ñ
|
|||
|
|
ô
|
|||
|
|
ð
|
|||
|
|
ð 48ˆÔ Ð /Ñ0õ ‰^Œ^×
|
|||
|
|
.Ò
|
|||
|
|
.Ñ
|
|||
|
|
0Ô
|
|||
|
|
0ð S ð S à )× -Ò -Ý$¨tÔ/DÐKqð .ñ ô ˆMõ <Ø ˜tÔ 4Ð;Tð ñ ô ˆMð *× -Ò -Ý)Ø&Ð(8Ð9Ø Ô.Ø>ð .ñ ô ˆMð Ð'Ø +× /Ò /Ý(°4Ô3HÐOtð 0ñ ô � õ ?Ø $Ô"7Ð>Vð ñ ô � ð ,× /Ò /Ý-Ø *Ð,<Ð =Ø !Ô 2ØAð 0ñ ô � ð *× -Ò -ÝØ Ø&¨Ô(=Ð>Ø Ô.Ø/ð .ñ ô ˆMð Ø&*Ô&=Ø !Ô 2Ø "œoØ#'Ô#7Ø&*Ô&=Ø%)Ô%;Ø)-Ô)Cð ð ˆIð *× -Ò -Ý Ø#Ø Ô.Ø9ð .ñ ô ˆMð Ð'Ø +× /Ò /Ý Ø *¨DÔ,AÐ BØ Ø !Ô 2Ø2ð 0ñ ô � ð ,× /Ò /Ý#Ø 'Ø !Ô 2Ø<ð 0ñ ô � ð Ô ñ /
|
|||
|
|
aØÔ3°qÒ8Ð8Ý$ð bñô ð ð $1×#4Ò#4Ý#Ø Ø #Ô ?Ø !Ô 2Ø6ð $5ñ $ô $Ð ð ',� ˜(Ñ#Ø#3×#7Ò#7Ý#Ø 'Ø !Ô 2Ø#pÐ#pÐ#pÐ#pÐ/?Ô/LÐ#pÑ#pÔ#pØ@ð $8ñ $ô $Ð õ !5°mÐEUÐ5VÐ]^Ð _Ñ _Ô _�
|
|||
|
|
à Ð+à&2×&6Ò&6Ý'Ø $Ø#'Ô#CØ!%Ô!6Ø 9ð '7ñ 'ô '�Oð '6×&9Ò&9Ý'Ø"+Ø!%Ô!6Ø'rÐ'rÐ'rÐ'r°?Ô3OÐ'rÑ'rÔ'rØ Cð ':ñ 'ô '�Oõ $8¸ ÀÐ8WÐ^_Ð#`Ñ#`Ô#`�Lõ ¥ M°'Ô$:Ñ ;Ô ;¸QÑ ?Ô ?ˆMÝ !¥# m°GÔ&<Ñ"=Ô"=À
|
|||
|
|
Ñ"MÈqÑ QÔ QˆOà Ò/Ð/å).°À$ÔBYÑ0YÐ\iÑ0iÐmnÑ/nÐpqÑ)rÔ)rÐ&Ý).°À$ÔBYÑ0YÐ\iÑ0iÐmqÑ/qÐstÑ)uÔ)uÐ&Ý).°
|
|||
|
|
ÀÔ@UÑ0UÐXgÑ0gÐkoÑ/oÐqrÑ)sÔ)sÐ&Ý).°
|
|||
|
|
ÀÔ@UÑ0UÐXgÑ0gÐklÑ/lÐnoÑ)pÔ)pÐ&à&<ÀÔ@UÐ&oÐ&oÒ&oÐ&oÐYoÒ&oÐ&oÐ&oÐ&oÐ#Ø&<ÀÔ@WÐ&qÐ&qÒ&qÐ&qÐ[qÒ&qÐ&qÐ&qÐ&qÐ#à+ð Ð/Bð Ý ”MðWð 1GðWð Wð KaðWð Wð 3Ið Wð Wð Mcð Wð Wð Wõ $ñô ð ðWS ð S ð S ñ S ô S ð S ð S ð S ð S ð S ð S øøøð S ð S ð S ð S õj ‰Œ×ÒØØØ'Ø'Ø%Ø -Ø!Ø +Ø Ø!Ø*Gð ñ
|
|||
|
|
ô
|
|||
|
|
ð
|
|||
|
|
ð" */ˆÔ&õ �4”:Ð /Ñ 0Ô 0ð 7Ø ŒJ× %Ò % d¤oÑ 6Ô 6Ð 6å�t˜]Ñ+Ô+ð Ý Øjñô ð
|
|||
|
|
ð
|
|||
|
|
Ô $ð ØÔ Ô%Ô6ÔAÀQÒFÐFÈ4ÔKhÐFÝ ð Iñô ð ð Œ>Ð !ØÔ&ð
|
|||
|
|
¨$Ô*Gð
|
|||
|
|
Ý Øsñô ð øð Ô(ð
|
|||
|
|
fÝ!2°4´>À4ÔCSÑ!TÔ!T�”�à!%Ô!1×!?Ò!?ÀÄÐ`dÐ!?Ñ!eÔ!e�”ð Œ9Ô #ð Ý,Ñ.Ô.ð
|
|||
|
|
Ý!ðTñô ð ð Œ~Ð!6Ð6Ð6Ý ðDñô ð ð Ô,ð
|
|||
|
|
Ý ð8ñô ð ð Ô!ð
|
|||
|
|
TÔ%:Ð%FÝ Øoñô ð õ 7Ø !Ô 4¸4¼9ÐUYÔUcÐkoÐUoð ñ ô ˆDÔ Ð Ð ð) ð s Ü4N2k2ë2k6ë9k6c # óz K — | j r8| j s1| j | j ¦ « ¦ « n
|
|||
|
|
t
|
|||
|
|
¦ « 5 | j r | j | j ¦ « dV — | j r!| j | j pd¦ « ddd¦ « dS # 1 swxY w Y dS )zWContext manager for handling null reference model (that is, peft adapter manipulation).Nrv ) r— rS r‚ Ú unwrap_modelr™ Údisable_adapterr; Ú set_adapterrR )r7 s rp Únull_ref_contextz#_UnslothKTOTrainer.null_ref_context¥ s è è € ð
|
|||
|
|
Ô!ð
|
|||
|
|
Ø*.Ô*?ð
|
|||
|
|
ˆDÔ × )Ò )¨$¬*Ñ 5Ô 5× EÒ EÑ GÔ GÐ Gå ‘ ” ð Mð Mð
|
|||
|
|
Ô$ð
|
|||
|
|
>Ø”
|
|||
|
|
×&Ò& tÔ'<Ñ=Ô=Ð=Ø ˆEˆEˆEØÔ$ð
|
|||
|
|
MØ”
|
|||
|
|
×&Ò& tÔ'>Ð'KÀ)ÑLÔLÐLð Mð Mð Mñ Mô Mð Mð Mð Mð Mð Mð Mð Møøøð Mð Mð Mð Mð Mð Ms ÁAB0Â0B4Â7B4Úreturnc óø •— | j �rÒ| j �sÊ| j j | j | j j | j j ddœ}| j t | j
|
|||
|
|
fi |¤Ž¦ « }g }g }t |d¬¦ « D ]£}| |¦ « \ }}| j
|
|||
|
|
|¦ « }| | ¦ « ¦ « | j rA| j
|
|||
|
|
|¦ « }| | ¦ « ¦ « Œ¤| j
|
|||
|
|
dt% j |¦ « ¦ « ¦ « ¬¦ « | _
|
|||
|
|
| j rW| j
|
|||
|
|
dt% j |¦ « ¦ « ¦ « ¬¦ « | _
|
|||
|
|
d| _ t- ¦ « ¦ « S ) z·
|
|||
|
|
Returns the training [`~torch.utils.data.DataLoader`].
|
|||
|
|
|
|||
|
|
Subclass of transformers.src.transformers.trainer.get_train_dataloader to precompute `ref_log_probs`.
|
|||
|
|
F©ru Ú
|
|||
|
|
collate_fnÚ num_workersÚ
|
|||
|
|
pin_memoryÚshufflez!Train dataset reference log probs©Úiterablerp Úreference_logps©ÚnameÚcolumnÚreference_KL_logpsT)r* rœ rG r¤ rK rÛ rö r‚ Úpreparer rH rF Úcompute_reference_log_probsÚgather_for_metricsre Úcpur› Ú
|
|||
|
|
add_columnrE ÚcatÚfloatÚnumpyr5 Úget_train_dataloader) r7 Údataloader_paramsÚ data_loaderÚreference_completion_logpsrÑ Ú padded_batchÚreference_completion_logpÚreference_KL_logpr9 s €rp rÚ z'_UnslothKTOTrainer.get_train_dataloader³ s÷ ø€ ð Ô (ñ ! 9°Ô1Vñ ! 9à "œiÔ CØ "Ô 0Ø #œyÔ ?Ø "œiÔ =Ø ð !ð !Ð ð Ô*×2Ò2µ:¸dÔ>PÐ3fÐ3fÐTeÐ3fÐ3fÑgÔgˆKØ)+Ð &Ø!#Ð å $¨kÐ@cÐ dÑ dÔ dð
|
|||
|
|
Gð
|
|||
|
|
G� Ø?C×?_Ò?_Ð`lÑ?mÔ?mÑ<Ð)Ð+<à,0Ô,<×,OÒ,OÐPiÑ,jÔ,jÐ)Ø*×1Ò1Ð2K×2OÒ2OÑ2QÔ2QÑRÔRÐRàÔ$ð GØ(,Ô(8×(KÒ(KÐL]Ñ(^Ô(^Ð%Ø&×-Ò-Ð.?×.CÒ.CÑ.EÔ.EÑFÔFÐFøà!%Ô!3×!>Ò!>Ø&u¬yÐ9SÑ/TÔ/T×/ZÒ/ZÑ/\Ô/\×/bÒ/bÑ/dÔ/dð "?ñ "ô "ˆDÔ ð Ô ð
|
|||
|
|
Ø%)Ô%7×%BÒ%BØ-µe´iÐ@RÑ6SÔ6S×6YÒ6YÑ6[Ô6[×6aÒ6aÑ6cÔ6cð &Cñ &ô &�Ô"ð 59ˆDÔ 1å‰wŒw×+Ò+Ñ-Ô-Ð-rr c ó, •— |€| j €t d¦ « ‚|�|n| j }| j �rÇ| j �s¿| j j | j | j j | j j ddœ}| j
|
|||
|
|
t |fi |¤Ž¦ « }g }g }t |d¬¦ « D ]£}|
|
|||
|
|
|¦ « \ }}| j |¦ « }| | ¦ « ¦ « | j rA| j |¦ « }| | ¦ « ¦ « Œ¤| dt' j |¦ « ¦ « ¦ « ¬¦ « }| j rM| d t' j |¦ « ¦ « ¦ « ¬¦ « }| j �|| _ d
|
|||
|
|
| _ t/ ¦ « |¬ ¦ « S ) aé
|
|||
|
|
Returns the evaluation [`~torch.utils.data.DataLoader`].
|
|||
|
|
|
|||
|
|
Subclass of transformers.src.transformers.trainer.get_eval_dataloader to precompute `ref_log_probs`.
|
|||
|
|
|
|||
|
|
Args:
|
|||
|
|
eval_dataset (`torch.utils.data.Dataset`, *optional*):
|
|||
|
|
If provided, will override `self.eval_dataset`. If it is a [`~datasets.Dataset`], columns not accepted
|
|||
|
|
by the `model.forward()` method are automatically removed. It must implement `__len__`.
|
|||
|
|
Nz-Trainer: evaluation requires an eval_dataset.FrÆ z Eval dataset reference log probsrË rÍ rÎ rÑ T)rI )rI rˆ r* r� rG r¥ rK rÛ rö r‚ rÒ r rF rÓ rÔ re rÕ r› rÖ rE r× rØ rÙ r5 Úget_eval_dataloader)
|
|||
|
|
r7 rI rÛ rÜ rÝ rÑ rÞ rß rà r9 s
|
|||
|
|
€rp râ z&_UnslothKTOTrainer.get_eval_dataloaderß s+ ø€ ð Ð DÔ$5Ð$=Ý Ð LÑMÔMÐ MØ'3Ð'?�|�|ÀTÔEVˆ à Ô (ñ $ 8°Ô1Uñ $ 8à "œiÔ BØ "Ô 0Ø #œyÔ ?Ø "œiÔ =Ø ð !ð !Ð ð Ô*×2Ò2µ:¸lÐ3`Ð3`ÐN_Ð3`Ð3`ÑaÔaˆKà)+Ð &Ø!#Ð å $¨kÐ@bÐ cÑ cÔ cð
|
|||
|
|
Gð
|
|||
|
|
G� Ø?C×?_Ò?_Ð`lÑ?mÔ?mÑ<Ð)Ð+<à,0Ô,<×,OÒ,OÐPiÑ,jÔ,jÐ)Ø*×1Ò1Ð2K×2OÒ2OÑ2QÔ2QÑRÔRÐRàÔ$ð GØ(,Ô(8×(KÒ(KÐL]Ñ(^Ô(^Ð%Ø&×-Ò-Ð.?×.CÒ.CÑ.EÔ.EÑFÔFÐFøà'×2Ò2Ø&u¬yÐ9SÑ/TÔ/T×/ZÒ/ZÑ/\Ô/\×/bÒ/bÑ/dÔ/dð 3ñ ô ˆLð Ô ð
|
|||
|
|
Ø +× 6Ò 6Ø-µe´iÐ@RÑ6SÔ6S×6YÒ6YÑ6[Ô6[×6aÒ6aÑ6cÔ6cð 7ñ ô � ð
|
|||
|
|
Ô Ð,Ø$0�Ô!Ø37ˆDÔ 0å‰wŒw×*Ò*¸ Ð*ÑEÔEÐErr rÞ c ó6 — t j ¦ « 5 | j �€| ¦ « 5 | j rŽ| |d |d | d¦ « |d ¬¦ « j }| j rC| |d |d | d ¦ « |d
|
|||
|
|
¬¦ « j }nW| |d |d ¬
|
|||
|
|
¦ « j }| j r(| |d |d ¬
|
|||
|
|
¦ « j }ddd¦ « n # 1 swxY w Y nì| j rŽ| |d |d | d¦ « |d ¬¦ « j }| j rC| |d |d | d ¦ « |d
|
|||
|
|
¬¦ « j }nW| |d |d ¬
|
|||
|
|
¦ « j }| j r(| |d |d ¬
|
|||
|
|
¦ « j }ddd¦ « n # 1 swxY w Y | ||d d| j | j
|
|||
|
|
¬¦ « }| j r+| ||d
|
|||
|
|
d| j | j
|
|||
|
|
¬¦ « }nd}||fS )zfComputes log probabilities of the reference model for a single padded batch of a KTO specific dataset.NÚprompt_input_idsÚprompt_attention_maskÚ completion_decoder_input_idsÚcompletion_labels)Úattention_maskÚdecoder_input_idsÚlabelsÚKL_prompt_input_idsÚKL_prompt_attention_maskÚ KL_completion_decoder_input_idsÚKL_completion_labelsÚcompletion_input_idsÚcompletion_attention_mask)rè ÚKL_completion_input_idsÚ KL_completion_attention_maskF©Úaverage_log_probr( r$ ) rE Úno_gradrF rà r( r™ r‹ rg r› Úget_batch_logpsr$ )r7 rÞ Úcompletion_logitsÚ KL_logitsÚcompletion_logpsÚKL_logpss rp rÓ z._UnslothKTOTrainer.compute_reference_log_probs s® € å
|
|||
|
|
Œ]‰_Œ_ð 6 !ð 6 !ØŒ~Ñ%Ø×*Ò*Ñ,Ô,ð %ð %ØÔ.ð %Ø,0¯JªJØ (Ð);Ô <Ø+7Ð8OÔ+PØ.:×.>Ò.>Ð?]Ñ.^Ô.^Ø#/Ð0CÔ#Dð -7ñ -ô -ô
|
|||
|
|
!ð *ð Ô,ð %Ø(,¯
|
|||
|
|
ª
|
|||
|
|
Ø ,Ð-BÔ CØ/;Ð<VÔ/WØ2>×2BÒ2BÐCdÑ2eÔ2eØ'3Ð4JÔ'Kð )3ñ ) ô ) ô
|
|||
|
|
%ð &øð -1¯JªJØ (Ð)?Ô @Ø+7Ð8SÔ+Tð -7ñ -ô -ô !ð *ð
|
|||
|
|
Ô,ð %Ø(,¯
|
|||
|
|
ª
|
|||
|
|
Ø ,Ð-FÔ GØ/;Ð<ZÔ/[ð )3ñ ) ô ) ô %ð &ð/%ð %ð %ñ %ô %ð %ð %ð %ð %ð %ð %øøøð %ð %ð %ð %øð8 Ô*ð !Ø(,¯ªØ$Ð%7Ô8Ø'3Ð4KÔ'LØ*6×*:Ò*:Ð;YÑ*ZÔ*ZØ +Ð,?Ô @ð )7ñ )ô )ô
|
|||
|
|
ð &ð Ô(ð !Ø$(§N¢NØ (Ð)>Ô ?Ø+7Ð8RÔ+SØ.:×.>Ò.>Ð?`Ñ.aÔ.aØ#/Ð0FÔ#Gð %3ñ %ô %ô
|
|||
|
|
!ð "øð )-¯ªØ$Ð%;Ô<È\ÐZuÔMvð )7ñ )ô )ä ð &ð Ô(ð !Ø$(§N¢NØ (Ð)BÔ CØ+7Ð8VÔ+Wð %3ñ %ô %ô !ð "ðg6 !ð 6 !ð 6 !ñ 6 !ô 6 !ð 6 !ð 6 !ð 6 !ð 6 !ð 6 !ð 6 !øøøð 6 !ð 6 !ð 6 !ð 6 !ðp ×/Ò/Ø Ø Ð,Ô -Ø "Ø #Ô 6Ø #Ô 6ð 0ñ
|
|||
|
|
ô
|
|||
|
|
Ðð Ô ð Ø×+Ò+ØØ Ð 3Ô4Ø!&Ø#'Ô#:Ø#'Ô#:ð ,ñ ô ˆHˆHð ˆHà Ð)Ð)s6 ” H.±C-D*Ä H.Ä*D. Ä.H.Ä1D. Ä2C0H.È.H2È5H2Fr— rg rê rô r$ r( c ó® — | j dd… |j k rt d¦ « ‚|s2|dd…dd…f ¦ « }| dd…dd…dd…f } n| ¦ « }||k }d|||k < t | |¦ « }|r.||z d¦ « | d¦ « z S ||z d¦ « S )a Compute the log probabilities of the given labels under the given logits.
|
|||
|
|
|
|||
|
|
Args:
|
|||
|
|
logits:
|
|||
|
|
Logits of the model (unnormalized). Shape: (batch_size, sequence_length, vocab_size)
|
|||
|
|
labels:
|
|||
|
|
Labels for which to compute the log probabilities. Label tokens with a value of label_pad_token_id are
|
|||
|
|
ignored. Shape: (batch_size, sequence_length)
|
|||
|
|
average_log_prob:
|
|||
|
|
If True, return the average log probability per (non-masked) token. Otherwise, return the sum of the
|
|||
|
|
log probabilities of the (non-masked) tokens.
|
|||
|
|
|
|||
|
|
Returns:
|
|||
|
|
A tensor of shape (batch_size,) containing the average/sum log probabilities of the given labels under the
|
|||
|
|
given logits.
|
|||
|
|
NrU zKLogits (batch and sequence length dim) and labels must have the same shape.rZ r )r] rˆ ÚclonerC r¥ )rg rê rô r$ r( Ú loss_maskro s rp rö z"_UnslothKTOTrainer.get_batch_logpse sø € ð0 Œ<˜˜˜Ô ¤ Ò ,Ð ,Ý Ð jÑkÔkÐ kà!ð $ؘA˜A˜A˜q˜r˜r˜E”]×(Ò(Ñ*Ô*ˆFؘA˜A˜A˜s ˜s A A A˜IÔ&ˆFˆFð —\’\‘^”^ˆFàÐ 0Ò0ˆ ð 01ˆˆvÐ+Ò+Ñ,å/°¸Ñ?Ô?ˆà ð 9Ø# iÑ/×4Ò4°RÑ8Ô8¸9¿=º=ÈÑ;LÔ;LÑLÐ Là# iÑ/×4Ò4°RÑ8Ô8Ð 8rr Úbatchc óª ‡— | |‰¦ « }| j r ‰d ‰ d¦ « dœni }| j rd|d< |‰d fd‰d i|¤Ž}|j }| |‰d d | j | j ¬
|
|||
|
|
¦ « }|j d t ‰d ¦ « k rt d
|
|||
|
|
¦ « ‚ˆfd„t |j d ¦ « D ¦ « }ˆfd„t |j d ¦ « D ¦ « } ||df }
|
|||
|
|
|| df } ||df } || df }
|
|||
|
|
| j r
|
|||
|
|
|
|
|||
|
|
| | |
|
|||
|
|
||j fS |
|
|||
|
|
| | |
|
|||
|
|
|fS )Nrç ræ ©rê ré Trl rï rè rð Fró r r€ z‡There is a mismatch between the number of examples in this batch and the number of examples for which an output sequence was predicted.c ó4 •— g | ]}‰d | du ¯|‘ŒS ©r€ Tr0 ©rz Úirþ s €rp r| z._UnslothKTOTrainer.forward.<locals>.<listcomp>¸ s. ø€ Ð_Ð_Ð_˜AÀUÈ7Ä^ÐTUÔEVÐZ^ÐE^ÐE^�aÐE^ÐE^ÐE^rr c ó4 •— g | ]}‰d | du ¯|‘ŒS ©r€ Fr0 r s €rp r| z._UnslothKTOTrainer.forward.<locals>.<listcomp>¹ s. ø€ ÐbÐbÐb˜aÀuÈWÄ~ÐVWÔGXÐ\aÐGaÐGa˜ÐGaÐGaÐGarr .) Ú_compute_kl_logpsr( r‹ rŸ rg rö r$ r] r¦ rˆ ÚrangeÚaux_loss)r7 r™ rþ rú Ú model_kwargsÚoutputsr÷ rù Ú
|
|||
|
|
chosen_idxÚ rejected_idxÚ chosen_logpsÚrejected_logpsÚ
|
|||
|
|
chosen_logitsÚrejected_logitss ` rp Úforwardz_UnslothKTOTrainer.forward“ sè ø€ ð ×)Ò)¨%°Ñ7Ô7ˆð Ô&ð
|
|||
|
|
Ø Ð 3Ô4Ø%*§Y¢YÐ/MÑ%NÔ%Nð
|
|||
|
|
ð
|
|||
|
|
ð
|
|||
|
|
ð
|
|||
|
|
ð
|
|||
|
|
ð Ô ð 8Ø37ˆLÐ/Ñ 0à�%Ø Ð(Ô )ð
|
|||
|
|
ð
|
|||
|
|
à Ð!<Ô=ð
|
|||
|
|
ð ð
|
|||
|
|
ð
|
|||
|
|
ˆð
|
|||
|
|
$œNÐà ×/Ò/Ø Ø Ð%Ô &Ø "Ø #Ô 6Ø #Ô 6ð 0ñ
|
|||
|
|
ô
|
|||
|
|
Ðð Ô ! !Ô $¨E°'¬NÑ(;Ô(;Ò ;Ð ;Ý ðGñô ð
|
|||
|
|
ð
|
|||
|
|
`Ð_Ð_Ð_¥Ð'7Ô'=¸aÔ'@Ñ!AÔ!AÐ_Ñ_Ô_ˆ
|
|||
|
|
ØbÐbÐbÐb¥5Ð)9Ô)?ÀÔ)BÑ#CÔ#CÐbÑbÔbˆ à'¨
|
|||
|
|
°C¨Ô8ˆ Ø)¨,¸Ð*;Ô<ˆà)¨*°c¨/Ô:ˆ
|
|||
|
|
Ø+¨L¸#Ð,=Ô>ˆà Ô ð \Ø .°-À ÐRZÐ\cÔ\lÐmÐ mà .°-À ÐRZÐ[Ð [rr Úpolicy_chosen_logpsÚpolicy_rejected_logpsÚpolicy_KL_logpsÚreference_chosen_logpsÚreference_rejected_logpsrÑ c óˆ — | j rj||z
|
|||
|
|
¦ « ¦ « }| j |¦ « ¦ « d¬¦ « }n,t
|
|||
|
|
j d¦ « |j ¦ « }|j
|
|||
|
|
d dk s|j
|
|||
|
|
d dk rz||z
|
|||
|
|
}| j dk r#dt j
|
|||
|
|
| j ||z
|
|||
|
|
z ¦ « z
|
|||
|
|
} n*| j dk r dt j
|
|||
|
|
| j |z ¦ « z
|
|||
|
|
} | j | ¦ « z }
|
|||
|
|
nbt
|
|||
|
|
j g ¦ « | j j ¦ « } t
|
|||
|
|
j g ¦ « | j j ¦ « }
|
|||
|
|
|j
|
|||
|
|
d dk s|j
|
|||
|
|
d dk rw||z
|
|||
|
|
} | j dk r#dt j
|
|||
|
|
| j || z
|
|||
|
|
z ¦ « z
|
|||
|
|
} n'| j dk r t j
|
|||
|
|
| j | z ¦ « } | j | ¦ « z }
|
|||
|
|
nbt
|
|||
|
|
j g ¦ « | j j ¦ « } t
|
|||
|
|
j g ¦ « | j j ¦ « }
|
|||
|
|
t
|
|||
|
|
j | j | z | j | z fd¦ « }||
|
|||
|
|
|
|
|||
|
|
|fS )av Compute the KTO loss for a batch of policy and reference model log probabilities.
|
|||
|
|
|
|||
|
|
Args:
|
|||
|
|
policy_chosen_logps:
|
|||
|
|
Log probabilities of the policy model for the chosen responses. Shape: (num(chosen) in batch_size,)
|
|||
|
|
policy_rejected_logps:
|
|||
|
|
Log probabilities of the policy model for the rejected responses. Shape: (num(rejected) in batch_size,)
|
|||
|
|
policy_KL_logps: Log probabilities of the policy model for the KL responses. Shape: (batch_size,)
|
|||
|
|
reference_chosen_logps:
|
|||
|
|
Log probabilities of the reference model for the chosen responses. Shape: (num(chosen) in batch_size,)
|
|||
|
|
reference_rejected_logps:
|
|||
|
|
Log probabilities of the reference model for the rejected responses. Shape: (num(rejected) in
|
|||
|
|
batch_size,)
|
|||
|
|
reference_KL_logps: Log probabilities of the reference model for the KL responses. Shape: (batch_size,)
|
|||
|
|
|
|||
|
|
Returns:
|
|||
|
|
A tuple of four tensors: (losses, chosen_rewards, rejected_rewards, KL). The losses tensor contains the KTO
|
|||
|
|
loss for each example in the batch. The chosen_rewards and rejected_rewards tensors contain the rewards for
|
|||
|
|
the chosen and rejected responses, respectively. The KL tensor contains the detached KL divergence estimate
|
|||
|
|
between the policy and reference models.
|
|||
|
|
r ©r4 rZ r– rg )r› ÚmeanÚdetachr‚ rÔ ÚclamprE Úzerosr_ Údevicer] r! r Úsigmoidr r r× r" r# )r7 r r r r r rÑ ÚklÚchosen_logratiosÚ
|
|||
|
|
chosen_lossesÚchosen_rewardsÚrejected_logratiosÚrejected_lossesÚrejected_rewardsÚlossess rp Úkto_lossz_UnslothKTOTrainer.kto_lossÆ s© € ð< Ô ð ?Ø!Ð$6Ñ6×<Ò<Ñ>Ô>×EÒEÑGÔGˆBØÔ!×4Ò4°RÑ8Ô8×=Ò=Ñ?Ô?×EÒEÈ!ÐEÑLÔLˆBˆB唘Q‘ ” ×"Ò"Ð#6Ô#=Ñ>Ô>ˆBð Ô $ QÔ '¨1Ò ,Ð ,Ð0FÔ0LÈQÔ0OÐSTÒ0TÐ0TØ 2Ð5KÑ KÐ àŒ~ Ò&Ð&à !¥A¤I¨d¬iÐ;KÈbÑ;PÑ.QÑ$RÔ$RÑ R�
|
|||
|
|
�
|
|||
|
|
Ø” Ð#6Ò6Ð6ð !"¥A¤I¨d¬iÐ:JÑ.JÑ$KÔ$KÑ K�
|
|||
|
|
à !œYÐ)9×)@Ò)@Ñ)BÔ)BÑ BˆNˆNõ "œL¨Ñ ,Ô ,× /Ò /°Ô0@Ô0GÑ HÔ HˆMÝ "œ\¨"Ñ -Ô -× 0Ò 0°Ô1AÔ1HÑ IÔ IˆNð !Ô & qÔ )¨QÒ .Ð .Ð2JÔ2PÐQRÔ2SÐWXÒ2XÐ2XØ!6Ð9QÑ!QÐ àŒ~ Ò&Ð&Ø"#¥a¤i°´ ¸RÐBTÑ=TÑ0UÑ&VÔ&VÑ"V��Ø” Ð#6Ò6Ð6Ý"#¤)¨D¬IÐ8JÑ,JÑ"KÔ"K�à #œyÐ+=×+DÒ+DÑ+FÔ+FÑ FÐ Ð õ $œl¨2Ñ .Ô .× 1Ò 1°$Ô2BÔ2IÑ JÔ JˆOÝ $œ|¨BÑ /Ô /× 2Ò 2°4Ô3CÔ3JÑ KÔ KÐ å”Ø
|
|||
|
|
Ô
|
|||
|
|
" ]Ñ
|
|||
|
|
2°DÔ4KÈoÑ4]Ð ^Ø
|
|||
|
|
ñ
|
|||
|
|
ô
|
|||
|
|
ˆð
|
|||
|
|
�~Ð'7¸Ð;Ð;rr c óf — d}| j r§| j r-|d |d |d | d¦ « dœ}n|d |d d œ}t j ¦ « 5 |d i |¤Žj }ddd¦ « n # 1 swxY w Y | ||d d
|
|||
|
|
| j | j ¬ ¦ « }|S )
|
|||
|
|
z/Compute KL log probabilities for a given batch.Nrë rì rî rí )Ú input_idsrè rê ré rñ rò )r* rè Fró r0 )r› r( r‹ rE rõ rg rö r$ )r7 r™ rþ rú ÚKL_model_kwargsrø s rp r z$_UnslothKTOTrainer._compute_kl_logps s/ € àˆØ Ô ð ØÔ&ð
|
|||
|
|
à!&Ð'<Ô!=Ø&+Ð,FÔ&GØ #Ð$:Ô ;Ø).¯ªÐ3TÑ)UÔ)Uð #ð #��ð "'Ð'@Ô!AØ&+Ð,JÔ&Kð#ð #�õ
|
|||
|
|
” ‘ ” ð
|
|||
|
|
<ð
|
|||
|
|
<Ø !˜EÐ 4Ð 4 OÐ 4Ð 4Ô ;� ð
|
|||
|
|
<ð
|
|||
|
|
<ð
|
|||
|
|
<ñ
|
|||
|
|
<ô
|
|||
|
|
<ð
|
|||
|
|
<ð
|
|||
|
|
<ð
|
|||
|
|
<ð
|
|||
|
|
<ð
|
|||
|
|
<ð
|
|||
|
|
<øøøð
|
|||
|
|
<ð
|
|||
|
|
<ð
|
|||
|
|
<ð
|
|||
|
|
<ð ×+Ò+ØØÐ,Ô-Ø!&Ø#'Ô#:Ø#'Ô#:ð ,ñ ô ˆHð ˆs Á"A<Á<B ÂB c
|
|||
|
|
óp — | ||¦ « }| | j |¦ « }| j rj||z
|
|||
|
|
¦ « ¦ « }| j |¦ « ¦ « d¬¦ « }n1t j d¦ «
|