412 lines
75 KiB
Plaintext
412 lines
75 KiB
Plaintext
|
§
|
|||
|
|
5$�hË( ã ól — d Z ddlmZ ddlZddlmZ ddlmZ ddlmZm Z m
|
|||
|
|
Z
|
|||
|
|
m Z m Z m
|
|||
|
|
Z
|
|||
|
|
mZmZ ddlmZmZmZmZmZmZmZmZmZmZmZmZmZmZm
|
|||
|
|
Z
|
|||
|
|
m Z m Z m Z m Z m Z m!Z!m"Z"m#Z#m Z m$Z$m%Z%m&Z&m'Z'm(Z(m)Z)m*Z*m+Z+m,Z,m-Z-m.Z.m/Z/m0Z0m1Z1m2Z2m3Z3mZm4Z4m5Z5m6Z6m7Z7m8Z8m9Z9m:Z:m;Z;m<Z<m=Z=mZm>Z>mZm
|
|||
|
|
Z
|
|||
|
|
m Z m Z m"Z"m-Z-m6Z6mZ ddl6Z6ddlT ddl?m@Z@mAZA dd lBmCZC ddlZddlDZ4dd
|
|||
|
|
lEm5Z5 ddlmZ dd lFmGZGmHZI d d
|
|||
|
|
d d
|
|||
|
|
d
|
|||
|
|
dœZJ ejK d d eJ¬¦ « d„ ¦ « ZLe@ G d„ de¦ « ¦ « ZM G d„ de"¦ « ZN G d„ deN¦ « ZOdS )z8
|
|||
|
|
2025.8.4
|
|||
|
|
2025.8.5
|
|||
|
|
4.55.1
|
|||
|
|
0.21.0
|
|||
|
|
__UNSLOTH_VERSIONING__
|
|||
|
|
é )ÚTensorN)Ú
|
|||
|
|
functional)ÚAnyÚListÚOptionalÚTupleÚUnionÚDictÚSetÚCallable)=r ÚAutoModelForCausalLMÚBaseImageProcessorr ÚDPODataCollatorWithPaddingÚ DataCollatorÚ
|
|||
|
|
DataLoaderÚDatasetÚEvalLoopOutputÚFÚFeatureExtractionMixinÚLiteralÚ
|
|||
|
|
ORPOConfigÚ ORPOTrainerr Ú PartialStateÚPathÚ PeftModelÚPreTrainedModelÚPreTrainedTokenizerBaseÚProcessorMixinÚTrainerÚTrainerCallbackr Úadd_bos_token_if_neededÚadd_eos_token_if_neededÚautocastÚ defaultdictÚdisable_dropout_in_modelÚgenerate_model_cardÚget_comet_experiment_urlÚinspectÚis_comet_availableÚis_peft_availableÚis_torch_fx_proxyÚis_torch_xla_availableÚis_wandb_availableÚ log_table_to_comet_experimentÚmaybe_apply_chat_templateÚmaybe_extract_promptÚnnÚnpÚ nullcontextÚosÚ
|
|||
|
|
pad_to_lengthÚpdÚpeft_module_casting_to_bf16Ú prepare_model_for_kbit_trainingÚrandomÚselective_log_softmaxÚtextwrapÚtorchÚwarningsr r r r r r* r4 r< )Ú*)Ú dataclassÚfield)ÚVersion)r3 )ÚDataCollatorForSeq2SeqÚ DataCollatorForLanguageModelingTF)Úepilogue_fusionÚ max_autotuneÚ
|
|||
|
|
shape_paddingz
|
|||
|
|
trace.enabledztriton.cudagraphs)ÚdynamicÚ fullgraphÚoptionsc ó’ — t j | d| j d ¦ « dd¬¦ « }t j | d¦ « dd¬¦ « }g }t ||¦ « D ]‘\ }}| t j ¦ « }t j |d| d¦ « ¬¦ « d¦ « }t j
|
|||
|
|
|d¬¦ « }||z
|
|||
|
|
} | | ¦ « Œ’ t j |¦ « }| | j d | j d f¦ « }|S )Néÿÿÿÿé r )ÚchunksÚdim)rN Úindex©rN é )
|
|||
|
|
r< ÚchunkÚreshapeÚshapeÚzipÚtoÚfloat32ÚgatherÚ unsqueezeÚsqueezeÚ logsumexpÚappendÚconcat)
|
|||
|
|
ÚlogitsrO Úchunked_logitsÚ
|
|||
|
|
chunked_indexÚall_per_token_logpsÚ chunk_logitsÚ chunk_indexÚselected_logitsÚlogsumexp_valuesÚper_token_logpss
|
|||
|
|
ú^/workspace/Fine-tuning/DS-LLM-TEMPLATE-FINETUNING/unsloth_compiled_cache/UnslothORPOTrainer.pyÚ chunked_selective_log_softmaxrh " s5 € õ ”[ § ¢ °°F´LÀÔ4DÑ!EÔ!EÐPQÐYZÐ[Ñ[Ô[€NÝ”[ § ¢ ¨rÑ!2Ô!2¸QÀaÐHÑHÔH€MØ Ðå%(¨ ¸ Ñ%GÔ%Gð 4ð 4Ñ!ˆ �kØ#—’¥u¤}Ñ5Ô5ˆ Ý œ, |¸2À{×G\ÒG\Ð]_ÑG`ÔG`ÐaÑaÔa×iÒiÐjlÑmÔmˆÝ œ?¨<¸rÐBÑBÔBÐØ)Ð,<Ñ<ˆØ×"Ò" ?Ñ3Ô3Ð3Ð3ØÝ œ,Ð':Ñ;Ô;ÐØ-×5Ò5°v´|ÀA´ÈÌ ÐUVÌÐ6XÑYÔYÐØ Ð ó c óª ‡ — e Zd ZU dZ edddi¬¦ « Zee ed< edddi¬¦ « Z ee
|
|||
|
|
ed < d.ˆ fd-„ Z ˆ xZ S )/ÚUnslothORPOConfiguB
|
|||
|
|
|
|||
|
|
Configuration class for the [`ORPOTrainer`].
|
|||
|
|
|
|||
|
|
This class includes only the parameters that are specific to ORPO training. For a full list of training arguments,
|
|||
|
|
please refer to the [`~transformers.TrainingArguments`] documentation. Note that default values in this class may
|
|||
|
|
differ from those in [`~transformers.TrainingArguments`].
|
|||
|
|
|
|||
|
|
Using [`~transformers.HfArgumentParser`] we can turn this class into
|
|||
|
|
[argparse](https://docs.python.org/3/library/argparse#module-argparse) arguments that can be specified on the
|
|||
|
|
command line.
|
|||
|
|
|
|||
|
|
Parameters:
|
|||
|
|
max_length (`int` or `None`, *optional*, defaults to `1024`):
|
|||
|
|
Maximum length of the sequences (prompt + completion) in the batch. This argument is required if you want
|
|||
|
|
to use the default data collator.
|
|||
|
|
max_prompt_length (`int` or `None`, *optional*, defaults to `512`):
|
|||
|
|
Maximum length of the prompt. This argument is required if you want to use the default data collator.
|
|||
|
|
max_completion_length (`int` or `None`, *optional*, defaults to `None`):
|
|||
|
|
Maximum length of the completion. This argument is required if you want to use the default data collator
|
|||
|
|
and your model is an encoder-decoder.
|
|||
|
|
beta (`float`, *optional*, defaults to `0.1`):
|
|||
|
|
Parameter controlling the relative ratio loss weight in the ORPO loss. In the
|
|||
|
|
[paper](https://huggingface.co/papers/2403.07691), it is denoted by λ. In the
|
|||
|
|
[code](https://github.com/xfactlab/orpo), it is denoted by `alpha`.
|
|||
|
|
disable_dropout (`bool`, *optional*, defaults to `True`):
|
|||
|
|
Whether to disable dropout in the model.
|
|||
|
|
label_pad_token_id (`int`, *optional*, defaults to `-100`):
|
|||
|
|
Label pad token id. This argument is required if you want to use the default data collator.
|
|||
|
|
padding_value (`int` or `None`, *optional*, defaults to `None`):
|
|||
|
|
Padding value to use. If `None`, the padding value of the tokenizer is used.
|
|||
|
|
truncation_mode (`str`, *optional*, defaults to `"keep_end"`):
|
|||
|
|
Truncation mode to use when the prompt is too long. Possible values are `"keep_end"` or `"keep_start"`.
|
|||
|
|
This argument is required if you want to use the default data collator.
|
|||
|
|
generate_during_eval (`bool`, *optional*, defaults to `False`):
|
|||
|
|
If `True`, generates and logs completions from the model to W&B or Comet during evaluation.
|
|||
|
|
is_encoder_decoder (`bool` or `None`, *optional*, defaults to `None`):
|
|||
|
|
When using the `model_init` argument (callable) to instantiate the model instead of the `model` argument,
|
|||
|
|
you need to specify if the model returned by the callable is an encoder-decoder model.
|
|||
|
|
model_init_kwargs (`dict[str, Any]` or `None`, *optional*, defaults to `None`):
|
|||
|
|
Keyword arguments to pass to `AutoModelForCausalLM.from_pretrained` when instantiating the model from a
|
|||
|
|
string.
|
|||
|
|
dataset_num_proc (`int` or `None`, *optional*, defaults to `None`):
|
|||
|
|
Number of processes to use for processing the dataset.
|
|||
|
|
|
|||
|
|
NÚhelpzvLLM SamplingParams)ÚdefaultÚmetadataÚvllm_sampling_paramsrK z8Chunk size to reduce memory usage. -1 is most efficient.Úunsloth_num_chunksFÚnorL é r éú ç-C ëâ6
|
|||
|
|
?ç{®Gáz„?çÍÌÌÌÌÌì?ç+‡ÙÎ÷ï?ç:Œ0âŽyE>ç ð?ç @Úlinearçš™™™™™¹?ÚpassiveÚwarningTÚstepsrQ éô éO
|
|||
|
|
ÚO1ÚautoÚ ç Ú
|
|||
|
|
adamw_8bitÚlengthÚ
|
|||
|
|
every_saveÚlasté é é éœÿÿÿÚkeep_endc� ól •— |dk rt d|› d�¦ « ‚|dk rt d|› d�¦ « ‚|€|#dk r
|
|||
|
|
|$dk rd}d }#|Œ€!d
|
|||
|
|
d lm}� t |�¦ « d z d ¦ « }Œ t ¦ « j d™i d
|
|||
|
|
|“d|“d|“d|“d|“d|“d|“d|“d| “d|
|
|||
|
|
“d| “d| “d|
|
|||
|
|
“d|“d|“d |“d |“d |“d |“d |“d!|“d"|“d#|“d$|“d%|“d&|“d'|“d(| “d)| “d*| “d+| “d,| “d-|!“d.|"“d/|#“d0|$“d1|%“d2|&“d3|'“d4|(“d5|)“d6|*“d7|+“d8|,“d9|-“d:|.“d;|/“d<|0“d=|1“d>|2“d?|3“d@|4“dA|5“dB|6“dC|7“dD|8“dE|9“dF|:“dG|;“dH|<“dI|=“dJ|>“dK|?“dL|@“dM|A“dN|B“dO|C“dP|D“dQ|E“dR|F“dS|G“dT|H“dU|I“dV|J“dW|K“dX|L“dY|M“dZ|N“d[|O“d\|P“d]|Q“d^|R“d_|S“d`|T“da|U“db|V“dc|W“dd|X“de|Y“df|Z“dg|[“dh|\“di|]“dj|^“dk|_“dl|`“dm|a“dn|b“do|c“dp|d“dq|e“dr|f“ds|g“dt|h“du|i“dv|j“dw|k“dx|l“dy|m“dz|n“d{|o“d||p“d}|q“d~|r“d|s“d€|t“d�|u“d‚|v“dƒ|w“d„|x“d…|y“d†|z“d‡|{“dˆ||“d‰|}“dŠ|~“d‹|“dŒ|€“d�|�“dŽ|‚“d�|ƒ“d�|„“d‘|…“d’|†“d“|‡“d”|ˆ“d•|‰“d–|Š“d—|‹“d˜|Œ“|�¤Ž |�| _ |Ž| _ d S )šNgH¯¼šò×z>z Unsloth: Your learning rate of `zi` is too small and less than 1e-7! Consider increasing it, otherwise gradient updates will be close to 0!rQ za` is way too larger > 1! Consider decreasing it to 1e-1, otherwise gradient updates will explode!r r€ Ú unsloth_training_checkpointsrq r )Ú cpu_countrr Ú
|
|||
|
|
output_dirÚoverwrite_output_dirÚdo_trainÚdo_evalÚ
|
|||
|
|
do_predictÚ
|
|||
|
|
eval_strategyÚprediction_loss_onlyÚper_device_train_batch_sizeÚper_device_eval_batch_sizeÚper_gpu_train_batch_sizeÚper_gpu_eval_batch_sizeÚgradient_accumulation_stepsÚeval_accumulation_stepsÚ
|
|||
|
|
eval_delayÚtorch_empty_cache_stepsÚ
|
|||
|
|
learning_rateÚ weight_decayÚ
|
|||
|
|
adam_beta1Ú
|
|||
|
|
adam_beta2Ú adam_epsilonÚ
|
|||
|
|
max_grad_normÚnum_train_epochsÚ max_stepsÚlr_scheduler_typeÚ warmup_ratioÚ warmup_stepsÚ log_levelÚlog_level_replicaÚlog_on_each_nodeÚ logging_dirÚlogging_strategyÚlogging_first_stepÚ
|
|||
|
|
logging_stepsÚlogging_nan_inf_filterÚ
|
|||
|
|
save_strategyÚ
|
|||
|
|
save_stepsÚsave_total_limitÚsave_safetensorsÚsave_on_each_nodeÚsave_only_modelÚ'restore_callback_states_from_checkpointÚno_cudaÚuse_cpuÚuse_mps_deviceÚseedÚ data_seedÚ
|
|||
|
|
jit_mode_evalÚuse_ipexÚbf16Úfp16Úfp16_opt_levelÚhalf_precision_backendÚbf16_full_evalÚfp16_full_evalÚtf32Ú
|
|||
|
|
local_rankÚ ddp_backendÚ
|
|||
|
|
tpu_num_coresÚtpu_metrics_debugÚdebugÚdataloader_drop_lastÚ
|
|||
|
|
eval_stepsÚdataloader_num_workersÚdataloader_prefetch_factorÚ
|
|||
|
|
past_indexÚrun_nameÚ disable_tqdmÚremove_unused_columnsÚ label_namesÚload_best_model_at_endÚmetric_for_best_modelÚgreater_is_betterÚignore_data_skipÚfsdpÚfsdp_min_num_paramsÚ fsdp_configÚ"fsdp_transformer_layer_cls_to_wrapÚaccelerator_configÚ deepspeedÚlabel_smoothing_factorÚoptimÚ
|
|||
|
|
optim_argsÚ adafactorÚgroup_by_lengthÚlength_column_nameÚ report_toÚddp_find_unused_parametersÚddp_bucket_cap_mbÚddp_broadcast_buffersÚdataloader_pin_memoryÚ dataloader_persistent_workersÚskip_memory_metricsÚuse_legacy_prediction_loopÚ push_to_hubÚresume_from_checkpointÚ hub_model_idÚ hub_strategyÚ hub_tokenÚhub_private_repoÚhub_always_pushÚ hub_revisionÚgradient_checkpointingÚ gradient_checkpointing_kwargsÚinclude_inputs_for_metricsÚeval_do_concat_batchesÚ fp16_backendÚpush_to_hub_model_idÚpush_to_hub_organizationÚpush_to_hub_tokenÚ
|
|||
|
|
mp_parametersÚauto_find_batch_sizeÚfull_determinismÚ torchdynamoÚ ray_scopeÚ ddp_timeoutÚ
|
|||
|
|
torch_compileÚtorch_compile_backendÚtorch_compile_modeÚinclude_tokens_per_secondÚ include_num_input_tokens_seenÚneftune_noise_alphaÚoptim_target_modulesÚbatch_eval_metricsÚ
|
|||
|
|
eval_on_startÚuse_liger_kernelÚliger_kernel_configÚeval_use_gather_objectÚ average_tokens_across_devicesÚ
|
|||
|
|
max_lengthÚmax_prompt_lengthÚmax_completion_lengthÚbetaÚdisable_dropoutÚlabel_pad_token_idÚ
|
|||
|
|
padding_valueÚtruncation_modeÚgenerate_during_evalÚis_encoder_decoderÚmodel_init_kwargsÚdataset_num_proc© ) ÚFloatingPointErrorÚ
|
|||
|
|
OverflowErrorÚmultiprocessingr‘ ÚminÚsuperÚ__init__ro rp )’Úselfr’ r“ r” r• r– r— r˜ r™ rš r› rœ r� rž rŸ r r¡ r¢ r£ r¤ r¥ r¦ r§ r¨ r© rª r« r¬ r r® r¯ r° r± r² r³ r´ rµ r¶ r· r¸ r¹ rº r» r¼ r½ r¾ r¿ rÀ rÁ r rà rÄ rÅ rÆ rÇ rÈ rÉ rÊ rË rÌ rÍ rÎ rÏ rÐ rÑ rÒ rÓ rÔ rÕ rÖ r× rØ rÙ rÚ rÛ rÜ rÝ rÞ rß rà rá râ rã rä rå ræ rç rè ré rê rë rì rí rî rï rð rñ rò ró rô rõ rö r÷ rø rù rú rû rü rý rþ rÿ r r r r r r r r r r r
|
|||
|
|
r r r
|
|||
|
|
r r r r r r r r r r r r r r r r ro rp Úkwargsr‘ Ú __class__s’ €rg r$ zUnslothORPOConfig.__init__k s_ ø€ ðd ˜4Ò Ð Õ'9ð ;VÐ]jð ;Vð ;Vð ;Vñ (Wô (Wð "WØ ˜1Ò Ð ¥Mð 3FÐUbð 3Fð 3Fð 3Fñ %Gô %Gð GØ Ð -°7Ò":Ð":¸zÈSÒ?PÐ?PØ7ˆJØ ˆMØ Ð #Ø 1Ð 1Ð 1Ð 1Ð 1Ð 1Ý " 9 9¡;¤;¨q¡=°!Ñ 4Ô 4Ð à�‰ŒÔð L :ð L :ð L :Ø#˜ðL :à#7Ð#7ðL :ð �xðL :ð �gð L :ð
|
|||
|
|
$˜ð L :ð *˜Mð
|
|||
|
|
L :ð $8Ð#7ðL :ð +FÐ*EðL :ð *DÐ)CðL :ð (@Ð'?ðL :ð '>Ð&=ðL :ð +FÐ*EðL :ð '>Ð&=ðL :ð $˜ð L :ð '>Ð&=ð L :ð *˜Mð!L :ð" (˜<ð#L :ð$ $˜ð%L :ð& $˜ð'L :ð( (˜<ð)L :ð* *˜Mð+L :ð, 0Ð /ð-L :ð. "˜ ð/L :ð0 !2Ð 1ð1L :ð2 (˜<ð3L :ð4 (˜<ð5L :ð6 "˜ ð7L :ð8 !2Ð 1ð9L :ð: 0Ð /ð;L :ð< &˜+ð=L :ð> 0Ð /ð?L :ð@ "4Ð!3ðAL :ðB *˜MðCL :ðD &<Ð%;ðEL :ðF *˜MðGL :ðH $˜ðIL :ðJ 0Ð /ðKL :ðL 0Ð /ðML :ðN !2Ð 1ðOL :ðP .˜oðQL :ðR 7^Ð6]ðSL :ðT �gðUL :ðV �gðWL :ðX ,˜^ðYL :ðZ �4ð[L :ð\ "˜ ð]L :ð^ *˜Mð_L :ð` �xðaL :ðb �4ðcL :ðd �4ðeL :ðf ,˜^ðgL :ðh &<Ð%;ðiL :ðj ,˜^ðkL :ðl ,˜^ðmL :ðn �4ðoL :ðp $˜ðqL :ðr &˜+ðsL :ðt *˜MðuL :ðv !2Ð 1ðwL :ðx �EðyL :ðz $8Ð#7ð{L :ð| $˜ð}L :ð~ &<Ð%;ðL :ð@ *DÐ)CðAL :ðB $˜ðCL :ðD �xðEL :ðF (˜<ðGL :ðH %:Ð$9ðIL :ðJ &˜+ðKL :ðL &<Ð%;ðML :ðN %:Ð$9ðOL :ðP !2Ð 1ðQL :ðR 0Ð /ðSL :ðT �4ðUL :ðV #6Ð"5ðWL :ðX &˜+ðYL :ðZ 2TÐ1Sð[L :ð\ "4Ð!3ð]L :ð^ "˜ ð_L :ð` &<Ð%;ðaL :ðb �EðcL :ðd $˜ðeL :ðf "˜ ðgL :ðh .˜oðiL :ðj "4Ð!3ðkL :ðl "˜ ðmL :ðn *DÐ)CðoL :ðp !2Ð 1ðqL :ðr %:Ð$9ðsL :ðt %:Ð$9ðuL :ðv -JÐ,IðwL :ðx #6Ð"5ðyL :ðz *DÐ)Cð{L :ð| &˜+ð}L :ð~ &<Ð%;ðL :ð@ (˜<ðAL :ðB (˜<ðCL :ðD "˜ ðEL :ðF 0Ð /ðGL :ðH .˜oðIL :ðJ (˜<ðKL :ðL &<Ð%;ðML :ðN -JÐ,IðOL :ðP *DÐ)CðQL :ðR &<Ð%;ðSL :ðT (˜<ðUL :ðV $8Ð#7ðWL :ðX (@Ð'?ðYL :ðZ !2Ð 1ð[L :ð\ *˜Mð]L :ð^ $8Ð#7ð_L :ð` 0Ð /ðaL :ðb &˜+ðcL :ðd "˜ ðeL :ðf &˜+ðgL :ðh *˜MðiL :ðj %:Ð$9ðkL :ðl "4Ð!3ðmL :ðn )BÐ(AðoL :ðp -JÐ,IðqL :ðr #6Ð"5ðsL :ðt $8Ð#7ðuL :ðv "4Ð!3ðwL :ðx *˜MðyL :ðz 0Ð /ð{L :ð| #6Ð"5ð}L :ð~ &<Ð%;ðL :ð@ -JÐ,IðAL :ðB $˜ðCL :ðD !2Ð 1ðEL :ðF %:Ð$9ðGL :ðH �4ðIL :ðJ .˜oðKL :ðL "4Ð!3ðML :ðN *˜MðOL :ðP .˜oðQL :ðR $8Ð#7ðSL :ðT "4Ð!3ðUL :ðV !2Ð 1ðWL :ðX 0Ð /°&ðYL :ð L :ð L :ðZ %9ˆÔ!Ø"4ˆÔ Ð Ð ri )ŽNNFFFrq FrL rL NNrr rr r rs rt ru rv rw rx ry rz rK r{ r| r r} r~ TNr FrQ Fr r€ NTFFFFFFr� r� FFFFr‚ rƒ FFNrK NNFr„ FNr NrK NNTNFNNFr„ r NNNNr… r† NFFr‡ NNNNTFTFFNNrˆ NNFNFNFTrƒ NNNr„ TFNr‰ rŠ FNNFFNNFFFNFTr‹ rŒ Nr| Tr� NrŽ FNNNNrK )
|
|||
|
|
Ú__name__Ú
|
|||
|
|
__module__Ú __qualname__Ú__doc__r@ ro r r Ú__annotations__rp Úintr$ Ú
|
|||
|
|
__classcell__©r' s @rg rk rk 3 su ø€ € € € € € ð-ð -ð\ +0¨%ØØÐ 1Ð2ð+ñ +ô +И( 3œ-ð ð ñ ð */¨ØØÐ VÐWð*ñ *ô *И #œ ð ð ñ ð Ø #ØØØØ Ø $Ø&'Ø%&Ø#'Ø"&Ø&'Ø"#ØØ"%Ø ØØØØ ØØ ØØ $ØØØ Ø %Ø ØØ"Ø "ØØ!&Ø ØØ Ø Ø !Ø Ø27ØØØ ØØØ ØØØØ Ø!'Ø Ø ØØØØ Ø !ØØ $ØØ!"Ø%)ØØØØ $ØØ!&Ø $Ø Ø ØØ ØØ-1Ø !ØØ!$Ø ØØØ Ø %ØØ%)Ø Ø $Ø $Ø(-Ø "Ø%*ØØ!%ØØ#ØØ Ø ØØ!&Ø(,Ø%*Ø!%Ø Ø #Ø#'Ø ØØ #Ø ØØØØ Ø $Ø !Ø$)Ø(-Ø "Ø #Ø "Ø Ø Ø "Ø!&Ø(,ØØ Ø $ØØ Ø !Ø Ø$Ø $Ø !Ø Ø Ø #Ø ð_i5ð i5ð i5ð i5ð i5ð i5ð i5ð i5ð i5ð i5ri rk c óº ‡ — e Zd ZdZddgZ dAdeeeej e
|
|||
|
|
f dee dee d ee
|
|||
|
|
d
|
|||
|
|
eee
|
|||
|
|
ee
|
|||
|
|
e
|
|||
|
|
f f d eeeeeef d eeg ef d
|
|||
|
|
eee deej j ej j j f deeej ej gej f dee deee gef fˆ fd„
|
|||
|
|
Z d„ Z dBdeeeej f defd„Z e! dCdee
|
|||
|
|
eeej" f f de#de$d e$d eej% dee
|
|||
|
|
ej" f f d „¦ « Z&d ej' d ej' deej' ej' ej' ej' ej' f fd!„Z(e! dDd"ej' d#ej" d$e#de$de#dej' f d%„¦ « Z)dej dee
|
|||
|
|
eeej" f f deej' ej' ej' ej' f fd&„Z* dEdee
|
|||
|
|
eeej" f f d(e+d) fd*„Z, dFdeeej f d+ee
|
|||
|
|
eej e-f f deej eej ee
|
|||
|
|
ej f f f fd,„Z.dee
|
|||
|
|
ej" f de
|
|||
|
|
fd-„Z/ dBdeeej f d+ee
|
|||
|
|
eej e-f f d.e#d/eee
|
|||
|
|
fd0„Z0dEd1ee
|
|||
|
|
e1f d(e+d) ddfd2„Z2 dGd4e3d5e
|
|||
|
|
d.ee# d/eee
|
|||
|
|
d6e
|
|||
|
|
de f ˆ fd7„
|
|||
|
|
Z4dBd8ee
|
|||
|
|
e1f d9ee1 ddfˆ fd:„
|
|||
|
|
Z5d;„ Z6ˆ fd<„Z7 dHd=ee
|
|||
|
|
d>ee
|
|||
|
|
d?ee
|
|||
|
|
ee
|
|||
|
|
df fd@„Z8ˆ xZ9S )IÚ_UnslothORPOTrainerr„ ÚtrlÚorpoN©NNÚmodelÚargsÚ
|
|||
|
|
data_collatorÚ
|
|||
|
|
train_datasetÚ eval_datasetÚprocessing_classÚ
|
|||
|
|
model_initÚ callbacksÚ
|
|||
|
|
optimizersÚ preprocess_logits_for_metricsÚ peft_configÚcompute_metricsc
|
|||
|
|
|
|||
|
|
ó^
|
|||
|
|
•— |j €i }
|
|||
|
|
nªt |t ¦ « st d¦ « ‚|j }
|
|||
|
|
|
|
|||
|
|
d¦ « }|�ht |t ¦ « r|dk rt t |¦ « }|dk r-t |t j ¦ « st d|› d�¦ « ‚||
|
|||
|
|
d< t |t ¦ « rt j |fi |
|
|||
|
|
¤Ž}d| _
|
|||
|
|
t ¦ « s| �t d¦ « ‚t ¦ « �r5| ��2t |t ¦ « r|
|
|||
|
|
¦ « }t |dd¦ « st |d d¦ « r`t |d
|
|||
|
|
¦ « o,d
|
|||
|
|
t t! j t$ ¦ « j ¦ « v }d |j i}|r
|
|||
|
|
|j |d
|
|||
|
|
< t% |fi |¤Ž}nV|j rOt |d ¦ « r| ¦ « n*d
|
|||
|
|
„ }| ¦ « |¦ « |}|j r't |d d¦ « rt5 |¦ « d| _
|
|||
|
|
nV|j rOt |d ¦ « r| ¦ « n*d„ }| ¦ « |¦ « |j r+t9 ¦ « s t; ¦ « st d¦ « ‚|�|j j | _ n"|j €t d¦ « ‚|j | _ | j r"|j j | _ |j j! | _! |€t d¦ « ‚|j" € tG j$ dtJ ¦ « d}n|j" }|j& € tG j$ dtJ ¦ « d}n|j& }|j' €)| j r"tG j$ dtJ ¦ « d| _' n |j' | _' |€QtQ |j! |j) | j ¬¦ « }|j* r!d|_* tG j$ dtJ ¦ « d| _+ nd| _+ |j, rt[ |¦ « || _" |j | _ |j) | _) |j. �|j. n|j! | _. || _& |j/ | _/ || _0 |j1 | _1 t |j dd¦ « | _2 t |j dd ¦ « | _3 | j2 r%| j3 d k rtG j$ d tJ ¦ « ti d „ ¦ « | _5 d|j6 d < to ¦ « 8 ¦ « 5 | 9 tt |j; ¬ ¦ « }| 9 tx d!|i|j; ¬"¦ « }| 9 | j= |j; ¬ ¦ « }|�f| 9 tt |j; ¬ ¦ « }| 9 tx d!|i|j; ¬"¦ « }| 9 | j= |j; ¬ ¦ « }d d d ¦ « n # 1 swxY w Y t} ¦ « ? |||||||| || |
|
|||
|
|
¬#¦ « d| _@ t | jA d$¦ « r | jA B | jC ¦ « t | d%¦ « st‰ d&¦ « ‚d S )'NzSYou passed model_kwargs to the ORPOTrainer. But your model is already instantiated.Ú torch_dtyperƒ zoInvalid `torch_dtype` passed to the ORPOConfig. Expected a string with either `torch.dtype` or 'auto', but got ú.FzvPEFT is not installed and you passed a `peft_config` in the trainer's kwargs, please install it to use the PEFT modelsÚis_loaded_in_8bitÚis_loaded_in_4bitrø Úuse_gradient_checkpointingÚenable_input_require_gradsc ó0 — | d¦ « d S ©NT©Úrequires_grad_©ÚmoduleÚinputÚoutputs rg Úmake_inputs_require_gradz>_UnslothORPOTrainer.__init__.<locals>.make_inputs_require_gradá s € Ø ×-Ò-¨dÑ3Ô3Ð3Ð3Ð3ri Tc ó0 — | d¦ « d S rI rJ rL s rg rP z>_UnslothORPOTrainer.__init__.<locals>.make_inputs_require_gradö s € Ø×)Ò)¨$Ñ/Ô/Ð/Ð/Ð/ri z‚`generate_during_eval=True` requires Weights and Biases or Comet to be installed. Please install `wandb` or `comet-ml` to resolve.zMWhen no model is provided, you need to pass the parameter is_encoder_decoder.z>processing_class must be specified to tokenize a ORPO dataset.z‚`max_length` is not set in the ORPOConfig's init it will default to `512` by default, but you should do it yourself in the future.rŒ z‰`max_prompt_length` is not set in the ORPOConfig's init it will default to `128` by default, but you should do it yourself in the future.é€ z½When using an encoder decoder architecture, you should set `max_completion_length` in the ORPOConfig's init it will default to `128` by default, but you should do it yourself in the future.)Ú pad_token_idr r z²When using DPODataCollatorWithPadding, you should set `remove_unused_columns=False` in your TrainingArguments we have set it for you, but you should do it yourself in the future.Úoutput_router_logitsÚrouter_aux_loss_coefr… a- You set `output_router_logits` to `True` in the model config, but `router_aux_loss_coef` is set to `0.0`, meaning the auxiliary loss will not be used. Either set `router_aux_loss_coef` to a value greater than `0.0`, or set `output_router_logits` to `False` if you don't want to use the auxiliary loss.c ó* — t t ¦ « S ©N)r$ Úlistr ri rg ú<lambda>z._UnslothORPOTrainer.__init__.<locals>.<lambda>W s € µ;½tÑ3DÔ3D€ ri Úestimate_tokens)Únum_procÚ tokenizer)Ú fn_kwargsr[ ) r5 r6 r7 r8 r9 r: r; r@ r< r= r> Úadd_model_tagsÚ acceleratorzXYour `Trainer` does not have an `accelerator` object. Consider upgrading `transformers`.)Er Ú
|
|||
|
|
isinstanceÚstrÚ
|
|||
|
|
ValueErrorÚgetÚgetattrr< Údtyper
|
|||
|
|
Úfrom_pretrainedÚ _peft_has_been_casted_to_bf16r* r Úmerge_and_unloadÚhasattrrX r( Ú signaturer8 Ú
|
|||
|
|
parametersr÷ rø rG Úget_input_embeddingsÚregister_forward_hookr r7 r r- r) Úconfigr Údecoder_start_token_idrS r r= ÚwarnÚ UserWarningr r r r rÕ Úuse_dpo_data_collatorr r% r r r: r Úaux_loss_enabledÚ
|
|||
|
|
aux_loss_coefr$ Ú_stored_metricsÚwarnings_issuedr Úmain_process_firstÚmapr0 r r/ Ú tokenize_rowr# r$ Úmodel_accepts_loss_kwargsr5 r^ Ú
|
|||
|
|
_tag_namesÚAttributeError)r% r5 r6 r7 r8 r9 r: r; r< r= r> r? r@ r rB Ú_support_gc_kwargsÚprepare_model_kwargsrP r r r' s €rg r$ z _UnslothORPOTrainer.__init__œ s ø€ ð" Ô !Ð )Ø "РРݘE¥3Ñ'Ô'ð
|
|||
|
|
?Ý Ð rÑsÔsÐ sà $Ô 6Ð Ø+×/Ò/°
|
|||
|
|
Ñ>Ô>ˆKØÐ&å ˜k3Ñ/Ô/ð >°KÀ6Ò4IÐ4IÝ")%°Ñ"=Ô"=�KØ &Ò(Ð(µ¸KÍÌÑ1UÔ1UÐ(Ý$ð Yð KVð Yð Yð Yñô ð ð 4?Ð! -Ñ0å �e�SÑ !Ô !ð UÝ(Ô8¸ÐTÐTÐBSÐTÐTˆEð .3ˆÔ*å Ñ"Ô"ð 4 ] {Ð'>Ý ð Iñô ð
|
|||
|
|
õ Ñ
|
|||
|
|
Ô
|
|||
|
|
ñ 0 ] [Ñ%<å˜%¥Ñ+Ô+ð
|
|||
|
|
1Ø ×.Ò.Ñ0Ô0�å�uÐ 1°5Ñ9Ô9ð
|
|||
|
|
a½WÀUÐL_ÐafÑ=gÔ=gð
|
|||
|
|
aÝ%,ØÐ9ñ&ô &ð &à5½ÝÔ%Õ&EÑFÔFÔQñ:ô :ð ð #ð )EÀdÔFaÐ'bÐ$à%ð oØLPÔLnÐ(Ð)HÑIå7¸ÐVÐVÐAUÐVÐV��ØÔ,ð
|
|||
|
|
aå˜5Ð">Ñ?Ô?ð aØ×4Ò4Ñ6Ô6Ð6Ð6ð4ð 4ð 4ð ×.Ò.Ñ0Ô0×FÒFÐG_Ñ`Ô`Ð`ð ˆEØŒyð
|
|||
|
|
:�W UÐ,?ÀÑ GÔ Gð
|
|||
|
|
:Ý+¨EÑ2Ô2Ð2à59�Ô2øð
|
|||
|
|
Ô
|
|||
|
|
(ð ]å�uÐ :Ñ;Ô;ð
|
|||
|
|
]Ø×0Ò0Ñ2Ô2Ð2Ð2ð0ð 0ð 0ð ×*Ò*Ñ,Ô,×BÒBÐC[Ñ\Ô\Ð\à Ô $ð Õ.@Ñ.BÔ.Bð ÕFXÑFZÔFZð Ý ðDñô ð
|
|||
|
|
ð
|
|||
|
|
Ð Ø&+¤lÔ&EˆDÔ #Ð #Ø
|
|||
|
|
Ô
|
|||
|
|
$Ð
|
|||
|
|
,Ý Ð lÑmÔmÐ mà&*Ô&=ˆDÔ #à Ô "ð :Ø*/¬,Ô*MˆDÔ 'Ø %¤ Ô 9ˆDÔ à Ð #Ý Ð ]Ñ^Ô^Ð ^Ø Œ?Ð "Ý ŒMðeåñ
|
|||
|
|
ô
|
|||
|
|
ð
|
|||
|
|
ð
|
|||
|
|
ˆJˆJà œ ˆJØ Ô !Ð )Ý ŒMðeåñ
|
|||
|
|
ô
|
|||
|
|
ð
|
|||
|
|
ð
|
|||
|
|
!$Ð Ð à $Ô 6Ð à Ô %Ð -°$Ô2IÐ -Ý ŒMðeåñ
|
|||
|
|
ô
|
|||
|
|
ð
|
|||
|
|
ð
|
|||
|
|
*-ˆDÔ &Ð &à)-Ô)CˆDÔ &à Ð Ý 6Ø -Ô :Ø#'Ô#:Ø#'Ô#:ð ñ ô ˆMð Ô)ð
|
|||
|
|
Ø-2�Ô*å”
|
|||
|
|
ð\å ñô ð ð *.ˆDÔ &Ð &à).ˆDÔ &ð Ô ð ,Ý $ UÑ +Ô +Ð +à$ˆŒØ$(Ô$=ˆÔ!Ø"&Ô"9ˆÔ Ø37Ô3EÐ3Q˜TÔ /Ð /ÐWgÔWtˆÔØ!2ˆÔ Ø #Ô 3ˆÔ Ø 0ˆÔ à”IˆŒ Ý '¨¬ Ð6LÈeÑ TÔ TˆÔ Ý $ U¤\Ð3IÈ3Ñ OÔ OˆÔØ Ô ð TÔ%7¸3Ò%>Ð%>Ý ŒMðõ ñ
|
|||
|
|
ô
|
|||
|
|
ð
|
|||
|
|
õ +Ð+DÐ+DÑ EÔ EˆÔ ð 48ˆÔ Ð /Ñ0õ ‰^Œ^×
|
|||
|
|
.Ò
|
|||
|
|
.Ñ
|
|||
|
|
0Ô
|
|||
|
|
0ð cð cà )× -Ò -Õ.BÈTÔMbÐ -Ñ cÔ cˆMØ )× -Ò -Ý)°kÐCSÐ5TÐ_cÔ_tð .ñ ô ˆMð *× -Ò -¨dÔ.?È$ÔJ_Ð -Ñ `Ô `ˆMØÐ'Ø +× /Ò /Õ0DÈtÔOdÐ /Ñ eÔ e� Ø +× /Ò /Ý-Ø *Ð,<Ð =Ø !Ô 2ð 0ñ ô � ð
|
|||
|
|
,× /Ò /°Ô0AÈDÔLaÐ /Ñ bÔ b� ð cð cð cñ cô cð cð cð cð cð cð cøøøð cð cð cð cõ ‰Œ×ÒØØØ'Ø'Ø%Ø -Ø!Ø +Ø Ø!Ø*Gð ñ
|
|||
|
|
ô
|
|||
|
|
ð
|
|||
|
|
ð" */ˆÔ&õ �4”:Ð /Ñ 0Ô 0ð 7Ø ŒJ× %Ò % d¤oÑ 6Ô 6Ð 6å�t˜]Ñ+Ô+ð Ý Øjñô ð
|
|||
|
|
ð ð s ÕCX Ø X Ø#X c óö — | ||z d¬¦ « }| |d¬¦ « d }|d t |¦ « d… }|d t |¦ « d… }t j ||g¦ « }t j |d ¦ « }t |¦ « t |¦ « k rt d¦ « ‚t |¦ « } ||d d| … k r| dz } |d d| … }|d d| … }
|
|||
|
|
t |¦ « t |
|
|||
|
|
¦ « k rt d¦ « ‚|d | d… }|d | d… }t
|
|||
|
|
||
|
|||
|
|
||¬ ¦ « S )
|
|||
|
|
a
|
|||
|
|
Llama tokenizer does satisfy `enc(a + b) = enc(a) + enc(b)`. It does ensure `enc(a + b) = enc(a) + enc(a +
|
|||
|
|
b)[len(enc(a)):]`. Reference:
|
|||
|
|
https://github.com/EleutherAI/lm-evaluation-harness/pull/531#issuecomment-1595586257
|
|||
|
|
F©Úadd_special_tokensÚ input_idsNÚattention_maskzBPrompt input ids and answer input ids should have the same length.rQ z@Prompt input ids and attention mask should have the same length.)Úprompt_input_idsÚprompt_attention_maskr‚ rƒ )r: Úlenr2 Ú concatenateÚarrayrb Údict) r% ÚpromptÚanswerÚfull_tokenizedr„ Úanswer_input_idsÚanswer_attention_maskÚfull_concat_input_idsÚfull_input_idsÚ response_token_ids_start_idxr… s rg Úbuild_tokenized_answerz*_UnslothORPOTrainer.build_tokenized_answer� sÀ € ð ×.Ò.¨v¸©ÐSXÐ.ÑYÔYˆØ ×0Ò0°ÈEÐ0ÑRÔRÐS^Ô_Ðà)¨+Ô6µsÐ;KÑ7LÔ7LÐ7NÐ7NÔOÐØ .Ð/?Ô @ÅÐEUÑAVÔAVÐAXÐAXÔ YÐ õ !#¤Ð0@ÐBRÐ/SÑ TÔ TÐ õ œ .°Ô"=Ñ>Ô>ˆå ˆ~Ñ Ô ¥#Ð&;Ñ"<Ô"<Ò <Ð <Ý Ð aÑbÔbÐ bõ (+Ð+;Ñ'<Ô'<Ð$ð ˜~¨kÔ :Ð;XÐ<XÐ;XÔ YÒ YÐ YØ (¨AÑ -Ð (à)¨+Ô6Ð7TÐ8TÐ7TÔUÐØ .Ð/?Ô @ÐA^ÐB^ÐA^Ô _Ð å Ð Ñ Ô ¥CÐ(=Ñ$>Ô$>Ò >Ð >Ý Ð _Ñ`Ô`Ð `à)¨+Ô6Ð7SÐ7TÐ7TÔUÐØ .Ð/?Ô @ÐA]ÐA^ÐA^Ô _Ð åØ -Ø"7Ø&Ø0ð
|
|||
|
|
ñ
|
|||
|
|
ô
|
|||
|
|
ð
|
|||
|
|
ri Úreturnc óª ‡‡— i }|d }|d }|d }| j �s-t |t ¦ « s t dt |¦ « › �¦ « ‚| |d¬¦ « }d„ | ¦ « D ¦ « }t |t ¦ « s t dt |¦ « › �¦ « ‚| ||¦ « Št |t ¦ « s t d t |¦ « › �¦ « ‚| ||¦ « Št |d
|
|||
|
|
¦ « }t ‰d
|
|||
|
|
¦ « } t ‰d
|
|||
|
|
¦ « }
|
|||
|
|
t | |
|
|||
|
|
¦ « }| ¦ « D ]\ } } | d |… || < Œt d „ t ‰d
|
|||
|
|
‰d
|
|||
|
|
¦ « D ¦ « ¦ « }
|
|||
|
|
t | |
|
|||
|
|
z
|
|||
|
|
¦ « }|
|
|||
|
|
d
|
|||
|
|
k s|d
|
|||
|
|
k rt d¦ « ‚t | j j ||| ‰|
|
|||
|
|
‰¦ « \ }ŠŠt | j j ‰‰¦ « \ ŠŠt# t ‰d ¦ « t ‰d ¦ « ¦ « }‰‰|fD ]�}t |d
|
|||
|
|
¦ « |z | j k rj| j dk r dD ]} || d | j … || < ŒŒL| j dk r dD ]} || | j d … || < Œ Œvt d| j › �¦ « ‚ŒŽ‰‰fD ]H}t |d
|
|||
|
|
¦ « |z | j k r%dD ]"} || d | j | j z
|
|||
|
|
… || < Œ#ŒIˆfd„dD ¦ « }ˆfd„dD ¦ « }|d d d … |d< | j gt ‰d
|
|||
|
|
¦ « z |d d t ‰d
|
|||
|
|
¦ « …< |d d d … |d< | j gt ‰d
|
|||
|
|
¦ « z |d d t ‰d
|
|||
|
|
¦ « …< |||dœ ¦ « D ]/\ } }| ¦ « D ]\ }}|dk rŒ ||| › |› �< ŒŒ0nú| |d| j d¬¦ « Š| |d| j d¬¦ « Š| |d| j d¬¦ « }‰d |d < ‰d |d < |d |d
|
|||
|
|
< |d |d < |�rt/ |d ¦ « rb| t3 j |d ¦ « ¬!¦ « |d"< | t3 j |d ¦ « ¬!¦ « |d#< t7 ¦ « ry|D ]v} d| v s| j r| j }n4| d$¦ « r| j }n| d%¦ « rd&}|| |g| j t || ¦ « z
|
|||
|
|
z z || < Œw|S )'a/ Tokenize a single row from a ORPO specific dataset.
|
|||
|
|
|
|||
|
|
At this stage, we don't convert to PyTorch tensors yet; we just handle the truncation in case the prompt +
|
|||
|
|
chosen or prompt + rejected responses is/are too long. First we truncate the prompt; if we're still too long,
|
|||
|
|
we truncate the chosen/rejected.
|
|||
|
|
|
|||
|
|
We also create the labels for the chosen/rejected responses, which are of length equal to the sum of the length
|
|||
|
|
of the prompt and the chosen/rejected response, with label_pad_token_id for the prompt tokens.
|
|||
|
|
rŠ ÚchosenÚrejectedz prompt should be an str but got Fr€ c ó — i | ] \ }}d |› �|“Œ S ©Úprompt_r )Ú.0ÚkÚvs rg ú
|
|||
|
|
<dictcomp>z4_UnslothORPOTrainer.tokenize_row.<locals>.<dictcomp>Ù s$ € Ð PÐ PÐ P±$°!°Q˜] q˜]˜]¨AÐ PÐ PÐ Pri z chosen should be an str but got z"rejected should be an str but got r„ Nc ó — g | ] \ }}||k ‘Œ S r r )rš ÚaÚbs rg ú
|
|||
|
|
<listcomp>z4_UnslothORPOTrainer.tokenize_row.<locals>.<listcomp>ñ s € ÐpÐpÐp™D˜A˜q��a’ÐpÐpÐpri rQ zdChosen and rejected prompt_input_ids might only differ on the last token due to tokenizer merge ops.r‚ Ú
|
|||
|
|
keep_start)r„ r… rŽ zUnknown truncation mode: )r‚ rƒ c ó: •— i | ]}|‰d |› � ‰| z “ŒS r˜ r )rš r› Ú
|
|||
|
|
chosen_tokenss €rg r� z4_UnslothORPOTrainer.tokenize_row.<locals>.<dictcomp> s= ø€ ð &ð &ð &ØGH��= ¨1 Ô/°-ÀÔ2BÑBð&ð &ð &ri c ó: •— i | ]}|‰d |› � ‰| z “ŒS r˜ r )rš r› Úrejected_tokenss €rg r� z4_UnslothORPOTrainer.tokenize_row.<locals>.<dictcomp>" s= ø€ ð (ð (ð (ØKL��? =¨Q = =Ô1°OÀAÔ4FÑFð(ð (ð (ri Úlabels)Úchosen_Ú rejected_r„ Útoken_type_idsT)Ú
|
|||
|
|
truncationr r� Ú
|
|||
|
|
chosen_labelsÚrejected_labelsrƒ r… Ú%prepare_decoder_input_ids_from_labels)r§ Úrejected_decoder_input_idsÚchosen_decoder_input_idsÚ
|
|||
|
|
_input_idsÚ_attention_maskr ) r r` ra rb Útyper: Úitemsr’ r† r" ÚsumrU Úabsr! Ú bos_token_idr" Ú eos_token_idÚmaxr r r r r ri r® r< Útensorr, Úendswithr )r% Úfeaturer5 ÚbatchrŠ r• r– Ú
|
|||
|
|
prompt_tokensÚprompt_len_input_idsÚchosen_prompt_len_input_idsÚ rejected_prompt_len_input_idsr› rœ Únum_diff_tokensÚ num_diff_lenÚlonger_response_lengthÚ
|
|||
|
|
answer_tokensÚchosen_sequence_tokensÚrejected_sequence_tokensÚtoksÚtype_keyÚtokensÚ pad_valuer¤ r¦ s @@rg ry z _UnslothORPOTrainer.tokenize_rowÁ s‰ øø€ ð ˆØ˜Ô"ˆØ˜Ô"ˆØ˜:Ô&ˆàÔ&ñ ~ õ ˜f¥cÑ*Ô*ð
|
|||
|
|
TÝ Ð!RÅDÈÁLÄLÐ!RÐ!RÑSÔSÐSØ × 1Ò 1°&ÈUÐ 1Ñ SÔ SˆMØ PÐ P¸-×:MÒ:MÑ:OÔ:OÐ PÑ PÔ PˆMå ˜f¥cÑ*Ô*ð
|
|||
|
|
TÝ Ð!RÅDÈÁLÄLÐ!RÐ!RÑSÔSÐSØ × 7Ò 7¸ÀÑ GÔ GˆMå ˜hÑ,Ô,ð
|
|||
|
|
XÝ Ð!VÅdÈ8ÁnÄnÐ!VÐ!VÑWÔWÐWØ "× 9Ò 9¸&À(Ñ KÔ KˆOõ $' }Ð5GÔ'HÑ#IÔ#IÐ å*-¨mÐ<NÔ.OÑ*PÔ*PÐ 'Ý,/°Ð@RÔ0SÑ,TÔ,TÐ )Ý#&Ð'BÐDaÑ#bÔ#bÐ à%×+Ò+Ñ-Ô-ð
|
|||
|
|
<ð
|
|||
|
|
<‘��1Ø#$Ð%:Ð&:Ð%:Ô#;�
|
|||
|
|
˜aÑ Ð õ "ØpÐp¥C¨
|
|||
|
|
Ð6HÔ(IÈ?Ð[mÔKnÑ$oÔ$oÐpÑpÔpñ ô ˆOõ Ð :Ð=ZÑ ZÑ[Ô[ˆLØ Ò"Ð" l°QÒ&6Ð&6Ý ð=ñô ð õ =TØÔ%Ô2Ø$Ø Ø+Ø Ø-Ø ñ=ô =Ñ 9ˆM˜=¨/õ .EØÔ%Ô2°MÀ?ñ.ô .Ñ *ˆM˜?õ &)¨]¸;Ô-GÑ)HÔ)HÍ#ÈoÐ^iÔNjÑJkÔJkÑ%lÔ%lÐ "ð #0° À-Ð!Pð
|
|||
|
|
]ð
|
|||
|
|
]�
|
|||
|
|
Ý�}Ð%7Ô8Ñ9Ô9Ð<RÑRÐUYÔUdÒdÐdØÔ+¨|Ò;Ð;Ø!Nð Zð Z˜AØ/<¸QÔ/?Ð@XÀ$ÔBXÐ@XÔ/Y˜M¨!Ñ ,Ð ,ðZà Ô-°Ò;Ð;Ø!Nð [ð [˜AØ/<¸QÔ/?ÀÔAWÐ@WÐ@YÐ@YÔ/Z˜M¨!Ñ ,Ð ,ð[õ )Ð)[ÀTÔEYÐ)[Ð)[Ñ \Ô \Ð\ð eð #0° Ð!Að
|
|||
|
|
hð
|
|||
|
|
h�
|
|||
|
|
Ý�}Ð%7Ô8Ñ9Ô9Ð<RÑRÐUYÔUdÒdÐdØ <ð hð h˜Ø+8¸Ô+;Ð<f¸d¼oÐPTÔPfÑ>fÐ<fÔ+g˜
|
|||
|
|
aÑ(Ð(øð&ð &ð &ð &ØLkð&ñ &ô &Ð "ð(ð (ð (ð (ØPoð(ñ (ô (Ð $ð 0FÀkÔ/RÐSTÐSTÐSTÔ/UÐ " 8Ñ ,àÔ'ðZå�MÐ"4Ô5Ñ6Ô6ñZ7Ð " 8Ô ,Ð-Us°=ÐASÔ3TÑ/UÔ/UÐ-UÑ Vð 2JÈ+Ô1VÐWXÐWXÐWXÔ1YÐ $ XÑ .àÔ'ð^å�OÐ$6Ô7Ñ8Ô8ñ^9Ð $ XÔ .Ð/Yµ°_ÐEWÔ5XÑ1YÔ1YÐ/YÑ Zð
|
|||
|
|
2Ø 5Ø!ð ð ÷ Še‰gŒgð
|
|||
|
|
5ð
|
|||
|
|
5‘��4ð
|
|||
|
|
)-¯
|
|||
|
|
ª
|
|||
|
|
© ¬ ð 5ð 5Ñ$�H˜fØ Ð#3Ò3Ð3Ø Ø.4�E˜QÐ* Ð*Ð*Ñ+Ð+ð5ð
|
|||
|
|
5ð !× 1Ò 1Ø 4°DÔ4NÐcgð 2ñ ô ˆMð #× 3Ò 3Ø T°dÔ6PÐeið 4ñ ô ˆOð !× 1Ò 1Ø 4°DÔ4JÐ_cð 2ñ ô ˆMð &3°;Ô%?ˆE�/Ñ "Ø'6°{Ô'CˆEÐ#Ñ $Ø(5°kÔ(BˆEÐ$Ñ %Ø-:Ð;KÔ-LˆEÐ)Ñ *àÐ ¥W¨UÐ4[Ñ%\Ô%\Ð Ø6;×6aÒ6aÝ œ<¨Ð.?Ô(@ÑAÔAð 7bñ 7ô 7�Ð2Ñ3ð 5:×4_Ò4_Ý œ<¨¨oÔ(>Ñ?Ô?ð 5`ñ 5ô 5�Ð0Ñ1õ "Ñ #Ô #ð Vàð
|
|||
|
|
Vð
|
|||
|
|
V�ؘq�=�= DÔ$;�=Ø $Ô 7�I�IØ—Z’Z Ñ-Ô-ð "Ø $Ô 2�I�IØ—Z’ZÐ 1Ñ2Ô2ð "Ø !�IØ œ8 y k°T´_ÅsÈ5ÐQRÌ8Á}Ä}Ñ5TÑ&UÑU��a‘�؈ ri Fr� r r½ r r r Údevicec ó” — i }|r3t | d j d | d j d ¦ « }n2t | d j d | d j d ¦ « }| D ] }| d¦ « r‰t | | t j ¦ « rid|v s|r|}n/| d¦ « r|}n| d ¦ « rd
|
|||
|
|
}| dd ¦ « } t | | ||¬ ¦ « || < Œ¡| D ]Ð}| d
|
|||
|
|
¦ « r¹t | | t j ¦ « r™d|v s|r|}n/| d¦ « r|}n| d ¦ « rd
|
|||
|
|
}| d
|
|||
|
|
d ¦ « } t j || t | | ||¬ ¦ « fd
|
|||
|
|
¬¦ «
|
|||
|
|
|¬¦ « || < ŒÑ|rf| d dd¦ «
|
|||
|
|
|¬¦ « |d< | d dd¦ «
|
|||
|
|
|¬¦ « |d< |S )aç Concatenate the chosen and rejected inputs into a single tensor.
|
|||
|
|
|
|||
|
|
Args:
|
|||
|
|
batch:
|
|||
|
|
A batch of data. Must contain the keys 'chosen_input_ids' and 'rejected_input_ids', which are tensors
|
|||
|
|
of shape (batch_size, sequence_length).
|
|||
|
|
is_encoder_decoder:
|
|||
|
|
Whether the model is an encoder-decoder model.
|
|||
|
|
label_pad_token_id:
|
|||
|
|
The label pad token id.
|
|||
|
|
padding_value:
|
|||
|
|
The padding value to use for the concatenated inputs_ids.
|
|||
|
|
device:
|
|||
|
|
The device for the concatenated inputs.
|
|||
|
|
|
|||
|
|
Returns:
|
|||
|
|
A dictionary containing the concatenated inputs under the key 'concatenated_input_ids'.
|
|||
|
|
r¬ rQ r Úchosen_input_idsÚrejected_input_idsr• r§ r± r² r Ú concatenated)rË r– rP ©rÌ r„ rr Úconcatenated_input_idsr… Úconcatenated_attention_mask) r¹ rT Ú
|
|||
|
|
startswithr` r< r r» Úreplacer5 ÚcatrV Úrepeat)
|
|||
|
|
r½ r r r rÌ Úconcatenated_batchr r› rË Úconcatenated_keys
|
|||
|
|
rg Úconcatenated_inputsz'_UnslothORPOTrainer.concatenated_inputs\ s˜ € ð4 Ðà ð gÝ ˜U ?Ô 3Ô 9¸!Ô <¸eÐDUÔ>VÔ>\Ð]^Ô>_Ñ`Ô`ˆJˆJå ˜UÐ#5Ô 6Ô <¸QÔ ?ÀÐG[ÔA\ÔAbÐcdÔAeÑfÔfˆJàð pð pˆAØ�|Š|˜HÑ%Ô%ð
|
|||
|
|
p*°U¸1´X½u¼|Ñ*LÔ*Lð
|
|||
|
|
pؘq�=�=Ð$6�=Ø 2�I�IØ—Z’Z Ñ-Ô-ð "Ø -�I�IØ—Z’ZÐ 1Ñ2Ô2ð "Ø !�IØ#$§9¢9¨X°~Ñ#FÔ#FÐ Ý7DÀUÈ1ÄXÈzÐenÐ7oÑ7oÔ7oÐ"Ð#3Ñ4øØð $ð $ˆAØ�|Š|˜JÑ'Ô'ð
|
|||
|
|
$J°u¸Q´xÅÄ Ñ,NÔ,Nð
|
|||
|
|
$ؘq�=�=Ð$6�=Ø 2�I�IØ—Z’Z Ñ-Ô-ð "Ø -�I�IØ—Z’ZÐ 1Ñ2Ô2ð "Ø !�IØ#$§9¢9¨Z¸ Ñ#HÔ#HÐ Ý7<´yà*Ð+;Ô<Ý% e¨A¤h°
|
|||
|
|
ÀiÐPÑPÔPðð ð 8ñ 8ô 8÷ ’"˜F�"Ñ#Ô#ð
|
|||
|
|
#Ð#3Ñ4øð ð Ø;@ÐASÔ;T×;[Ò;[Ð\]Ð_`Ñ;aÔ;a×;dÒ;dÐlrÐ;dÑ;sÔ;sÐ Ð 7Ñ 8àÐ-Ô.×5Ò5°a¸Ñ;Ô;×>Ò>ÀfÐ>ÑMÔMð
|
|||
|
|
Ð <Ñ =ð "Ð!ri Úpolicy_chosen_logpsÚpolicy_rejected_logpsc ó" — ||z
|
|||
|
|
t j t j |¦ « ¦ « t j t j |¦ « ¦ « z
|
|||
|
|
z
|
|||
|
|
}t j |¦ « }| j |z }| j | | j j ¦ « ¦ « z }| j | | j j ¦ « ¦ « z }|||t j
|
|||
|
|
|¦ « t j
|
|||
|
|
|¦ « fS )an Compute ORPO's odds ratio (OR) loss for a batch of policy and reference model log probabilities.
|
|||
|
|
|
|||
|
|
Args:
|
|||
|
|
policy_chosen_logps:
|
|||
|
|
Log probabilities of the policy model for the chosen responses. Shape: (batch_size,)
|
|||
|
|
policy_rejected_logps:
|
|||
|
|
Log probabilities of the policy model for the rejected responses. Shape: (batch_size,)
|
|||
|
|
|
|||
|
|
Returns:
|
|||
|
|
A tuple of three tensors: (losses, chosen_rewards, rejected_rewards). The losses tensor contains the ORPO
|
|||
|
|
loss for each example in the batch. The chosen_rewards and rejected_rewards tensors contain the rewards for
|
|||
|
|
the chosen and rejected responses, respectively. The log odds ratio of the chosen responses over the
|
|||
|
|
rejected responses ratio for logging purposes. The `log(sigmoid(log_odds_chosen))` for logging purposes.
|
|||
|
|
) r< Úlog1pÚexpr Ú
|
|||
|
|
logsigmoidr rV r_ rÌ ÚdetachÚmean)r% rÛ rÜ Úlog_oddsÚratioÚlossesÚchosen_rewardsÚrejected_rewardss rg Úodds_ratio_lossz#_UnslothORPOTrainer.odds_ratio_loss së € ð* (Ð*?Ñ?Ý ŒK�œÐ#6Ñ7Ô7Ð7Ñ 8Ô 8½5¼;ÍÌ ÐRgÑHhÔHhÐGhÑ;iÔ;iÑ iñ
|
|||
|
|
ˆõ ” ˜XÑ&Ô&ˆØ”˜UÑ"ˆà œÐ&9×&<Ò&<¸TÔ=MÔ=TÑ&UÔ&U×%]Ò%]Ñ%_Ô%_Ñ_ˆØ œ9Ð(=×(@Ò(@ÀÔAQÔAXÑ(YÔ(Y×'aÒ'aÑ'cÔ'cÑcÐà�~Ð'7½¼ÀEÑ9JÔ9JÍEÌJÐW_ÑL`ÔL`Ð`Ð`ri r^ r§ Úaverage_log_probc ó¦ — | j dd… |j k rt d¦ « ‚|s1|dd…dd…f ¦ « }| dd…dd…dd…f } ||k }t j ||k d|¦ « }t | |¦ « }|r.||z d¦ « | d¦ « z S ||z d¦ « S )aŽ Compute the log probabilities of the given labels under the given logits.
|
|||
|
|
|
|||
|
|
Args:
|
|||
|
|
logits: Logits of the model (unnormalized). Shape: (batch_size, sequence_length, vocab_size)
|
|||
|
|
labels:
|
|||
|
|
Labels for which to compute the log probabilities. Label tokens with a value of label_pad_token_id are
|
|||
|
|
ignored. Shape: (batch_size, sequence_length)
|
|||
|
|
average_log_prob:
|
|||
|
|
If True, return the average log probability per (non-masked) token. Otherwise, return the sum of the
|
|||
|
|
log probabilities of the (non-masked) tokens.
|
|||
|
|
label_pad_token_id: The label pad token id.
|
|||
|
|
is_encoder_decoder: Whether the model is an encoder-decoder model.
|
|||
|
|
|
|||
|
|
Returns:
|
|||
|
|
A tensor of shape (batch_size,) containing the average/sum log probabilities of the given labels under the
|
|||
|
|
given logits.
|
|||
|
|
NrK zKLogits (batch and sequence length dim) and labels must have the same shape.rQ r )rT rb Úcloner< Úwherer: rµ )r^ r§ ré r r Ú loss_maskrf s rg Úget_batch_logpsz#_UnslothORPOTrainer.get_batch_logpsÀ sð € ð2 Œ<˜˜˜Ô ¤ Ò ,Ð ,Ý Ð jÑkÔkÐ kà!ð 'ؘA˜A˜A˜q˜r˜r˜E”]×(Ò(Ñ*Ô*ˆFؘA˜A˜A˜s ˜s A A A˜IÔ&ˆFØÐ 0Ò0ˆ õ ”˜VÐ'9Ò 9¸1¸fÑEÔEˆå/°¸Ñ?Ô?ˆà ð 9Ø# iÑ/×4Ò4°RÑ8Ô8¸9¿=º=ÈÑ;LÔ;LÑLÐ Là# iÑ/×4Ò4°RÑ8Ô8Ð 8ri c óh ‡ — ‰ |‰ j ‰ j ‰ j ‰ j j ¬¦ « }|d j d }‰ j r d‰ |d ¦ « ini }‰ j rd|d< ||d f|d d
|
|||
|
|
d œ|¤Ž}|j }ˆ fd „}‰ j r|d
|
|||
|
|
¦ « } nA|d
|
|||
|
|
¦ « } |d }
|
|||
|
|
t j |
|
|||
|
|
d
|
|||
|
|
k | ‰ j ¦ « } ||d|… | d|… ¦ « } ‰
|
|||
|
|
||d d‰ j ‰ j ¬¦ « } | d|… }
|
|||
|
|
| |d… }‰ j s#|d|…dd…dd…f }||d…dd…dd…f }n|d|… }||d… }‰ j r
|
|||
|
|
|
|
|||
|
|
|||| |j fS |
|
|||
|
|
|||| fS )zÆRun the given model on the given batch of inputs, concatenating the chosen and rejected inputs together.
|
|||
|
|
|
|||
|
|
We do this to avoid doing two forward passes, because it's faster for FSDP.
|
|||
|
|
)r r r rÌ r¬ r Údecoder_input_idsÚconcatenated_labelsTrT rÒ rÓ F)rƒ Ú use_cachec ór •— ‰j s?| dd d…d d …f ¦ « } |ddd …f ¦ « }t j ¦ « }| d| j d ¦ « } | d¦ « }| | j ¦ « } || |¦ « }|S )N.rK rQ )r Ú
|
|||
|
|
contiguousr1 ÚCrossEntropyLossÚviewrT rV rÌ )r^ r§ Úloss_fctÚlossr% s €rg Úcross_entropy_losszD_UnslothORPOTrainer.concatenated_forward.<locals>.cross_entropy_loss s¯ ø€ ØÔ*ð
|
|||
|
|
6à S b S¨!¨!¨! Ô,×7Ò7Ñ9Ô9�Ø Q R R œ ×3Ò3Ñ5Ô5�åÔ*Ñ,Ô,ˆHØ—[’[ V¤\°"Ô%5Ñ6Ô6ˆFØ—[’[ ‘_”_ˆFà—Y’Y˜vœ}Ñ-Ô-ˆFØ�8˜F FÑ+Ô+ˆD؈Kri rQ N)ré r r rK )rÚ r r r r_ rÌ rT Ú _shift_rightrs r^ rë r< rì rî Úaux_loss)r% r5 r½ rØ Ú
|
|||
|
|
len_chosenÚ model_kwargsÚoutputsÚ
|
|||
|
|
all_logitsrù r§ rƒ Úchosen_nll_lossÚ all_logpsÚ chosen_logpsÚrejected_logpsÚ
|
|||
|
|
chosen_logitsÚrejected_logitss ` rg Úconcatenated_forwardz(_UnslothORPOTrainer.concatenated_forwardë st ø€ ð "× 5Ò 5Ø Ø #Ô 6Ø #Ô 6Ø Ô,ØÔ#Ô*ð 6ñ
|
|||
|
|
ô
|
|||
|
|
Ðð ˜?Ô+Ô1°!Ô4ˆ
|
|||
|
|
ð Ô&ð
|
|||
|
|
Ø# T×%6Ò%6Ð7IÐJ_Ô7`Ñ%aÔ%að
|
|||
|
|
ð
|
|||
|
|
ð ð ð Ô ð 8Ø37ˆLÐ/Ñ 0à�%Ø Ð 7Ô 8ð
|
|||
|
|
à-Ð.KÔLØð
|
|||
|
|
ð
|
|||
|
|
ð ð
|
|||
|
|
ð
|
|||
|
|
ˆð ”^ˆ
|
|||
|
|
ð ð ð ð ð ð Ô "ð WØ'Ð(=Ô>×DÒDÑFÔFˆFˆFà'Ð(@ÔA×GÒGÑIÔIˆFØ /Ð0MÔ NˆNÝ”[ °1Ò!4°f¸dÔ>UÑVÔVˆFà,Ð,¨Z¸ ¸¸ Ô-DÀfÈ[ÈjÈ[ÔFYÑZÔZˆà×(Ò(Ø Ø Ð 4Ô 5Ø !Ø #Ô 6Ø #Ô 6ð )ñ
|
|||
|
|
ô
|
|||
|
|
ˆ ð ! * Ô-ˆ Ø" : ; ;Ô/ˆàÔ&ð 6Ø & {¨
|
|||
|
|
{°C°R°C¸¸¸Ð':Ô ;ˆMØ (¨¨¨°c°r°c¸1¸1¸1Ð)<Ô =ˆOˆOà & {¨
|
|||
|
|
{Ô 3ˆMØ (¨¨¨Ô 5ˆOà Ô ð uØ .°-À ÐRaÐcjÔcsÐtÐ tà ˜n¨m¸_ÈoÐ^Ð^ri ÚtrainÚ
|
|||
|
|
train_eval)r Úevalc ó� — i }| ||¦ « }|dd… \ }}}} }
|
|||
|
|
| j r|d } | ||¦ « \ } }
|
|||
|
|
}}}|
|
|||
|
|
| ¦ « z
|
|||
|
|
}|
|
|||
|
|
|k ¦ « }|dk rdnd}| j |
|
|||
|
|
¦ « ¦ « ||› d�< | j |¦ « ¦ « ||› d�< | j |¦ « ¦ « ||› d�< | j |
|
|||
|
|
|z
|
|||
|
|
¦ « ¦ « ||› d �< | j |¦ « ¦ « ¦ « ||› d
|
|||
|
|
�< | j |¦ « ¦ « ¦ « ||› d �< | j | ¦ « ¦ « ¦ « ¦ « ||› d �< | j | ¦ « ¦ « ¦ « ¦ « ||› d
|
|||
|
|
�< | j |
|
|||
|
|
¦ « ¦ « ¦ « ||› d�< | j |¦ « ¦ « ¦ « ||› d�< | j |¦ « ¦ « ¦ « ||› d�< t ¦ « rt
|