418 lines
77 KiB
Plaintext
418 lines
77 KiB
Plaintext
|
§
|
|||
|
|
3$�hÚ/ ã óh — d Z ddlmZ ddlZddlmZ ddlmZ ddlmZm Z m
|
|||
|
|
Z
|
|||
|
|
m Z m Z m
|
|||
|
|
Z
|
|||
|
|
mZmZ ddlmZmZmZmZmZmZmZmZmZmZmZmZmZmZm
|
|||
|
|
Z
|
|||
|
|
m Z m Z m Z m Z m Z m!Z!m"Z"m#Z#m Z m$Z$m%Z%m&Z&m'Z'm(Z(m)Z)m*Z*m+Z+m,Z,m-Z-m.Z.m/Z/m0Z0m1Z1m2Z2mZm3Z3m4Z4m5Z5m6Z6m7Z7m8Z8m9Z9m:Z:m;Z;m<Z<mZm=Z=mZm
|
|||
|
|
Z
|
|||
|
|
m Z m Z m"Z"m-Z-m5Z5mZ ddl5Z5ddlT ddl>m?Z?m@Z@ dd lAmBZB ddlZddlCZ3dd
|
|||
|
|
lDm4Z4 ddlmZ dd lEmFZFmGZH d d
|
|||
|
|
d d
|
|||
|
|
d
|
|||
|
|
dœZI ejJ d d eI¬¦ « d„ ¦ « ZKe? G d„ de¦ « ¦ « ZL G d„ de"¦ « ZM G d„ deM¦ « ZNdS )z8
|
|||
|
|
2025.8.4
|
|||
|
|
2025.8.5
|
|||
|
|
4.55.1
|
|||
|
|
0.21.0
|
|||
|
|
__UNSLOTH_VERSIONING__
|
|||
|
|
é )ÚTensorN)Ú
|
|||
|
|
functional)ÚAnyÚListÚOptionalÚTupleÚUnionÚDictÚSetÚCallable)<r ÚAutoModelForCausalLMÚBaseImageProcessorÚ CPOConfigÚ
|
|||
|
|
CPOTrainerr ÚDPODataCollatorWithPaddingÚ DataCollatorÚ
|
|||
|
|
DataLoaderÚDatasetÚEvalLoopOutputÚFÚFeatureExtractionMixinÚLiteralr Ú PartialStateÚPathÚ PeftModelÚPreTrainedModelÚPreTrainedTokenizerBaseÚProcessorMixinÚTrainerÚTrainerCallbackr Úadd_bos_token_if_neededÚadd_eos_token_if_neededÚautocastÚ defaultdictÚdisable_dropout_in_modelÚgenerate_model_cardÚget_comet_experiment_urlÚinspectÚis_comet_availableÚis_peft_availableÚis_torch_fx_proxyÚis_wandb_availableÚ log_table_to_comet_experimentÚmaybe_apply_chat_templateÚmaybe_extract_promptÚnnÚnpÚ nullcontextÚosÚ
|
|||
|
|
pad_to_lengthÚpdÚpeft_module_casting_to_bf16Ú prepare_model_for_kbit_trainingÚrandomÚselective_log_softmaxÚtextwrapÚtorchÚwarningsr r r r r r* r3 r; )Ú*)Ú dataclassÚfield)ÚVersion)r2 )ÚDataCollatorForSeq2SeqÚ DataCollatorForLanguageModelingTF)Úepilogue_fusionÚ max_autotuneÚ
|
|||
|
|
shape_paddingz
|
|||
|
|
trace.enabledztriton.cudagraphs)ÚdynamicÚ fullgraphÚoptionsc ó’ — t j | d| j d ¦ « dd¬¦ « }t j | d¦ « dd¬¦ « }g }t ||¦ « D ]‘\ }}| t j ¦ « }t j |d| d¦ « ¬¦ « d¦ « }t j
|
|||
|
|
|d¬¦ « }||z
|
|||
|
|
} | | ¦ « Œ’ t j |¦ « }| | j d | j d f¦ « }|S )Néÿÿÿÿé r )ÚchunksÚdim)rM Úindex©rM é )
|
|||
|
|
r; ÚchunkÚreshapeÚshapeÚzipÚtoÚfloat32ÚgatherÚ unsqueezeÚsqueezeÚ logsumexpÚappendÚconcat)
|
|||
|
|
ÚlogitsrN Úchunked_logitsÚ
|
|||
|
|
chunked_indexÚall_per_token_logpsÚ chunk_logitsÚ chunk_indexÚselected_logitsÚlogsumexp_valuesÚper_token_logpss
|
|||
|
|
ú]/workspace/Fine-tuning/DS-LLM-TEMPLATE-FINETUNING/unsloth_compiled_cache/UnslothCPOTrainer.pyÚ chunked_selective_log_softmaxrg " s5 € õ ”[ § ¢ °°F´LÀÔ4DÑ!EÔ!EÐPQÐYZÐ[Ñ[Ô[€NÝ”[ § ¢ ¨rÑ!2Ô!2¸QÀaÐHÑHÔH€MØ Ðå%(¨ ¸ Ñ%GÔ%Gð 4ð 4Ñ!ˆ �kØ#—’¥u¤}Ñ5Ô5ˆ Ý œ, |¸2À{×G\ÒG\Ð]_ÑG`ÔG`ÐaÑaÔa×iÒiÐjlÑmÔmˆÝ œ?¨<¸rÐBÑBÔBÐØ)Ð,<Ñ<ˆØ×"Ò" ?Ñ3Ô3Ð3Ð3ØÝ œ,Ð':Ñ;Ô;ÐØ-×5Ò5°v´|ÀA´ÈÌ ÐUVÌÐ6XÑYÔYÐØ Ð ó c ó² ‡ — e Zd ZU dZ edddi¬¦ « Zee ed< edddi¬¦ « Z ee
|
|||
|
|
ed < d0ˆ fd/„ Z ˆ xZ S )1ÚUnslothCPOConfiguy
|
|||
|
|
|
|||
|
|
Configuration class for the [`CPOTrainer`].
|
|||
|
|
|
|||
|
|
This class includes only the parameters that are specific to CPO training. For a full list of training arguments,
|
|||
|
|
please refer to the [`~transformers.TrainingArguments`] documentation. Note that default values in this class may
|
|||
|
|
differ from those in [`~transformers.TrainingArguments`].
|
|||
|
|
|
|||
|
|
Using [`~transformers.HfArgumentParser`] we can turn this class into
|
|||
|
|
[argparse](https://docs.python.org/3/library/argparse#module-argparse) arguments that can be specified on the
|
|||
|
|
command line.
|
|||
|
|
|
|||
|
|
Parameters:
|
|||
|
|
max_length (`int` or `None`, *optional*, defaults to `1024`):
|
|||
|
|
Maximum length of the sequences (prompt + completion) in the batch. This argument is required if you want
|
|||
|
|
to use the default data collator.
|
|||
|
|
max_prompt_length (`int` or `None`, *optional*, defaults to `512`):
|
|||
|
|
Maximum length of the prompt. This argument is required if you want to use the default data collator.
|
|||
|
|
max_completion_length (`int` or `None`, *optional*, defaults to `None`):
|
|||
|
|
Maximum length of the completion. This argument is required if you want to use the default data collator
|
|||
|
|
and your model is an encoder-decoder.
|
|||
|
|
beta (`float`, *optional*, defaults to `0.1`):
|
|||
|
|
Parameter controlling the deviation from the reference model. Higher β means less deviation from the
|
|||
|
|
reference model. For the IPO loss (`loss_type="ipo"`), β is the regularization parameter denoted by τ in
|
|||
|
|
the [paper](https://huggingface.co/papers/2310.12036).
|
|||
|
|
label_smoothing (`float`, *optional*, defaults to `0.0`):
|
|||
|
|
Label smoothing factor. This argument is required if you want to use the default data collator.
|
|||
|
|
loss_type (`str`, *optional*, defaults to `"sigmoid"`):
|
|||
|
|
Type of loss to use. Possible values are:
|
|||
|
|
|
|||
|
|
- `"sigmoid"`: sigmoid loss from the original [DPO](https://huggingface.co/papers/2305.18290) paper.
|
|||
|
|
- `"hinge"`: hinge loss on the normalized likelihood from the
|
|||
|
|
[SLiC](https://huggingface.co/papers/2305.10425) paper.
|
|||
|
|
- `"ipo"`: IPO loss from the [IPO](https://huggingface.co/papers/2310.12036) paper.
|
|||
|
|
- `"simpo"`: SimPO loss from the [SimPO](https://huggingface.co/papers/2405.14734) paper.
|
|||
|
|
|
|||
|
|
disable_dropout (`bool`, *optional*, defaults to `True`):
|
|||
|
|
Whether to disable dropout in the model.
|
|||
|
|
cpo_alpha (`float`, *optional*, defaults to `1.0`):
|
|||
|
|
Weight of the BC regularizer in CPO training.
|
|||
|
|
simpo_gamma (`float`, *optional*, defaults to `0.5`):
|
|||
|
|
Target reward margin for the SimPO loss, used only when the `loss_type="simpo"`.
|
|||
|
|
label_pad_token_id (`int`, *optional*, defaults to `-100`):
|
|||
|
|
Label pad token id. This argument is required if you want to use the default data collator.
|
|||
|
|
padding_value (`int` or `None`, *optional*, defaults to `None`):
|
|||
|
|
Padding value to use. If `None`, the padding value of the tokenizer is used.
|
|||
|
|
truncation_mode (`str`,*optional*, defaults to `"keep_end"`):
|
|||
|
|
Truncation mode to use when the prompt is too long. Possible values are `"keep_end"` or `"keep_start"`.
|
|||
|
|
This argument is required if you want to use the default data collator.
|
|||
|
|
generate_during_eval (`bool`, *optional*, defaults to `False`):
|
|||
|
|
If `True`, generates and logs completions from the model to W&B or Comet during evaluation.
|
|||
|
|
is_encoder_decoder (`bool` or `None`, *optional*, defaults to `None`):
|
|||
|
|
When using the `model_init` argument (callable) to instantiate the model instead of the `model` argument,
|
|||
|
|
you need to specify if the model returned by the callable is an encoder-decoder model.
|
|||
|
|
model_init_kwargs (`dict[str, Any]` or `None`, *optional*, defaults to `None`):
|
|||
|
|
Keyword arguments to pass to `AutoModelForCausalLM.from_pretrained` when instantiating the model from a
|
|||
|
|
string.
|
|||
|
|
dataset_num_proc (`int` or `None`, *optional*, defaults to `None`):
|
|||
|
|
Number of processes to use for processing the dataset.
|
|||
|
|
|
|||
|
|
NÚhelpzvLLM SamplingParams)ÚdefaultÚmetadataÚvllm_sampling_paramsrJ z8Chunk size to reduce memory usage. -1 is most efficient.Úunsloth_num_chunksFÚnorK é r éú ç-C ëâ6
|
|||
|
|
?ç{®Gáz„?çÍÌÌÌÌÌì?ç+‡ÙÎ÷ï?ç:Œ0âŽyE>ç ð?ç @Úlinearçš™™™™™¹?ÚpassiveÚwarningTÚstepsrP éô éO
|
|||
|
|
ÚO1ÚautoÚ ç Ú
|
|||
|
|
adamw_8bitÚlengthÚ
|
|||
|
|
every_saveÚlasté é é Úsigmoidç à?éœÿÿÿÚkeep_endc“ ó„ •— |dk rt d|› d�¦ « ‚|dk rt d|› d�¦ « ‚|€|#dk r
|
|||
|
|
|$dk rd}d }#|�€!d
|
|||
|
|
d lm}” t |”¦ « d z d ¦ « }� t ¦ « j d�i d
|
|||
|
|
|“d|“d|“d|“d|“d|“d|“d|“d| “d|
|
|||
|
|
“d| “d| “d|
|
|||
|
|
“d|“d|“d |“d |“d |“d |“d |“d!|“d"|“d#|“d$|“d%|“d&|“d'|“d(| “d)| “d*| “d+| “d,| “d-|!“d.|"“d/|#“d0|$“d1|%“d2|&“d3|'“d4|(“d5|)“d6|*“d7|+“d8|,“d9|-“d:|.“d;|/“d<|0“d=|1“d>|2“d?|3“d@|4“dA|5“dB|6“dC|7“dD|8“dE|9“dF|:“dG|;“dH|<“dI|=“dJ|>“dK|?“dL|@“dM|A“dN|B“dO|C“dP|D“dQ|E“dR|F“dS|G“dT|H“dU|I“dV|J“dW|K“dX|L“dY|M“dZ|N“d[|O“d\|P“d]|Q“d^|R“d_|S“d`|T“da|U“db|V“dc|W“dd|X“de|Y“df|Z“dg|[“dh|\“di|]“dj|^“dk|_“dl|`“dm|a“dn|b“do|c“dp|d“dq|e“dr|f“ds|g“dt|h“du|i“dv|j“dw|k“dx|l“dy|m“dz|n“d{|o“d||p“d}|q“d~|r“d|s“d€|t“d�|u“d‚|v“dƒ|w“d„|x“d…|y“d†|z“d‡|{“dˆ||“d‰|}“dŠ|~“d‹|“dŒ|€“d�|�“dŽ|‚“d�|ƒ“d�|„“d‘|…“d’|†“d“|‡“d”|ˆ“d•|‰“d–|Š“d—|‹“d˜|Œ“d™|�“dš|Ž“d›|�“dœ|�“|“¤Ž |‘| _ |’| _ d S )žNgH¯¼šò×z>z Unsloth: Your learning rate of `zi` is too small and less than 1e-7! Consider increasing it, otherwise gradient updates will be close to 0!rP za` is way too larger > 1! Consider decreasing it to 1e-1, otherwise gradient updates will explode!r~ r Ú unsloth_training_checkpointsrp r )Ú cpu_countrq Ú
|
|||
|
|
output_dirÚoverwrite_output_dirÚdo_trainÚdo_evalÚ
|
|||
|
|
do_predictÚ
|
|||
|
|
eval_strategyÚprediction_loss_onlyÚper_device_train_batch_sizeÚper_device_eval_batch_sizeÚper_gpu_train_batch_sizeÚper_gpu_eval_batch_sizeÚgradient_accumulation_stepsÚeval_accumulation_stepsÚ
|
|||
|
|
eval_delayÚtorch_empty_cache_stepsÚ
|
|||
|
|
learning_rateÚ weight_decayÚ
|
|||
|
|
adam_beta1Ú
|
|||
|
|
adam_beta2Ú adam_epsilonÚ
|
|||
|
|
max_grad_normÚnum_train_epochsÚ max_stepsÚlr_scheduler_typeÚ warmup_ratioÚ warmup_stepsÚ log_levelÚlog_level_replicaÚlog_on_each_nodeÚ logging_dirÚlogging_strategyÚlogging_first_stepÚ
|
|||
|
|
logging_stepsÚlogging_nan_inf_filterÚ
|
|||
|
|
save_strategyÚ
|
|||
|
|
save_stepsÚsave_total_limitÚsave_safetensorsÚsave_on_each_nodeÚsave_only_modelÚ'restore_callback_states_from_checkpointÚno_cudaÚuse_cpuÚuse_mps_deviceÚseedÚ data_seedÚ
|
|||
|
|
jit_mode_evalÚuse_ipexÚbf16Úfp16Úfp16_opt_levelÚhalf_precision_backendÚbf16_full_evalÚfp16_full_evalÚtf32Ú
|
|||
|
|
local_rankÚ ddp_backendÚ
|
|||
|
|
tpu_num_coresÚtpu_metrics_debugÚdebugÚdataloader_drop_lastÚ
|
|||
|
|
eval_stepsÚdataloader_num_workersÚdataloader_prefetch_factorÚ
|
|||
|
|
past_indexÚrun_nameÚ disable_tqdmÚremove_unused_columnsÚ label_namesÚload_best_model_at_endÚmetric_for_best_modelÚgreater_is_betterÚignore_data_skipÚfsdpÚfsdp_min_num_paramsÚ fsdp_configÚ"fsdp_transformer_layer_cls_to_wrapÚaccelerator_configÚ deepspeedÚlabel_smoothing_factorÚoptimÚ
|
|||
|
|
optim_argsÚ adafactorÚgroup_by_lengthÚlength_column_nameÚ report_toÚddp_find_unused_parametersÚddp_bucket_cap_mbÚddp_broadcast_buffersÚdataloader_pin_memoryÚ dataloader_persistent_workersÚskip_memory_metricsÚuse_legacy_prediction_loopÚ push_to_hubÚresume_from_checkpointÚ hub_model_idÚ hub_strategyÚ hub_tokenÚhub_private_repoÚhub_always_pushÚ hub_revisionÚgradient_checkpointingÚ gradient_checkpointing_kwargsÚinclude_inputs_for_metricsÚeval_do_concat_batchesÚ fp16_backendÚpush_to_hub_model_idÚpush_to_hub_organizationÚpush_to_hub_tokenÚ
|
|||
|
|
mp_parametersÚauto_find_batch_sizeÚfull_determinismÚ torchdynamoÚ ray_scopeÚ ddp_timeoutÚ
|
|||
|
|
torch_compileÚtorch_compile_backendÚtorch_compile_modeÚinclude_tokens_per_secondÚ include_num_input_tokens_seenÚneftune_noise_alphaÚoptim_target_modulesÚbatch_eval_metricsÚ
|
|||
|
|
eval_on_startÚuse_liger_kernelÚliger_kernel_configÚeval_use_gather_objectÚ average_tokens_across_devicesÚ
|
|||
|
|
max_lengthÚmax_prompt_lengthÚmax_completion_lengthÚbetaÚlabel_smoothingÚ loss_typeÚdisable_dropoutÚ cpo_alphaÚ simpo_gammaÚlabel_pad_token_idÚ
|
|||
|
|
padding_valueÚtruncation_modeÚgenerate_during_evalÚis_encoder_decoderÚmodel_init_kwargsÚdataset_num_proc© ) ÚFloatingPointErrorÚ
|
|||
|
|
OverflowErrorÚmultiprocessingr’ ÚminÚsuperÚ__init__rn ro )–Úselfr“ r” r• r– r— r˜ r™ rš r› rœ r� rž rŸ r r¡ r¢ r£ r¤ r¥ r¦ r§ r¨ r© rª r« r¬ r r® r¯ r° r± r² r³ r´ rµ r¶ r· r¸ r¹ rº r» r¼ r½ r¾ r¿ rÀ rÁ r rà rÄ rÅ rÆ rÇ rÈ rÉ rÊ rË rÌ rÍ rÎ rÏ rÐ rÑ rÒ rÓ rÔ rÕ rÖ r× rØ rÙ rÚ rÛ rÜ rÝ rÞ rß rà rá râ rã rä rå ræ rç rè ré rê rë rì rí rî rï rð rñ rò ró rô rõ rö r÷ rø rù rú rû rü rý rþ rÿ r r r r r r r r r r r
|
|||
|
|
r r r
|
|||
|
|
r r r r r r r r r r r r r r r r r r r r! r" rn ro Úkwargsr’ Ú __class__s– €rf r) zUnslothCPOConfig.__init__z s› ø€ ðl ˜4Ò Ð Õ'9ð ;VÐ]jð ;Vð ;Vð ;Vñ (Wô (Wð "WØ ˜1Ò Ð ¥Mð 3FÐUbð 3Fð 3Fð 3Fñ %Gô %Gð GØ Ð -°7Ò":Ð":¸zÈSÒ?PÐ?PØ7ˆJØ ˆMØ Ð #Ø 1Ð 1Ð 1Ð 1Ð 1Ð 1Ý " 9 9¡;¤;¨q¡=°!Ñ 4Ô 4Ð à�‰ŒÔð P :ð P :ð P :Ø#˜ðP :à#7Ð#7ðP :ð �xðP :ð �gð P :ð
|
|||
|
|
$˜ð P :ð *˜Mð
|
|||
|
|
P :ð $8Ð#7ðP :ð +FÐ*EðP :ð *DÐ)CðP :ð (@Ð'?ðP :ð '>Ð&=ðP :ð +FÐ*EðP :ð '>Ð&=ðP :ð $˜ð P :ð '>Ð&=ð P :ð *˜Mð!P :ð" (˜<ð#P :ð$ $˜ð%P :ð& $˜ð'P :ð( (˜<ð)P :ð* *˜Mð+P :ð, 0Ð /ð-P :ð. "˜ ð/P :ð0 !2Ð 1ð1P :ð2 (˜<ð3P :ð4 (˜<ð5P :ð6 "˜ ð7P :ð8 !2Ð 1ð9P :ð: 0Ð /ð;P :ð< &˜+ð=P :ð> 0Ð /ð?P :ð@ "4Ð!3ðAP :ðB *˜MðCP :ðD &<Ð%;ðEP :ðF *˜MðGP :ðH $˜ðIP :ðJ 0Ð /ðKP :ðL 0Ð /ðMP :ðN !2Ð 1ðOP :ðP .˜oðQP :ðR 7^Ð6]ðSP :ðT �gðUP :ðV �gðWP :ðX ,˜^ðYP :ðZ �4ð[P :ð\ "˜ ð]P :ð^ *˜Mð_P :ð` �xðaP :ðb �4ðcP :ðd �4ðeP :ðf ,˜^ðgP :ðh &<Ð%;ðiP :ðj ,˜^ðkP :ðl ,˜^ðmP :ðn �4ðoP :ðp $˜ðqP :ðr &˜+ðsP :ðt *˜MðuP :ðv !2Ð 1ðwP :ðx �EðyP :ðz $8Ð#7ð{P :ð| $˜ð}P :ð~ &<Ð%;ðP :ð@ *DÐ)CðAP :ðB $˜ðCP :ðD �xðEP :ðF (˜<ðGP :ðH %:Ð$9ðIP :ðJ &˜+ðKP :ðL &<Ð%;ðMP :ðN %:Ð$9ðOP :ðP !2Ð 1ðQP :ðR 0Ð /ðSP :ðT �4ðUP :ðV #6Ð"5ðWP :ðX &˜+ðYP :ðZ 2TÐ1Sð[P :ð\ "4Ð!3ð]P :ð^ "˜ ð_P :ð` &<Ð%;ðaP :ðb �EðcP :ðd $˜ðeP :ðf "˜ ðgP :ðh .˜oðiP :ðj "4Ð!3ðkP :ðl "˜ ðmP :ðn *DÐ)CðoP :ðp !2Ð 1ðqP :ðr %:Ð$9ðsP :ðt %:Ð$9ðuP :ðv -JÐ,IðwP :ðx #6Ð"5ðyP :ðz *DÐ)Cð{P :ð| &˜+ð}P :ð~ &<Ð%;ðP :ð@ (˜<ðAP :ðB (˜<ðCP :ðD "˜ ðEP :ðF 0Ð /ðGP :ðH .˜oðIP :ðJ (˜<ðKP :ðL &<Ð%;ðMP :ðN -JÐ,IðOP :ðP *DÐ)CðQP :ðR &<Ð%;ðSP :ðT (˜<ðUP :ðV $8Ð#7ðWP :ðX (@Ð'?ðYP :ðZ !2Ð 1ð[P :ð\ *˜Mð]P :ð^ $8Ð#7ð_P :ð` 0Ð /ðaP :ðb &˜+ðcP :ðd "˜ ðeP :ðf &˜+ðgP :ðh *˜MðiP :ðj %:Ð$9ðkP :ðl "4Ð!3ðmP :ðn )BÐ(AðoP :ðp -JÐ,IðqP :ðr #6Ð"5ðsP :ðt $8Ð#7ðuP :ðv "4Ð!3ðwP :ðx *˜MðyP :ðz 0Ð /ð{P :ð| #6Ð"5ð}P :ð~ &<Ð%;ðP :ð@ -JÐ,IðAP :ðB $˜ðCP :ðD !2Ð 1ðEP :ðF %:Ð$9ðGP :ðH �4ðIP :ðJ .˜oðKP :ðL "˜ ðMP :ðN .˜oðOP :ðP "˜ ðQP :ðR &˜+ðSP :ðT "4Ð!3ðUP :ðV *˜MðWP :ðX .˜oðYP :ðZ $8Ð#7ð[P :ð\ "4Ð!3ð]P :ð^ !2Ð 1ð_P :ð` 0Ð /°&ðaP :ð P :ð P :ðb %9ˆÔ!Ø"4ˆÔ Ð Ð rh )’NNFFFrp FrK rK NNrq rq r rr rs rt ru rv rw rx ry rJ rz r{ r r| r} TNr~ FrP Fr~ r NTFFFFFFr€ r€ FFFFr� r‚ FFNrJ NNFrƒ FNr NrJ NNTNFNNFrƒ r NNNNr„ r… NFFr† NNNNTFTFFNNr‡ NNFNFNFTr‚ NNNrƒ TFNrˆ r‰ FNNFFNNFFFNFTrŠ r‹ Nr{ r„ rŒ Trx r� rŽ Nr� FNNNNrJ )
|
|||
|
|
Ú__name__Ú
|
|||
|
|
__module__Ú __qualname__Ú__doc__r? rn r r Ú__annotations__ro Úintr) Ú
|
|||
|
|
__classcell__©r, s @rf rj rj 3 s� ø€ € € € € € ð<ð <ðz +0¨%ØØÐ 1Ð2ð+ñ +ô +И( 3œ-ð ð ñ ð */¨ØØÐ VÐWð*ñ *ô *И #œ ð ð ñ ð Ø #ØØØØ Ø $Ø&'Ø%&Ø#'Ø"&Ø&'Ø"#ØØ"%Ø ØØØØ ØØ ØØ $ØØØ Ø %Ø ØØ"Ø "ØØ!&Ø ØØ Ø Ø !Ø Ø27ØØØ ØØØ ØØØØ Ø!'Ø Ø ØØØØ Ø !ØØ $ØØ!"Ø%)ØØØØ $ØØ!&Ø $Ø Ø ØØ ØØ-1Ø !ØØ!$Ø ØØØ Ø %ØØ%)Ø Ø $Ø $Ø(-Ø "Ø%*ØØ!%ØØ#ØØ Ø ØØ!&Ø(,Ø%*Ø!%Ø Ø #Ø#'Ø ØØ #Ø ØØØØ Ø $Ø !Ø$)Ø(-Ø "Ø #Ø "Ø Ø Ø "Ø!&Ø(,ØØ Ø $ØØ Ø Ø ØØØ !Ø Ø$Ø $Ø !Ø Ø Ø #Ø ðgq5ð q5ð q5ð q5ð q5ð q5ð q5ð q5ð q5ð q5rh rj c ó¢ ‡ — e Zd ZdZddgZ dAdeeeej e
|
|||
|
|
f dee dee d ee
|
|||
|
|
d
|
|||
|
|
eee
|
|||
|
|
ee
|
|||
|
|
e
|
|||
|
|
f f d eeeeeef d eeg ef d
|
|||
|
|
eee deej j ej j j f deeej ej gej f dee deee gef fˆ fd„
|
|||
|
|
Z d„ Z dBdeeeej f defd„Z e! dCdee
|
|||
|
|
eeej" f f de#de$d e$d eej% dee
|
|||
|
|
ej" f f d „¦ « Z&d ej' d ej' deej' ej' ej' f fd!„Z(e! dDd"ej' d#ej" d$e#de$de#dej' f d%„¦ « Z)dej dee
|
|||
|
|
eeej" f f deej' ej' ej' ej' f fd&„Z* dEdee
|
|||
|
|
eeej" f f d(e+d) fd*„Z, dFdeeej f d+ee
|
|||
|
|
eej e-f f deej eej ee
|
|||
|
|
ej f f f fd,„Z.dee
|
|||
|
|
ej" f de
|
|||
|
|
fd-„Z/ dBdeeej f d+ee
|
|||
|
|
eej e-f f d.e#d/eee
|
|||
|
|
fd0„Z0dEd1ee
|
|||
|
|
e1f d(e+d) ddfd2„Z2 dGd4e3d5e
|
|||
|
|
d.ee# d/eee
|
|||
|
|
d6e
|
|||
|
|
de f ˆ fd7„
|
|||
|
|
Z4dBd8ee
|
|||
|
|
e1f d9ee1 ddfˆ fd:„
|
|||
|
|
Z5d;„ Z6ˆ fd<„Z7 dHd=ee
|
|||
|
|
d>ee
|
|||
|
|
d?ee
|
|||
|
|
ee
|
|||
|
|
df fd@„Z8ˆ xZ9S )IÚ_UnslothCPOTrainerrƒ ÚtrlÚcpoN©NNÚmodelÚargsÚ
|
|||
|
|
data_collatorÚ
|
|||
|
|
train_datasetÚ eval_datasetÚprocessing_classÚ
|
|||
|
|
model_initÚ callbacksÚ
|
|||
|
|
optimizersÚ preprocess_logits_for_metricsÚ peft_configÚcompute_metricsc
|
|||
|
|
|
|||
|
|
ó¬ •— |j €i }
|
|||
|
|
nªt |t ¦ « st d¦ « ‚|j }
|
|||
|
|
|
|
|||
|
|
d¦ « }|�ht |t ¦ « r|dk rt t |¦ « }|dk r-t |t j ¦ « st d|› d�¦ « ‚||
|
|||
|
|
d< t |t ¦ « rt j |fi |
|
|||
|
|
¤Ž}d| _
|
|||
|
|
t ¦ « s| �t d¦ « ‚t ¦ « �r5| ��2t |t ¦ « r|
|
|||
|
|
¦ « }t |dd¦ « st |d d¦ « r`t |d
|
|||
|
|
¦ « o,d
|
|||
|
|
t t! j t$ ¦ « j ¦ « v }d |j i}|r
|
|||
|
|
|j |d
|
|||
|
|
< t% |fi |¤Ž}nV|j rOt |d ¦ « r| ¦ « n*d
|
|||
|
|
„ }| ¦ « |¦ « |}|j r't |d d¦ « rt5 |¦ « d| _
|
|||
|
|
nV|j rOt |d ¦ « r| ¦ « n*d„ }| ¦ « |¦ « |j r+t9 ¦ « s t; ¦ « st d¦ « ‚|�|j j | _ n"|j €t d¦ « ‚|j | _ | j r"|j j | _ |j j! | _! |€t d¦ « ‚|j" € tG j$ dtJ ¦ « d}n|j" }|j& € tG j$ dtJ ¦ « d}n|j& }||k st d|› d|› d�¦ « ‚|j' €$| j r tG j$ dtJ ¦ « d}n|j' }|€QtQ |j! |j) | j ¬¦ « }|j* r!d|_* tG j$ d tJ ¦ « d| _+ nd| _+ |j, rt[ |¦ « || _" |j | _ |j) | _) |j. �|j. n|j! | _. || _& |j/ | _/ || _' || _0 |j1 d v r.|j2 d k r#tG j$ d |j1 › d �tJ ¦ « |j1 d!k rt d"¦ « ‚|j3 | _3 |j2 | _2 |j1 | _1 |j4 | _4 t |j d#d¦ « | _5 t |j d$d%¦ « | _6 | j5 r%| j6 d%k rtG j$ d&tJ ¦ « |j1 d'k r |j7 | _7 tq d(„ ¦ « | _9 d|j: d)< tw ¦ « < ¦ « 5 | = t| |j? ¬*¦ « }| = t€ d+|i|j? ¬,¦ « }|�E| = t| |j? ¬*¦ « }| = t€ d+|i|j? ¬,¦ « }| = | jA |j? ¬*¦ « }|�!| = | jA |j? ¬*¦ « }d d d ¦ « n # 1 swxY w Y t… ¦ « C |||||||| || |
|
|||
|
|
¬-¦ « d| _D t | jE d.¦ « r | jE F | jG ¦ « t | d/¦ « st‘ d0¦ « ‚d S )1NzRYou passed model_kwargs to the CPOTrainer. But your model is already instantiated.Ú torch_dtyper‚ znInvalid `torch_dtype` passed to the CPOConfig. Expected a string with either `torch.dtype` or 'auto', but got ú.FzvPEFT is not installed and you passed a `peft_config` in the trainer's kwargs, please install it to use the PEFT modelsÚis_loaded_in_8bitÚis_loaded_in_4bitrù Úuse_gradient_checkpointingÚenable_input_require_gradsc ó0 — | d¦ « d S ©NT©Úrequires_grad_©ÚmoduleÚinputÚoutputs rf Úmake_inputs_require_gradz=_UnslothCPOTrainer.__init__.<locals>.make_inputs_require_gradø s € Ø ×-Ò-¨dÑ3Ô3Ð3Ð3Ð3rh Tc ó0 — | d¦ « d S rN rO rQ s rf rU z=_UnslothCPOTrainer.__init__.<locals>.make_inputs_require_grad
|
|||
|
|
s € Ø×)Ò)¨$Ñ/Ô/Ð/Ð/Ð/rh z‚`generate_during_eval=True` requires Weights and Biases or Comet to be installed. Please install `wandb` or `comet-ml` to resolve.zMWhen no model is provided, you need to pass the parameter is_encoder_decoder.z=processing_class must be specified to tokenize a CPO dataset.z�`max_length` is not set in the CPOConfig's init it will default to `512` by default, but you should do it yourself in the future.r‹ zˆ`max_prompt_length` is not set in the CPOConfig's init it will default to `128` by default, but you should do it yourself in the future.é€ zmax_prompt_length (z+) should be strictly less than max_length (z).z¼When using an encoder decoder architecture, you should set `max_completion_length` in the CPOConfig's init it will default to `128` by default, but you should do it yourself in the future.)Ú pad_token_idr r z²When using DPODataCollatorWithPadding, you should set `remove_unused_columns=False` in your TrainingArguments we have set it for you, but you should do it yourself in the future.)ÚhingeÚipor zYou are using the z™ loss type that does not support label smoothing. The `label_smoothing` parameter will be ignored. Set `label_smoothing` to `0.0` to remove this warning.Úkto_pairzKSupport for kto_pair has been removed in CPOTrainer. Please use KTOTrainer.Úoutput_router_logitsÚrouter_aux_loss_coefr„ a- You set `output_router_logits` to `True` in the model config, but `router_aux_loss_coef` is set to `0.0`, meaning the auxiliary loss will not be used. Either set `router_aux_loss_coef` to a value greater than `0.0`, or set `output_router_logits` to `False` if you don't want to use the auxiliary loss.Úsimpoc ó* — t t ¦ « S ©N)r$ Úlistr# rh rf ú<lambda>z-_UnslothCPOTrainer.__init__.<locals>.<lambda>ƒ s € µ;½tÑ3DÔ3D€ rh Úestimate_tokens)Únum_procÚ tokenizer)Ú fn_kwargsrd ) r: r; r< r= r> r? r@ rE rA rB rC Úadd_model_tagsÚ acceleratorzXYour `Trainer` does not have an `accelerator` object. Consider upgrading `transformers`.)Ir! Ú
|
|||
|
|
isinstanceÚstrÚ
|
|||
|
|
ValueErrorÚgetÚgetattrr; Údtyper
|
|||
|
|
Úfrom_pretrainedÚ _peft_has_been_casted_to_bf16r* r Úmerge_and_unloadÚhasattrra r( Ú signaturer7 Ú
|
|||
|
|
parametersrø rù rL Úget_input_embeddingsÚregister_forward_hookrà r6 r r, r) Úconfigr Údecoder_start_token_idrX r r< ÚwarnÚ UserWarningr r r r rÖ Úuse_dpo_data_collatorr r% r r r? r r r r Úaux_loss_enabledÚ
|
|||
|
|
aux_loss_coefr r$ Ú_stored_metricsÚwarnings_issuedr Úmain_process_firstÚmapr/ r" r. Ú tokenize_rowr( r) Úmodel_accepts_loss_kwargsr: rg Ú
|
|||
|
|
_tag_namesÚAttributeError)r* r: r; r< r= r> r? r@ rA rB rC rD rE r! rG Ú_support_gc_kwargsÚprepare_model_kwargsrU r r r r, s €rf r) z_UnslothCPOTrainer.__init__³ s÷ ø€ ð" Ô !Ð )Ø "РРݘE¥3Ñ'Ô'ð
|
|||
|
|
?Ý Ð qÑrÔrÐ rà $Ô 6Ð Ø+×/Ò/°
|
|||
|
|
Ñ>Ô>ˆKØÐ&å ˜k3Ñ/Ô/ð >°KÀ6Ò4IÐ4IÝ")%°Ñ"=Ô"=�KØ &Ò(Ð(µ¸KÍÌÑ1UÔ1UÐ(Ý$ð Xð JUð Xð Xð Xñô ð ð 4?Ð! -Ñ0å �e�SÑ !Ô !ð UÝ(Ô8¸ÐTÐTÐBSÐTÐTˆEð .3ˆÔ*å Ñ"Ô"ð 4 ] {Ð'>Ý ð Iñô ð
|
|||
|
|
õ Ñ
|
|||
|
|
Ô
|
|||
|
|
ñ 0 ] [Ñ%<å˜%¥Ñ+Ô+ð
|
|||
|
|
1Ø ×.Ò.Ñ0Ô0�å�uÐ 1°5Ñ9Ô9ð
|
|||
|
|
a½WÀUÐL_ÐafÑ=gÔ=gð
|
|||
|
|
aÝ%,ØÐ9ñ&ô &ð &à5½ÝÔ%Õ&EÑFÔFÔQñ:ô :ð ð #ð )EÀdÔFaÐ'bÐ$à%ð oØLPÔLnÐ(Ð)HÑIå7¸ÐVÐVÐAUÐVÐV��ØÔ,ð
|
|||
|
|
aå˜5Ð">Ñ?Ô?ð aØ×4Ò4Ñ6Ô6Ð6Ð6ð4ð 4ð 4ð ×.Ò.Ñ0Ô0×FÒFÐG_Ñ`Ô`Ð`ð ˆEØŒyð
|
|||
|
|
:�W UÐ,?ÀÑ GÔ Gð
|
|||
|
|
:Ý+¨EÑ2Ô2Ð2à59�Ô2øð
|
|||
|
|
Ô
|
|||
|
|
(ð ]å�uÐ :Ñ;Ô;ð
|
|||
|
|
]Ø×0Ò0Ñ2Ô2Ð2Ð2ð0ð 0ð 0ð ×*Ò*Ñ,Ô,×BÒBÐC[Ñ\Ô\Ð\à Ô $ð Õ.@Ñ.BÔ.Bð ÕFXÑFZÔFZð Ý ðDñô ð
|
|||
|
|
ð
|
|||
|
|
Ð Ø&+¤lÔ&EˆDÔ #Ð #Ø
|
|||
|
|
Ô
|
|||
|
|
$Ð
|
|||
|
|
,Ý Ð lÑmÔmÐ mà&*Ô&=ˆDÔ #à Ô "ð :Ø*/¬,Ô*MˆDÔ 'Ø %¤ Ô 9ˆDÔ à Ð #Ý Ð \Ñ]Ô]Ð ]Ø Œ?Ð "Ý ŒMðeåñ
|
|||
|
|
ô
|
|||
|
|
ð
|
|||
|
|
ð
|
|||
|
|
ˆJˆJà œ ˆJØ Ô !Ð )Ý ŒMðeåñ
|
|||
|
|
ô
|
|||
|
|
ð
|
|||
|
|
ð
|
|||
|
|
!$Ð Ð à $Ô 6Ð à :Ò-Ð-Ý ØrÐ&7ÐrÐrÐdnÐrÐrÐrñô ð
|
|||
|
|
ð Ô %Ð -°$Ô2IÐ -Ý ŒMðeåñ
|
|||
|
|
ô
|
|||
|
|
ð
|
|||
|
|
ð
|
|||
|
|
%(Ð !Ð !à$(Ô$>Ð !à Ð Ý 6Ø -Ô :Ø#'Ô#:Ø#'Ô#:ð ñ ô ˆMð Ô)ð
|
|||
|
|
Ø-2�Ô*å”
|
|||
|
|
ð\å ñô ð ð *.ˆDÔ &Ð &à).ˆDÔ &ð Ô ð ,Ý $ UÑ +Ô +Ð +à$ˆŒØ$(Ô$=ˆÔ!Ø"&Ô"9ˆÔ Ø37Ô3EÐ3Q˜TÔ /Ð /ÐWgÔWtˆÔØ!2ˆÔ Ø #Ô 3ˆÔ Ø%:ˆÔ"Ø 0ˆÔ à Œ>Ð -Ð -Ð -°$Ô2FÈÒ2JÐ2JÝ ŒMðv T¤^ð vð vð våñ
|
|||
|
|
ô
|
|||
|
|
ð
|
|||
|
|
ð
|
|||
|
|
Œ>˜ZÒ 'Ð 'Ý Ð jÑkÔkÐ kà”IˆŒ Ø #Ô 3ˆÔ Ø œ ˆŒØ œ ˆŒÝ '¨¬ Ð6LÈeÑ TÔ TˆÔ Ý $ U¤\Ð3IÈ3Ñ OÔ OˆÔØ Ô ð TÔ%7¸3Ò%>Ð%>Ý ŒMðõ ñ
|
|||
|
|
ô
|
|||
|
|
ð
|
|||
|
|
ð Œ>˜WÒ $Ð $Ø #Ô /ˆDÔ å *Ð+DÐ+DÑ EÔ EˆÔ ð 48ˆÔ Ð /Ñ0õ ‰^Œ^×
|
|||
|
|
.Ò
|
|||
|
|
.Ñ
|
|||
|
|
0Ô
|
|||
|
|
0ð cð cà )× -Ò -Õ.BÈTÔMbÐ -Ñ cÔ cˆMØ )× -Ò -Ý)°kÐCSÐ5TÐ_cÔ_tð .ñ ô ˆMð Ð'Ø +× /Ò /Õ0DÈtÔOdÐ /Ñ eÔ e� Ø +× /Ò /Ý-Ø *Ð,<Ð =Ø !Ô 2ð 0ñ ô � ð *× -Ò -¨dÔ.?È$ÔJ_Ð -Ñ `Ô `ˆMØÐ'Ø +× /Ò /°Ô0AÈDÔLaÐ /Ñ bÔ b� ð# cð cð cñ cô cð cð cð cð cð cð cøøøð cð cð cð cõ& ‰Œ×ÒØØØ'Ø'Ø%Ø -Ø!Ø +Ø Ø!Ø*Gð ñ
|
|||
|
|
ô
|
|||
|
|
ð
|
|||
|
|
ð" */ˆÔ&õ �4”:Ð /Ñ 0Ô 0ð 7Ø ŒJ× %Ò % d¤oÑ 6Ô 6Ð 6å�t˜]Ñ+Ô+ð Ý Øjñô ð
|
|||
|
|
ð ð s ×&C[Û[Û
|
|||
|
|
[c óö — | ||z d¬¦ « }| |d¬¦ « d }|d t |¦ « d… }|d t |¦ « d… }t j ||g¦ « }t j |d ¦ « }t |¦ « t |¦ « k rt d¦ « ‚t |¦ « } ||d d| … k r| dz } |d d| … }|d d| … }
|
|||
|
|
t |¦ « t |
|
|||
|
|
¦ « k rt d¦ « ‚|d | d… }|d | d… }t
|
|||
|
|
||
|
|||
|
|
||¬ ¦ « S )
|
|||
|
|
a
|
|||
|
|
Llama tokenizer does satisfy `enc(a + b) = enc(a) + enc(b)`. It does ensure `enc(a + b) = enc(a) + enc(a +
|
|||
|
|
b)[len(enc(a)):]`. Reference:
|
|||
|
|
https://github.com/EleutherAI/lm-evaluation-harness/pull/531#issuecomment-1595586257
|
|||
|
|
F©Úadd_special_tokensÚ input_idsNÚattention_maskzBPrompt input ids and answer input ids should have the same length.rP z@Prompt input ids and attention mask should have the same length.)Úprompt_input_idsÚprompt_attention_maskr‹ rŒ )r? Úlenr1 Ú concatenateÚarrayrk Údict) r* ÚpromptÚanswerÚfull_tokenizedr� Úanswer_input_idsÚanswer_attention_maskÚfull_concat_input_idsÚfull_input_idsÚ response_token_ids_start_idxrŽ s rf Úbuild_tokenized_answerz)_UnslothCPOTrainer.build_tokenized_answer¿ sÀ € ð ×.Ò.¨v¸©ÐSXÐ.ÑYÔYˆØ ×0Ò0°ÈEÐ0ÑRÔRÐS^Ô_Ðà)¨+Ô6µsÐ;KÑ7LÔ7LÐ7NÐ7NÔOÐØ .Ð/?Ô @ÅÐEUÑAVÔAVÐAXÐAXÔ YÐ õ !#¤Ð0@ÐBRÐ/SÑ TÔ TÐ õ œ .°Ô"=Ñ>Ô>ˆå ˆ~Ñ Ô ¥#Ð&;Ñ"<Ô"<Ò <Ð <Ý Ð aÑbÔbÐ bõ (+Ð+;Ñ'<Ô'<Ð$ð ˜~¨kÔ :Ð;XÐ<XÐ;XÔ YÒ YÐ YØ (¨AÑ -Ð (à)¨+Ô6Ð7TÐ8TÐ7TÔUÐØ .Ð/?Ô @ÐA^ÐB^ÐA^Ô _Ð å Ð Ñ Ô ¥CÐ(=Ñ$>Ô$>Ò >Ð >Ý Ð _Ñ`Ô`Ð `à)¨+Ô6Ð7SÐ7TÐ7TÔUÐØ .Ð/?Ô @ÐA]ÐA^ÐA^Ô _Ð åØ -Ø"7Ø&Ø0ð
|
|||
|
|
ñ
|
|||
|
|
ô
|
|||
|
|
ð
|
|||
|
|
rh Úreturnc óœ
|
|||
|
|
‡‡— i }|d }|d }|d }| j �s-t |t ¦ « s t dt |¦ « › �¦ « ‚| |d¬¦ « }d„ | ¦ « D ¦ « }t |t ¦ « s t dt |¦ « › �¦ « ‚| ||¦ « Št |t ¦ « s t d t |¦ « › �¦ « ‚| ||¦ « Št |d
|
|||
|
|
¦ « }t ‰d
|
|||
|
|
¦ « } t ‰d
|
|||
|
|
¦ « }
|
|||
|
|
t | |
|
|||
|
|
¦ « }| ¦ « D ]\ } } | d |… || < Œt d „ t ‰d
|
|||
|
|
‰d
|
|||
|
|
¦ « D ¦ « ¦ « }
|
|||
|
|
t | |
|
|||
|
|
z
|
|||
|
|
¦ « }|
|
|||
|
|
d
|
|||
|
|
k s|d
|
|||
|
|
k rt d¦ « ‚t | j j ||| ‰|
|
|||
|
|
‰¦ « \ }ŠŠt | j j ‰‰¦ « \ ŠŠt# t ‰d ¦ « t ‰d ¦ « ¦ « }‰‰|fD ]�}t |d
|
|||
|
|
¦ « |z | j k rj| j dk r dD ]} || d | j … || < ŒŒL| j dk r dD ]} || | j d … || < Œ Œvt d| j › �¦ « ‚ŒŽ‰‰fD ]H}t |d
|
|||
|
|
¦ « |z | j k r%dD ]"} || d | j | j z
|
|||
|
|
… || < Œ#ŒIˆfd„dD ¦ « }ˆfd„dD ¦ « }|d d d … |d< | j gt ‰d
|
|||
|
|
¦ « z |d d t ‰d
|
|||
|
|
¦ « …< |d d d … |d< | j gt ‰d
|
|||
|
|
¦ « z |d d t ‰d
|
|||
|
|
¦ « …< |||dœ ¦ « D ]/\ } }| ¦ « D ]\ }}|dk rŒ ||| › |› �< ŒŒ0nú| |d| j d¬¦ « Š| |d| j d¬¦ « Š| |d| j d¬¦ « }‰d |d < ‰d |d < |d |d
|
|||
|
|
< |d |d < |�rt/ |d ¦ « rb| t3 j |d ¦ « ¬!¦ « |d"< | t3 j |d ¦ « ¬!¦ « |d#< |S )$a. Tokenize a single row from a CPO specific dataset.
|
|||
|
|
|
|||
|
|
At this stage, we don't convert to PyTorch tensors yet; we just handle the truncation in case the prompt +
|
|||
|
|
chosen or prompt + rejected responses is/are too long. First we truncate the prompt; if we're still too long,
|
|||
|
|
we truncate the chosen/rejected.
|
|||
|
|
|
|||
|
|
We also create the labels for the chosen/rejected responses, which are of length equal to the sum of the length
|
|||
|
|
of the prompt and the chosen/rejected response, with label_pad_token_id for the prompt tokens.
|
|||
|
|
r“ ÚchosenÚrejectedz prompt should be an str but got Fr‰ c ó — i | ] \ }}d |› �|“Œ S ©Úprompt_r# )Ú.0ÚkÚvs rf ú
|
|||
|
|
<dictcomp>z3_UnslothCPOTrainer.tokenize_row.<locals>.<dictcomp> s$ € Ð PÐ PÐ P±$°!°Q˜] q˜]˜]¨AÐ PÐ PÐ Prh z chosen should be an str but got z"rejected should be an str but got r� Nc ó — g | ] \ }}||k ‘Œ S r# r# )r£ ÚaÚbs rf ú
|
|||
|
|
<listcomp>z3_UnslothCPOTrainer.tokenize_row.<locals>.<listcomp> s € ÐpÐpÐp™D˜A˜q��a’ÐpÐpÐprh rP zdChosen and rejected prompt_input_ids might only differ on the last token due to tokenizer merge ops.r‹ Ú
|
|||
|
|
keep_start)r� rŽ r� zUnknown truncation mode: )r‹ rŒ c ó: •— i | ]}|‰d |› � ‰| z “ŒS r¡ r# )r£ r¤ Ú
|
|||
|
|
chosen_tokenss €rf r¦ z3_UnslothCPOTrainer.tokenize_row.<locals>.<dictcomp>N s= ø€ ð &ð &ð &ØGH��= ¨1 Ô/°-ÀÔ2BÑBð&ð &ð &rh c ó: •— i | ]}|‰d |› � ‰| z “ŒS r¡ r# )r£ r¤ Úrejected_tokenss €rf r¦ z3_UnslothCPOTrainer.tokenize_row.<locals>.<dictcomp>Q s= ø€ ð (ð (ð (ØKL��? =¨Q = =Ô1°OÀAÔ4FÑFð(ð (ð (rh Úlabels)Úchosen_Ú rejected_rƒ Útoken_type_idsT)Ú
|
|||
|
|
truncationr rŠ Ú
|
|||
|
|
chosen_labelsÚrejected_labelsrŒ rŽ Ú%prepare_decoder_input_ids_from_labels)r° Úrejected_decoder_input_idsÚchosen_decoder_input_ids)r ri rj rk Útyper? Úitemsr› r� r' ÚsumrT Úabsr! Ú bos_token_idr" Ú eos_token_idÚmaxr r r r r rr r· r; Útensor)r* Úfeaturer: Úbatchr“ rž rŸ Ú
|
|||
|
|
prompt_tokensÚprompt_len_input_idsÚchosen_prompt_len_input_idsÚ rejected_prompt_len_input_idsr¤ r¥ Únum_diff_tokensÚ num_diff_lenÚlonger_response_lengthÚ
|
|||
|
|
answer_tokensÚchosen_sequence_tokensÚrejected_sequence_tokensÚtoksÚtype_keyÚtokensr r¯ s @@rf r‚ z _UnslothCPOTrainer.tokenize_rowð sê øø€ ð ˆØ˜Ô"ˆØ˜Ô"ˆØ˜:Ô&ˆàÔ&ñ ~ õ ˜f¥cÑ*Ô*ð
|
|||
|
|
TÝ Ð!RÅDÈÁLÄLÐ!RÐ!RÑSÔSÐSØ × 1Ò 1°&ÈUÐ 1Ñ SÔ SˆMØ PÐ P¸-×:MÒ:MÑ:OÔ:OÐ PÑ PÔ PˆMå ˜f¥cÑ*Ô*ð
|
|||
|
|
TÝ Ð!RÅDÈÁLÄLÐ!RÐ!RÑSÔSÐSØ × 7Ò 7¸ÀÑ GÔ GˆMå ˜hÑ,Ô,ð
|
|||
|
|
XÝ Ð!VÅdÈ8ÁnÄnÐ!VÐ!VÑWÔWÐWØ "× 9Ò 9¸&À(Ñ KÔ KˆOõ $' }Ð5GÔ'HÑ#IÔ#IÐ å*-¨mÐ<NÔ.OÑ*PÔ*PÐ 'Ý,/°Ð@RÔ0SÑ,TÔ,TÐ )Ý#&Ð'BÐDaÑ#bÔ#bÐ à%×+Ò+Ñ-Ô-ð
|
|||
|
|
<ð
|
|||
|
|
<‘��1Ø#$Ð%:Ð&:Ð%:Ô#;�
|
|||
|
|
˜aÑ Ð õ "ØpÐp¥C¨
|
|||
|
|
Ð6HÔ(IÈ?Ð[mÔKnÑ$oÔ$oÐpÑpÔpñ ô ˆOõ Ð :Ð=ZÑ ZÑ[Ô[ˆLØ Ò"Ð" l°QÒ&6Ð&6Ý ð=ñô ð õ =TØÔ%Ô2Ø$Ø Ø+Ø Ø-Ø ñ=ô =Ñ 9ˆM˜=¨/õ .EØÔ%Ô2°MÀ?ñ.ô .Ñ *ˆM˜?õ &)¨]¸;Ô-GÑ)HÔ)HÍ#ÈoÐ^iÔNjÑJkÔJkÑ%lÔ%lÐ "ð #0° À-Ð!Pð
|
|||
|
|
]ð
|
|||
|
|
]�
|
|||
|
|
Ý�}Ð%7Ô8Ñ9Ô9Ð<RÑRÐUYÔUdÒdÐdØÔ+¨|Ò;Ð;Ø!Nð Zð Z˜AØ/<¸QÔ/?Ð@XÀ$ÔBXÐ@XÔ/Y˜M¨!Ñ ,Ð ,ðZà Ô-°Ò;Ð;Ø!Nð [ð [˜AØ/<¸QÔ/?ÀÔAWÐ@WÐ@YÐ@YÔ/Z˜M¨!Ñ ,Ð ,ð[õ )Ð)[ÀTÔEYÐ)[Ð)[Ñ \Ô \Ð\ð eð #0° Ð!Að
|
|||
|
|
hð
|
|||
|
|
h�
|
|||
|
|
Ý�}Ð%7Ô8Ñ9Ô9Ð<RÑRÐUYÔUdÒdÐdØ <ð hð h˜Ø+8¸Ô+;Ð<f¸d¼oÐPTÔPfÑ>fÐ<fÔ+g˜
|
|||
|
|
aÑ(Ð(øð&ð &ð &ð &ØLkð&ñ &ô &Ð "ð(ð (ð (ð (ØPoð(ñ (ô (Ð $ð 0FÀkÔ/RÐSTÐSTÐSTÔ/UÐ " 8Ñ ,àÔ'ðZå�MÐ"4Ô5Ñ6Ô6ñZ7Ð " 8Ô ,Ð-Us°=ÐASÔ3TÑ/UÔ/UÐ-UÑ Vð 2JÈ+Ô1VÐWXÐWXÐWXÔ1YÐ $ XÑ .àÔ'ð^å�OÐ$6Ô7Ñ8Ô8ñ^9Ð $ XÔ .Ð/Yµ°_ÐEWÔ5XÑ1YÔ1YÐ/YÑ Zð
|
|||
|
|
2Ø 5Ø!ð ð ÷ Še‰gŒgð
|
|||
|
|
5ð
|
|||
|
|
5‘��4ð
|
|||
|
|
)-¯
|
|||
|
|
ª
|
|||
|
|
© ¬ ð 5ð 5Ñ$�H˜fØ Ð#3Ò3Ð3Ø Ø.4�E˜QÐ* Ð*Ð*Ñ+Ð+ð5ð
|
|||
|
|
5ð !× 1Ò 1Ø 4°DÔ4NÐcgð 2ñ ô ˆMð #× 3Ò 3Ø T°dÔ6PÐeið 4ñ ô ˆOð !× 1Ò 1Ø 4°DÔ4JÐ_cð 2ñ ô ˆMð &3°;Ô%?ˆE�/Ñ "Ø'6°{Ô'CˆEÐ#Ñ $Ø(5°kÔ(BˆEÐ$Ñ %Ø-:Ð;KÔ-LˆEÐ)Ñ *àÐ ¥W¨UÐ4[Ñ%\Ô%\Ð Ø6;×6aÒ6aÝ œ<¨Ð.?Ô(@ÑAÔAð 7bñ 7ô 7�Ð2Ñ3ð 5:×4_Ò4_Ý œ<¨¨oÔ(>Ñ?Ô?ð 5`ñ 5ô 5�Ð0Ñ1ð ˆ rh FrŽ r rà r r r Údevicec ó” — i }|r3t | d j d | d j d ¦ « }n2t | d j d | d j d ¦ « }| D ] }| d¦ « r‰t | | t j ¦ « rid|v s|r|}n/| d¦ « r|}n| d ¦ « rd
|
|||
|
|
}| dd ¦ « } t | | ||¬ ¦ « || < Œ¡| D ]Ð}| d
|
|||
|
|
¦ « r¹t | | t j ¦ « r™d|v s|r|}n/| d¦ « r|}n| d ¦ « rd
|
|||
|
|
}| d
|
|||
|
|
d ¦ « } t j || t | | ||¬ ¦ « fd
|
|||
|
|
¬¦ «
|
|||
|
|
|¬¦ « || < ŒÑ|rf| d dd¦ «
|
|||
|
|
|¬¦ « |d< | d dd¦ «
|
|||
|
|
|¬¦ « |d< |S )aç Concatenate the chosen and rejected inputs into a single tensor.
|
|||
|
|
|
|||
|
|
Args:
|
|||
|
|
batch:
|
|||
|
|
A batch of data. Must contain the keys 'chosen_input_ids' and 'rejected_input_ids', which are tensors
|
|||
|
|
of shape (batch_size, sequence_length).
|
|||
|
|
is_encoder_decoder:
|
|||
|
|
Whether the model is an encoder-decoder model.
|
|||
|
|
label_pad_token_id:
|
|||
|
|
The label pad token id.
|
|||
|
|
padding_value:
|
|||
|
|
The padding value to use for the concatenated inputs_ids.
|
|||
|
|
device:
|
|||
|
|
The device for the concatenated inputs.
|
|||
|
|
|
|||
|
|
Returns:
|
|||
|
|
A dictionary containing the concatenated inputs under the key 'concatenated_input_ids'.
|
|||
|
|
rµ rP r¶ Úchosen_input_idsÚrejected_input_idsrž r° Ú
|
|||
|
|
_input_idsÚ_attention_maskr Ú concatenated)Ú pad_valuerŸ rO ©rÑ r� rq Úconcatenated_input_idsrŽ Úconcatenated_attention_mask) rÀ rS Ú
|
|||
|
|
startswithri r; r ÚendswithÚreplacer4 ÚcatrU Úrepeat)
|
|||
|
|
rà r r r rÑ Úconcatenated_batchr r¤ rØ Úconcatenated_keys
|
|||
|
|
rf Úconcatenated_inputsz&_UnslothCPOTrainer.concatenated_inputs� s˜ € ð4 Ðà ð gÝ ˜U ?Ô 3Ô 9¸!Ô <¸eÐDUÔ>VÔ>\Ð]^Ô>_Ñ`Ô`ˆJˆJå ˜UÐ#5Ô 6Ô <¸QÔ ?ÀÐG[ÔA\ÔAbÐcdÔAeÑfÔfˆJàð pð pˆAØ�|Š|˜HÑ%Ô%ð
|
|||
|
|
p*°U¸1´X½u¼|Ñ*LÔ*Lð
|
|||
|
|
pؘq�=�=Ð$6�=Ø 2�I�IØ—Z’Z Ñ-Ô-ð "Ø -�I�IØ—Z’ZÐ 1Ñ2Ô2ð "Ø !�IØ#$§9¢9¨X°~Ñ#FÔ#FÐ Ý7DÀUÈ1ÄXÈzÐenÐ7oÑ7oÔ7oÐ"Ð#3Ñ4øØð $ð $ˆAØ�|Š|˜JÑ'Ô'ð
|
|||
|
|
$J°u¸Q´xÅÄ Ñ,NÔ,Nð
|
|||
|
|
$ؘq�=�=Ð$6�=Ø 2�I�IØ—Z’Z Ñ-Ô-ð "Ø -�I�IØ—Z’ZÐ 1Ñ2Ô2ð "Ø !�IØ#$§9¢9¨Z¸ Ñ#HÔ#HÐ Ý7<´yà*Ð+;Ô<Ý% e¨A¤h°
|
|||
|
|
ÀiÐPÑPÔPðð ð 8ñ 8ô 8÷ ’"˜F�"Ñ#Ô#ð
|
|||
|
|
#Ð#3Ñ4øð ð Ø;@ÐASÔ;T×;[Ò;[Ð\]Ð_`Ñ;aÔ;a×;dÒ;dÐlrÐ;dÑ;sÔ;sÐ Ð 7Ñ 8àÐ-Ô.×5Ò5°a¸Ñ;Ô;×>Ò>ÀfÐ>ÑMÔMð
|
|||
|
|
Ð <Ñ =ð "Ð!rh Úpolicy_chosen_logpsÚpolicy_rejected_logpsc óˆ — ||z
|
|||
|
|
| j j ¦ « }| j dk rc| j | j z }||z
|
|||
|
|
}t
|
|||
|
|
j | j |z ¦ « d| j z
|
|||
|
|
z t
|
|||
|
|
j | j |z ¦ « | j z z
|
|||
|
|
}n¼| j dk rOt
|
|||
|
|
j | j |z ¦ « d| j z
|
|||
|
|
z t
|
|||
|
|
j | j |z ¦ « | j z z
|
|||
|
|
}nb| j dk r t j
|
|||
|
|
d| j |z z
|
|||
|
|
¦ « }n7| j dk r|dd| j z z z
|
|||
|
|
dz }nt d| j › d�¦ « ‚| j | | j j ¦ « ¦ « z }| j | | j j ¦ « ¦ « z }|||fS ) aµ Compute the CPO loss for a batch of policy and reference model log probabilities.
|
|||
|
|
|
|||
|
|
Args:
|
|||
|
|
policy_chosen_logps:
|
|||
|
|
Log probabilities of the policy model for the chosen responses. Shape: (batch_size,)
|
|||
|
|
policy_rejected_logps:
|
|||
|
|
Log probabilities of the policy model for the rejected responses. Shape: (batch_size,)
|
|||
|
|
|
|||
|
|
Returns:
|
|||
|
|
A tuple of three tensors: (losses, chosen_rewards, rejected_rewards). The losses tensor contains the CPO
|
|||
|
|
loss for each example in the batch. The chosen_rewards and rejected_rewards tensors contain the rewards for
|
|||
|
|
the chosen and rejected responses, respectively.
|
|||
|
|
r^ rP rΠrY rZ rq zUnknown loss type: z7. Should be one of ['sigmoid', 'hinge', 'ipo', 'simpo'])
|
|||
|
|
rU rh rÑ r r r r Ú
|
|||
|
|
logsigmoidr r; Úrelurk Údetach)r* rä rå r] Úgamma_logratiosÚlossesÚchosen_rewardsÚrejected_rewardss rf Úcpo_lossz_UnslothCPOTrainer.cpo_lossÅ sÞ € ð$ &Ð(=Ñ=×AÒAÀ$ÔBRÔBYÑZÔZˆð Œ>˜WÒ $Ð $Ø "Ô .°´Ñ :ˆOؘoÑ-ˆFõ ” ˜dœi¨&Ñ 0Ñ1Ô1Ð1°Q¸Ô9MÑ5MÑNÝ”, ¤ ˜z¨FÑ 2Ñ3Ô3°dÔ6JÑJñKð
|
|||
|
|
ˆFð Œ^˜yÒ
|
|||
|
|
(Ð
|
|||
|
|
(õ ” ˜dœi¨&Ñ 0Ñ1Ô1Ð1°Q¸Ô9MÑ5MÑNÝ”, ¤ ˜z¨FÑ 2Ñ3Ô3°dÔ6JÑJñKð
|
|||
|
|
ˆFð Œ^˜wÒ
|
|||
|
|
&Ð
|
|||
|
|
&Ý”Z D¤I°Ñ$6Ñ 6Ñ7Ô7ˆFˆFØ
|
|||
|
|
Œ^˜uÒ
|
|||
|
|
$Ð
|
|||
|
|
$à ˜q A¨¬ ¡MÑ 2Ñ2°qÑ8ˆFˆFå Øm d¤nÐmÐmÐmñô ð
|
|||
|
|
ð œÐ&9×&<Ò&<¸TÔ=MÔ=TÑ&UÔ&U×%]Ò%]Ñ%_Ô%_Ñ_ˆØ œ9Ð(=×(@Ò(@ÀÔAQÔAXÑ(YÔ(Y×'aÒ'aÑ'cÔ'cÑcÐà�~Ð'7Ð7Ð7rh r] r° Úaverage_log_probc ó„ — | j dd… |j k rt d¦ « ‚|s1|dd…dd…f ¦ « }| dd…dd…dd…f } ||k }d|||k < t | |¦ « }|r.||z d¦ « | d¦ « z S ||z d¦ « S )aŽ Compute the log probabilities of the given labels under the given logits.
|
|||
|
|
|
|||
|
|
Args:
|
|||
|
|
logits: Logits of the model (unnormalized). Shape: (batch_size, sequence_length, vocab_size)
|
|||
|
|
labels:
|
|||
|
|
Labels for which to compute the log probabilities. Label tokens with a value of label_pad_token_id are
|
|||
|
|
ignored. Shape: (batch_size, sequence_length)
|
|||
|
|
average_log_prob:
|
|||
|
|
If True, return the average log probability per (non-masked) token. Otherwise, return the sum of the
|
|||
|
|
log probabilities of the (non-masked) tokens.
|
|||
|
|
label_pad_token_id: The label pad token id.
|
|||
|
|
is_encoder_decoder: Whether the model is an encoder-decoder model.
|
|||
|
|
|
|||
|
|
Returns:
|
|||
|
|
A tensor of shape (batch_size,) containing the average/sum log probabilities of the given labels under the
|
|||
|
|
given logits.
|
|||
|
|
NrJ zKLogits (batch and sequence length dim) and labels must have the same shape.rP r )rS rk Úcloner9 r¼ )r] r° rï r r Ú loss_maskre s rf Úget_batch_logpsz"_UnslothCPOTrainer.get_batch_logpsú sç € ð2 Œ<˜˜˜Ô ¤ Ò ,Ð ,Ý Ð jÑkÔkÐ kà!ð 'ؘA˜A˜A˜q˜r˜r˜E”]×(Ò(Ñ*Ô*ˆFؘA˜A˜A˜s ˜s A A A˜IÔ&ˆFØÐ 0Ò0ˆ ð 01ˆˆvÐ+Ò+Ñ,å/°¸Ñ?Ô?ˆà ð 9Ø# iÑ/×4Ò4°RÑ8Ô8¸9¿=º=ÈÑ;LÔ;LÑLÐ Là# iÑ/×4Ò4°RÑ8Ô8Ð 8rh c ó
|
|||
|
|
‡ — ‰ |‰ j ‰ j ‰ j ‰ j j ¬¦ « }|d j d }‰ j r d‰ |d ¦ « ini }‰ j rd|d< ||d f|d d
|
|||
|
|
d œ|¤Ž}|j }ˆ fd „}|d
|
|||
|
|
¦ « } ‰ j dk r2t j
|
|||
|
|
d
|
|||
|
|
¦ « ‰ j j ¦ « }
|
|||
|
|
n ||d|… | d|… ¦ « }
|
|||
|
|
‰ ||d ‰ j dv ‰ j ‰ j ¬¦ « } | d|… } | |d… }
|
|||
|
|
|d|… }||d… }‰ j r
|
|||
|
|
| |
|
|||
|
|
|||
|
|||
|
|
|j fS | |
|
|||
|
|
|||
|
|||
|
|
fS )zÆRun the given model on the given batch of inputs, concatenating the chosen and rejected inputs together.
|
|||
|
|
|
|||
|
|
We do this to avoid doing two forward passes, because it's faster for FSDP.
|
|||
|
|
)r r r rÑ rµ r Údecoder_input_idsÚconcatenated_labelsTr\ rÚ rÛ F)rŒ Ú use_cachec ór •— ‰j s?| dd d…d d …f ¦ « } |ddd …f ¦ « }t j ¦ « }| d| j d ¦ « } | d¦ « }| | j ¦ « } || |¦ « }|S )N.rJ rP )r Ú
|
|||
|
|
contiguousr0 ÚCrossEntropyLossÚviewrS rU rÑ )r] r° Úloss_fctÚlossr* s €rf Úcross_entropy_losszC_UnslothCPOTrainer.concatenated_forward.<locals>.cross_entropy_lossH s¯ ø€ ØÔ*ð
|
|||
|
|
6à S b S¨!¨!¨! Ô,×7Ò7Ñ9Ô9�Ø Q R R œ ×3Ò3Ñ5Ô5�åÔ*Ñ,Ô,ˆHØ—[’[ V¤\°"Ô%5Ñ6Ô6ˆFØ—[’[ ‘_”_ˆFà—Y’Y˜vœ}Ñ-Ô-ˆFØ�8˜F FÑ+Ô+ˆD؈Krh r„ N)rZ r^ )rï r r )rã r r r rh rÑ rS Ú _shift_rightr| r] rñ r r; rÁ rU ró r Úaux_loss)r* r: rà rá Ú
|