189 lines
38 KiB
Plaintext
189 lines
38 KiB
Plaintext
|
§
|
|||
|
|
3$�h¦œ ã óü — d Z ddlmZ ddlZddlmZ ddlmZ ddlmZm Z m
|
|||
|
|
Z
|
|||
|
|
m Z m Z m
|
|||
|
|
Z
|
|||
|
|
mZmZ ddlmZmZmZmZmZmZmZmZmZmZmZmZmZm
|
|||
|
|
Z
|
|||
|
|
mZm Z m Z m Z m Z m Z m Z m!Z!m"Z"m#Z#m$Z$m%Z%mZm&Z&m'Z'm(Z(m)Z)mZm*Z* ddl&Z&ddlT ddl+m,Z,m-Z- dd l.m/Z/ ddlZddl0Z1dd
|
|||
|
|
l2m3Z3 ddlmZ dd l4m5Z5m6Z7 d d
|
|||
|
|
d d
|
|||
|
|
d
|
|||
|
|
dœZ8 ej9 d d e8¬¦ « d„ ¦ « Z:e, G d„ de¦ « ¦ « Z; G d„ de ¦ « Z< G d„ de<¦ « Z=dS )z8
|
|||
|
|
2025.8.4
|
|||
|
|
2025.8.5
|
|||
|
|
4.55.1
|
|||
|
|
0.21.0
|
|||
|
|
__UNSLOTH_VERSIONING__
|
|||
|
|
é )ÚTensorN)Ú
|
|||
|
|
functional)ÚAnyÚListÚOptionalÚTupleÚUnionÚDictÚSetÚCallable)!r ÚAutoModelForCausalLMÚBaseImageProcessorr Ú DataCollatorÚDataCollatorForChatMLÚDatasetÚEvalPredictionÚFÚFeatureExtractionMixinÚ GKDConfigÚ
|
|||
|
|
GKDTrainerÚGenerationConfigr Ú
|
|||
|
|
PeftConfigÚPreTrainedModelÚPreTrainedTokenizerBaseÚProcessorMixinÚ
|
|||
|
|
SFTTrainerÚTrainerCallbackr Údisable_dropout_in_modelÚ empty_cacheÚgenerate_model_cardÚget_comet_experiment_urlÚis_wandb_availableÚnnÚosÚprepare_deepspeedÚrandomÚtextwrapÚtorchÚunwrap_model_for_generation)Ú*)Ú dataclassÚfield)ÚVersion)Ú nullcontext)ÚDataCollatorForSeq2SeqÚ DataCollatorForLanguageModelingTF)Úepilogue_fusionÚ max_autotuneÚ
|
|||
|
|
shape_paddingz
|
|||
|
|
trace.enabledztriton.cudagraphs)ÚdynamicÚ fullgraphÚoptionsc ó’ — t j | d| j d ¦ « dd¬¦ « }t j | d¦ « dd¬¦ « }g }t ||¦ « D ]‘\ }}| t j ¦ « }t j |d| d¦ « ¬¦ « d¦ « }t j
|
|||
|
|
|d¬¦ « }||z
|
|||
|
|
} | | ¦ « Œ’ t j |¦ « }| | j d | j d f¦ « }|S )Néÿÿÿÿé r )ÚchunksÚdim)r; Úindex©r; é )
|
|||
|
|
r( ÚchunkÚreshapeÚshapeÚzipÚtoÚfloat32ÚgatherÚ unsqueezeÚsqueezeÚ logsumexpÚappendÚconcat)
|
|||
|
|
Úlogitsr< Úchunked_logitsÚ
|
|||
|
|
chunked_indexÚall_per_token_logpsÚ chunk_logitsÚ chunk_indexÚselected_logitsÚlogsumexp_valuesÚper_token_logpss
|
|||
|
|
ú]/workspace/Fine-tuning/DS-LLM-TEMPLATE-FINETUNING/unsloth_compiled_cache/UnslothGKDTrainer.pyÚ chunked_selective_log_softmaxrU " s5 € õ ”[ § ¢ °°F´LÀÔ4DÑ!EÔ!EÐPQÐYZÐ[Ñ[Ô[€NÝ”[ § ¢ ¨rÑ!2Ô!2¸QÀaÐHÑHÔH€MØ Ðå%(¨ ¸ Ñ%GÔ%Gð 4ð 4Ñ!ˆ �kØ#—’¥u¤}Ñ5Ô5ˆ Ý œ, |¸2À{×G\ÒG\Ð]_ÑG`ÔG`ÐaÑaÔa×iÒiÐjlÑmÔmˆÝ œ?¨<¸rÐBÑBÔBÐØ)Ð,<Ñ<ˆØ×"Ò" ?Ñ3Ô3Ð3Ð3ØÝ œ,Ð':Ñ;Ô;ÐØ-×5Ò5°v´|ÀA´ÈÌ ÐUVÌÐ6XÑYÔYÐØ Ð ó c ó ‡ — e Zd ZU dZ edddi¬¦ « Zee ed< edddi¬¦ « Z ee
|
|||
|
|
ed < d/ˆ fd.„ Z ˆ xZ S )0ÚUnslothGKDConfigaB
|
|||
|
|
|
|||
|
|
Configuration class for [`GKDTrainer`].
|
|||
|
|
|
|||
|
|
This class includes only the parameters that are specific to GKD training. For a full list of training arguments,
|
|||
|
|
please refer to the [`~transformers.TrainingArguments`] and [`SFTConfig`] documentation.
|
|||
|
|
|
|||
|
|
Args:
|
|||
|
|
temperature (`float`, *optional*, defaults to `0.9`):
|
|||
|
|
Temperature for sampling. The higher the temperature, the more random the completions.
|
|||
|
|
lmbda (`float`, *optional*, defaults to `0.5`):
|
|||
|
|
Lambda parameter that controls the student data fraction (i.e., the proportion of on-policy
|
|||
|
|
student-generated outputs).
|
|||
|
|
beta (`float`, *optional*, defaults to `0.5`):
|
|||
|
|
Interpolation coefficient between `0.0` and `1.0` of the Generalized Jensen-Shannon Divergence loss. When
|
|||
|
|
beta is `0.0`, the loss is the KL divergence. When beta is `1.0`, the loss is the Inverse KL Divergence.
|
|||
|
|
max_new_tokens (`int`, *optional*, defaults to `128`):
|
|||
|
|
Maximum number of tokens to generate per completion.
|
|||
|
|
teacher_model_name_or_path (`str` or `None`, *optional*, defaults to `None`):
|
|||
|
|
Model name or path of the teacher model. If `None`, the teacher model will be the same as the model being
|
|||
|
|
trained.
|
|||
|
|
teacher_model_init_kwargs (`dict[str, Any]]` or `None`, *optional*, defaults to `None`):
|
|||
|
|
Keyword arguments to pass to `AutoModelForCausalLM.from_pretrained` when instantiating the teacher model
|
|||
|
|
from a string.
|
|||
|
|
disable_dropout (`bool`, *optional*, defaults to `True`):
|
|||
|
|
Whether to disable dropout in the model.
|
|||
|
|
seq_kd (`bool`, *optional*, defaults to `False`):
|
|||
|
|
Seq_kd parameter that controls whether to perform Sequence-Level KD (can be viewed as supervised FT on
|
|||
|
|
teacher-generated output).
|
|||
|
|
|
|||
|
|
NÚhelpzvLLM SamplingParams)ÚdefaultÚmetadataÚvllm_sampling_paramsr8 z8Chunk size to reduce memory usage. -1 is most efficient.Úunsloth_num_chunksFÚnor9 é r éú ç-C ëâ6
|
|||
|
|
?ç{®Gáz„?çÍÌÌÌÌÌì?ç+‡ÙÎ÷ï?ç:Œ0âŽyE>ç ð?ç @Úlinearçš™™™™™¹?ÚpassiveÚwarningTÚstepsr> éô éO
|
|||
|
|
ÚO1ÚautoÚ ç Ú
|
|||
|
|
adamw_8bitÚlengthÚ
|
|||
|
|
every_saveÚlasté Útexté Úbfdç à?é€ c› ó •— |dk rt d|› d�¦ « ‚|dk rt d|› d�¦ « ‚|€|#dk r
|
|||
|
|
|$dk rd}d }#|…€!d
|
|||
|
|
d lm}œ t |œ¦ « d z d ¦ « }…|‘d
|
|||
|
|
k rt d
|
|||
|
|
¦ « ‚|‘dk rt d¦ « ‚ t
|
|||
|
|
¦ « j d¨i d|“d|“d|“d|“d|“d|“d|“d|“d| “d|
|
|||
|
|
“d| “d| “d |
|
|||
|
|
“d |“d |“d |“d |“d!|“d"|“d#|“d$|“d%|“d&|“d'|“d(|“d)|“d*|“d+| “d,| “d-| “d.| “d/| “d0|!“d1|"“d2|#“d3|$“d4|%“d5|&“d6|'“d7|(“d8|)“d9|*“d:|+“d;|,“d<|-“d=|.“d>|/“d?|0“d@|1“dA|2“dB|3“dC|4“dD|5“dE|6“dF|7“dG|8“dH|9“dI|:“dJ|;“dK|<“dL|=“dM|>“dN|?“dO|@“dP|A“dQ|B“dR|C“dS|D“dT|E“dU|F“dV|G“dW|H“dX|I“dY|J“dZ|K“d[|L“d\|M“d]|N“d^|O“d_|P“d`|Q“da|R“db|S“dc|T“dd|U“de|V“df|W“dg|X“dh|Y“di|Z“dj|[“dk|\“dl|]“dm|^“dn|_“do|`“dp|a“dq|b“dr|c“ds|d“dt|e“du|f“dv|g“dw|h“dx|i“dy|j“dz|k“d{|l“d||m“d}|n“d~|o“d|p“d€|q“d�|r“d‚|s“dƒ|t“d„|u“d…|v“d†|w“d‡|x“dˆ|y“d‰|z“dŠ|{“d‹||“dŒ|}“d�|~“dŽ|“d�|€“d�|�“d‘|‚“d’|ƒ“d“|„“d”|…“d•|†“d–|‡“d—|ˆ“d˜|‰“d™|Š“dš|‹“d›|Œ“dœ|�“d�|Ž“dž|�“dŸ|�“d |‘“d¡|’“d¢|““d£|”“d¤|•“d¥|–“d¦|—“d§|˜“|›¤Ž |™| _ |š| _ d S )©NgH¯¼šò×z>z Unsloth: Your learning rate of `zi` is too small and less than 1e-7! Consider increasing it, otherwise gradient updates will be close to 0!r> za` is way too larger > 1! Consider decreasing it to 1e-1, otherwise gradient updates will explode!rl rm Ú unsloth_training_checkpointsr^ r )Ú cpu_countr_ zUUnsloth: Please set a positive non-zero temperature since your results will be wrong.é
|
|||
|
|
zgUnsloth: Please set a positive non-zero temperature less than 10, since sampling will be quite erratic.Ú
|
|||
|
|
output_dirÚoverwrite_output_dirÚdo_trainÚdo_evalÚ
|
|||
|
|
do_predictÚ
|
|||
|
|
eval_strategyÚprediction_loss_onlyÚper_device_train_batch_sizeÚper_device_eval_batch_sizeÚper_gpu_train_batch_sizeÚper_gpu_eval_batch_sizeÚgradient_accumulation_stepsÚeval_accumulation_stepsÚ
|
|||
|
|
eval_delayÚtorch_empty_cache_stepsÚ
|
|||
|
|
learning_rateÚ weight_decayÚ
|
|||
|
|
adam_beta1Ú
|
|||
|
|
adam_beta2Ú adam_epsilonÚ
|
|||
|
|
max_grad_normÚnum_train_epochsÚ max_stepsÚlr_scheduler_typeÚ warmup_ratioÚ warmup_stepsÚ log_levelÚlog_level_replicaÚlog_on_each_nodeÚ logging_dirÚlogging_strategyÚlogging_first_stepÚ
|
|||
|
|
logging_stepsÚlogging_nan_inf_filterÚ
|
|||
|
|
save_strategyÚ
|
|||
|
|
save_stepsÚsave_total_limitÚsave_safetensorsÚsave_on_each_nodeÚsave_only_modelÚ'restore_callback_states_from_checkpointÚno_cudaÚuse_cpuÚuse_mps_deviceÚseedÚ data_seedÚ
|
|||
|
|
jit_mode_evalÚuse_ipexÚbf16Úfp16Úfp16_opt_levelÚhalf_precision_backendÚbf16_full_evalÚfp16_full_evalÚtf32Ú
|
|||
|
|
local_rankÚ ddp_backendÚ
|
|||
|
|
tpu_num_coresÚtpu_metrics_debugÚdebugÚdataloader_drop_lastÚ
|
|||
|
|
eval_stepsÚdataloader_num_workersÚdataloader_prefetch_factorÚ
|
|||
|
|
past_indexÚrun_nameÚ disable_tqdmÚremove_unused_columnsÚ label_namesÚload_best_model_at_endÚmetric_for_best_modelÚgreater_is_betterÚignore_data_skipÚfsdpÚfsdp_min_num_paramsÚ fsdp_configÚ"fsdp_transformer_layer_cls_to_wrapÚaccelerator_configÚ deepspeedÚlabel_smoothing_factorÚoptimÚ
|
|||
|
|
optim_argsÚ adafactorÚgroup_by_lengthÚlength_column_nameÚ report_toÚddp_find_unused_parametersÚddp_bucket_cap_mbÚddp_broadcast_buffersÚdataloader_pin_memoryÚ dataloader_persistent_workersÚskip_memory_metricsÚuse_legacy_prediction_loopÚ push_to_hubÚresume_from_checkpointÚ hub_model_idÚ hub_strategyÚ hub_tokenÚhub_private_repoÚhub_always_pushÚ hub_revisionÚgradient_checkpointingÚ gradient_checkpointing_kwargsÚinclude_inputs_for_metricsÚeval_do_concat_batchesÚ fp16_backendÚpush_to_hub_model_idÚpush_to_hub_organizationÚpush_to_hub_tokenÚ
|
|||
|
|
mp_parametersÚauto_find_batch_sizeÚfull_determinismÚ torchdynamoÚ ray_scopeÚ ddp_timeoutÚ
|
|||
|
|
torch_compileÚtorch_compile_backendÚtorch_compile_modeÚinclude_tokens_per_secondÚ include_num_input_tokens_seenÚneftune_noise_alphaÚoptim_target_modulesÚbatch_eval_metricsÚ
|
|||
|
|
eval_on_startÚuse_liger_kernelÚliger_kernel_configÚeval_use_gather_objectÚ average_tokens_across_devicesÚmodel_init_kwargsÚchat_template_pathÚdataset_text_fieldÚdataset_kwargsÚdataset_num_procÚ eos_tokenÚ pad_tokenÚ
|
|||
|
|
max_lengthÚpackingÚpacking_strategyÚ padding_freeÚpad_to_multiple_ofÚ eval_packingÚcompletion_only_lossÚassistant_only_lossÚactivation_offloadingÚ temperatureÚlmbdaÚbetaÚmax_new_tokensÚteacher_model_name_or_pathÚteacher_model_init_kwargsÚdisable_dropoutÚseq_kd© )
|
|||
|
|
ÚFloatingPointErrorÚ
|
|||
|
|
OverflowErrorÚmultiprocessingr ÚminÚ MathErrorÚsuperÚ__init__r\ r] )žÚselfr� r‚ rƒ r„ r… r† r‡ rˆ r‰ rŠ r‹ rŒ r� rŽ r� r� r‘ r’ r“ r” r• r– r— r˜ r™ rš r› rœ r� rž rŸ r r¡ r¢ r£ r¤ r¥ r¦ r§ r¨ r© rª r« r¬ r r® r¯ r° r± r² r³ r´ rµ r¶ r· r¸ r¹ rº r» r¼ r½ r¾ r¿ rÀ rÁ r rà rÄ rÅ rÆ rÇ rÈ rÉ rÊ rË rÌ rÍ rÎ rÏ rÐ rÑ rÒ rÓ rÔ rÕ rÖ r× rØ rÙ rÚ rÛ rÜ rÝ rÞ rß rà rá râ rã rä rå ræ rç rè ré rê rë rì rí rî rï rð rñ rò ró rô rõ rö r÷ rø rù rú rû rü rý rþ rÿ r r r r r r r r r r r
|
|||
|
|
r r r
|
|||
|
|
r r r r r r r r r r r r\ r] Úkwargsr Ú __class__sž €rT r zUnslothGKDConfig.__init__\ s[
|
|||
|
|
ø€ ð| ˜4Ò Ð Õ'9ð ;VÐ]jð ;Vð ;Vð ;Vñ (Wô (Wð "WØ ˜1Ò Ð ¥Mð 3FÐUbð 3Fð 3Fð 3Fñ %Gô %Gð GØ Ð -°7Ò":Ð":¸zÈSÒ?PÐ?PØ7ˆJØ ˆMØ Ð #Ø 1Ð 1Ð 1Ð 1Ð 1Ð 1Ý " 9 9¡;¤;¨q¡=°!Ñ 4Ô 4Ð Ø ˜!Ò Ð ÝÐ sÑtÔtÐ tØ
|
|||
|
|
˜BÒ
|
|||
|
|
Ð
|
|||
|
|
Ýð Fñ Gô Gð
|
|||
|
|
Gð �‰ŒÔð X &ð X &ð X &Ø#˜ðX &à#7Ð#7ðX &ð �xðX &ð �gð X &ð
|
|||
|
|
$˜ð X &ð *˜Mð
|
|||
|
|
X &ð $8Ð#7ðX &ð +FÐ*EðX &ð *DÐ)CðX &ð (@Ð'?ðX &ð '>Ð&=ðX &ð +FÐ*EðX &ð '>Ð&=ðX &ð $˜ð X &ð '>Ð&=ð X &ð *˜Mð!X &ð" (˜<ð#X &ð$ $˜ð%X &ð& $˜ð'X &ð( (˜<ð)X &ð* *˜Mð+X &ð, 0Ð /ð-X &ð. "˜ ð/X &ð0 !2Ð 1ð1X &ð2 (˜<ð3X &ð4 (˜<ð5X &ð6 "˜ ð7X &ð8 !2Ð 1ð9X &ð: 0Ð /ð;X &ð< &˜+ð=X &ð> 0Ð /ð?X &ð@ "4Ð!3ðAX &ðB *˜MðCX &ðD &<Ð%;ðEX &ðF *˜MðGX &ðH $˜ðIX &ðJ 0Ð /ðKX &ðL 0Ð /ðMX &ðN !2Ð 1ðOX &ðP .˜oðQX &ðR 7^Ð6]ðSX &ðT �gðUX &ðV �gðWX &ðX ,˜^ðYX &ðZ �4ð[X &ð\ "˜ ð]X &ð^ *˜Mð_X &ð` �xðaX &ðb �4ðcX &ðd �4ðeX &ðf ,˜^ðgX &ðh &<Ð%;ðiX &ðj ,˜^ðkX &ðl ,˜^ðmX &ðn �4ðoX &ðp $˜ðqX &ðr &˜+ðsX &ðt *˜MðuX &ðv !2Ð 1ðwX &ðx �EðyX &ðz $8Ð#7ð{X &ð| $˜ð}X &ð~ &<Ð%;ðX &ð@ *DÐ)CðAX &ðB $˜ðCX &ðD �xðEX &ðF (˜<ðGX &ðH %:Ð$9ðIX &ðJ &˜+ðKX &ðL &<Ð%;ðMX &ðN %:Ð$9ðOX &ðP !2Ð 1ðQX &ðR 0Ð /ðSX &ðT �4ðUX &ðV #6Ð"5ðWX &ðX &˜+ðYX &ðZ 2TÐ1Sð[X &ð\ "4Ð!3ð]X &ð^ "˜ ð_X &ð` &<Ð%;ðaX &ðb �EðcX &ðd $˜ðeX &ðf "˜ ðgX &ðh .˜oðiX &ðj "4Ð!3ðkX &ðl "˜ ðmX &ðn *DÐ)CðoX &ðp !2Ð 1ðqX &ðr %:Ð$9ðsX &ðt %:Ð$9ðuX &ðv -JÐ,IðwX &ðx #6Ð"5ðyX &ðz *DÐ)Cð{X &ð| &˜+ð}X &ð~ &<Ð%;ðX &ð@ (˜<ðAX &ðB (˜<ðCX &ðD "˜ ðEX &ðF 0Ð /ðGX &ðH .˜oðIX &ðJ (˜<ðKX &ðL &<Ð%;ðMX &ðN -JÐ,IðOX &ðP *DÐ)CðQX &ðR &<Ð%;ðSX &ðT (˜<ðUX &ðV $8Ð#7ðWX &ðX (@Ð'?ðYX &ðZ !2Ð 1ð[X &ð\ *˜Mð]X &ð^ $8Ð#7ð_X &ð` 0Ð /ðaX &ðb &˜+ðcX &ðd "˜ ðeX &ðf &˜+ðgX &ðh *˜MðiX &ðj %:Ð$9ðkX &ðl "4Ð!3ðmX &ðn )BÐ(AðoX &ðp -JÐ,IðqX &ðr #6Ð"5ðsX &ðt $8Ð#7ðuX &ðv "4Ð!3ðwX &ðx *˜MðyX &ðz 0Ð /ð{X &ð| #6Ð"5ð}X &ð~ &<Ð%;ðX &ð@ -JÐ,IðAX &ðB !2Ð 1ðCX &ðD "4Ð!3ðEX &ðF "4Ð!3ðGX &ðH ,˜^ðIX &ðJ 0Ð /ðKX &ðL "˜ ðMX &ðN "˜ ðOX &ðP $˜ðQX &ðR �gðSX &ðT 0Ð /ðUX &ðV (˜<ðWX &ðX "4Ð!3ðYX &ðZ (˜<ð[X &ð\ $8Ð#7ð]X &ð^ #6Ð"5ð_X &ð` %:Ð$9ðaX &ðb &˜+ðcX &ðd �EðeX &ðf �4ðgX &ðh ,˜^ðiX &ðj *DÐ)CðkX &ðl )BÐ(AðmX &ðn .˜oðoX &ðp �V˜fðqX &ð X &ð X &ðr %9ˆÔ!Ø"4ˆÔ Ð Ð rV )šNNFFFr^ Fr9 r9 NNr_ r_ r r` ra rb rc rd re rf rg r8 rh ri r rj rk TNrl Fr> Frl rm NTFFFFFFrn rn FFFFro rp FFNr8 NNFrq FNr Nr8 NNTNFNNFrq r NNNNrr rs NFFrt NNNNTFTFFNNru NNFNFNFTrp NNNrq TFNrv rw FNNFFNNFFFNFTNNrx NNNNry Frz FNNNFFrc r{ r{ r| NNTFNr8 )
|
|||
|
|
Ú__name__Ú
|
|||
|
|
__module__Ú __qualname__Ú__doc__r, r\ r r Ú__annotations__r] Úintr Ú
|
|||
|
|
__classcell__©r# s @rT rX rX 3 s˜ ø€ € € € € € ð ð ð> +0¨%ØØÐ 1Ð2ð+ñ +ô +И( 3œ-ð ð ñ ð */¨ØØÐ VÐWð*ñ *ô *И #œ ð ð ñ ð Ø #ØØØØ Ø $Ø&'Ø%&Ø#'Ø"&Ø&'Ø"#ØØ"%Ø ØØØØ ØØ ØØ $ØØØ Ø %Ø ØØ"Ø "ØØ!&Ø ØØ Ø Ø !Ø Ø27ØØØ ØØØ ØØØØ Ø!'Ø Ø ØØØØ Ø !ØØ $ØØ!"Ø%)ØØØØ $ØØ!&Ø $Ø Ø ØØ ØØ-1Ø !ØØ!$Ø ØØØ Ø %ØØ%)Ø Ø $Ø $Ø(-Ø "Ø%*ØØ!%ØØ#ØØ Ø ØØ!&Ø(,Ø%*Ø!%Ø Ø #Ø#'Ø ØØ #Ø ØØØØ Ø $Ø !Ø$)Ø(-Ø "Ø #Ø "Ø Ø Ø "Ø!&Ø(,Ø Ø !Ø #Ø Ø ØØØØØ Ø Ø !ØØ #Ø #Ø %ØØØØ Ø%)Ø$(Ø ØØ #Ø ðwF5ð F5ð F5ð F5ð F5ð F5ð F5ð F5ð F5ð F5rV rX c óð ‡ — e Zd ZddgZ d#deeeej e f deeej e f dee
|
|||
|
|
dee d ee d
|
|||
|
|
eee e
|
|||
|
|
e e f f d eeeeeef d eeege
|
|||
|
|
f d
|
|||
|
|
eee deej j ej j j f deeej ej gej f ded dee fˆ fd„
|
|||
|
|
Z e d$d„¦ « Z d%d„Z e d&d„¦ « Z! d&dej de
|
|||
|
|
e eej e"f f d ee# d ej fˆ fd „
|
|||
|
|
Z$ d'd ee d ee d!ee ee df fd"„Z%ˆ xZ&S )(Ú_UnslothGKDTrainerÚtrlÚgkdN©NNÚmodelÚ
|
|||
|
|
teacher_modelÚargsÚ
|
|||
|
|
data_collatorÚ
|
|||
|
|
train_datasetÚ eval_datasetÚprocessing_classÚcompute_metricsÚ callbacksÚ
|
|||
|
|
optimizersÚ preprocess_logits_for_metricsÚ peft_configr Úformatting_funcc óÒ •— d|_ t ||j ¬¦ « }t ¦ « |||||||| |
|
|||
|
|
| | |
|
|||
|
|
¬¦ « |j €i }n[t
|
|||
|
|
|t ¦ « st d¦ « ‚|j }|d dv r|d nt t |d ¦ « |d< t
|
|||
|
|
|t ¦ « rt j |fi |¤Ž}|j
|
|||
|
|
rt | j ¦ « | j rt# || j ¦ « | _ n!| j |d¬¦ « | _ |j | _ |j | _ |j | _ |j | _ t3 |j |j dd |j rdnd| j j ¬
|
|||
|
|
¦ « | _ t? | j j d ¦ « r.| j j j � | j j j | j _ d S d S d S ) NF)Ú tokenizerr ) r3 r4 r5 r6 r7 r8 r9 r: r; r< r= zfYou passed teacher_model_init_kwargs to the GKDConfig, but your teacher_model is already instantiated.Ú torch_dtype)rp NT)Úevaluation_moder )r r Ú do_sampleÚtop_kÚ use_cacheÚ pad_token_idÚ eos_token_id)!rÄ r r r r r Ú
|
|||
|
|
isinstanceÚstrÚ
|
|||
|
|
ValueErrorÚgetattrr( r
|
|||
|
|
Úfrom_pretrainedr r r1 Úis_deepspeed_enabledr% Ú acceleratorr2 Ú
|
|||
|
|
prepare_modelr r r r r r ræ r7 rE Úgeneration_configÚhasattrrF )r! r1 r2 r3 r4 r5 r6 r7 r8 r9 r: r; r< r= r r# s €rT r z_UnslothGKDTrainer.__init__¨ s3 ø€ ð& &+ˆÔ"Ý-Ð8HÐUYÔUdÐeÑeÔeˆ
|
|||
|
|
å
|
|||
|
|
‰Œ×ÒØ ØØ'Ø'Ø%Ø -Ø +Ø Ø!Ø*GØ#Ø +ð ñ
|
|||
|
|
|
|||
|
|
ô
|
|||
|
|
|
|||
|
|
ð
|
|||
|
|
|
|||
|
|
ð Ô )Ð 1Ø(*Ð %Ð %ݘM3Ñ/Ô/ð
|
|||
|
|
Ý Øxñô ð
|
|||
|
|
ð )-Ô(FÐ %ð -¨]Ô;¸~ÐMÐMð *¨-Ô8Ð8å �UÐ$=¸mÔ$LÑMÔMð
|
|||
|
|
& mÑ 4õ �m¥SÑ )Ô )ð mÝ 0Ô @À Ð lÐ lÐRkÐ lÐ lˆMð Ô ð 1Ý $ T¤ZÑ 0Ô 0Ð 0à Ô $ð eÝ!2°=À$ÔBRÑ!SÔ!SˆDÔ Ð à!%Ô!1×!?Ò!?À
|
|||
|
|
Ð_cÐ!?Ñ!dÔ!dˆDÔ à”ZˆŒ
|
|||
|
|
Ø”IˆŒ Ø Ô+ˆÔØ”kˆŒ å!1Ø Ô.Ø Ô(ØØØ #Ô :ÐD�e�eÀØ Ô.Ô;ð
|
|||
|
|
"
|
|||
|
|
ñ "
|
|||
|
|
ô "
|
|||
|
|
ˆÔ õ
|
|||
|
|
�D”JÔ0°.Ñ AÔ Að \à”
|
|||
|
|
Ô,Ô9ÐEà26´*Ô2NÔ2[ˆDÔ "Ô /Ð /Ð /ð \ð \àEÐErV r{ rf Ú batchmeanc ó¸ — | |z } ||z }t j | d¬¦ « }t j |d¬¦ « }|dk rt j ||dd¬¦ « }nÏ|dk rt j ||dd¬¦ « }n°t j ||j ¬¦ « }t j t j |t j d|z
|
|||
|
|
¦ « z |t j |¦ « z g¦ « d¬¦ « } t j | |dd¬¦ « }
|
|||
|
|
t j | |dd¬¦ « } ||
|
|||
|
|
z d|z
|
|||
|
|
| z z }|�|d
|
|||
|
|
k } || }|d k rk|�)| ¦ « | ¦ « z n?| ¦ « |
|
|||
|
|
d¦ « |
|
|||
|
|
d¦ « z z S |d k r| ¦ « S |d
|
|||
|
|
k r| ¦ « S |S )a£
|
|||
|
|
Compute the generalized Jensen-Shannon Divergence loss for knowledge distillation using F.kl_div. See Eq. (1)
|
|||
|
|
of https://huggingface.co/papers/2306.13649 for the definition.
|
|||
|
|
|
|||
|
|
Args:
|
|||
|
|
student_logits:
|
|||
|
|
Tensor of shape (batch_size, sequence_length, vocab_size)
|
|||
|
|
teacher_logits:
|
|||
|
|
Tensor of shape (batch_size, sequence_length, vocab_size)
|
|||
|
|
labels:
|
|||
|
|
Tensor of shape (batch_size, sequence_length) with -100 for padding tokens to ignore when computing
|
|||
|
|
loss
|
|||
|
|
beta:
|
|||
|
|
Interpolation coefficient between 0 and 1 (default: 0.5)
|
|||
|
|
temperature:
|
|||
|
|
Softmax temperature (default: 1.0)
|
|||
|
|
reduction:
|
|||
|
|
Specifies the reduction to apply to the output (default: 'batchmean')
|
|||
|
|
|
|||
|
|
Returns:
|
|||
|
|
loss: Scalar tensor with the generalized JSD loss
|
|||
|
|
r8 r= r ÚnoneT)Ú reductionÚ
|
|||
|
|
log_targetr> )ÚdtypeNéœÿÿÿrQ ÚsumÚmean) r Ú log_softmaxÚkl_divr( ÚtensorrV rH ÚstackÚlogrX ÚsizerY )
|
|||
|
|
Ústudent_logitsÚteacher_logitsÚlabelsr r rT Ústudent_log_probsÚteacher_log_probsÚjsdÚmixture_log_probsÚ
|
|||
|
|
kl_teacherÚ
|
|||
|
|
kl_studentÚmasks
|
|||
|
|
rT Úgeneralized_jsd_lossz'_UnslothGKDTrainer.generalized_jsd_lossþ s € ð8 (¨+Ñ5ˆØ'¨+Ñ5ˆõ œM¨.¸bÐ AÑ AÔ AÐÝ œM¨.¸bÐ AÑ AÔ AÐà �1Š9ˆ9Ý”(Ð,Ð.?È6Ð^bÐcÑcÔcˆCˆCØ
|
|||
|
|
�QŠYˆYÝ”(Ð,Ð.?È6Ð^bÐcÑcÔcˆCˆCõ ”< Ð,=Ô,CÐDÑDÔDˆDÝ %¤Ý” Ð .µ´¸1¸t¹8Ñ1DÔ1DÑ DÐFWÕZ_ÔZcÐdhÑZiÔZiÑFiÐ jÑkÔkØð!ñ !ô !Ð õ œÐ"3Ð5FÐRXÐeiÐjÑjÔjˆJÝœÐ"3Ð5FÐRXÐeiÐjÑjÔjˆJð ˜Ñ# q¨4¡x°:Ñ&=Ñ=ˆCð РؘT’>ˆDØ�d”)ˆCð ˜ Ò #Ð #Ø-3Ð-?�3—7’7‘9”9˜tŸxšx™zœzÑ)Ð)ÀSÇWÂWÁYÄYÐRU×RZÒRZÐ[\ÑR]ÔR]Ð`c×`hÒ`hÐijÑ`kÔ`kÑRkÑElÐ lØ
|
|||
|
|
˜%Ò
|
|||
|
|
Ð
|
|||
|
|
Ø—7’7‘9”9Ð Ø
|
|||
|
|
˜&Ò
|
|||
|
|
Ð
|