199 lines
41 KiB
Plaintext
199 lines
41 KiB
Plaintext
|
§
|
|||
|
|
5$�hL¡ ã óD — d Z ddlmZ ddlZddlmZ ddlmZ ddlmZm Z m
|
|||
|
|
Z
|
|||
|
|
m Z m Z m
|
|||
|
|
Z
|
|||
|
|
mZmZ ddlmZmZmZmZmZmZmZmZm
|
|||
|
|
Z
|
|||
|
|
mZmZmZmZmZm Z m Z m Z m Z m Z m!Z!m Z m"Z"m#Z#m$Z$m%Z%m&Z&m'Z'm(Z(m)Z)m*Z*m+Z+m,Z,m-Z-m.Z.m/Z/m0Z0mZm1Z1m2Z2m3Z3m4Z4m5Z5mZm6Z6m
|
|||
|
|
Z
|
|||
|
|
mZmZm Z m+Z+m1Z1mZ ddl1Z1ddlT ddl7m8Z8m9Z9 dd l:m;Z; ddlZddl<Z=dd
|
|||
|
|
l>m?Z? ddlmZ dd l@mAZAmBZC d d
|
|||
|
|
d d
|
|||
|
|
d
|
|||
|
|
dœZD ejE d d eD¬¦ « d„ ¦ « ZFe8 G d„ de ¦ « ¦ « ZG G d„ de ¦ « ZH G d„ deH¦ « ZIdS )z8
|
|||
|
|
2025.8.4
|
|||
|
|
2025.8.5
|
|||
|
|
4.55.1
|
|||
|
|
0.21.0
|
|||
|
|
__UNSLOTH_VERSIONING__
|
|||
|
|
é )ÚTensorN)Ú
|
|||
|
|
functional)ÚAnyÚListÚOptionalÚTupleÚUnionÚDictÚSetÚCallable)3r ÚBaseImageProcessorr Ú DataCollatorÚDatasetÚEvalPredictionÚFeatureExtractionMixinÚFrozenInstanceErrorr Ú PartialStateÚPathÚ PeftModelÚPreTrainedModelÚPreTrainedTokenizerBaseÚProcessorMixinÚ RewardConfigÚ RewardDataCollatorWithPaddingÚ
|
|||
|
|
RewardTrainerÚTrainerÚTrainerCallbackr Ú _tokenizeÚcompute_accuracyÚdecode_and_strip_paddingÚ defaultdictÚdisable_dropout_in_modelÚ
|
|||
|
|
gather_objectÚgenerate_model_cardÚget_comet_experiment_urlÚinspectÚis_peft_availableÚis_rich_availableÚis_wandb_availableÚ log_table_to_comet_experimentÚmaybe_apply_chat_templateÚ
|
|||
|
|
nested_detachÚnnÚosÚpdÚ prepare_model_for_kbit_trainingÚprint_rich_tableÚreplaceÚtorchÚwarningsr r r r r' r. r3 )Ú*)Ú dataclassÚfield)ÚVersion)Ú nullcontext)ÚDataCollatorForSeq2SeqÚ DataCollatorForLanguageModelingTF)Úepilogue_fusionÚ max_autotuneÚ
|
|||
|
|
shape_paddingz
|
|||
|
|
trace.enabledztriton.cudagraphs)ÚdynamicÚ fullgraphÚoptionsc ó’ — t j | d| j d ¦ « dd¬¦ « }t j | d¦ « dd¬¦ « }g }t ||¦ « D ]‘\ }}| t j ¦ « }t j |d| d¦ « ¬¦ « d¦ « }t j
|
|||
|
|
|d¬¦ « }||z
|
|||
|
|
} | | ¦ « Œ’ t j |¦ « }| | j d | j d f¦ « }|S )Néÿÿÿÿé r )ÚchunksÚdim)rF Úindex©rF é )
|
|||
|
|
r3 ÚchunkÚreshapeÚshapeÚzipÚtoÚfloat32ÚgatherÚ unsqueezeÚsqueezeÚ logsumexpÚappendÚconcat)
|
|||
|
|
ÚlogitsrG Úchunked_logitsÚ
|
|||
|
|
chunked_indexÚall_per_token_logpsÚ chunk_logitsÚ chunk_indexÚselected_logitsÚlogsumexp_valuesÚper_token_logpss
|
|||
|
|
ú`/workspace/Fine-tuning/DS-LLM-TEMPLATE-FINETUNING/unsloth_compiled_cache/UnslothRewardTrainer.pyÚ chunked_selective_log_softmaxr` " s5 € õ ”[ § ¢ °°F´LÀÔ4DÑ!EÔ!EÐPQÐYZÐ[Ñ[Ô[€NÝ”[ § ¢ ¨rÑ!2Ô!2¸QÀaÐHÑHÔH€MØ Ðå%(¨ ¸ Ñ%GÔ%Gð 4ð 4Ñ!ˆ �kØ#—’¥u¤}Ñ5Ô5ˆ Ý œ, |¸2À{×G\ÒG\Ð]_ÑG`ÔG`ÐaÑaÔa×iÒiÐjlÑmÔmˆÝ œ?¨<¸rÐBÑBÔBÐØ)Ð,<Ñ<ˆØ×"Ò" ?Ñ3Ô3Ð3Ð3ØÝ œ,Ð':Ñ;Ô;ÐØ-×5Ò5°v´|ÀA´ÈÌ ÐUVÌÐ6XÑYÔYÐØ Ð ó c óš ‡ — e Zd ZU dZ edddi¬¦ « Zee ed< edddi¬¦ « Z ee
|
|||
|
|
ed < d+ˆ fd*„ Z ˆ xZ S ),ÚUnslothRewardConfigaI
|
|||
|
|
|
|||
|
|
Configuration class for the [`RewardTrainer`].
|
|||
|
|
|
|||
|
|
This class includes only the parameters that are specific to Reward training. For a full list of training
|
|||
|
|
arguments, please refer to the [`~transformers.TrainingArguments`] documentation. Note that default values in this
|
|||
|
|
class may differ from those in [`~transformers.TrainingArguments`].
|
|||
|
|
|
|||
|
|
Using [`~transformers.HfArgumentParser`] we can turn this class into
|
|||
|
|
[argparse](https://docs.python.org/3/library/argparse#module-argparse) arguments that can be specified on the
|
|||
|
|
command line.
|
|||
|
|
|
|||
|
|
Parameters:
|
|||
|
|
max_length (`int` or `None`, *optional*, defaults to `1024`):
|
|||
|
|
Maximum length of the sequences (prompt + completion) in the batch, filters out entries that exceed the
|
|||
|
|
limit. This argument is required if you want to use the default data collator.
|
|||
|
|
disable_dropout (`bool`, *optional*, defaults to `True`):
|
|||
|
|
Whether to disable dropout in the model.
|
|||
|
|
dataset_num_proc (`int`, *optional*, defaults to `None`):
|
|||
|
|
Number of processes to use for processing the dataset.
|
|||
|
|
center_rewards_coefficient (`float`, *optional*, defaults to `None`):
|
|||
|
|
Coefficient to incentivize the reward model to output mean-zero rewards (proposed by
|
|||
|
|
https://huggingface.co/papers/2312.09244, Eq. 2). Recommended value: `0.01`.
|
|||
|
|
remove_unused_columns (`bool`, *optional*, defaults to `False`):
|
|||
|
|
Whether to remove the columns that are not used by the model's forward pass. Can be `True` only if the
|
|||
|
|
dataset is pretokenized.
|
|||
|
|
|
|||
|
|
NÚhelpzvLLM SamplingParams)ÚdefaultÚmetadataÚvllm_sampling_paramsrC z8Chunk size to reduce memory usage. -1 is most efficient.Úunsloth_num_chunksFÚnorD é r éú ç-C ëâ6
|
|||
|
|
?ç{®Gáz„?çÍÌÌÌÌÌì?ç+‡ÙÎ÷ï?ç:Œ0âŽyE>ç ð?ç @Úlinearçš™™™™™¹?ÚpassiveÚwarningTÚstepsrI éô éO
|
|||
|
|
ÚO1ÚautoÚ ç Ú
|
|||
|
|
adamw_8bitÚlengthÚ
|
|||
|
|
every_saveÚlasté é c‡ ó< •— |dk rt d|› d�¦ « ‚|dk rt d|› d�¦ « ‚|€|#dk r
|
|||
|
|
|$dk rd}d }#|ƒ€!d
|
|||
|
|
d lm}ˆ t |ˆ¦ « d z d ¦ « }ƒ t ¦ « j d‘i d
|
|||
|
|
|“d|“d|“d|“d|“d|“d|“d|“d| “d|
|
|||
|
|
“d| “d| “d|
|
|||
|
|
“d|“d|“d |“d |“d |“d |“d |“d!|“d"|“d#|“d$|“d%|“d&|“d'|“d(| “d)| “d*| “d+| “d,| “d-|!“d.|"“d/|#“d0|$“d1|%“d2|&“d3|'“d4|(“d5|)“d6|*“d7|+“d8|,“d9|-“d:|.“d;|/“d<|0“d=|1“d>|2“d?|3“d@|4“dA|5“dB|6“dC|7“dD|8“dE|9“dF|:“dG|;“dH|<“dI|=“dJ|>“dK|?“dL|@“dM|A“dN|B“dO|C“dP|D“dQ|E“dR|F“dS|G“dT|H“dU|I“dV|J“dW|K“dX|L“dY|M“dZ|N“d[|O“d\|P“d]|Q“d^|R“d_|S“d`|T“da|U“db|V“dc|W“dd|X“de|Y“df|Z“dg|[“dh|\“di|]“dj|^“dk|_“dl|`“dm|a“dn|b“do|c“dp|d“dq|e“dr|f“ds|g“dt|h“du|i“dv|j“dw|k“dx|l“dy|m“dz|n“d{|o“d||p“d}|q“d~|r“d|s“d€|t“d�|u“d‚|v“dƒ|w“d„|x“d…|y“d†|z“d‡|{“dˆ||“d‰|}“dŠ|~“d‹|“dŒ|€“d�|�“dŽ|‚“d�|ƒ“d�|„“|‡¤Ž |…| _ |†| _ d S )’NgH¯¼šò×z>z Unsloth: Your learning rate of `zi` is too small and less than 1e-7! Consider increasing it, otherwise gradient updates will be close to 0!rI za` is way too larger > 1! Consider decreasing it to 1e-1, otherwise gradient updates will explode!rw rx Ú unsloth_training_checkpointsri r )Ú cpu_countrj Ú
|
|||
|
|
output_dirÚoverwrite_output_dirÚdo_trainÚdo_evalÚ
|
|||
|
|
do_predictÚ
|
|||
|
|
eval_strategyÚprediction_loss_onlyÚper_device_train_batch_sizeÚper_device_eval_batch_sizeÚper_gpu_train_batch_sizeÚper_gpu_eval_batch_sizeÚgradient_accumulation_stepsÚeval_accumulation_stepsÚ
|
|||
|
|
eval_delayÚtorch_empty_cache_stepsÚ
|
|||
|
|
learning_rateÚ weight_decayÚ
|
|||
|
|
adam_beta1Ú
|
|||
|
|
adam_beta2Ú adam_epsilonÚ
|
|||
|
|
max_grad_normÚnum_train_epochsÚ max_stepsÚlr_scheduler_typeÚ warmup_ratioÚ warmup_stepsÚ log_levelÚlog_level_replicaÚlog_on_each_nodeÚ logging_dirÚlogging_strategyÚlogging_first_stepÚ
|
|||
|
|
logging_stepsÚlogging_nan_inf_filterÚ
|
|||
|
|
save_strategyÚ
|
|||
|
|
save_stepsÚsave_total_limitÚsave_safetensorsÚsave_on_each_nodeÚsave_only_modelÚ'restore_callback_states_from_checkpointÚno_cudaÚuse_cpuÚuse_mps_deviceÚseedÚ data_seedÚ
|
|||
|
|
jit_mode_evalÚuse_ipexÚbf16Úfp16Úfp16_opt_levelÚhalf_precision_backendÚbf16_full_evalÚfp16_full_evalÚtf32Ú
|
|||
|
|
local_rankÚ ddp_backendÚ
|
|||
|
|
tpu_num_coresÚtpu_metrics_debugÚdebugÚdataloader_drop_lastÚ
|
|||
|
|
eval_stepsÚdataloader_num_workersÚdataloader_prefetch_factorÚ
|
|||
|
|
past_indexÚrun_nameÚ disable_tqdmÚremove_unused_columnsÚ label_namesÚload_best_model_at_endÚmetric_for_best_modelÚgreater_is_betterÚignore_data_skipÚfsdpÚfsdp_min_num_paramsÚ fsdp_configÚ"fsdp_transformer_layer_cls_to_wrapÚaccelerator_configÚ deepspeedÚlabel_smoothing_factorÚoptimÚ
|
|||
|
|
optim_argsÚ adafactorÚgroup_by_lengthÚlength_column_nameÚ report_toÚddp_find_unused_parametersÚddp_bucket_cap_mbÚddp_broadcast_buffersÚdataloader_pin_memoryÚ dataloader_persistent_workersÚskip_memory_metricsÚuse_legacy_prediction_loopÚ push_to_hubÚresume_from_checkpointÚ hub_model_idÚ hub_strategyÚ hub_tokenÚhub_private_repoÚhub_always_pushÚ hub_revisionÚgradient_checkpointingÚ gradient_checkpointing_kwargsÚinclude_inputs_for_metricsÚeval_do_concat_batchesÚ fp16_backendÚpush_to_hub_model_idÚpush_to_hub_organizationÚpush_to_hub_tokenÚ
|
|||
|
|
mp_parametersÚauto_find_batch_sizeÚfull_determinismÚ torchdynamoÚ ray_scopeÚ ddp_timeoutÚ
|
|||
|
|
torch_compileÚtorch_compile_backendÚtorch_compile_modeÚinclude_tokens_per_secondÚ include_num_input_tokens_seenÚneftune_noise_alphaÚoptim_target_modulesÚbatch_eval_metricsÚ
|
|||
|
|
eval_on_startÚuse_liger_kernelÚliger_kernel_configÚeval_use_gather_objectÚ average_tokens_across_devicesÚ
|
|||
|
|
max_lengthÚdisable_dropoutÚdataset_num_procÚcenter_rewards_coefficient© ) ÚFloatingPointErrorÚ
|
|||
|
|
OverflowErrorÚmultiprocessingr† ÚminÚsuperÚ__init__rg rh )ŠÚselfr‡ rˆ r‰ rŠ r‹ rŒ r� rŽ r� r� r‘ r’ r“ r” r• r– r— r˜ r™ rš r› rœ r� rž rŸ r r¡ r¢ r£ r¤ r¥ r¦ r§ r¨ r© rª r« r¬ r r® r¯ r° r± r² r³ r´ rµ r¶ r· r¸ r¹ rº r» r¼ r½ r¾ r¿ rÀ rÁ r rà rÄ rÅ rÆ rÇ rÈ rÉ rÊ rË rÌ rÍ rÎ rÏ rÐ rÑ rÒ rÓ rÔ rÕ rÖ r× rØ rÙ rÚ rÛ rÜ rÝ rÞ rß rà rá râ rã rä rå ræ rç rè ré rê rë rì rí rî rï rð rñ rò ró rô rõ rö r÷ rø rù rú rû rü rý rþ rÿ r r r r r r r r r r r
|
|||
|
|
rg rh Úkwargsr† Ú __class__sŠ €r_ r z UnslothRewardConfig.__init__Y sl ø€ ðT ˜4Ò Ð Õ'9ð ;VÐ]jð ;Vð ;Vð ;Vñ (Wô (Wð "WØ ˜1Ò Ð ¥Mð 3FÐUbð 3Fð 3Fð 3Fñ %Gô %Gð GØ Ð -°7Ò":Ð":¸zÈSÒ?PÐ?PØ7ˆJØ ˆMØ Ð #Ø 1Ð 1Ð 1Ð 1Ð 1Ð 1Ý " 9 9¡;¤;¨q¡=°!Ñ 4Ô 4Ð à�‰ŒÔð D Nð D Nð D NØ#˜ðD Nà#7Ð#7ðD Nð �xðD Nð �gð D Nð
|
|||
|
|
$˜ð D Nð *˜Mð
|
|||
|
|
D Nð $8Ð#7ðD Nð +FÐ*EðD Nð *DÐ)CðD Nð (@Ð'?ðD Nð '>Ð&=ðD Nð +FÐ*EðD Nð '>Ð&=ðD Nð $˜ð D Nð '>Ð&=ð D Nð *˜Mð!D Nð" (˜<ð#D Nð$ $˜ð%D Nð& $˜ð'D Nð( (˜<ð)D Nð* *˜Mð+D Nð, 0Ð /ð-D Nð. "˜ ð/D Nð0 !2Ð 1ð1D Nð2 (˜<ð3D Nð4 (˜<ð5D Nð6 "˜ ð7D Nð8 !2Ð 1ð9D Nð: 0Ð /ð;D Nð< &˜+ð=D Nð> 0Ð /ð?D Nð@ "4Ð!3ðAD NðB *˜MðCD NðD &<Ð%;ðED NðF *˜MðGD NðH $˜ðID NðJ 0Ð /ðKD NðL 0Ð /ðMD NðN !2Ð 1ðOD NðP .˜oðQD NðR 7^Ð6]ðSD NðT �gðUD NðV �gðWD NðX ,˜^ðYD NðZ �4ð[D Nð\ "˜ ð]D Nð^ *˜Mð_D Nð` �xðaD Nðb �4ðcD Nðd �4ðeD Nðf ,˜^ðgD Nðh &<Ð%;ðiD Nðj ,˜^ðkD Nðl ,˜^ðmD Nðn �4ðoD Nðp $˜ðqD Nðr &˜+ðsD Nðt *˜MðuD Nðv !2Ð 1ðwD Nðx �EðyD Nðz $8Ð#7ð{D Nð| $˜ð}D Nð~ &<Ð%;ðD Nð@ *DÐ)CðAD NðB $˜ðCD NðD �xðED NðF (˜<ðGD NðH %:Ð$9ðID NðJ &˜+ðKD NðL &<Ð%;ðMD NðN %:Ð$9ðOD NðP !2Ð 1ðQD NðR 0Ð /ðSD NðT �4ðUD NðV #6Ð"5ðWD NðX &˜+ðYD NðZ 2TÐ1Sð[D Nð\ "4Ð!3ð]D Nð^ "˜ ð_D Nð` &<Ð%;ðaD Nðb �EðcD Nðd $˜ðeD Nðf "˜ ðgD Nðh .˜oðiD Nðj "4Ð!3ðkD Nðl "˜ ðmD Nðn *DÐ)CðoD Nðp !2Ð 1ðqD Nðr %:Ð$9ðsD Nðt %:Ð$9ðuD Nðv -JÐ,IðwD Nðx #6Ð"5ðyD Nðz *DÐ)Cð{D Nð| &˜+ð}D Nð~ &<Ð%;ðD Nð@ (˜<ðAD NðB (˜<ðCD NðD "˜ ðED NðF 0Ð /ðGD NðH .˜oðID NðJ (˜<ðKD NðL &<Ð%;ðMD NðN -JÐ,IðOD NðP *DÐ)CðQD NðR &<Ð%;ðSD NðT (˜<ðUD NðV $8Ð#7ðWD NðX (@Ð'?ðYD NðZ !2Ð 1ð[D Nð\ *˜Mð]D Nð^ $8Ð#7ð_D Nð` 0Ð /ðaD Nðb &˜+ðcD Nðd "˜ ðeD Nðf &˜+ðgD Nðh *˜MðiD Nðj %:Ð$9ðkD Nðl "4Ð!3ðmD Nðn )BÐ(AðoD Nðp -JÐ,IðqD Nðr #6Ð"5ðsD Nðt $8Ð#7ðuD Nðv "4Ð!3ðwD Nðx *˜MðyD Nðz 0Ð /ð{D Nð| #6Ð"5ð}D Nð~ &<Ð%;ðD Nð@ -JÐ,IðAD NðB $˜ðCD NðD .˜oðED NðF 0Ð /ðGD NðH *DÐ)CÀfðID Nð D Nð D NðJ %9ˆÔ!Ø"4ˆÔ Ð Ð ra )†NNFFFri FrD rD NNrj rj r rk rl rm rn ro rp rq rr rC rs rt r ru rv TNrw FrI Frw rx NTFFFFFFry ry FFFFrz r{ FFNrC NNFr| FNr NrC NNFNFNNFr| r NNNNr} r~ NFFr NNNNTFTFFNNr€ NNFNFNFTr{ NNNr| TFNr� r‚ FNNFFNNFFFNFTrƒ TNNNrC )
|
|||
|
|
Ú__name__Ú
|
|||
|
|
__module__Ú __qualname__Ú__doc__r7 rg r r Ú__annotations__rh Úintr Ú
|
|||
|
|
__classcell__©r s @r_ rc rc 3 s\ ø€ € € € € € ðð ð8 +0¨%ØØÐ 1Ð2ð+ñ +ô +И( 3œ-ð ð ñ ð */¨ØØÐ VÐWð*ñ *ô *И #œ ð ð ñ ð Ø #ØØØØ Ø $Ø&'Ø%&Ø#'Ø"&Ø&'Ø"#ØØ"%Ø ØØØØ ØØ ØØ $ØØØ Ø %Ø ØØ"Ø "ØØ!&Ø ØØ Ø Ø !Ø Ø27ØØØ ØØØ ØØØØ Ø!'Ø Ø ØØØØ Ø !ØØ $ØØ!"Ø%)ØØØØ %ØØ!&Ø $Ø Ø ØØ ØØ-1Ø !ØØ!$Ø ØØØ Ø %ØØ%)Ø Ø $Ø $Ø(-Ø "Ø%*ØØ!%ØØ#ØØ Ø ØØ!&Ø(,Ø%*Ø!%Ø Ø #Ø#'Ø ØØ #Ø ØØØØ Ø $Ø !Ø$)Ø(-Ø "Ø #Ø "Ø Ø Ø "Ø!&Ø(,ØØ Ø Ø%)Ø #Ø ðOY5ð Y5ð Y5ð Y5ð Y5ð Y5ð Y5ð Y5ð Y5ð Y5ra rc c óÔ ‡ — e Zd ZddgZ d!deeeej f dee dee
|
|||
|
|
dee d eee e e
|
|||
|
|
e f f d
|
|||
|
|
eeeeeef d eeg ef d eeege f d
|
|||
|
|
eee deej j ej j j f deeej ej gej f dee fˆ fd„
|
|||
|
|
Z d"deeej f de e
|
|||
|
|
eej e f f deej eej e e
|
|||
|
|
ej f f f fd„Z d#deeej f de e
|
|||
|
|
eej e f f de deee
|
|||
|
|
deeej eej eej f f
|
|||
|
|
d„Z!ˆ fd„Z"de#fd„Z$ˆ fd „Z% d$d ee
|
|||
|
|
d ee
|
|||
|
|
d ee
|
|||
|
|
ee
|
|||
|
|
df fd „Z&ˆ xZ'S )%Ú_UnslothRewardTrainerÚtrlzreward-trainerN©NNÚmodelÚargsÚ
|
|||
|
|
data_collatorÚ
|
|||
|
|
train_datasetÚ eval_datasetÚprocessing_classÚ
|
|||
|
|
model_initÚcompute_metricsÚ callbacksÚ
|
|||
|
|
optimizersÚ preprocess_logits_for_metricsÚ peft_configc
|
|||
|
|
|
|||
|
|
óø •‡— t ¦ « s| �t d¦ « ‚t ¦ « rµ| �³t |t ¦ « sžt |dd¦ « st |dd¦ « rzdt t
|
|||
|
|
j t ¦ « j ¦ « v }
|
|||
|
|
d|j
|
|||
|
|
i}|
|
|||
|
|
s"|j �t j
|
|||
|
|
dt ¦ « n|
|
|||
|
|
r|j �
|
|||
|
|
|j |d< t |fi |¤Ž}|}|j rt! |¦ « |€t" }|€z|€t d ¦ « ‚|j Št' |¦ « }|j rD d|_ n!# t* $ r t- |d¬
|
|||
|
|
¦ « }Y nw xY wt j
|
|||
|
|
d t ¦ « d | _ nd| _ d |j d
|
|||
|
|
< d|j vrþt5 ¦ « ¦ « 5 d|i}| t: d|i¬¦ « }| t< d ||j ¬¦ « }| ˆfd„|j ¬¦ « }|�`| t: d|i¬¦ « }| t< |d |j ¬¦ « }| ˆfd„|j ¬¦ « }ddd¦ « n # 1 swxY w Y tC ¦ « " ||||||||| |
|
|||
|
|
| ¬¦ « tG | j$ d¦ « r!| j$ % | j& ¦ « dS dS )aÂ
|
|||
|
|
Initialize RewardTrainer.
|
|||
|
|
|
|||
|
|
Args:
|
|||
|
|
model (`transformers.PreTrainedModel`):
|
|||
|
|
The model to train, preferably an `AutoModelForSequenceClassification`.
|
|||
|
|
args (`RewardConfig`):
|
|||
|
|
The arguments to use for training.
|
|||
|
|
data_collator (`transformers.DataCollator`):
|
|||
|
|
The data collator to use for training. If None is specified, the default data collator
|
|||
|
|
(`RewardDataCollatorWithPadding`) will be used which will pad the sequences to the maximum length of
|
|||
|
|
the sequences in the batch, given a dataset of paired sequences.
|
|||
|
|
train_dataset (`datasets.Dataset`):
|
|||
|
|
The dataset to use for training.
|
|||
|
|
eval_dataset (`datasets.Dataset`):
|
|||
|
|
The dataset to use for evaluation.
|
|||
|
|
processing_class ([`~transformers.PreTrainedTokenizerBase`], [`~transformers.BaseImageProcessor`], [`~transformers.FeatureExtractionMixin`] or [`~transformers.ProcessorMixin`], *optional*, defaults to `None`):
|
|||
|
|
Processing class used to process the data. If provided, will be used to automatically process the
|
|||
|
|
inputs for the model, and it will be saved along the model to make it easier to rerun an interrupted
|
|||
|
|
training or reuse the fine-tuned model.
|
|||
|
|
model_init (`Callable[[], transformers.PreTrainedModel]`):
|
|||
|
|
The model initializer to use for training. If None is specified, the default model initializer will be
|
|||
|
|
used.
|
|||
|
|
compute_metrics (`Callable[[transformers.EvalPrediction], dict]`, *optional* defaults to `compute_accuracy`):
|
|||
|
|
The metrics to use for evaluation. If no metrics are specified, the default metric (`compute_accuracy`)
|
|||
|
|
will be used.
|
|||
|
|
callbacks (`list[transformers.TrainerCallback]`):
|
|||
|
|
The callbacks to use for training.
|
|||
|
|
optimizers (`tuple[torch.optim.Optimizer, torch.optim.lr_scheduler.LambdaLR]`):
|
|||
|
|
The optimizer and scheduler to use for training.
|
|||
|
|
preprocess_logits_for_metrics (`Callable[[torch.Tensor, torch.Tensor], torch.Tensor]`):
|
|||
|
|
The function to use to preprocess the logits before computing the metrics.
|
|||
|
|
peft_config (`dict`, defaults to `None`):
|
|||
|
|
The PEFT configuration to use for training. If you pass a PEFT configuration, the model will be wrapped
|
|||
|
|
in a PEFT model.
|
|||
|
|
NzvPEFT is not installed and you passed a `peft_config` in the trainer's kwargs, please install it to use the PEFT modelsÚis_loaded_in_8bitFÚ is_quantizedrí Úuse_gradient_checkpointingzÂYou passed `gradient_checkpointing_kwargs` in the trainer's kwargs, but your peft version does not support it. please update to the latest version of peft to use `gradient_checkpointing_kwargs`.zYA processing_class must be specified when using the default RewardDataCollatorWithPadding)rÊ z°When using RewardDataCollatorWithPadding, you should set `remove_unused_columns=False` in your RewardConfig we have set it for you, but you should do it yourself in the future.TÚestimate_tokensÚinput_ids_chosenÚ tokenizer)Ú fn_kwargs)Úbatchedr4 Únum_procc óh •— t | d ¦ « ‰k ot | d ¦ « ‰k S ©Nr2 Úinput_ids_rejected©Úlen©Úxr s €r_ ú<lambda>z0_UnslothRewardTrainer.__init__.<locals>.<lambda> s3 ø€ �c !Ð$6Ô"7Ñ 8Ô 8¸JÒ FÐ uÍ3ÈqÐQeÔOfÑKgÔKgÐkuÒKu€ ra )r6 )r4 r5 r6 c óh •— t | d ¦ « ‰k ot | d ¦ « ‰k S r8 r: r< s €r_ r> z0_UnslothRewardTrainer.__init__.<locals>.<lambda> s9 ø€ ¥# aÐ(:Ô&;Ñ"<Ô"<À
|
|||
|
|
Ò"Jð #GÝ Ð"6Ô 7Ñ 8Ô 8¸JÒ Fð ra ) r! r" r# r$ r% r& r' r( r) r* r+ Úadd_model_tags)'r' Ú
|
|||
|
|
ValueErrorÚ
|
|||
|
|
isinstancer ÚgetattrÚlistr& Ú signaturer0 Ú
|
|||
|
|
parametersrì rí r4 ÚwarnÚ UserWarningr r" r r r rÊ r r2 Úuse_reward_data_collatorÚwarnings_issuedÚ column_namesr Úmain_process_firstÚmapr+ r r Úfilterr r Úhasattrr! r@ Ú
|
|||
|
|
_tag_names)r r! r" r# r$ r% r& r' r( r) r* r+ r, Ú_supports_gc_kwargsÚprepare_model_kwargsr4 r r s @€r_ r z _UnslothRewardTrainer.__init__x s5 øø€ õp !Ñ"Ô"ð {Ð'>Ý ð Iñô ð
|
|||
|
|
õ Ñ
|
|||
|
|
Ô
|
|||
|
|
ð [Ð%<Ý ˜e¥YÑ/Ô/ð
|
|||
|
|
ݘ5Ð"5°uÑ=Ô=ð [ÅÈÐP^Ð`eÑAfÔAfð [Ø*IÍTÝ Ô)Õ*IÑJÔJÔUñNô Nð +Ð'ð -IÈ$ÔJeÐ+fÐ(à.ð s°4Ô3UÐ3aÝ œ
|
|||
|
|
ð rå 'ñô ð ð ð
|
|||
|
|
-ð s°Ô1SÐ1_ØPTÔPrÐ,Ð-LÑMå ;¸EÐ ZÐ ZÐEYÐ ZÐ Z�Eà �ð Ô ð ,Ý $ UÑ +Ô +Ð +à Ð "Ý .ˆOà Ð Ø Ð'Ý Øoñô ð ð œ ˆJå 9Ð:JÑ KÔ KˆMàÔ)ð
|
|||
|
|
|
|||
|
|
ðFØ16�DÔ.Ð.øÝ*ð Fð Fð FÝ" 4¸uÐEÑEÔE�D�D�DðFøøøõ ”
|
|||
|
|
ð\å ñô ð ð -1ˆDÔ )Ð )à,1ˆDÔ )ð 48ˆÔ Ð /Ñ0à ]Ô%?Ð ?Ð ?Ý ‘ ” ×2Ò2Ñ4Ô4ð !
|
|||
|
|
ð !
|
|||
|
|
Ø (Ð*:Ð ;� Ø -× 1Ò 1Õ2KÐXcÐeuÐWvÐ 1Ñ wÔ w�
|
|||
|
|
Ø -× 1Ò 1Ý Ø Ø 'Ø !Ô 2ð !2ñ !ô !�
|
|||
|
|
ð !.× 4Ò 4ØuÐuÐuÐuØ !Ô 2ð !5ñ !ô !�
|
|||
|
|
ð Ð+Ø#/×#3Ò#3Ý1¸kÐK[Ð=\ð $4ñ $ô $�Lð $0×#3Ò#3Ý!Ø"+Ø $Ø!%Ô!6ð $4ñ $ô $�Lð $0×#6Ò#6ðGð Gð Gð Gà!%Ô!6ð $7ñ $ô $�Lð;!
|
|||
|
|
ð !
|
|||
|
|
ð !
|
|||
|
|
ñ !
|
|||
|
|
ô !
|
|||
|
|
ð !
|
|||
|
|
ð !
|
|||
|
|
ð !
|
|||
|
|
ð !
|
|||
|
|
ð !
|
|||
|
|
ð !
|
|||
|
|
øøøð !
|
|||
|
|
ð !
|
|||
|
|
ð !
|
|||
|
|
ð !
|
|||
|
|
õF ‰Œ×ÒØØØ'Ø'Ø%Ø -Ø!Ø +Ø Ø!Ø*Gð ñ
|
|||
|
|
ô
|
|||
|
|
ð
|