307 lines
65 KiB
Plaintext
307 lines
65 KiB
Plaintext
|
§
|
|||
|
|
5$�h ã ó” — d Z ddlmZ ddlZddlmZ ddlmZ ddlmZm Z m
|
|||
|
|
Z
|
|||
|
|
m Z m Z m
|
|||
|
|
Z
|
|||
|
|
mZmZ ddlmZmZmZmZmZmZmZmZmZmZmZm Z m Z m
|
|||
|
|
Z
|
|||
|
|
m Z m Z m Z m!Z!m"Z"m#Z#m$Z$m%Z%m&Z&m'Z'm(Z(m)Z)m Z m*Z*m+Z+m,Z,m-Z-m.Z.m/Z/m0Z0m1Z1m2Z2m3Z3m4Z4m5Z5m6Z6m7Z7m8Z8m9Z9m:Z:m;Z;m<Z<m=Z=m>Z>m?Z?m@Z@mAZAmZmBZBmCZCmDZDmEZEmFZFmGZGmHZHmIZImJZJmKZKmZmLZLmMZMm
|
|||
|
|
Z
|
|||
|
|
m"Z"m'Z'm;Z;mDZDmZ ddlDZDddlT ddlNmOZOmPZP dd lQmRZR ddlZddlSZBdd
|
|||
|
|
lTmCZC ddlmZ dd lUmVZVmWZX d d
|
|||
|
|
d d
|
|||
|
|
d
|
|||
|
|
dœZY ejZ d d eY¬¦ « d„ ¦ « Z[eO G d„ de ¦ « ¦ « Z\ G d„ de'¦ « Z] G d„ de]¦ « Z^dS )z8
|
|||
|
|
2025.8.4
|
|||
|
|
2025.8.5
|
|||
|
|
4.55.1
|
|||
|
|
0.21.0
|
|||
|
|
__UNSLOTH_VERSIONING__
|
|||
|
|
é )ÚTensorN)Ú
|
|||
|
|
functional)ÚAnyÚListÚOptionalÚTupleÚUnionÚDictÚSetÚCallable)GÚ AcceleratorÚBaseImageProcessorÚCallbackHandlerÚDEFAULT_CALLBACKSÚDEFAULT_PROGRESS_CALLBACKÚDataCollatorWithPaddingÚ
|
|||
|
|
DataLoaderÚDatasetÚExportableStateÚFeatureExtractionMixinÚGenerationConfigÚINVALID_LOGPROBÚOnlineTrainerStater Ú PPOConfigÚ
|
|||
|
|
PPOTrainerÚPathÚ
|
|||
|
|
PeftConfigÚ PeftModelÚPolicyAndValueWrapperÚPreTrainedTokenizerBaseÚPrinterCallbackÚProcessorMixinÚTrainerÚTrainerCallbackÚTrainerControlr Úbatch_generationÚ broadcastÚcontextmanagerÚcreate_reference_modelÚ defaultdictÚdisable_dropout_in_modelÚ empty_cacheÚ exact_divÚfirst_true_indicesÚforwardÚ
|
|||
|
|
gather_objectÚgcÚgenerate_model_cardÚget_comet_experiment_urlÚget_peft_modelÚ#get_reporting_integration_callbacksÚ
|
|||
|
|
get_rewardÚis_peft_availableÚis_rich_availableÚis_wandb_availableÚ log_table_to_comet_experimentÚ masked_meanÚ
|
|||
|
|
masked_whitenÚmathÚnnÚnpÚ nullcontextÚosÚpdÚpeft_module_casting_to_bf16Úprepare_deepspeedÚprint_rich_tableÚselective_log_softmaxÚtextwrapÚtimeÚtorchÚtruncate_responseÚunwrap_model_for_generationr r r# r7 rA rI )Ú*)Ú dataclassÚfield)ÚVersion)r@ )ÚDataCollatorForSeq2SeqÚ DataCollatorForLanguageModelingTF)Úepilogue_fusionÚ max_autotuneÚ
|
|||
|
|
shape_paddingz
|
|||
|
|
trace.enabledztriton.cudagraphs)ÚdynamicÚ fullgraphÚoptionsc ó’ — t j | d| j d ¦ « dd¬¦ « }t j | d¦ « dd¬¦ « }g }t ||¦ « D ]‘\ }}| t j ¦ « }t j |d| d¦ « ¬¦ « d¦ « }t j
|
|||
|
|
|d¬¦ « }||z
|
|||
|
|
} | | ¦ « Œ’ t j |¦ « }| | j d | j d f¦ « }|S )Néÿÿÿÿé r )ÚchunksÚdim)r\ Úindex©r\ é )
|
|||
|
|
rI ÚchunkÚreshapeÚshapeÚzipÚtoÚfloat32ÚgatherÚ unsqueezeÚsqueezeÚ logsumexpÚappendÚconcat)
|
|||
|
|
Úlogitsr] Úchunked_logitsÚ
|
|||
|
|
chunked_indexÚall_per_token_logpsÚ chunk_logitsÚ chunk_indexÚselected_logitsÚlogsumexp_valuesÚper_token_logpss
|
|||
|
|
ú]/workspace/Fine-tuning/DS-LLM-TEMPLATE-FINETUNING/unsloth_compiled_cache/UnslothPPOTrainer.pyÚ chunked_selective_log_softmaxrv " s5 € õ ”[ § ¢ °°F´LÀÔ4DÑ!EÔ!EÐPQÐYZÐ[Ñ[Ô[€NÝ”[ § ¢ ¨rÑ!2Ô!2¸QÀaÐHÑHÔH€MØ Ðå%(¨ ¸ Ñ%GÔ%Gð 4ð 4Ñ!ˆ �kØ#—’¥u¤}Ñ5Ô5ˆ Ý œ, |¸2À{×G\ÒG\Ð]_ÑG`ÔG`ÐaÑaÔa×iÒiÐjlÑmÔmˆÝ œ?¨<¸rÐBÑBÔBÐØ)Ð,<Ñ<ˆØ×"Ò" ?Ñ3Ô3Ð3Ð3ØÝ œ,Ð':Ñ;Ô;ÐØ-×5Ò5°v´|ÀA´ÈÌ ÐUVÌÐ6XÑYÔYÐØ Ð ó c óÒ ‡ — e Zd ZU dZ edddi¬¦ « Zee ed< edddi¬¦ « Z ee
|
|||
|
|
ed < d4ˆ fd3„ Z ˆ xZ S )5ÚUnslothPPOConfigaþ
|
|||
|
|
|
|||
|
|
|
|||
|
|
Configuration class for the [`PPOTrainer`].
|
|||
|
|
|
|||
|
|
This class includes only the parameters that are specific to PPO training. For a full list of training arguments,
|
|||
|
|
please refer to the [`~transformers.TrainingArguments`] and [`OnPolicyConfig`] documentation. Note that default
|
|||
|
|
values in this class may differ from those in [`~transformers.TrainingArguments`].
|
|||
|
|
|
|||
|
|
Using [`~transformers.HfArgumentParser`] we can turn this class into
|
|||
|
|
[argparse](https://docs.python.org/3/library/argparse#module-argparse) arguments that can be specified on the
|
|||
|
|
command line.
|
|||
|
|
|
|||
|
|
Parameters:
|
|||
|
|
exp_name (`str`, *optional*, defaults to `os.path.basename(__file__)[:-3]`):
|
|||
|
|
Name of this experiment.
|
|||
|
|
reward_model_path (`str`, *optional*, defaults to `"EleutherAI/pythia-160m"`):
|
|||
|
|
Path to the reward model.
|
|||
|
|
model_adapter_name (`str` or `None`, *optional*, defaults to `None`):
|
|||
|
|
Name of the train target PEFT adapter, when using LoRA with multiple adapters.
|
|||
|
|
ref_adapter_name (`str` or `None`, *optional*, defaults to `None`):
|
|||
|
|
Name of the reference PEFT adapter, when using LoRA with multiple adapters.
|
|||
|
|
num_ppo_epochs (`int`, *optional*, defaults to `4`):
|
|||
|
|
Number of epochs to train.
|
|||
|
|
whiten_rewards (`bool`, *optional*, defaults to `False`):
|
|||
|
|
Whether to whiten the rewards.
|
|||
|
|
kl_coef (`float`, *optional*, defaults to `0.05`):
|
|||
|
|
KL coefficient.
|
|||
|
|
kl_estimator (`Literal["k1", "k3"]`, *optional*, defaults to `"k1"`):
|
|||
|
|
Which estimator for KL-Divergence to use from [Approximating KL
|
|||
|
|
Divergence](http://joschu.net/blog/kl-approx.html). Defaults to "k1", a straightforward, unbiased
|
|||
|
|
estimator. Can be set to "k3", an unbiased estimator with lower variance which "appears to be a strictly
|
|||
|
|
better estimator". Cannot be set to "k2", as it is used for logging purposes.
|
|||
|
|
cliprange (`float`, *optional*, defaults to `0.2`):
|
|||
|
|
Clip range.
|
|||
|
|
vf_coef (`float`, *optional*, defaults to `0.1`):
|
|||
|
|
Value function coefficient.
|
|||
|
|
cliprange_value (`float`, *optional*, defaults to `0.2`):
|
|||
|
|
Clip range for the value function.
|
|||
|
|
gamma (`float`, *optional*, defaults to `1.0`):
|
|||
|
|
Discount factor.
|
|||
|
|
lam (`float`, *optional*, defaults to `0.95`):
|
|||
|
|
Lambda value for GAE.
|
|||
|
|
ds3_gather_for_generation (`bool`, *optional*, defaults to `True`):
|
|||
|
|
This setting applies to DeepSpeed ZeRO-3. If enabled, the policy model weights are gathered for generation,
|
|||
|
|
improving generation speed. However, disabling this option allows training models that exceed the VRAM
|
|||
|
|
capacity of a single GPU, albeit at the cost of slower generation.
|
|||
|
|
|
|||
|
|
NÚhelpzvLLM SamplingParams)ÚdefaultÚmetadataÚvllm_sampling_paramsrY z8Chunk size to reduce memory usage. -1 is most efficient.Úunsloth_num_chunksFÚnorZ é r éú ç-C ëâ6
|
|||
|
|
?ç{®Gáz„?çÍÌÌÌÌÌì?ç+‡ÙÎ÷ï?ç:Œ0âŽyE>ç ð?ç @Úlinearçš™™™™™¹?ÚpassiveÚwarningTÚstepsr_ éô éO
|
|||
|
|
ÚO1ÚautoÚ ç Ú
|
|||
|
|
adamw_8bitÚlengthÚ
|
|||
|
|
every_saveÚlasté é@ é
|
|||
|
|
é5 çffffffæ?úEleutherAI/pythia-160mÚ
|
|||
|
|
ppo_configçš™™™™™©?Úk1çš™™™™™É?çffffffî?c£ ó8 •— |dk rt d|› d�¦ « ‚|dk rt d|› d�¦ « ‚|€|#dk r
|
|||
|
|
|$dk rd}d }#|�€!d
|
|||
|
|
d lm}¤ t |¤¦ « d z d ¦ « }�|‰d
|
|||
|
|
k rt d
|
|||
|
|
¦ « ‚|‰dk rt d¦ « ‚ t
|
|||
|
|
¦ « j d°i d|“d|“d|“d|“d|“d|“d|“d|“d| “d|
|
|||
|
|
“d| “d| “d |
|
|||
|
|
“d |“d |“d |“d |“d!|“d"|“d#|“d$|“d%|“d&|“d'|“d(|“d)|“d*|“d+| “d,| “d-| “d.| “d/| “d0|!“d1|"“d2|#“d3|$“d4|%“d5|&“d6|'“d7|(“d8|)“d9|*“d:|+“d;|,“d<|-“d=|.“d>|/“d?|0“d@|1“dA|2“dB|3“dC|4“dD|5“dE|6“dF|7“dG|8“dH|9“dI|:“dJ|;“dK|<“dL|=“dM|>“dN|?“dO|@“dP|A“dQ|B“dR|C“dS|D“dT|E“dU|F“dV|G“dW|H“dX|I“dY|J“dZ|K“d[|L“d\|M“d]|N“d^|O“d_|P“d`|Q“da|R“db|S“dc|T“dd|U“de|V“df|W“dg|X“dh|Y“di|Z“dj|[“dk|\“dl|]“dm|^“dn|_“do|`“dp|a“dq|b“dr|c“ds|d“dt|e“du|f“dv|g“dw|h“dx|i“dy|j“dz|k“d{|l“d||m“d}|n“d~|o“d|p“d€|q“d�|r“d‚|s“dƒ|t“d„|u“d…|v“d†|w“d‡|x“dˆ|y“d‰|z“dŠ|{“d‹||“dŒ|}“d�|~“dŽ|“d�|€“d�|�“d‘|‚“d’|ƒ“d“|„“d”|…“d•|†“d–|‡“d—|ˆ“d˜|‰“d™|Š“dš|‹“d›|Œ“dœ|�“d�|Ž“dž|�“dŸ|�“d |‘“d¡|’“d¢|““d£|”“d¤|•“d¥|–“d¦|—“d§|˜“d¨|™“d©|š“dª|›“d«|œ“d¬|�“d|ž“d®|Ÿ“d¯| “|£¤Ž |¡| _ |¢| _ d S )±NçH¯¼šò×z>z Unsloth: Your learning rate of `zi` is too small and less than 1e-7! Consider increasing it, otherwise gradient updates will be close to 0!r_ za` is way too larger > 1! Consider decreasing it to 1e-1, otherwise gradient updates will explode!r� rŽ Ú unsloth_training_checkpointsr r )Ú cpu_countr€ zUUnsloth: Please set a positive non-zero temperature since your results will be wrong.rš zgUnsloth: Please set a positive non-zero temperature less than 10, since sampling will be quite erratic.Ú
|
|||
|
|
output_dirÚoverwrite_output_dirÚdo_trainÚdo_evalÚ
|
|||
|
|
do_predictÚ
|
|||
|
|
eval_strategyÚprediction_loss_onlyÚper_device_train_batch_sizeÚper_device_eval_batch_sizeÚper_gpu_train_batch_sizeÚper_gpu_eval_batch_sizeÚgradient_accumulation_stepsÚeval_accumulation_stepsÚ
|
|||
|
|
eval_delayÚtorch_empty_cache_stepsÚ
|
|||
|
|
learning_rateÚ weight_decayÚ
|
|||
|
|
adam_beta1Ú
|
|||
|
|
adam_beta2Ú adam_epsilonÚ
|
|||
|
|
max_grad_normÚnum_train_epochsÚ max_stepsÚlr_scheduler_typeÚ warmup_ratioÚ warmup_stepsÚ log_levelÚlog_level_replicaÚlog_on_each_nodeÚ logging_dirÚlogging_strategyÚlogging_first_stepÚ
|
|||
|
|
logging_stepsÚlogging_nan_inf_filterÚ
|
|||
|
|
save_strategyÚ
|
|||
|
|
save_stepsÚsave_total_limitÚsave_safetensorsÚsave_on_each_nodeÚsave_only_modelÚ'restore_callback_states_from_checkpointÚno_cudaÚuse_cpuÚuse_mps_deviceÚseedÚ data_seedÚ
|
|||
|
|
jit_mode_evalÚuse_ipexÚbf16Úfp16Úfp16_opt_levelÚhalf_precision_backendÚbf16_full_evalÚfp16_full_evalÚtf32Ú
|
|||
|
|
local_rankÚ ddp_backendÚ
|
|||
|
|
tpu_num_coresÚtpu_metrics_debugÚdebugÚdataloader_drop_lastÚ
|
|||
|
|
eval_stepsÚdataloader_num_workersÚdataloader_prefetch_factorÚ
|
|||
|
|
past_indexÚrun_nameÚ disable_tqdmÚremove_unused_columnsÚ label_namesÚload_best_model_at_endÚmetric_for_best_modelÚgreater_is_betterÚignore_data_skipÚfsdpÚfsdp_min_num_paramsÚ fsdp_configÚ"fsdp_transformer_layer_cls_to_wrapÚaccelerator_configÚ deepspeedÚlabel_smoothing_factorÚoptimÚ
|
|||
|
|
optim_argsÚ adafactorÚgroup_by_lengthÚlength_column_nameÚ report_toÚddp_find_unused_parametersÚddp_bucket_cap_mbÚddp_broadcast_buffersÚdataloader_pin_memoryÚ dataloader_persistent_workersÚskip_memory_metricsÚuse_legacy_prediction_loopÚ push_to_hubÚresume_from_checkpointÚ hub_model_idÚ hub_strategyÚ hub_tokenÚhub_private_repoÚhub_always_pushÚ hub_revisionÚgradient_checkpointingÚ gradient_checkpointing_kwargsÚinclude_inputs_for_metricsÚeval_do_concat_batchesÚ fp16_backendÚpush_to_hub_model_idÚpush_to_hub_organizationÚpush_to_hub_tokenÚ
|
|||
|
|
mp_parametersÚauto_find_batch_sizeÚfull_determinismÚ torchdynamoÚ ray_scopeÚ ddp_timeoutÚ
|
|||
|
|
torch_compileÚtorch_compile_backendÚtorch_compile_modeÚinclude_tokens_per_secondÚ include_num_input_tokens_seenÚneftune_noise_alphaÚoptim_target_modulesÚbatch_eval_metricsÚ
|
|||
|
|
eval_on_startÚuse_liger_kernelÚliger_kernel_configÚeval_use_gather_objectÚ average_tokens_across_devicesÚdataset_num_procÚnum_mini_batchesÚtotal_episodesÚ local_rollout_forward_batch_sizeÚnum_sample_generationsÚresponse_lengthÚ
|
|||
|
|
stop_tokenÚ
|
|||
|
|
stop_token_idÚ temperatureÚmissing_eos_penaltyÚsft_model_pathÚ
|
|||
|
|
world_sizeÚnum_total_batchesÚmicro_batch_sizeÚlocal_batch_sizeÚ
|
|||
|
|
batch_sizeÚlocal_mini_batch_sizeÚmini_batch_sizeÚexp_nameÚreward_model_pathÚmodel_adapter_nameÚref_adapter_nameÚnum_ppo_epochsÚwhiten_rewardsÚkl_coefÚ kl_estimatorÚ cliprangeÚvf_coefÚcliprange_valueÚgammaÚlamÚds3_gather_for_generation© )
|
|||
|
|
ÚFloatingPointErrorÚ
|
|||
|
|
OverflowErrorÚmultiprocessingr¦ ÚminÚ MathErrorÚsuperÚ__init__r} r~ )¦Úselfr§ r¨ r© rª r« r¬ r r® r¯ r° r± r² r³ r´ rµ r¶ r· r¸ r¹ rº r» r¼ r½ r¾ r¿ rÀ rÁ r rà rÄ rÅ rÆ rÇ rÈ rÉ rÊ rË rÌ rÍ rÎ rÏ rÐ rÑ rÒ rÓ rÔ rÕ rÖ r× rØ rÙ rÚ rÛ rÜ rÝ rÞ rß rà rá râ rã rä rå ræ rç rè ré rê rë rì rí rî rï rð rñ rò ró rô rõ rö r÷ rø rù rú rû rü rý rþ rÿ r r r r r r r r r r r
|
|||
|
|
r r r
|
|||
|
|
r r r r r r r r r r r r r r r r r r r r! r" r# r$ r% r& r' r( r) r* r+ r, r- r. r/ r0 r1 r2 r3 r4 r5 r6 r7 r8 r9 r: r; r< r= r> r? r@ rA rB rC rD rE rF r} r~ Úkwargsr¦ Ú __class__s¦ €ru rN zUnslothPPOConfig.__init__m sz ø€ ðL ˜4Ò Ð Õ'9ð ;VÐ]jð ;Vð ;Vð ;Vñ (Wô (Wð "WØ ˜1Ò Ð ¥Mð 3FÐUbð 3Fð 3Fð 3Fñ %Gô %Gð GØ Ð -°7Ò":Ð":¸zÈSÒ?PÐ?PØ7ˆJØ ˆMØ Ð #Ø 1Ð 1Ð 1Ð 1Ð 1Ð 1Ý " 9 9¡;¤;¨q¡=°!Ñ 4Ô 4Ð Ø ˜!Ò Ð ÝÐ sÑtÔtÐ tØ
|
|||
|
|
˜BÒ
|
|||
|
|
Ð
|
|||
|
|
Ýð Fñ Gô Gð
|
|||
|
|
Gð �‰ŒÔð ` Lð ` Lð ` LØ#˜ð` Là#7Ð#7ð` Lð �xð` Lð �gð ` Lð
|
|||
|
|
$˜ð ` Lð *˜Mð
|
|||
|
|
` Lð $8Ð#7ð` Lð +FÐ*Eð` Lð *DÐ)Cð` Lð (@Ð'?ð` Lð '>Ð&=ð` Lð +FÐ*Eð` Lð '>Ð&=ð` Lð $˜ð ` Lð '>Ð&=ð ` Lð *˜Mð!` Lð" (˜<ð#` Lð$ $˜ð%` Lð& $˜ð'` Lð( (˜<ð)` Lð* *˜Mð+` Lð, 0Ð /ð-` Lð. "˜ ð/` Lð0 !2Ð 1ð1` Lð2 (˜<ð3` Lð4 (˜<ð5` Lð6 "˜ ð7` Lð8 !2Ð 1ð9` Lð: 0Ð /ð;` Lð< &˜+ð=` Lð> 0Ð /ð?` Lð@ "4Ð!3ðA` LðB *˜MðC` LðD &<Ð%;ðE` LðF *˜MðG` LðH $˜ðI` LðJ 0Ð /ðK` LðL 0Ð /ðM` LðN !2Ð 1ðO` LðP .˜oðQ` LðR 7^Ð6]ðS` LðT �gðU` LðV �gðW` LðX ,˜^ðY` LðZ �4ð[` Lð\ "˜ ð]` Lð^ *˜Mð_` Lð` �xða` Lðb �4ðc` Lðd �4ðe` Lðf ,˜^ðg` Lðh &<Ð%;ði` Lðj ,˜^ðk` Lðl ,˜^ðm` Lðn �4ðo` Lðp $˜ðq` Lðr &˜+ðs` Lðt *˜Mðu` Lðv !2Ð 1ðw` Lðx �Eðy` Lðz $8Ð#7ð{` Lð| $˜ð}` Lð~ &<Ð%;ð` Lð@ *DÐ)CðA` LðB $˜ðC` LðD �xðE` LðF (˜<ðG` LðH %:Ð$9ðI` LðJ &˜+ðK` LðL &<Ð%;ðM` LðN %:Ð$9ðO` LðP !2Ð 1ðQ` LðR 0Ð /ðS` LðT �4ðU` LðV #6Ð"5ðW` LðX &˜+ðY` LðZ 2TÐ1Sð[` Lð\ "4Ð!3ð]` Lð^ "˜ ð_` Lð` &<Ð%;ða` Lðb �Eðc` Lðd $˜ðe` Lðf "˜ ðg` Lðh .˜oði` Lðj "4Ð!3ðk` Lðl "˜ ðm` Lðn *DÐ)Cðo` Lðp !2Ð 1ðq` Lðr %:Ð$9ðs` Lðt %:Ð$9ðu` Lðv -JÐ,Iðw` Lðx #6Ð"5ðy` Lðz *DÐ)Cð{` Lð| &˜+ð}` Lð~ &<Ð%;ð` Lð@ (˜<ðA` LðB (˜<ðC` LðD "˜ ðE` LðF 0Ð /ðG` LðH .˜oðI` LðJ (˜<ðK` LðL &<Ð%;ðM` LðN -JÐ,IðO` LðP *DÐ)CðQ` LðR &<Ð%;ðS` LðT (˜<ðU` LðV $8Ð#7ðW` LðX (@Ð'?ðY` LðZ !2Ð 1ð[` Lð\ *˜Mð]` Lð^ $8Ð#7ð_` Lð` 0Ð /ða` Lðb &˜+ðc` Lðd "˜ ðe` Lðf &˜+ðg` Lðh *˜Mði` Lðj %:Ð$9ðk` Lðl "4Ð!3ðm` Lðn )BÐ(Aðo` Lðp -JÐ,Iðq` Lðr #6Ð"5ðs` Lðt $8Ð#7ðu` Lðv "4Ð!3ðw` Lðx *˜Mðy` Lðz 0Ð /ð{` Lð| #6Ð"5ð}` Lð~ &<Ð%;ð` Lð@ -JÐ,IðA` LðB 0Ð /ðC` LðD 0Ð /ðE` LðF ,˜^ðG` LðH 0PÐ/OðI` LðJ &<Ð%;ðK` LðL .˜oðM` LðN $˜ðO` LðP *˜MðQ` LðR &˜+ðS` LðT #6Ð"5ðU` LðV ,˜^ðW` LðX $˜ðY` LðZ !2Ð 1ð[` Lð\ 0Ð /ð]` Lð^ 0Ð /ð_` Lð` $˜ða` Lðb %:Ð$9ðc` Lðd .˜oðe` Lðf �xðg` Lðh !2Ð 1ði` Lðj "4Ð!3ðk` Lðl 0Ð /ðm` Lðn ,˜^ðo` Lðp ,˜^ðq` Lðr �gðs` Lðt (˜<ðu` Lðv "˜ ðw` Lðx �gðy` Lðz .˜oð{` Lð| �Eð}` Lð~ �#ð` Lð@ )BÐ(AÀFðA` Lð ` Lð ` LðB %9ˆÔ!Ø"4ˆÔ Ð Ð rw )¢NNFFFr FrZ rZ NNr€ r€ r r� r‚ rƒ r„ r… r† r‡ rˆ rY r‰ rŠ r r‹ rŒ TNr� Fr_ Fr� rŽ NTFFFFFFr� r� FFFFr� r‘ FFNrY NNFr’ FNr NrY NNTNFNNFr’ r NNNNr“ r” NFFr• NNNNTFTFFNNr– NNFNFNFTr‘ NNNr’ TFNr— r˜ FNNFFNNFFFNFTNr_ Nr™ rš r› NNrœ Nr� NNNNNNNrž r� NNrZ FrŸ r r¡ rŠ r¡ r‡ r¢ TNrY )
|
|||
|
|
Ú__name__Ú
|
|||
|
|
__module__Ú __qualname__Ú__doc__rN r} r r Ú__annotations__r~ ÚintrN Ú
|
|||
|
|
__classcell__©rQ s @ru ry ry 3 s± ø€ € € € € € ð/ð /ð` +0¨%ØØÐ 1Ð2ð+ñ +ô +И( 3œ-ð ð ñ ð */¨ØØÐ VÐWð*ñ *ô *И #œ ð ð ñ ð Ø #ØØØØ Ø $Ø&'Ø%&Ø#'Ø"&Ø&'Ø"#ØØ"%Ø ØØØØ ØØ ØØ $ØØØ Ø %Ø ØØ"Ø "ØØ!&Ø ØØ Ø Ø !Ø Ø27ØØØ ØØØ ØØØØ Ø!'Ø Ø ØØØØ Ø !ØØ $ØØ!"Ø%)ØØØØ $ØØ!&Ø $Ø Ø ØØ ØØ-1Ø !ØØ!$Ø ØØØ Ø %ØØ%)Ø Ø $Ø $Ø(-Ø "Ø%*ØØ!%ØØ#ØØ Ø ØØ!&Ø(,Ø%*Ø!%Ø Ø #Ø#'Ø ØØ #Ø ØØØØ Ø $Ø !Ø$)Ø(-Ø "Ø #Ø "Ø Ø Ø "Ø!&Ø(,Ø Ø Ø Ø+-Ø!#Ø ØØ ØØ "Ø1ØØ Ø Ø ØØ $Ø Ø Ø 4Ø !Ø ØØ ØØØØØ ØØØ$(Ø #Ø ðGV5ð V5ð V5ð V5ð V5ð V5ð V5ð V5ð V5ð V5rw ry c ó ‡ — e Zd ZddgZ d#dedeeeee e
|
|||
|
|
f de j dee j d e j d
|
|||
|
|
e
|
|||
|
|
d e j d ee d
|
|||
|
|
eee
|
|||
|
|
eee
|
|||
|
|
f f deej j ej j j f deee ded ddfd„Zdefd„Zdefd„Z e d„ ¦ « Z d$dee de fˆ fd„
|
|||
|
|
Z d„ Z!d%d e fd „Z"ˆ fd „Z# d&d ee d ee d!eeee df fd"„Z$ˆ xZ%S )'Ú_UnslothPPOTrainerÚtrlÚppoN©NNÚargsÚprocessing_classÚmodelÚ ref_modelÚ reward_modelÚ
|
|||
|
|
train_datasetÚ value_modelÚ
|
|||
|
|
data_collatorÚ eval_datasetÚ
|
|||
|
|
optimizersÚ callbacksÚ peft_configr Úreturnc
|
|||
|
|
ó — ||u rt d¦ « ‚|| _ || _ || _ |€t | j ¦ « }|j r|j rt d¦ « ‚|j rA|j dk r |j x| j j _ | _ n5t d|j › d�¦ « ‚|j x| j j _ | _ | j j dvrt d¦ « ‚t ¦ « s| �t d¦ « ‚t ¦ « r…| �ƒt | j t ¦ « r | j ¦ « | _ t | j | ¦ « | _ |j r*t# | j d d
|
|||
|
|
¦ « rt% | j ¦ « t ¦ « ot | j t ¦ « | _ |j | _ |j | _ |r|| _ n(| j rd | _ nt/ | j ¦ « | _ || _ || _ t5 |¦ « | _ || _ || _ | | _ |
|
|||
|
|
\ | _ | _ d | _! |j" €!tG |j$ | j z ¦ « |_" tK |j& ¬ ¦ « }
|
|||
|
|
|
|
|||
|
|
| _' |
|
|||
|
|
j( |_) |j* |j& z |_+ tG |j* |j) z ¦ « |_, tG |j+ |j) z ¦ « |_- t] |j- |j/ d ¦ « |_0 t] |j+ |j/ d
|
|||
|
|
¦ « |_1 |j2 r |j1 dk sJ d|j1 › d�¦ « ‚tg j4 |j" |j- z ¦ « |_5 tm j7 tG tq j8 ¦ « ¦ « |
|
|||
|
|
j9 ¬¦ « }tu |d¦ « ; ¦ « }|j< › d|j= › d|› �|_> |j= |
|
|||
|
|
j? dz z | _@ |jA dk r"t… d|j5 |jA z ¦ « | _C |j+ | _D | j | j | j | j fD ]}|�t‹ |¦ « Œt� | j | j ¦ « | _G | j jH | jG _H | I |j5 ¬¦ « t” t— | j jL ¦ « z }| €|n|| z | _M t� | jM | jG | j | j | j ¦ « | _O | P | j jQ rt¤ nt¦ ¦ « t© ¦ « | _U t | W ¦ « | X ¦ « d„ | jO jM | jU gz D ¦ « ¬¦ « | _Y d| _Z d | _[ t# | j' jY dd ¦ « d u| _\ t# | j' jY dd ¦ « d u| _] d | _^ | j j_ r| ` ¦ « | j ja r tÅ jc | j jd d¬ ¦ « tË | jG d ¦ « r | jG f | jg ¦ « tÑ | j | jD d| j d¬ ¦ « | _i tm jj |j= ¦ « |
|
|||
|
|
k | jG | j | ji ¦ « \ | _G | _ | _i tm jj | j@ ¦ « tÑ | j |jl | j d¬ ¦ « | _m |
|
|||
|
|
k | jm ¦ « | _m | j\ rwtÝ | j |j* |jo |j ¦ « | _ | j €| j st d ¦ « ‚d S tÝ | j |j* |jo |j ¦ « | _ d S | j €| j st d ¦ « ‚n)| j p | j' j9 ¦ « | _ | j p | j' j9 ¦ « | _ d S )!Nzœ`model` and `ref_model` cannot be the same object. If you want `ref_model` to be the same as `model`, you must make a copy of it, or `None` if you use peft.z5You cannot set both `stop_token` and `stop_token_id`.ÚeoszUnknown `stop_token` z9. Allowed values are: `'eos'` and `None` (no stop token).> r Úk3zákl_estimator must be either 'k1' (straightforward, unbiased) or 'k3' (lower variance, unbiased, appears to be a strictly better estimator). See [Approximating KL Divergence](http://joschu.net/blog/kl-approx.html) for details.zvPEFT is not installed and you passed a `peft_config` in the trainer's kwargs, please install it to use the PEFT modelsÚis_loaded_in_4bitF)r² z5`batch_size` must be a multiple of `num_mini_batches`z;`local_batch_size` must be a multiple of `num_mini_batches`é zPer-rank minibatch size z is insufficient for whitening©Údevicer Ú__i£† r_ )Únum_training_stepsc ó< — g | ]
|
|||
|
|
isinstancer )Ú.0Úcbs ru ú
|
|||
|
|
<listcomp>z/_UnslothPPOTrainer.__init__.<locals>.<listcomp>\ s: € ð ð ð ØÕQ[Ð\^Õ`oÑQpÔQpð Øð ð ð rw )Úis_local_process_zeroÚis_world_process_zeroÚstateful_callbacksÚdeepspeed_pluginÚ fsdp_pluginT)Úexist_okÚadd_model_tags)r6 ÚshuffleÚ
|
|||
|
|
collate_fnÚ drop_last)r6 r‚ rƒ z1No reference model and model is not a Peft model.)qÚ
|
|||
|
|
ValueErrorr_ r` Ú policy_modelr r- r. Ú eos_token_idÚgeneration_configr@ r7 Ú ImportErrorrv r Úmerge_and_unloadr4 r× ÚgetattrrC Ú
|
|||
|
|
is_peft_modelr; r< rb r) rc rd ÚlenÚtrain_dataset_lenre rf rg Ú optimizerÚ lr_schedulerÚoptimizer_cls_and_kwargsr) rW r¼ r
|
|||
|
|
r² Ú acceleratorÚ
|
|||
|
|
num_processesr2 r® r5 r4 r6 r- r( r8 r7 r> r= Úceilr3 rI ÚtensorrH rr r' Úitemr9 rÓ rè Ú
|
|||
|
|
process_indexÚ
|
|||
|
|
local_seedr+ ÚmaxÚsample_generations_freqÚlocal_dataloader_batch_sizer+ r ra ÚconfigÚ create_optimizer_and_schedulerr r5 rü ri r Úcallback_handlerÚ add_callbackré r! r r% Úcontrolr rz r{ ÚstateÚ current_flosÚhp_search_backendÚis_deepspeed_enabledÚis_fsdp_enabledr r Ú init_hf_repoÚ should_saverA Úmakedirsr§ Úhasattrr€ Ú
|
|||
|
|
_tag_namesr Ú
|
|||
|
|
dataloaderÚ manual_seedÚpreparer¯ Úeval_dataloaderrD rØ rd )rO r_ r` ra rb rc rd re rf rg rh ri rj r‘ Ú time_tensorÚtime_intÚmoduleÚdefault_callbackss ru rN z_UnslothPPOTrainer.__init__É s� € ð$ ˜Ð Ð Ý ðZñô ð
|
|||
|
|
ð
|
|||
|
|
ˆŒ Ø 0ˆÔ Ø !ˆÔð Ð Ý 3°DÔ4IÑ JÔ JˆMð Œ?ð
|
|||
|
|
g˜tÔ 1ð
|
|||
|
|
gÝ Ð TÑUÔUÐ UØ
|
|||
|
|
Œ_ð gØŒ %Ò'Ð'ØXhÔXuÐu�Ô!Ô3Ô@À4ÔCUÐCUå Øv¨D¬OÐvÐvÐvñô ð ð UYÔTfÐ fˆDÔ Ô /Ô <¸tÔ?Qð Œ9Ô !¨ Ð 5Ð 5Ý ðdñô ð
|
|||
|
|
õ !Ñ"Ô"ð ? {Ð'>Ý ð Iñô ð
|
|||
|
|
õ Ñ
|
|||
|
|
Ô
|
|||
|
|
ð ? [Ð%<å˜$Ô+YÑ7Ô7ð
|
|||
|
|
IØ$(Ô$5×$FÒ$FÑ$HÔ$H�Ô!õ !/¨tÔ/@À+Ñ NÔ NˆDÔ ØŒyð
|
|||
|
|
?�W TÔ%6Ð8KÈUÑ SÔ Sð
|
|||
|
|
?Ý+¨DÔ,=Ñ>Ô>Ð>å .Ñ 0Ô 0Ð ]µZÀÔ@QÕS\Ñ5]Ô5]ˆÔØ"&Ô"9ˆÔ Ø $Ô 5ˆÔ à ð GØ &ˆDŒNˆNØ
|
|||
|
|
Ô
|
|||
|
|
ð GØ !ˆDŒNˆNå 3°DÔ4EÑ FÔ FˆDŒNà (ˆÔØ *ˆÔÝ!$ ]Ñ!3Ô!3ˆÔ Ø&ˆÔØ *ˆÔØ (ˆÔØ,6Ñ)ˆŒ˜Ô)Ø(,ˆÔ%ð
|
|||
|
|
Ô Ð &Ý"% dÔ&;¸dÔ>TÑ&TÑ"UÔ"UˆDÔ Ý!¸dÔ>^Ð_Ñ_Ô_ˆ Ø&ˆÔØ%Ô3ˆŒØ $Ô @À4ÔCcÑ cˆÔ Ý # DÔ$DÀtÄÑ$VÑ WÔ WˆÔ Ý ˜dÔ 3°d´oÑ EÑFÔFˆŒÝ (Ø ŒO˜TÔ 2Ð4kñ
|
|||
|
|
ô
|
|||
|
|
ˆÔ õ &/Ø Ô ! 4Ô#8Ð:wñ&
|
|||
|
|
ô &
|
|||
|
|
ˆÔ"ð Ô ð ØÔ-°Ò2Ð2Ð2Øe¨4Ô+EÐeÐeÐeñ 3Ô2Ð2õ
|
|||
|
|
"&¤Ø Ô $¤/Ñ 1ñ"
|
|||
|
|
ô "
|
|||
|
|
ˆÔ õ ”l¥3¥t¤y¡{¤{Ñ#3Ô#3¸KÔ<NÐOÑOÔOˆ Ý ˜[¨!Ñ,Ô,×1Ò1Ñ3Ô3ˆØ œ=ÐCÐC¨D¬IÐCÐC¸ÐCÐCˆŒ
|
|||
|
|
Ø œ) kÔ&?À&Ñ&HÑHˆŒØ Ô &¨Ò *Ð *Ý+.¨q°$Ô2HÈDÔLgÑ2gÑ+hÔ+hˆDÔ (Ø+/Ô+@ˆÔ(ð
|
|||
|
|
Ô(¨$¬.¸$Ô:JÈDÔL]Ð^ð 1ð 1ˆFØÐ!Ý(¨Ñ0Ô0Ð0øÝ*¨4Ô+<¸dÔ>NÑOÔOˆŒ
|
|||
|
|
Ø Ô -Ô 4ˆŒ
|
|||
|
|
ÔØ ×+Ò+Ø #Ô 5ð ,ñ
|
|||
|
|
ô
|
|||
|
|
ð
|
|||
|
|
õ .Õ0SÐTXÔT]ÔTgÑ0hÔ0hÑ hÐØ.7Ð.?Ð*Ð*ÐEVÐYbÑEbˆŒÝ /Ø ŒN˜DœJ¨Ô(=¸t¼~ÈtÔO`ñ!
|
|||
|
|
ô !
|
|||
|
|
ˆÔ ð
|
|||
|
|
×Ò¨T¬YÔ-CÐb�/˜/ÕIbÑcÔcÐcÝ%Ñ'Ô'ˆŒ Ý'Ø"&×"<Ò"<Ñ">Ô">Ø"&×"<Ò"<Ñ">Ô">ð ð Ø !Ô 2Ô <ÀÄ ¸~Ñ Mð ñ ô ð
|
|||
|
|
ñ
|
|||
|
|
ô
|
|||
|
|
ˆŒ
|
|||
|
|
ð ˆÔØ!%ˆÔ Ý$+¨DÔ,<Ô,BÐDVÐX\Ñ$]Ô$]ÐeiÐ$iˆÔ!Ý & tÔ'7Ô'=¸}ÈdÑ SÔ SÐ[_Ð _ˆÔ à ˆÔØ Œ9Ô ð Ø × Ò Ñ Ô Ð Ø Œ9Ô ð =Ý ŒK˜œ Ô,°tÐ <Ñ <Ô <Ð <õ �4”:Ð /Ñ 0Ô 0ð 7Ø ŒJ× %Ò % d¤oÑ 6Ô 6Ð 6õ
|
|||
|
|
%Ø Ô ØÔ7ØØÔ)Øð
|
|||
|
|
ñ
|
|||
|
|
ô
|
|||
|
|
ˆŒõ Ô˜$œ)Ñ$Ô$Ð$Ø6A×6IÒ6IÈ$Ì*ÐVZÔVdÐfjÔfuÑ6vÔ6vÑ3ˆŒ
|
|||
|
|
�D”N D¤OÝ
|
|||
|
|
Ô˜$œ/Ñ*Ô*Ð*å )Ø Ô ØÔ6ØÔ)Øð
|
|||
|
|
ñ
|
|||
|
|
ô
|
|||
|
|
ˆÔ ð +× 2Ò 2°4Ô3GÑ HÔ HˆÔ à Ô $ð NÝ 1ØÔ! 4Ô#CÀTÄYÐPTÔPYñ!ô !ˆDÔ ð Œ~Ð%ØÔ)ð ZÝ$Ð%XÑYÔYÐYðZð Zõ "3Ø”N DÔ$DÀdÄiÐQUÔQZñ"ô "�”��ð Œ~Ð%ØÔ)ð ZÝ$Ð%XÑYÔYÐYðZð "&¤ ×!2Ò!2°4Ô3CÔ3JÑ!KÔ!K�”Ø $Ô 1× 4Ò 4°TÔ5EÔ5LÑ MÔ MˆDÔ Ð Ð rw c ó — | j S ©N©rª ©rO s ru Úget_train_dataloaderz'_UnslothPPOTrainer.get_train_dataloader› s
|
|||
|
|
€ ،Рrw c ó — | j S r³ )r rµ s ru Úget_eval_dataloaderz&_UnslothPPOTrainer.get_eval_dataloaderž s € ØÔ#Ð#rw c # ó˜ K — | j r=| j s6| j | j j ¦ « ¦ « n
|
|||
|
|
t ¦ « 5 | j r$| j j | j ¦ « dV — | j r&| j j | j pd¦ « ddd¦ « dS # 1 swxY w Y dS )zWContext manager for handling null reference model (that is, peft adapter manipulation).Nr{ )
|
|||
|
|
r‹ r< r‘ Ú unwrap_modelra ÚpolicyÚdisable_adapterr@ Ú set_adapterr; rµ s ru Únull_ref_contextz#_UnslothPPOTrainer.null_ref_context¡ s' è è € ð
|
|||
|
|
Ô!ð
|
|||
|
|
Ø*.Ô*?ð
|
|||
|
|
ˆDÔ × )Ò )¨$¬*Ô*;Ñ <Ô <× LÒ LÑ NÔ NÐ Nå ‘ ” ð Tð Tð
|
|||
|
|
Ô$ð
|
|||
|
|
EØ”
|
|||
|
|
Ô!×-Ò-¨dÔ.CÑDÔDÐDØ ˆEˆEˆEØÔ$ð
|
|||
|
|
TØ”
|
|||
|
|
Ô!×-Ò-¨dÔ.EÐ.RÈÑSÔSÐSð Tð Tð Tñ Tô Tð Tð Tð Tð Tð Tð Tð Tøøøð Tð Tð Tð Tð Tð Ts ÁA B?Â?CÃCFr§ Ú_internal_callc óÞ •— | j }| j j | _ | j r| j }| j | _ t ¦ « ||¦ « || _ | j r || _ d S d S r³ )ra r» r£ rõ rM Ú
|
|||
|
|
save_model)rO r§ r¿ Ú backup_modelÚbackup_deepspeedrQ s €ru rÁ z _UnslothPPOTrainer.save_model¯ ss ø€ Ø”zˆ Ø”ZÔ&ˆŒ
|
|||
|
|
à Ô $ð (Ø #œ~Ð Ø !œZˆDŒNå
|
|||
|
|
‰Œ×Ò˜: ~Ñ6Ô6Ð6à!ˆŒ
|
|||
|
|
à Ô $ð .Ø -ˆDŒNˆNˆNð .ð .rw c ó€* ‡r— | j }| j }| j }| j }| j }| j }| j }| j Šr|j }ˆrfd„} t | ¦ « ¦ « }
|
|||
|
|
t |j |j dz ddd¬¦ « } |
|
|||
|
|
d¦ « t j ¦ « } |j |j |j f}
|
|||
|
|
t% j |
|
|||
|
|
|¬¦ « }t% j |
|
|||
|
|
|¬¦ « }t% j |
|
|||
|
|
|¬¦ « }t% j |
|
|||
|
|
|¬¦ « }t% j |
|
|||
|
|
|¬¦ « }t% j |
|
|||
|
|
|¬¦ « }t% j |
|
|||
|
|
|¬¦ « }| ¦ « d | j _ d | j _ |j | j _ |j | j z | j _ |j �M|j d
|
|||
|
|
k r1t= j | j j |j z ¦ « | j _ n|j | j _ |j �M|j d
|
|||
|
|
k r1t= j | j j |j z ¦ « | j _ n|j | j _ |j! �M|j! d
|
|||
|
|
k r1t= j | j j |j! z ¦ « | j _! n|j! | j _! | j" # || j | j$ ¦ « | _$ | j% r| j | _& | j | _' tQ d
|
|||
|
|
|j d
|
|||
|
|
z ¦ « D �]³}| j xj d
|
|||
|
|
|j) z z
|
|||
|
|
c_ tU |
|
|||
|
|
¦ « }t% j+ ¦ « 5 |d , |¦ « }|j- d
|
|||
|
|
}g }g }g }g } g } g } g } t] | j | j | j j/ ¬ ¦ « 5 } ta | j1 ||j2 |j3 | ¦ « \ }!}"d d d ¦ « n # 1 swxY w Y tQ d |j- d |j2 ¦ « D �]f}#||#|#|j2 z … }$|!|#|#|j2 z … }%|%d d …|d …f }&|"|#|#|j2 z … }'ti |'|&¦ « }(~'tk ¦ « |€H| 6 ¦ « 5 to |j1 |%|j3 ¦ « })d d d ¦ « n # 1 swxY w Y nto ||%|j3 ¦ « })|)j8 d d …|d
|
|||
|
|
z
|
|||
|
|
d
|
|||
|
|
…f }*|*|j dz z }*ti |*|&¦ « }+~)~*tk ¦ « |&},| j9 �tu | j9 |j3 |&¦ « },t% j; |$|,fd
|
|||
|
|
¦ « }-ty |,|j3 k ¦ « d
|
|||
|
|
z
|
|||
|
|
}.| = |¦ « j> }/t |/|%|j3 |¦ « \ }0}1}1|0d d …|d
|
|||
|
|
z
|
|||
|
|
d
|
|||
|
|
…f @ d
|
|||
|
|
¦ « }2t ||-|j3 |¦ « \ }1}3}1| A |&¦ « | A |,¦ « | A |(¦ « | A |+¦ « | A |.¦ « | A |3¦ « | A |2¦ « �Œht% j; |d ¦ « }t% j; |d ¦ « }t% j; |d ¦ « }t% j; | d ¦ « } t% j; | d ¦ « } t% j; | d ¦ « } t% j; | d ¦ « } ~(~+~0~2~3~ tk ¦ « t… jC ¦ « t% jD || j jE k d
|
|||
|
|
¬¦ « }4| j jF �| |4 xx | j jF z cc< t% jG |j- d
|
|||
|
|
|j ¬¦ « H |j- d d
|
|||
|
|
¦ « }5|5| I d
|
|||
|
|
¦ « k }6t% jJ ||6t– ¦ « }t% jJ | |6t– ¦ « } | d
|
|||
|
|
z }7|5|7 I d
|
|||
|
|
¦ « k }8t% jJ | |8d ¦ « } | |z
|
|||
|
|
}9|jL dk r|9 n|9 M ¦ « d
|
|||
|
|
z
|
|||
|
|
|9z
|
|||
|
|
}:|jN |:z };|; O ¦ « }<t% jG |< P d ¦ « |<j ¬¦ « }=t% jQ |7|< P d
|
|||
|
|
¦ « k |7| ¦ « }>|<|=|>gxx | z
|
|||
|
|
cc< |jR r)t§ |<|8 d¬¦ « }<t% jJ |<|8d ¦ « }<d }?g }@|j- d
|
|||
|
|
}At© tQ |A¦ « ¦ « D ]j}B|B|Ad
|
|||
|
|
z
|
|||
|
|
k r| d d …|Bd
|
|||
|
|
z f nd}C|<d d …|Bf |jU |Cz z | d d …|Bf z
|
|||
|
|
}D|D|jU |jV z |?z z }?|@ A |?¦ « Œkt% jW |@d d d
|
|||
|
|
… d
|
|||
|
|
¬¦ « }E|E| z }Ft§ |E|6 ¦ « }Et% jJ |E|6d ¦ « }Etk ¦ « d d d ¦ « n # 1 swxY w Y tQ |j ¦ « D �]c}Gt° jY Z |j[ ¦ « }Hd }ItQ d |j[ |j\ ¦ « D �] }J|J|j\ z }K|H|J|K… }Ld }MtQ d |j\ |j] ¦ « D �]¹}N| ^ |¦ « 5 |N|j] z }O|L|N|O… }P|E|P }Q||P }R|!|P }S||P }T|F|P }U| |P }Vto ||S|j3 ¦ « \ }W}X|Wj8 d d …|d
|
|||
|
|
z
|
|||
|
|
d
|
|||
|
|
…f }'|'|j dz z }'ti |'|R¦ « }Yt% jJ |Y|6|P t– ¦ « }Y|Xd d …|d
|
|||
|
|
z
|
|||
|
|
d
|
|||
|
|
…f @ d
|
|||
|
|
¦ « }Zt% jJ |Z|8|P d ¦ « }Zt% j_ |Z|V|j` z
|
|||
|
|
|V|j` z ¦ « }[t% ja |Z|Uz
|
|||
|
|
¦ « }\t% ja |[|Uz
|
|||
|
|
¦ « }]t% jb |\|]¦ « }^dtÇ |^|8|P ¦ « z }_tÇ |]|\k d ¦ « |8|P ¦ « }`|Y|Tz
|
|||
|
|
}at% jM |a¦ « }b|Q |bz }c|Q t% j_ |bd|je z
|
|||
|
|
d|je z ¦ « z }dt% jb |c|d¦ « }etÇ |e|6|P ¦ « }f|f|jf |_z z }g| g |g¦ « | h ¦ « | i ¦ « t% j+ ¦ « 5 tÇ |d|ck d ¦ « |6|P ¦ « }ht$ jj jk l |'d
|
|||
|
|
¬¦ « }it% jm |'d
|
|||
|
|
¬¦ « t% jn |i|'z d
|
|||
|
|
¬¦ « z
|
|||
|
|
}jd|adz o ¦ « z }k|k||G|I|Mf< |h||G|I|Mf< |f||G|I|Mf< |_||G|I|Mf< |`||G|I|Mf< |j o ¦ « ||G|I|Mf< |b o ¦ « ||G|I|Mf< d d d ¦ « n # 1 swxY w Y d d d ¦ « n # 1 swxY w Y |Md
|
|||
|
|
z
|
|||
|
|
}M�Œ»|Id
|
|||
|
|
z
|
|||
|
|
}I~W~X~'~Y~Z~[~\~]~_~`~a~b~c~d~e~f~g~h~i~j~k~U~Q~V~R~S~Ttk ¦ « �Œ �Œet% j+ ¦ « 5 |: n d
|
|||
|
|
¦ « o ¦ « }l| n d
|
|||
|
|
¦ « o ¦ « }m|; n d
|
|||
|
|
¦ « o ¦ « }n|n| o ¦ « z }otá | j j t j ¦ « | z
|
|||
|
|
z ¦ « }pi }q|p|qd< | j q |l¦ « o ¦ « r ¦ « |qd< | j q |m¦ « o ¦ « r ¦ « |qd< | j q |n¦ « o ¦ « r ¦ « |qd< | j q |o¦ « o ¦ « r ¦ « |qd< | j q | o ¦ « ¦ « o ¦ « r ¦ « |qd< | j q |¦ « o ¦ « r ¦ « |qd< | j q |¦ « o ¦ « r ¦ « |qd < | j q |¦ « o ¦ « r ¦ « |qd < | j q |¦ « o ¦ « r ¦ « |qd < | j q |¦ « o ¦ « r ¦ « |qd < | j q |¦ « o ¦ « r ¦ « |qd < | j q |¦ « o ¦ « r ¦ « |qd!< | j q |¦ « s ¦ « r ¦ « |qd"< ||jE k n ¦ « r ¦ « |qd#< | jt u ¦ « d |qd$< | j j |qd%< | j j | j z | j _v | j xj d
|
|||
|
|
z
|
|||
|
|
c_ | w |q¦ « d d d ¦ « n # 1 swxY w Y | jt h ¦ « | j" x || j | j$ ¦ « | _$ | j$ jy rG| z |d ¬&¦ « | j" { | j | j | j$ ¦ « | _$ ~:~l~m~n~ ~q~;tk ¦ « t… jC ¦ « |j| d k r5|d
|
|||
|
|
z
|
|||
|
|
| j} z d k r$| ~ d¬'¦ « tk ¦ « ~!~~~~ ~ ~ ~4~7~5~6~8~<~=~>~E~Ftk ¦ « �Œµ| j" || j | j$ ¦ « | _$ | j$ jy rJ| z |d d ¬(¦ « | j" { | j | j | j$ ¦ « | _$ d S d S ))Nc 3 ó •K — ‰ E d {V —† Œ r³ rG r´ s €ru Úrepeat_generatorz2_UnslothPPOTrainer.train.<locals>.repeat_generatorÉ s) øè è € ð
|
|||
|
|
&Ø%Ð%Ð%Ð%Ð%Ð%Ð%Ð%ð
|
|||
|
|
&rw r¤ r“ r‡ T©Úmax_new_tokensr/ Útop_kÚtop_pÚ do_samplez===training policy===rq r r_ Ú input_ids©Úgather_deepspeed3_paramsrY r^ r F)ÚmaskÚ
|
|||
|
|
shift_mean)Úaxisg à?r€ Úepsz objective/klzobjective/entropyzobjective/non_score_rewardzobjective/rlhf_rewardzobjective/scoreszpolicy/approxkl_avgzpolicy/clipfrac_avgzloss/policy_avgzloss/value_avgzval/clipfrac_avgzpolicy/entropy_avgz val/ratioz
|
|||
|
|
val/ratio_varzval/num_eos_tokensÚlrÚepisode)Útrial)Úsampling)rÕ Úmetrics)€r_ r‘ rŽ ra rb rc r` rª rr Úiterr r, r/ ÚprintrH r= r( r² rI ÚzerosÚtrainr Ú global_steprÔ r3 r½ r) r� r¼ rÇ r= r“ rä rÊ r� Úon_train_beginrŸ r£ rõ Ú
|
|||
|
|
model_wrappedÚranger6 ÚnextÚno_gradrd rb rK rF r& r» r* Ú pad_token_idrF r, r¾ r/ rl r. rJ Úcatr. rº re r6 rh rj r1 ÚcollectÚanyr† r0 ÚarangeÚrepeatrg Ú masked_fillr r@ Úexpr? ÚcloneÚsizeÚwherer> r< ÚreversedrD rE Ústackr? ÚrandomÚ permutationr5 r7 r® Ú
|