259 lines
56 KiB
Plaintext
259 lines
56 KiB
Plaintext
|
§
|
|||
|
|
5$�hiÜ ã ód — d Z ddlmZ ddlZddlmZ ddlmZ ddlmZm Z m
|
|||
|
|
Z
|
|||
|
|
m Z m Z m
|
|||
|
|
Z
|
|||
|
|
mZmZ ddlmZmZmZmZmZmZmZmZmZmZmZmZm Z m Z m
|
|||
|
|
Z
|
|||
|
|
m Z m Z m Z m!Z!m"Z"m#Z#m$Z$m%Z%m&Z&m Z m'Z'm(Z(m)Z)m*Z*m+Z+m,Z,m-Z-m.Z.m/Z/m0Z0m1Z1m2Z2m3Z3m4Z4m5Z5m6Z6m7Z7m8Z8mZm9Z9m:Z:m;Z;m<Z<m=Z=m>Z>m?Z?m@Z@mZmAZAmBZBm
|
|||
|
|
Z
|
|||
|
|
m$Z$m:Z:mZ ddl:Z:ddlT ddlCmDZDmEZE dd lFmGZG ddlZddlHZ9dd
|
|||
|
|
lImJZJ ddlmZ dd lKmLZLmMZN d d
|
|||
|
|
d d
|
|||
|
|
d
|
|||
|
|
dœZO ejP d d eO¬¦ « d„ ¦ « ZQeD G d„ de"¦ « ¦ « ZR G d„ de$¦ « ZS G d„ deS¦ « ZTdS )z8
|
|||
|
|
2025.8.4
|
|||
|
|
2025.8.5
|
|||
|
|
4.55.1
|
|||
|
|
0.21.0
|
|||
|
|
__UNSLOTH_VERSIONING__
|
|||
|
|
é )ÚTensorN)Ú
|
|||
|
|
functional)ÚAnyÚListÚOptionalÚTupleÚUnionÚDictÚSetÚCallable);Ú AcceleratorÚBaseImageProcessorr ÚCallbackHandlerÚDEFAULT_CALLBACKSÚDEFAULT_PROGRESS_CALLBACKÚDataCollatorWithPaddingÚ
|
|||
|
|
DataLoaderÚDatasetÚExportableStateÚFeatureExtractionMixinÚGenerationConfigÚINVALID_LOGPROBÚOnlineTrainerStater ÚPathÚPreTrainedTokenizerBaseÚPrinterCallbackÚProcessorMixinÚ
|
|||
|
|
RLOOConfigÚ RLOOTrainerÚTrainerÚTrainerCallbackÚTrainerControlr Úbatch_generationÚ broadcastÚ defaultdictÚdisable_dropout_in_modelÚ empty_cacheÚ exact_divÚfirst_true_indicesÚforwardÚ
|
|||
|
|
gather_objectÚgcÚgenerate_model_cardÚget_comet_experiment_urlÚ#get_reporting_integration_callbacksÚ
|
|||
|
|
get_rewardÚis_rich_availableÚis_wandb_availableÚ log_table_to_comet_experimentÚmathÚnnÚnpÚosÚpdÚprepare_deepspeedÚprint_rich_tableÚselective_log_softmaxÚtextwrapÚtimeÚtorchÚtruncate_responseÚunwrap_model_for_generationr r r7 r> )Ú*)Ú dataclassÚfield)ÚVersion)Ú nullcontext)ÚDataCollatorForSeq2SeqÚ DataCollatorForLanguageModelingTF)Úepilogue_fusionÚ max_autotuneÚ
|
|||
|
|
shape_paddingz
|
|||
|
|
trace.enabledztriton.cudagraphs)ÚdynamicÚ fullgraphÚoptionsc ó’ — t j | d| j d ¦ « dd¬¦ « }t j | d¦ « dd¬¦ « }g }t ||¦ « D ]‘\ }}| t j ¦ « }t j |d| d¦ « ¬¦ « d¦ « }t j
|
|||
|
|
|d¬¦ « }||z
|
|||
|
|
} | | ¦ « Œ’ t j |¦ « }| | j d | j d f¦ « }|S )Néÿÿÿÿé r )ÚchunksÚdim)rR Úindex©rR é )
|
|||
|
|
r> ÚchunkÚreshapeÚshapeÚzipÚtoÚfloat32ÚgatherÚ unsqueezeÚsqueezeÚ logsumexpÚappendÚconcat)
|
|||
|
|
ÚlogitsrS Úchunked_logitsÚ
|
|||
|
|
chunked_indexÚall_per_token_logpsÚ chunk_logitsÚ chunk_indexÚselected_logitsÚlogsumexp_valuesÚper_token_logpss
|
|||
|
|
ú^/workspace/Fine-tuning/DS-LLM-TEMPLATE-FINETUNING/unsloth_compiled_cache/UnslothRLOOTrainer.pyÚ chunked_selective_log_softmaxrl " s5 € õ ”[ § ¢ °°F´LÀÔ4DÑ!EÔ!EÐPQÐYZÐ[Ñ[Ô[€NÝ”[ § ¢ ¨rÑ!2Ô!2¸QÀaÐHÑHÔH€MØ Ðå%(¨ ¸ Ñ%GÔ%Gð 4ð 4Ñ!ˆ �kØ#—’¥u¤}Ñ5Ô5ˆ Ý œ, |¸2À{×G\ÒG\Ð]_ÑG`ÔG`ÐaÑaÔa×iÒiÐjlÑmÔmˆÝ œ?¨<¸rÐBÑBÔBÐØ)Ð,<Ñ<ˆØ×"Ò" ?Ñ3Ô3Ð3Ð3ØÝ œ,Ð':Ñ;Ô;ÐØ-×5Ò5°v´|ÀA´ÈÌ ÐUVÌÐ6XÑYÔYÐØ Ð ó c óÎ ‡ — e Zd ZU dZ edddi¬¦ « Zee ed< edddi¬¦ « Z ee
|
|||
|
|
ed < d3ˆ fd2„ Z ˆ xZ S )4ÚUnslothRLOOConfigaÆ
|
|||
|
|
|
|||
|
|
Configuration class for the [`RLOOTrainer`].
|
|||
|
|
|
|||
|
|
This class includes only the parameters that are specific to RLOO training. For a full list of training arguments,
|
|||
|
|
please refer to the [`~transformers.TrainingArguments`] and [`OnPolicyConfig`] documentation. Note that default
|
|||
|
|
values in this class may differ from those in [`~transformers.TrainingArguments`].
|
|||
|
|
|
|||
|
|
Using [`~transformers.HfArgumentParser`] we can turn this class into
|
|||
|
|
[argparse](https://docs.python.org/3/library/argparse#module-argparse) arguments that can be specified on the
|
|||
|
|
command line.
|
|||
|
|
|
|||
|
|
Parameters:
|
|||
|
|
exp_name (`str`, *optional*, defaults to `os.path.basename(__file__)[: -len(".py")]`):
|
|||
|
|
Name of this experiment.
|
|||
|
|
reward_model_path (`str`, *optional*, defaults to `"EleutherAI/pythia-160m"`):
|
|||
|
|
Path to the reward model.
|
|||
|
|
num_ppo_epochs (`int`, *optional*, defaults to `4`):
|
|||
|
|
Number of epochs to train.
|
|||
|
|
whiten_rewards (`bool`, *optional*, defaults to `False`):
|
|||
|
|
Whether to whiten the rewards.
|
|||
|
|
kl_coef (`float`, *optional*, defaults to `0.05`):
|
|||
|
|
KL coefficient.
|
|||
|
|
cliprange (`float`, *optional*, defaults to `0.2`):
|
|||
|
|
Clip range.
|
|||
|
|
rloo_k (`int`, *optional*, defaults to `2`):
|
|||
|
|
REINFORCE Leave-One-Out (RLOO) number of online samples per prompt.
|
|||
|
|
normalize_reward (`bool`, *optional*, defaults to `False`):
|
|||
|
|
Whether to normalize rewards.
|
|||
|
|
reward_clip_range (`float`, *optional*, defaults to `10.0`):
|
|||
|
|
Clip range for rewards.
|
|||
|
|
normalize_advantage (`bool`, *optional*, defaults to `False`):
|
|||
|
|
Whether to normalize advantages.
|
|||
|
|
token_level_kl (`bool`, *optional*, defaults to `True`):
|
|||
|
|
Whether to use token-level KL penalty or sequence-level KL penalty.
|
|||
|
|
ds3_gather_for_generation (`bool`, *optional*, defaults to `True`):
|
|||
|
|
This setting applies to DeepSpeed ZeRO-3. If enabled, the policy model weights are gathered for generation,
|
|||
|
|
improving generation speed. However, disabling this option allows training models that exceed the VRAM
|
|||
|
|
capacity of a single GPU, albeit at the cost of slower generation.
|
|||
|
|
|
|||
|
|
NÚhelpzvLLM SamplingParams)ÚdefaultÚmetadataÚvllm_sampling_paramsrO z8Chunk size to reduce memory usage. -1 is most efficient.Úunsloth_num_chunksFÚnorP é r éú ç-C ëâ6
|
|||
|
|
?ç{®Gáz„?çÍÌÌÌÌÌì?ç+‡ÙÎ÷ï?ç:Œ0âŽyE>ç ð?ç @Úlinearçš™™™™™¹?ÚpassiveÚwarningTÚstepsrU éô éO
|
|||
|
|
ÚO1ÚautoÚ ç Ú
|
|||
|
|
adamw_8bitÚlengthÚ
|
|||
|
|
every_saveÚlasté é@ é
|
|||
|
|
é5 çffffffæ?úEleutherAI/pythia-160mÚ rloo_configçš™™™™™©?çš™™™™™É?ç $@c¡ ó, •— |dk rt d|› d�¦ « ‚|dk rt d|› d�¦ « ‚|€|#dk r
|
|||
|
|
|$dk rd}d }#|�€!d
|
|||
|
|
d lm}¢ t |¢¦ « d z d ¦ « }�|‰d
|
|||
|
|
k rt d
|
|||
|
|
¦ « ‚|‰dk rt d¦ « ‚ t
|
|||
|
|
¦ « j d®i d|“d|“d|“d|“d|“d|“d|“d|“d| “d|
|
|||
|
|
“d| “d| “d |
|
|||
|
|
“d |“d |“d |“d |“d!|“d"|“d#|“d$|“d%|“d&|“d'|“d(|“d)|“d*|“d+| “d,| “d-| “d.| “d/| “d0|!“d1|"“d2|#“d3|$“d4|%“d5|&“d6|'“d7|(“d8|)“d9|*“d:|+“d;|,“d<|-“d=|.“d>|/“d?|0“d@|1“dA|2“dB|3“dC|4“dD|5“dE|6“dF|7“dG|8“dH|9“dI|:“dJ|;“dK|<“dL|=“dM|>“dN|?“dO|@“dP|A“dQ|B“dR|C“dS|D“dT|E“dU|F“dV|G“dW|H“dX|I“dY|J“dZ|K“d[|L“d\|M“d]|N“d^|O“d_|P“d`|Q“da|R“db|S“dc|T“dd|U“de|V“df|W“dg|X“dh|Y“di|Z“dj|[“dk|\“dl|]“dm|^“dn|_“do|`“dp|a“dq|b“dr|c“ds|d“dt|e“du|f“dv|g“dw|h“dx|i“dy|j“dz|k“d{|l“d||m“d}|n“d~|o“d|p“d€|q“d�|r“d‚|s“dƒ|t“d„|u“d…|v“d†|w“d‡|x“dˆ|y“d‰|z“dŠ|{“d‹||“dŒ|}“d�|~“dŽ|“d�|€“d�|�“d‘|‚“d’|ƒ“d“|„“d”|…“d•|†“d–|‡“d—|ˆ“d˜|‰“d™|Š“dš|‹“d›|Œ“dœ|�“d�|Ž“dž|�“dŸ|�“d |‘“d¡|’“d¢|““d£|”“d¤|•“d¥|–“d¦|—“d§|˜“d¨|™“d©|š“dª|›“d«|œ“d¬|�“d|ž“|¡¤Ž |Ÿ| _ | | _ d S )¯NçH¯¼šò×z>z Unsloth: Your learning rate of `zi` is too small and less than 1e-7! Consider increasing it, otherwise gradient updates will be close to 0!rU za` is way too larger > 1! Consider decreasing it to 1e-1, otherwise gradient updates will explode!rƒ r„ Ú unsloth_training_checkpointsru r )Ú cpu_countrv zUUnsloth: Please set a positive non-zero temperature since your results will be wrong.r� zgUnsloth: Please set a positive non-zero temperature less than 10, since sampling will be quite erratic.Ú
|
|||
|
|
output_dirÚoverwrite_output_dirÚdo_trainÚdo_evalÚ
|
|||
|
|
do_predictÚ
|
|||
|
|
eval_strategyÚprediction_loss_onlyÚper_device_train_batch_sizeÚper_device_eval_batch_sizeÚper_gpu_train_batch_sizeÚper_gpu_eval_batch_sizeÚgradient_accumulation_stepsÚeval_accumulation_stepsÚ
|
|||
|
|
eval_delayÚtorch_empty_cache_stepsÚ
|
|||
|
|
learning_rateÚ weight_decayÚ
|
|||
|
|
adam_beta1Ú
|
|||
|
|
adam_beta2Ú adam_epsilonÚ
|
|||
|
|
max_grad_normÚnum_train_epochsÚ max_stepsÚlr_scheduler_typeÚ warmup_ratioÚ warmup_stepsÚ log_levelÚlog_level_replicaÚlog_on_each_nodeÚ logging_dirÚlogging_strategyÚlogging_first_stepÚ
|
|||
|
|
logging_stepsÚlogging_nan_inf_filterÚ
|
|||
|
|
save_strategyÚ
|
|||
|
|
save_stepsÚsave_total_limitÚsave_safetensorsÚsave_on_each_nodeÚsave_only_modelÚ'restore_callback_states_from_checkpointÚno_cudaÚuse_cpuÚuse_mps_deviceÚseedÚ data_seedÚ
|
|||
|
|
jit_mode_evalÚuse_ipexÚbf16Úfp16Úfp16_opt_levelÚhalf_precision_backendÚbf16_full_evalÚfp16_full_evalÚtf32Ú
|
|||
|
|
local_rankÚ ddp_backendÚ
|
|||
|
|
tpu_num_coresÚtpu_metrics_debugÚdebugÚdataloader_drop_lastÚ
|
|||
|
|
eval_stepsÚdataloader_num_workersÚdataloader_prefetch_factorÚ
|
|||
|
|
past_indexÚrun_nameÚ disable_tqdmÚremove_unused_columnsÚ label_namesÚload_best_model_at_endÚmetric_for_best_modelÚgreater_is_betterÚignore_data_skipÚfsdpÚfsdp_min_num_paramsÚ fsdp_configÚ"fsdp_transformer_layer_cls_to_wrapÚaccelerator_configÚ deepspeedÚlabel_smoothing_factorÚoptimÚ
|
|||
|
|
optim_argsÚ adafactorÚgroup_by_lengthÚlength_column_nameÚ report_toÚddp_find_unused_parametersÚddp_bucket_cap_mbÚddp_broadcast_buffersÚdataloader_pin_memoryÚ dataloader_persistent_workersÚskip_memory_metricsÚuse_legacy_prediction_loopÚ push_to_hubÚresume_from_checkpointÚ hub_model_idÚ hub_strategyÚ hub_tokenÚhub_private_repoÚhub_always_pushÚ hub_revisionÚgradient_checkpointingÚ gradient_checkpointing_kwargsÚinclude_inputs_for_metricsÚeval_do_concat_batchesÚ fp16_backendÚpush_to_hub_model_idÚpush_to_hub_organizationÚpush_to_hub_tokenÚ
|
|||
|
|
mp_parametersÚauto_find_batch_sizeÚfull_determinismÚ torchdynamoÚ ray_scopeÚ ddp_timeoutÚ
|
|||
|
|
torch_compileÚtorch_compile_backendÚtorch_compile_modeÚinclude_tokens_per_secondÚ include_num_input_tokens_seenÚneftune_noise_alphaÚoptim_target_modulesÚbatch_eval_metricsÚ
|
|||
|
|
eval_on_startÚuse_liger_kernelÚliger_kernel_configÚeval_use_gather_objectÚ average_tokens_across_devicesÚdataset_num_procÚnum_mini_batchesÚtotal_episodesÚ local_rollout_forward_batch_sizeÚnum_sample_generationsÚresponse_lengthÚ
|
|||
|
|
stop_tokenÚ
|
|||
|
|
stop_token_idÚ temperatureÚmissing_eos_penaltyÚsft_model_pathÚ
|
|||
|
|
world_sizeÚnum_total_batchesÚmicro_batch_sizeÚlocal_batch_sizeÚ
|
|||
|
|
batch_sizeÚlocal_mini_batch_sizeÚmini_batch_sizeÚexp_nameÚreward_model_pathÚnum_ppo_epochsÚwhiten_rewardsÚkl_coefÚ cliprangeÚrloo_kÚnormalize_rewardÚreward_clip_rangeÚnormalize_advantageÚtoken_level_klÚds3_gather_for_generation© )
|
|||
|
|
ÚFloatingPointErrorÚ
|
|||
|
|
OverflowErrorÚmultiprocessingr› ÚminÚ MathErrorÚsuperÚ__init__rs rt )¤Úselfrœ r� rž rŸ r r¡ r¢ r£ r¤ r¥ r¦ r§ r¨ r© rª r« r¬ r r® r¯ r° r± r² r³ r´ rµ r¶ r· r¸ r¹ rº r» r¼ r½ r¾ r¿ rÀ rÁ r rà rÄ rÅ rÆ rÇ rÈ rÉ rÊ rË rÌ rÍ rÎ rÏ rÐ rÑ rÒ rÓ rÔ rÕ rÖ r× rØ rÙ rÚ rÛ rÜ rÝ rÞ rß rà rá râ rã rä rå ræ rç rè ré rê rë rì rí rî rï rð rñ rò ró rô rõ rö r÷ rø rù rú rû rü rý rþ rÿ r r r r r r r r r r r
|
|||
|
|
r r r
|
|||
|
|
r r r r r r r r r r r r r r r r r r r r! r" r# r$ r% r& r' r( r) r* r+ r, r- r. r/ r0 r1 r2 r3 r4 r5 r6 r7 r8 r9 rs rt Úkwargsr› Ú __class__s¤ €rk rA zUnslothRLOOConfig.__init__f s[ ø€ ðH ˜4Ò Ð Õ'9ð ;VÐ]jð ;Vð ;Vð ;Vñ (Wô (Wð "WØ ˜1Ò Ð ¥Mð 3FÐUbð 3Fð 3Fð 3Fñ %Gô %Gð GØ Ð -°7Ò":Ð":¸zÈSÒ?PÐ?PØ7ˆJØ ˆMØ Ð #Ø 1Ð 1Ð 1Ð 1Ð 1Ð 1Ý " 9 9¡;¤;¨q¡=°!Ñ 4Ô 4Ð Ø ˜!Ò Ð ÝÐ sÑtÔtÐ tØ
|
|||
|
|
˜BÒ
|
|||
|
|
Ð
|
|||
|
|
Ýð Fñ Gô Gð
|
|||
|
|
Gð �‰ŒÔð ^ Lð ^ Lð ^ LØ#˜ð^ Là#7Ð#7ð^ Lð �xð^ Lð �gð ^ Lð
|
|||
|
|
$˜ð ^ Lð *˜Mð
|
|||
|
|
^ Lð $8Ð#7ð^ Lð +FÐ*Eð^ Lð *DÐ)Cð^ Lð (@Ð'?ð^ Lð '>Ð&=ð^ Lð +FÐ*Eð^ Lð '>Ð&=ð^ Lð $˜ð ^ Lð '>Ð&=ð ^ Lð *˜Mð!^ Lð" (˜<ð#^ Lð$ $˜ð%^ Lð& $˜ð'^ Lð( (˜<ð)^ Lð* *˜Mð+^ Lð, 0Ð /ð-^ Lð. "˜ ð/^ Lð0 !2Ð 1ð1^ Lð2 (˜<ð3^ Lð4 (˜<ð5^ Lð6 "˜ ð7^ Lð8 !2Ð 1ð9^ Lð: 0Ð /ð;^ Lð< &˜+ð=^ Lð> 0Ð /ð?^ Lð@ "4Ð!3ðA^ LðB *˜MðC^ LðD &<Ð%;ðE^ LðF *˜MðG^ LðH $˜ðI^ LðJ 0Ð /ðK^ LðL 0Ð /ðM^ LðN !2Ð 1ðO^ LðP .˜oðQ^ LðR 7^Ð6]ðS^ LðT �gðU^ LðV �gðW^ LðX ,˜^ðY^ LðZ �4ð[^ Lð\ "˜ ð]^ Lð^ *˜Mð_^ Lð` �xða^ Lðb �4ðc^ Lðd �4ðe^ Lðf ,˜^ðg^ Lðh &<Ð%;ði^ Lðj ,˜^ðk^ Lðl ,˜^ðm^ Lðn �4ðo^ Lðp $˜ðq^ Lðr &˜+ðs^ Lðt *˜Mðu^ Lðv !2Ð 1ðw^ Lðx �Eðy^ Lðz $8Ð#7ð{^ Lð| $˜ð}^ Lð~ &<Ð%;ð^ Lð@ *DÐ)CðA^ LðB $˜ðC^ LðD �xðE^ LðF (˜<ðG^ LðH %:Ð$9ðI^ LðJ &˜+ðK^ LðL &<Ð%;ðM^ LðN %:Ð$9ðO^ LðP !2Ð 1ðQ^ LðR 0Ð /ðS^ LðT �4ðU^ LðV #6Ð"5ðW^ LðX &˜+ðY^ LðZ 2TÐ1Sð[^ Lð\ "4Ð!3ð]^ Lð^ "˜ ð_^ Lð` &<Ð%;ða^ Lðb �Eðc^ Lðd $˜ðe^ Lðf "˜ ðg^ Lðh .˜oði^ Lðj "4Ð!3ðk^ Lðl "˜ ðm^ Lðn *DÐ)Cðo^ Lðp !2Ð 1ðq^ Lðr %:Ð$9ðs^ Lðt %:Ð$9ðu^ Lðv -JÐ,Iðw^ Lðx #6Ð"5ðy^ Lðz *DÐ)Cð{^ Lð| &˜+ð}^ Lð~ &<Ð%;ð^ Lð@ (˜<ðA^ LðB (˜<ðC^ LðD "˜ ðE^ LðF 0Ð /ðG^ LðH .˜oðI^ LðJ (˜<ðK^ LðL &<Ð%;ðM^ LðN -JÐ,IðO^ LðP *DÐ)CðQ^ LðR &<Ð%;ðS^ LðT (˜<ðU^ LðV $8Ð#7ðW^ LðX (@Ð'?ðY^ LðZ !2Ð 1ð[^ Lð\ *˜Mð]^ Lð^ $8Ð#7ð_^ Lð` 0Ð /ða^ Lðb &˜+ðc^ Lðd "˜ ðe^ Lðf &˜+ðg^ Lðh *˜Mði^ Lðj %:Ð$9ðk^ Lðl "4Ð!3ðm^ Lðn )BÐ(Aðo^ Lðp -JÐ,Iðq^ Lðr #6Ð"5ðs^ Lðt $8Ð#7ðu^ Lðv "4Ð!3ðw^ Lðx *˜Mðy^ Lðz 0Ð /ð{^ Lð| #6Ð"5ð}^ Lð~ &<Ð%;ð^ Lð@ -JÐ,IðA^ LðB 0Ð /ðC^ LðD 0Ð /ðE^ LðF ,˜^ðG^ LðH 0PÐ/OðI^ LðJ &<Ð%;ðK^ LðL .˜oðM^ LðN $˜ðO^ LðP *˜MðQ^ LðR &˜+ðS^ LðT #6Ð"5ðU^ LðV ,˜^ðW^ LðX $˜ðY^ LðZ !2Ð 1ð[^ Lð\ 0Ð /ð]^ Lð^ 0Ð /ð_^ Lð` $˜ða^ Lðb %:Ð$9ðc^ Lðd .˜oðe^ Lðf �xðg^ Lðh !2Ð 1ði^ Lðj ,˜^ðk^ Lðl ,˜^ðm^ Lðn �gðo^ Lðp "˜ ðq^ Lðr �Vðs^ Lðt 0Ð /ðu^ Lðv !2Ð 1ðw^ Lðx #6Ð"5ðy^ Lðz ,˜^ð{^ Lð| )BÐ(AÀFð}^ Lð ^ Lð ^ Lð~ %9ˆÔ!Ø"4ˆÔ Ð Ð rm ) NNFFFru FrP rP NNrv rv r rw rx ry rz r{ r| r} r~ rO r r€ r r� r‚ TNrƒ FrU Frƒ r„ NTFFFFFFr… r… FFFFr† r‡ FFNrO NNFrˆ FNr NrO NNTNFNNFrˆ r NNNNr‰ rŠ NFFr‹ NNNNTFTFFNNrŒ NNFNFNFTr‡ NNNrˆ TFNr� rŽ FNNFFNNFFFNFTNrU Nr� r� r‘ NNr’ Nr“ NNNNNNNr” r“ rP Fr• r– rv Fr— FFTNrO )
|
|||
|
|
Ú__name__Ú
|
|||
|
|
__module__Ú __qualname__Ú__doc__rC rs r r Ú__annotations__rt ÚintrA Ú
|
|||
|
|
__classcell__©rD s @rk ro ro 3 s« ø€ € € € € € ð(ð (ðR +0¨%ØØÐ 1Ð2ð+ñ +ô +И( 3œ-ð ð ñ ð */¨ØØÐ VÐWð*ñ *ô *И #œ ð ð ñ ð Ø #ØØØØ Ø $Ø&'Ø%&Ø#'Ø"&Ø&'Ø"#ØØ"%Ø ØØØØ ØØ ØØ $ØØØ Ø %Ø ØØ"Ø "ØØ!&Ø ØØ Ø Ø !Ø Ø27ØØØ ØØØ ØØØØ Ø!'Ø Ø ØØØØ Ø !ØØ $ØØ!"Ø%)ØØØØ $ØØ!&Ø $Ø Ø ØØ ØØ-1Ø !ØØ!$Ø ØØØ Ø %ØØ%)Ø Ø $Ø $Ø(-Ø "Ø%*ØØ!%ØØ#ØØ Ø ØØ!&Ø(,Ø%*Ø!%Ø Ø #Ø#'Ø ØØ #Ø ØØØØ Ø $Ø !Ø$)Ø(-Ø "Ø #Ø "Ø Ø Ø "Ø!&Ø(,Ø Ø Ø Ø+-Ø!#Ø ØØ ØØ "Ø1ØØ Ø Ø ØØ $Ø Ø Ø 4ØØ ØØØØ Ø Ø #Ø Ø$(Ø #Ø ðCR5ð R5ð R5ð R5ð R5ð R5ð R5ð R5ð R5ð R5rm ro c óì ‡ — e Zd ZddgZ d dedeeeee e
|
|||
|
|
f de j de j d ee j e
|
|||
|
|
ee gee f f d
|
|||
|
|
ed ee d eeeeeef f d
|
|||
|
|
eej j ej j j f deee ddfd„Zde fd„Z de fd„Z d„ Z d de fd„Z!ˆ fd„Z" d dee dee deeee df fd„Z#ˆ xZ$S ) Ú_UnslothRLOOTrainerÚtrlÚrlooN©NNÚconfigÚprocessing_classÚpolicyÚ
|
|||
|
|
ref_policyÚ reward_modelÚ
|
|||
|
|
train_datasetÚ
|
|||
|
|
data_collatorÚ eval_datasetÚ
|
|||
|
|
optimizersÚ callbacksÚreturnc ó„
|
|||
|
|
— ||u rt d¦ « ‚|| _ |} || _ || _ |€t | j ¦ « }d | j j _ d | j j _ || _ || _ || _
|
|||
|
|
t |¦ « | _ || _
|
|||
|
|
|| _ | \ | _ | _ d | _ | j €!t' | j | j z ¦ « | _ t+ | j ¬¦ « } | | _ | j | _ | j | j z | j z | _ t' | j | j z ¦ « | _ t' | j | j z ¦ « | _ t? | j | j d¦ « | _ t? | j | j d¦ « | _! tE j# | j | j z ¦ « | _$ tK j& t' tO j' ¦ « ¦ « | j( ¬¦ « }
|
|||
|
|
tS |
|
|||
|
|
d¦ « * ¦ « }| j+ › d| j, › d|› �| _- | j, | j. dz z | _/ | j0 dk r"tc d | j$ | j0 z ¦ « | _2 t? | j | j3 d
|
|||
|
|
¦ « | _4 |||fD ]+}tk |tl j7 ¦ « rtq |¦ « Œ,| j9 r | j9 d k r| j j | _: || _; | < | j$ ¬ ¦ « tz t} | j j? ¦ « z }|
|
|||
|
|
€|n||
|
|||
|
|
z | _@ tƒ | j@ | j; | j | j | j ¦ « | _B | C | j jD rtŠ ntŒ ¦ « t� ¦ « | _H t“ | J ¦ « | K ¦ « d
|
|||
|
|
„ | jB j@ | jH gz D ¦ « ¬¦ « | _L d| _M d | _N tŸ | j jL dd ¦ « d u| _P tŸ | j jL dd ¦ « d u| _Q d | _R | j jS r| T ¦ « | j jU r t jW | j jX d¬¦ « d | _Y tµ | j; d¦ « r | j; [ | j\ ¦ « t» | j
|
|||
|
|
| j4 d| j
|
|||
|
|
d¬¦ « | _^ tK j_ | j, ¦ « | ` | j; | j | j^ ¦ « \ | _; | _ | _^ tK j_ | j/ ¦ « t» | j | ja | j
|
|||
|
|
d¬¦ « | _b | ` | jb ¦ « | _b | jP rƒtk | j tl j7 ¦ « r+tÇ | j | j | jd | je ¦ « | _ tÇ | j | j | jd | je ¦ « | _ | j; | _f d S | j g | j j( ¦ « | _ tk | j tl j7 ¦ « r+| j g | j j( ¦ « | _ d S d S )Nz `policy` and `ref_policy` cannot be the same object. If you want `ref_policy` to be the same as `policy`, you must mass a copy of it, or `None` if you use peft.)r§ z5`batch_size` must be a multiple of `num_mini_batches`z;`local_batch_size` must be a multiple of `num_mini_batches`©Údevicer Ú__i£† rU z/`local_batch_size` must be a multiple of rloo_kÚeos)Únum_training_stepsc ó< — g | ]}t |t ¦ « ¯|‘ŒS r: )Ú
|
|||
|
|
isinstancer )Ú.0Úcbs rk ú
|
|||
|
|
<listcomp>z0_UnslothRLOOTrainer.__init__.<locals>.<listcomp>% s: € ð ð ð ØÕQ[Ð\^Õ`oÑQpÔQpð Øð ð ð rm )Úis_local_process_zeroÚis_world_process_zeroÚstateful_callbacksÚdeepspeed_pluginÚ fsdp_pluginT)Úexist_okÚadd_model_tags)r+ ÚshuffleÚ
|
|||
|
|
collate_fnÚ drop_last)r+ rp rq )hÚ
|
|||
|
|
ValueErrorÚargsrS rT r Úgeneration_configÚ eos_token_idÚ pad_token_idrU rV rW ÚlenÚtrain_dataset_lenrX rY Ú optimizerÚ lr_schedulerÚoptimizer_cls_and_kwargsr rJ r± r
|
|||
|
|
r§ Ú acceleratorÚ
|
|||
|
|
num_processesr' r£ r r* r) r+ r( r- r, r4 Úceilr( r> Útensorr= r_ r$ Úitemr. rÈ rÝ Ú
|
|||
|
|
process_indexÚ
|
|||
|
|
local_seedr ÚmaxÚsample_generations_freqr4 Úlocal_dataloader_batch_sizerd r5 ÚModuler& r" r# ÚmodelÚ create_optimizer_and_schedulerr r/ rñ r[ r Úcallback_handlerÚ add_callbackrÞ r r r" Úcontrolr rh ri ÚstateÚ current_flosÚhp_search_backendÚgetattrÚis_deepspeed_enabledÚis_fsdp_enabledrû rù Ú init_hf_repoÚ should_saver7 Úmakedirsrœ Ú backup_modelÚhasattrrn Ú
|
|||
|
|
_tag_namesr Ú
|
|||
|
|
dataloaderÚ manual_seedÚpreparer¤ Úeval_dataloaderr9 rÍ rÌ rê rZ )rB rR rS rT rU rV rW rX rY rZ r[ rs r| Ú time_tensorÚtime_intÚmoduleÚdefault_callbackss rk rA z _UnslothRLOOTrainer.__init__¾ s• € ð ˜Ð Ð Ý ð[ñô ð
|
|||
|
|
ð
|
|||
|
|
ˆŒ ØˆØ 0ˆÔ Ø ˆŒ ð Ð Ý 3°DÔ4IÑ JÔ JˆMð
|
|||
|
|
ð
|
|||
|
|
Œ Ô%Ô2ð 6:ˆŒ Ô%Ô2à$ˆŒØ (ˆÔØ *ˆÔÝ!$ ]Ñ!3Ô!3ˆÔ Ø *ˆÔØ (ˆÔØ,6Ñ)ˆŒ˜Ô)Ø(,ˆÔ%ð
|
|||
|
|
Ô Ð &Ý"% dÔ&;¸dÔ>TÑ&TÑ"UÔ"UˆDÔ Ý!¸dÔ>^Ð_Ñ_Ô_ˆ Ø&ˆÔØ%Ô3ˆŒà Ô ,¨tÔ/OÑ OÐRVÔRgÑ gð
|
|||
|
|
Ô õ !$ DÔ$DÀtÄÑ$VÑ WÔ WˆÔ Ý ˜dÔ 3°d´oÑ EÑFÔFˆŒÝ (Ø ŒO˜TÔ 2Ð4kñ
|
|||
|
|
ô
|
|||
|
|
ˆÔ õ &/Ø Ô ! 4Ô#8Ð:wñ&
|
|||
|
|
ô &
|
|||
|
|
ˆÔ"õ "&¤Ø Ô $¤/Ñ 1ñ"
|
|||
|
|
ô "
|
|||
|
|
ˆÔ õ ”l¥3¥t¤y¡{¤{Ñ#3Ô#3¸KÔ<NÐOÑOÔOˆ Ý ˜[¨!Ñ,Ô,×1Ò1Ñ3Ô3ˆØ œ=ÐCÐC¨D¬IÐCÐC¸ÐCÐCˆŒ
|
|||
|
|
Ø œ) kÔ&?À&Ñ&HÑHˆŒØ Ô &¨Ò *Ð *Ý+.¨q°$Ô2HÈDÔLgÑ2gÑ+hÔ+hˆDÔ (Ý+4Ø Ô ! 4¤;Ð0añ,
|
|||
|
|
ô ,
|
|||
|
|
ˆÔ(ð ˜z¨<Ð8ð 1ð 1ˆFݘ&¥"¤)Ñ,Ô,ð
|
|||
|
|
1Ý(¨Ñ0Ô0Ð0øØ Œ?ð D˜tœ°%Ò 7Ð 7Ø!%Ô!6Ô!CˆDÔ ØˆŒ
|
|||
|
|
Ø ×+Ò+Ø #Ô 5ð ,ñ
|
|||
|
|
ô
|
|||
|
|
ð
|
|||
|
|
õ .Õ0SÐTXÔT]ÔTgÑ0hÔ0hÑ hÐØ.7Ð.?Ð*Ð*ÐEVÐYbÑEbˆŒÝ /Ø ŒN˜DœJ¨Ô(=¸t¼~ÈtÔO`ñ!
|
|||
|
|
ô !
|
|||
|
|
ˆÔ ð
|
|||
|
|
×Ò¨T¬YÔ-CÐb�/˜/ÕIbÑcÔcÐcÝ%Ñ'Ô'ˆŒ Ý'Ø"&×"<Ò"<Ñ">Ô">Ø"&×"<Ò"<Ñ">Ô">ð ð Ø !Ô 2Ô <ÀÄ ¸~Ñ Mð ñ ô ð
|
|||
|
|
ñ
|
|||
|
|
ô
|
|||
|
|
ˆŒ
|
|||
|
|
ð ˆÔØ!%ˆÔ Ý$+¨DÔ,<Ô,BÐDVÐX\Ñ$]Ô$]ÐeiÐ$iˆÔ!Ý & tÔ'7Ô'=¸}ÈdÑ SÔ SÐ[_Ð _ˆÔ à ˆÔØ Œ9Ô ð Ø × Ò Ñ Ô Ð Ø Œ9Ô ð =Ý ŒK˜œ Ô,°tÐ <Ñ <Ô <Ð <Ø ˆÔõ �4”:Ð /Ñ 0Ô 0ð 7Ø ŒJ× %Ò % d¤oÑ 6Ô 6Ð 6õ
|
|||
|
|
%Ø Ô ØÔ7ØØÔ)Øð
|
|||
|
|
ñ
|
|||
|
|
ô
|
|||
|
|
ˆŒõ Ô˜$œ)Ñ$Ô$Ð$Ø6A×6IÒ6IÈ$Ì*ÐVZÔVdÐfjÔfuÑ6vÔ6vÑ3ˆŒ
|
|||
|
|
�D”N D¤OÝ
|
|||
|
|
Ô˜$œ/Ñ*Ô*Ð*å )Ø Ô ØÔ6ØÔ)Øð
|
|||
|
|
ñ
|
|||
|
|
ô
|
|||
|
|
ˆÔ ð +× 2Ò 2°4Ô3GÑ HÔ HˆÔ à Ô $ð Rݘ$Ô+R¬YÑ7Ô7ð
|
|||
|
|
Ý$5ØÔ% tÔ'GÈÌÐTXÔT]ñ%ô %�Ô!õ 0Ø” Ô!AÀ4Ä9ÈdÌiñ ô ˆDŒOð "œZˆDŒNˆNˆNà "œo× 0Ò 0°Ô1AÔ1HÑ IÔ IˆDŒOݘ$Ô+R¬YÑ7Ô7ð
|
|||
|
|
RØ$(Ô$5×$8Ò$8¸Ô9IÔ9PÑ$QÔ$Q�Ô!Ð!Ð!ð
|
|||
|
|
Rð
|
|||
|
|
Rrm c ó — | j S ©N©r˜ ©rB s rk Úget_train_dataloaderz(_UnslothRLOOTrainer.get_train_dataloader` s
|
|||
|
|
€ ،Рrm c ó — | j S r¡ )r› r£ s rk Úget_eval_dataloaderz'_UnslothRLOOTrainer.get_eval_dataloaderc s € ØÔ#Ð#rm c óŒ& ‡^— | j }| j }| j }| j }| j | _ | j }| j }| j }| j Š^|j }ˆ^fd„} t | ¦ « ¦ « }
|
|||
|
|
t |j |j
|
|||
|
|
dz ddd¬¦ « } | d¦ « t j ¦ « } |j |j |j f}
|
|||
|
|
t' j |
|
|||
|
|
|¬¦ « }t' j |
|
|||
|
|
|¬¦ « }t' j |
|
|||
|
|
|¬¦ « }t' j |
|
|||
|
|
|¬¦ « }t' j |
|
|||
|
|
|¬¦ « }t' j |
|
|||
|
|
|¬¦ « }| ¦ « d | j _ d | j _ |j |j z d
|
|||
|
|
z | j _ |j | j z | j _ |j �M|j d k r1t? j | j j |j z ¦ « | j _ n|j | j _ |j! �M|j! d k r1t? j | j j |j! z ¦ « | j _! n|j! | j _! |j" �M|j" d k r1t? j | j j |j" z ¦ « | j _" n|j" | j _" | j# $ || j | j% ¦ « | _% tM d |j d z ¦ « D �]×}| j xj d |j' z z
|
|||
|
|
c_ tQ |
|
|||
|
|
¦ « }t' j) ¦ « 5 |d * |¦ « }| + |j, d ¦ « }|j- d }g }g }g }g }g } g } t] | j | j | j j/ ¬
|
|||
|
|
¦ « 5 } ta | ||j1 |j2 | ¦ « \ } } d d d ¦ « n # 1 swxY w Y tM d |j- d |j1 ¦ « D �]}!||!|!|j1 z … }"| |!|!|j1 z … }#|#d d …|d …f }$| |!|!|j1 z … }%tg |%|$¦ « }&~%ti ¦ « tk ||#|j2 ¦ « }'|'j6 d d …|d z
|
|||
|
|
d…f }(|(|j
|
|||
|
|
dz z }(tg |(|$¦ « })~'~(ti ¦ « |$}*|j7 �tq |j7 |j2 |$¦ « }*t' j9 |"|*fd ¦ « }+tu |*|j2 k ¦ « d z
|
|||
|
|
},tw |tx j= ¦ « r t} ||+|j2 |¦ « \ }-}.}-nQt' j? || @ |+d¬¦ « ¦ « t& jA ¬¦ « * |¦ « }.| B |$¦ « | B |*¦ « | B |&¦ « | B |)¦ « | B |,¦ « | B |.¦ « �Œ t' j9 |d ¦ « }t' j9 |d ¦ « }t' j9 |d ¦ « }t' j9 |d ¦ « }t' j9 | d ¦ « } t' j9 | d ¦ « } ~&~)~.ti ¦ « t‡ jD ¦ « t' jE ||jF k d¬¦ « }/|jG �| |/ xx | j jG z cc< t' jH |j- d |j ¬¦ « + |j- d d ¦ « }0|0| I d ¦ « k }1t' jJ ||1t– ¦ « }t' jJ ||1t– ¦ « }||z
|
|||
|
|
}2|jL rP| | M ¦ « z
|
|||
|
|
| N ¦ « dz z } t' jO | |jP |jP ¦ « } |jQ ré|jR |2z }3|1 S d ¦ « d z
|
|||
|
|
|1 T ¦ « U ¦ « V d d¬¦ « z
|
|||
|
|
}4t' jW |2¦ « }5| X dd ¦ « * |2jY ¦ « }6|5 Z d |4|6¬¦ « |3 [ d ¦ « }7|5|3z }8|8 [ d ¦ « }9n%|2 [ d ¦ « }:|jR |:z }7|7| z }9|9 X |j, d¦ « }9|9 [ d ¦ « |9z
|
|||
|
|
|j, d z
|
|||
|
|
z };|9|;z
|
|||
|
|
}<|< \ ¦ « }<|j] r/|<|< M ¦ « z
|
|||
|
|
|< N ¦ « dz z }<ti ¦ « d d d ¦ « n # 1 swxY w Y tM |j ¦ « D �]v}=t¼ j_ ` |ja ¦ « }>d }?tM d |ja |jb ¦ « D �]0}@|@|jb z }A|>|@|A… }Bd }CtM d |jb |jc ¦ « D �]Ö}D| d |¦ « 5 |D|jc z }E|B|D|E… }F|<|F }G||F }H| |F }I||F }Jtk ||I|j2 ¦ « }K|Kj6 d d …|d z
|
|||
|
|
d…f }%|%|j
|
|||
|
|
dz z }%tg |%|H¦ « }Lt' jJ |L|1|F t– ¦ « }L|L|Jz
|
|||
|
|
e ¦ « }M|L [ d ¦ « }L|J [ d ¦ « }J|L|Jz
|
|||
|
|
}Nt' je |N¦ « }O|G |Oz }P|G t' jO |Od|jf z
|
|||
|
|
d|jf z ¦ « z }Qt' jg |P|Q¦ « }R|R M ¦ « }S|S}T| h |T¦ « | i ¦ « | j ¦ « t' j) ¦ « 5 |Q|Pk A ¦ « M ¦ « }Ut& j< jk l |%d¬¦ « }Vt' jm |%d¬¦ « t' j[ |V|%z d¬¦ « z
|
|||
|
|
}Wd|Nd
|
|||
|
|
z M ¦ « z }X|X||=|?|Cf< |U||=|?|Cf< |S||=|?|Cf< |W M ¦ « ||=|?|Cf< |M M ¦ « ||=|?|Cf< d d d ¦ « n # 1 swxY w Y d d d ¦ « n # 1 swxY w Y |Cd z
|
|||
|
|
}C�ŒØ|?d z
|
|||
|
|
}?~K~%~L~N~O~P~Q~S~T~U~V~W~X~G~H~I~Jti ¦ « �Œ2�Œxt' j) ¦ « 5 |2 [ d ¦ « M ¦ « }Y| [ d ¦ « M ¦ « }Z|7 M ¦ « }[tÝ | j j t j ¦ « | z
|
|||
|
|
z ¦ « }\i }]|\|]d< | j o |Y¦ « M ¦ « p ¦ « |]d< | j o |Z¦ « M ¦ « p ¦ « |]d< | j o |[¦ « M ¦ « p ¦ « |]d< | j o |9¦ « M ¦ « p ¦ « |]d< | j o | M ¦ « ¦ « M ¦ « p ¦ « |]d< | j o |¦ « M ¦ « p ¦ « |]d < | j o |¦ « M ¦ « p ¦ « |]d < | j o |¦ « M ¦ « p ¦ « |]d < | j o |¦ « M ¦ « p ¦ « |]d < | j o |¦ « M ¦ « p ¦ « |]d < | j o |¦ « M ¦ « p ¦ « |]d!< | j o |¦ « q ¦ « p ¦ « |]d"< ||jF k [ ¦ « p ¦ « |]d#< | jr s ¦ « d |]d$< | j j |]d%< | j j |j, | j z z | j _t | u |]¦ « d d d ¦ « n # 1 swxY w Y ~2~Y~Z~ | jr i ¦ « | j xj d z
|
|||
|
|
c_ | j# v || j | j% ¦ « | _% | j% jw rG| x |d ¬&¦ « | j# y | j | j | j% ¦ « | _% ti ¦ « t‡ jD ¦ « |jz d k r'|d z
|
|||
|
|
| j{ z d k r| | d¬'¦ « �ŒÙ| j# } || j | j% ¦ « | _% | j% jw rJ| x |d d ¬(¦ « | j# y | j | j | j% ¦ « | _% d S d S ))Nc 3 ó •K — ‰ E d {V —† Œ r¡ r: r¢ s €rk Úrepeat_generatorz3_UnslothRLOOTrainer.train.<locals>.repeat_generatorr s) øè è € ð
|
|||
|
|
&Ø%Ð%Ð%Ð%Ð%Ð%Ð%Ð%ð
|
|||
|
|
&rm r™ r‰ r} T©Úmax_new_tokensr$ Útop_kÚtop_pÚ do_samplez===training policy===r^ r rv rU Ú input_ids©Úgather_deepspeed3_paramsrO ©Úskip_special_tokens©ÚdtyperT r| )rR Úkeepdim)rR rS Úsrcg à?Úepsz objective/klzobjective/entropyzobjective/non_score_rewardzobjective/rlhf_rewardzobjective/scoreszpolicy/approxkl_avgzpolicy/clipfrac_avgzloss/policy_avgzval/clipfrac_avgzpolicy/entropy_avgz val/ratioz
|
|||
|
|
val/ratio_varzval/num_eos_tokensÚlrÚepisode)Útrial)Úsampling)r» Úmetrics)~rs r| ry r‡ Ú
|
|||
|
|
model_wrappedrU rV rS r˜ r_ Úiterr r! r$ Úprintr= r0 r r§ r> ÚzerosÚtrainrŒ Ú global_steprº r( r² r rx r± r¼ r4 r~ rÙ r¿ r‰ Úon_train_beginr‹ Úranger+ ÚnextÚno_gradrZ Úrepeatr4 rX r@ r9 r# r rv r; r' r* rb r# r? Úcatr) rd r5 r† r0 r Ú batch_decodeÚfloatr` r, ÚcollectÚanyru r% Úaranger] Ú masked_fillr r5 ÚmeanÚstdÚclampr6 r8 r2 ÚsizeÚlongÚfliplrÚargmaxÚ
|
|||
|
|
zeros_likerW rµ Úscatter_ÚsumÚflattenr7 r6 ÚrandomÚ permutationr* r, r£ Ú
|
|||
|
|
accumulateÚexpr3 rƒ ÚbackwardÚstepÚ zero_gradr Úsoftmaxr_ rJ Úgather_for_metricsr€ Úvarrz Ú get_last_lrÚepochÚlogÚ on_step_endr“ Ú_save_checkpointÚon_saver r„ Úgenerate_completionsÚ on_train_end)_rB rs r| ry r‡ rU rV rS r_ r© Úiter_dataloaderrt Ú
|
|||
|
|
start_timeÚ stats_shapeÚapproxkl_statsÚpg_clipfrac_statsÚ
|
|||
|
|
pg_loss_statsÚvf_clipfrac_statsÚ
|
|||
|
|
entropy_statsÚ ratio_statsÚupdateÚdataÚqueriesÚcontext_lengthÚ responsesÚpostprocessed_responsesÚlogprobsÚ ref_logprobsÚscoresÚsequence_lengthsÚunwrapped_modelÚquery_responsesÚlogitssÚiÚqueryÚquery_responseÚresponserb ÚlogprobÚ
|
|||
|
|
ref_outputÚ
|
|||
|
|
ref_logitsÚ ref_logprobÚpostprocessed_responseÚ postprocessed_query_responseÚsequence_lengthÚ_ÚscoreÚcontain_eos_tokenÚ
|
|||
|
|
response_idxsÚ padding_maskÚklÚ kl_rewardÚ eos_indicesÚ last_rewardÚ
|
|||
|
|
scores_shapedÚnon_score_rewardÚrewardÚ rlhf_rewardÚ sequence_klÚbaselineÚ
|
|||
|
|
advantagesÚ
|
|||
|
|
ppo_epoch_idxÚb_indsÚ
|
|||
|
|
minibatch_idxÚmini_batch_startÚmini_batch_endÚmini_batch_indsÚgradient_accumulation_idxÚmicro_batch_startÚmicro_batch_endÚmicro_batch_indsÚ mb_advantageÚ mb_responsesÚmb_query_responsesÚ mb_logprobsÚoutputÚ new_logprobsÚ new_ratioÚ
|
|||
|
|
logprobs_diffÚratioÚ pg_lossesÚ
|
|||
|
|
pg_losses2Ú pg_loss_maxÚpg_lossÚlossÚ pg_clipfracÚ prob_distÚentropyÚapproxklÚmean_klÚ mean_entropyÚmean_non_score_rewardr¸ r½ r˜ s_ @rk r z_UnslothRLOOTrainer.trainf s, ø€ ØŒyˆØÔ&ˆ Ø”Nˆ Ø”
|
|||
|
|
ˆØ !œZˆÔØ”_ˆ
|
|||
|
|
ØÔ(ˆ Ø Ô0ÐØ”_ˆ
|
|||
|
|
Ø Ô#ˆð &ð &ð &ð &ð &õ Ð /Ð /Ñ 1Ô 1Ñ2Ô2ˆÝ ,Ø Ô/Ø Ô)¨DÑ0ØØØð
|
|||
|
|
ñ
|
|||
|
|
ô
|
|||
|
|
Ðð ×ÒÐ1Ñ2Ô2Ð2Ý”Y‘[”[ˆ
|
|||
|
|
ØÔ*¨DÔ,AÀ4ÔCcÐdˆ Ý œ [¸Ð@Ñ@Ô@ˆÝ !œK¨ ¸FÐ CÑ CÔ CÐÝ œ K¸Ð?Ñ?Ô?ˆ
|
|||
|
|
Ý !œK¨ ¸FÐ CÑ CÔ CÐÝ œ K¸Ð?Ñ?Ô?ˆ
|
|||
|
|
Ý”k +°fÐ=Ñ=Ô=ˆ Ø
|
|||
|
|
� Š ‰
|
|||
|
|
Œ
|