308 lines
29 KiB
Plaintext
308 lines
29 KiB
Plaintext
|
o
|
|||
|
|
>—°hß ã @ s d Z ddlmZ ddlZddlmZ ddlmZ ddlmZm Z m
|
|||
|
|
Z
|
|||
|
|
m Z m Z m
|
|||
|
|
Z
|
|||
|
|
mZmZ ddlmZmZmZmZmZmZmZmZmZmZmZmZm Z m Z m
|
|||
|
|
Z
|
|||
|
|
m Z m Z m Z m!Z!m"Z"m#Z#m$Z$m%Z%m&Z&m Z m'Z'm(Z(m)Z)m*Z*m+Z+m,Z,m-Z-m.Z.m/Z/m0Z0m1Z1m2Z2m3Z3m4Z4m5Z5m6Z6m7Z7m8Z8mZm9Z9m:Z:m;Z;m<Z<m=Z=m>Z>m?Z?m@Z@mZmAZAmBZBm
|
|||
|
|
Z
|
|||
|
|
m$Z$m:Z:mZ ddl:Z:ddlT ddlCmDZDmEZE dd lFmGZG ddlZddlHZ9dd
|
|||
|
|
lImJZJ ddlmZ dd lKmLZLmMZN d d
|
|||
|
|
d d
|
|||
|
|
d
|
|||
|
|
dœZOejPd d eOd�dd„ ƒZQeDG dd„ de"ƒƒZR G dd„ de$ƒZSG dd„ deSƒZTdS )z9
|
|||
|
|
2025.8.9
|
|||
|
|
2025.8.10
|
|||
|
|
4.55.4
|
|||
|
|
0.21.0
|
|||
|
|
__UNSLOTH_VERSIONING__
|
|||
|
|
é )ÚTensorN)Ú
|
|||
|
|
functional)ÚAnyÚListÚOptionalÚTupleÚUnionÚDictÚSetÚCallable);Ú AcceleratorÚBaseImageProcessorr ÚCallbackHandlerÚDEFAULT_CALLBACKSÚDEFAULT_PROGRESS_CALLBACKÚDataCollatorWithPaddingÚ
|
|||
|
|
DataLoaderÚDatasetÚExportableStateÚFeatureExtractionMixinÚGenerationConfigÚINVALID_LOGPROBÚOnlineTrainerStater ÚPathÚPreTrainedTokenizerBaseÚPrinterCallbackÚProcessorMixinÚ
|
|||
|
|
RLOOConfigÚ RLOOTrainerÚTrainerÚTrainerCallbackÚTrainerControlr Úbatch_generationÚ broadcastÚ defaultdictÚdisable_dropout_in_modelÚ empty_cacheÚ exact_divÚfirst_true_indicesÚforwardÚ
|
|||
|
|
gather_objectÚgcÚgenerate_model_cardÚget_comet_experiment_urlÚ#get_reporting_integration_callbacksÚ
|
|||
|
|
get_rewardÚis_rich_availableÚis_wandb_availableÚ log_table_to_comet_experimentÚmathÚnnÚnpÚosÚpdÚprepare_deepspeedÚprint_rich_tableÚselective_log_softmaxÚtextwrapÚtimeÚtorchÚtruncate_responseÚunwrap_model_for_generationr r r6 r= )Ú*)Ú dataclassÚfield)ÚVersion)Ú nullcontext)ÚDataCollatorForSeq2SeqÚ DataCollatorForLanguageModelingTF)Úepilogue_fusionÚ max_autotuneÚ
|
|||
|
|
shape_paddingz
|
|||
|
|
trace.enabledztriton.cudagraphs)ÚdynamicÚ fullgraphÚoptionsc
|
|||
|
|
C s¾ t j| d| jd ¡ddd�}t j| d¡ddd�}g }t||ƒD ](\}}| t j¡}t j|d| d¡d� d¡}t j
|
|||
|
|
|dd�}|| } | | ¡ q! t |¡}| | jd | jd f¡}|S )Néÿÿÿÿé r )ÚchunksÚdim)rP Úindex©rP é )
|
|||
|
|
r= ÚchunkÚreshapeÚshapeÚzipÚtoÚfloat32ÚgatherÚ unsqueezeÚsqueezeÚ logsumexpÚappendÚconcat)
|
|||
|
|
ÚlogitsrQ Úchunked_logitsÚ
|
|||
|
|
chunked_indexÚall_per_token_logpsÚ chunk_logitsÚ chunk_indexÚselected_logitsÚlogsumexp_valuesÚper_token_logps© ri úR/workspace/DS-LLM-TEMPLATE-FINETUNING/unsloth_compiled_cache/UnslothRLOOTrainer.pyÚ chunked_selective_log_softmax" s
|
|||
|
|
rk c sœ e Zd ZU dZedddid�Zee ed< edddid�Z ee
|
|||
|
|
ed <
|
|||
|
|
|
|||
|
|
|
|||
|
|
|
|||
|
|
|
|||
|
|
|
|||
|
|
|
|||
|
|
|
|||
|
|
|
|||
|
|
|
|||
|
|
|
|||
|
|
|
|||
|
|
|
|||
|
|
|
|||
|
|
|
|||
|
|
|
|||
|
|
|
|||
|
|
|
|||
|
|
!
|
|||
|
|
|
|||
|
|
|
|||
|
|
"
|
|||
|
|
|
|||
|
|
|
|||
|
|
" # $
|
|||
|
|
|
|||
|
|
%
|
|||
|
|
|
|||
|
|
|
|||
|
|
&
|
|||
|
|
|
|||
|
|
|
|||
|
|
! "
|
|||
|
|
' (
|
|||
|
|
|
|||
|
|
|
|||
|
|
|
|||
|
|
|
|||
|
|
|
|||
|
|
|
|||
|
|
) * + , - . -
|
|||
|
|
/ 0
|
|||
|
|
|
|||
|
|
1
|
|||
|
|
|
|||
|
|
d4‡ fd2d3„ Z ‡ Z S )5ÚUnslothRLOOConfigaÆ
|
|||
|
|
|
|||
|
|
Configuration class for the [`RLOOTrainer`].
|
|||
|
|
|
|||
|
|
This class includes only the parameters that are specific to RLOO training. For a full list of training arguments,
|
|||
|
|
please refer to the [`~transformers.TrainingArguments`] and [`OnPolicyConfig`] documentation. Note that default
|
|||
|
|
values in this class may differ from those in [`~transformers.TrainingArguments`].
|
|||
|
|
|
|||
|
|
Using [`~transformers.HfArgumentParser`] we can turn this class into
|
|||
|
|
[argparse](https://docs.python.org/3/library/argparse#module-argparse) arguments that can be specified on the
|
|||
|
|
command line.
|
|||
|
|
|
|||
|
|
Parameters:
|
|||
|
|
exp_name (`str`, *optional*, defaults to `os.path.basename(__file__)[: -len(".py")]`):
|
|||
|
|
Name of this experiment.
|
|||
|
|
reward_model_path (`str`, *optional*, defaults to `"EleutherAI/pythia-160m"`):
|
|||
|
|
Path to the reward model.
|
|||
|
|
num_ppo_epochs (`int`, *optional*, defaults to `4`):
|
|||
|
|
Number of epochs to train.
|
|||
|
|
whiten_rewards (`bool`, *optional*, defaults to `False`):
|
|||
|
|
Whether to whiten the rewards.
|
|||
|
|
kl_coef (`float`, *optional*, defaults to `0.05`):
|
|||
|
|
KL coefficient.
|
|||
|
|
cliprange (`float`, *optional*, defaults to `0.2`):
|
|||
|
|
Clip range.
|
|||
|
|
rloo_k (`int`, *optional*, defaults to `2`):
|
|||
|
|
REINFORCE Leave-One-Out (RLOO) number of online samples per prompt.
|
|||
|
|
normalize_reward (`bool`, *optional*, defaults to `False`):
|
|||
|
|
Whether to normalize rewards.
|
|||
|
|
reward_clip_range (`float`, *optional*, defaults to `10.0`):
|
|||
|
|
Clip range for rewards.
|
|||
|
|
normalize_advantage (`bool`, *optional*, defaults to `False`):
|
|||
|
|
Whether to normalize advantages.
|
|||
|
|
token_level_kl (`bool`, *optional*, defaults to `True`):
|
|||
|
|
Whether to use token-level KL penalty or sequence-level KL penalty.
|
|||
|
|
ds3_gather_for_generation (`bool`, *optional*, defaults to `True`):
|
|||
|
|
This setting applies to DeepSpeed ZeRO-3. If enabled, the policy model weights are gathered for generation,
|
|||
|
|
improving generation speed. However, disabling this option allows training models that exceed the VRAM
|
|||
|
|
capacity of a single GPU, albeit at the cost of slower generation.
|
|||
|
|
|
|||
|
|
NÚhelpzvLLM SamplingParams)ÚdefaultÚmetadataÚvllm_sampling_paramsrM z8Chunk size to reduce memory usage. -1 is most efficient.Úunsloth_num_chunksFÚnorN é r éú ç-C ëâ6
|
|||
|
|
?ç{®Gáz„?çÍÌÌÌÌÌì?ç+‡ÙÎ÷ï?ç:Œ0âŽyE>ç ð?ç @Úlinearçš™™™™™¹?ÚpassiveÚwarningTÚstepsrS éô éO
|
|||
|
|
ÚO1ÚautoÚ ç Ú
|
|||
|
|
adamw_8bitÚlengthÚ
|
|||
|
|
every_saveÚlasté é@ é
|
|||
|
|
é5 çffffffæ?úEleutherAI/pythia-160mÚ rloo_configçš™™™™™©?çš™™™™™É?ç $@c¡ £ sj |dk r t d|› d�ƒ‚|dkrtd|› d�ƒ‚|d u r(|#dkr(|$dkr(d}d }#|�d u r:d
|
|||
|
|
d lm}¢ t|¢ƒ d d
|
|||
|
|
ƒ}�|‰d
|
|||
|
|
krBtdƒ‚|‰dkrJtdƒ‚tƒ jd¯i d|“d|“d|“d|“d|“d|“d|“d|“d| “d|
|
|||
|
|
“d| “d | “d |
|
|||
|
|
“d |“d |“d |“d!|“d"|“d#|“d$|“d%|“d&|“d'|“d(|“d)|“d*|“d+|“d,| “d-| “d.| “d/| “d0| “d1|!“d2|"“d3|#“d4|$“d5|%“d6|&“d7|'“d8|(“d9|)“d:|*“d;|+“d<|,“d=|-“d>|.“d?|/“d@|0“dA|1“dB|2“dC|3“dD|4“dE|5“dF|6“dG|7“dH|8“dI|9“dJ|:“dK|;“dL|<“dM|=“dN|>“dO|?“dP|@“dQ|A“dR|B“dS|C“dT|D“dU|E“dV|F“dW|G“dX|H“dY|I“dZ|J“d[|K“d\|L“d]|M“d^|N“d_|O“d`|P“da|Q“db|R“dc|S“dd|T“de|U“df|V“dg|W“dh|X“di|Y“dj|Z“dk|[“dl|\“dm|]“dn|^“do|_“dp|`“dq|a“dr|b“ds|c“dt|d“du|e“dv|f“dw|g“dx|h“dy|i“dz|j“d{|k“d||l“d}|m“d~|n“d|o“d€|p“d�|q“d‚|r“dƒ|s“d„|t“d…|u“d†|v“d‡|w“dˆ|x“d‰|y“dŠ|z“d‹|{“dŒ||“d�|}“dŽ|~“d�|“d�|€“d‘|�“d’|‚“d“|ƒ“d”|„“d•|…“d–|†“d—|‡“d˜|ˆ“d™|‰“dš|Š“d›|‹“dœ|Œ“d�|�“dž|Ž“dŸ|�“d |�“d¡|‘“d¢|’“d£|““d¤|”“d¥|•“d¦|–“d§|—“d¨|˜“d©|™“dª|š“d«|›“d¬|œ“d|�“d®|ž“|¡¤Ž |Ÿ| _| | _ d S )°NçH¯¼šò×z>z Unsloth: Your learning rate of `zi` is too small and less than 1e-7! Consider increasing it, otherwise gradient updates will be close to 0!rS za` is way too larger > 1! Consider decreasing it to 1e-1, otherwise gradient updates will explode!r€ r� Ú unsloth_training_checkpointsrr r )Ú cpu_countrN rs zUUnsloth: Please set a positive non-zero temperature since your results will be wrong.r� zgUnsloth: Please set a positive non-zero temperature less than 10, since sampling will be quite erratic.Ú
|
|||
|
|
output_dirÚoverwrite_output_dirÚdo_trainÚdo_evalÚ
|
|||
|
|
do_predictÚ
|
|||
|
|
eval_strategyÚprediction_loss_onlyÚper_device_train_batch_sizeÚper_device_eval_batch_sizeÚper_gpu_train_batch_sizeÚper_gpu_eval_batch_sizeÚgradient_accumulation_stepsÚeval_accumulation_stepsÚ
|
|||
|
|
eval_delayÚtorch_empty_cache_stepsÚ
|
|||
|
|
learning_rateÚ weight_decayÚ
|
|||
|
|
adam_beta1Ú
|
|||
|
|
adam_beta2Ú adam_epsilonÚ
|
|||
|
|
max_grad_normÚnum_train_epochsÚ max_stepsÚlr_scheduler_typeÚ warmup_ratioÚ warmup_stepsÚ log_levelÚlog_level_replicaÚlog_on_each_nodeÚ logging_dirÚlogging_strategyÚlogging_first_stepÚ
|
|||
|
|
logging_stepsÚlogging_nan_inf_filterÚ
|
|||
|
|
save_strategyÚ
|
|||
|
|
save_stepsÚsave_total_limitÚsave_safetensorsÚsave_on_each_nodeÚsave_only_modelÚ'restore_callback_states_from_checkpointÚno_cudaÚuse_cpuÚuse_mps_deviceÚseedÚ data_seedÚ
|
|||
|
|
jit_mode_evalÚuse_ipexÚbf16Úfp16Úfp16_opt_levelÚhalf_precision_backendÚbf16_full_evalÚfp16_full_evalÚtf32Ú
|
|||
|
|
local_rankÚ ddp_backendÚ
|
|||
|
|
tpu_num_coresÚtpu_metrics_debugÚdebugÚdataloader_drop_lastÚ
|
|||
|
|
eval_stepsÚdataloader_num_workersÚdataloader_prefetch_factorÚ
|
|||
|
|
past_indexÚrun_nameÚ disable_tqdmÚremove_unused_columnsÚ label_namesÚload_best_model_at_endÚmetric_for_best_modelÚgreater_is_betterÚignore_data_skipÚfsdpÚfsdp_min_num_paramsÚ fsdp_configÚ"fsdp_transformer_layer_cls_to_wrapÚaccelerator_configÚ deepspeedÚlabel_smoothing_factorÚoptimÚ
|
|||
|
|
optim_argsÚ adafactorÚgroup_by_lengthÚlength_column_nameÚ report_toÚddp_find_unused_parametersÚddp_bucket_cap_mbÚddp_broadcast_buffersÚdataloader_pin_memoryÚ dataloader_persistent_workersÚskip_memory_metricsÚuse_legacy_prediction_loopÚ push_to_hubÚresume_from_checkpointÚ hub_model_idÚ hub_strategyÚ hub_tokenÚhub_private_repoÚhub_always_pushÚ hub_revisionÚgradient_checkpointingÚ gradient_checkpointing_kwargsÚinclude_inputs_for_metricsÚeval_do_concat_batchesÚ fp16_backendÚpush_to_hub_model_idÚpush_to_hub_organizationÚpush_to_hub_tokenÚ
|
|||
|
|
mp_parametersÚauto_find_batch_sizeÚfull_determinismÚ torchdynamoÚ ray_scopeÚ ddp_timeoutÚ
|
|||
|
|
torch_compileÚtorch_compile_backendÚtorch_compile_modeÚinclude_tokens_per_secondÚ include_num_input_tokens_seenÚneftune_noise_alphaÚoptim_target_modulesÚbatch_eval_metricsÚ
|
|||
|
|
eval_on_startÚuse_liger_kernelÚliger_kernel_configÚeval_use_gather_objectÚ average_tokens_across_devicesÚdataset_num_procÚnum_mini_batchesÚtotal_episodesÚ local_rollout_forward_batch_sizeÚnum_sample_generationsÚresponse_lengthÚ
|
|||
|
|
stop_tokenÚ
|
|||
|
|
stop_token_idÚ temperatureÚmissing_eos_penaltyÚsft_model_pathÚ
|
|||
|
|
world_sizeÚnum_total_batchesÚmicro_batch_sizeÚlocal_batch_sizeÚ
|
|||
|
|
batch_sizeÚlocal_mini_batch_sizeÚmini_batch_sizeÚexp_nameÚreward_model_pathÚnum_ppo_epochsÚwhiten_rewardsÚkl_coefÚ cliprangeÚrloo_kÚnormalize_rewardÚreward_clip_rangeÚnormalize_advantageÚtoken_level_klÚds3_gather_for_generationri )
|
|||
|
|
ÚFloatingPointErrorÚ
|
|||
|
|
OverflowErrorÚmultiprocessingr— ÚmaxÚ MathErrorÚsuperÚ__init__rp rq )£Úselfr˜ r™ rš r› rœ r� rž rŸ r r¡ r¢ r£ r¤ r¥ r¦ r§ r¨ r© rª r« r¬ r r® r¯ r° r± r² r³ r´ rµ r¶ r· r¸ r¹ rº r» r¼ r½ r¾ r¿ rÀ rÁ r rà rÄ rÅ rÆ rÇ rÈ rÉ rÊ rË rÌ rÍ rÎ rÏ rÐ rÑ rÒ rÓ rÔ rÕ rÖ r× rØ rÙ rÚ rÛ rÜ rÝ rÞ rß rà rá râ rã rä rå ræ rç rè ré rê rë rì rí rî rï rð rñ rò ró rô rõ rö r÷ rø rù rú rû rü rý rþ rÿ r r r r r r r r r r r
|
|||
|
|
r r r
|
|||
|
|
r r r r r r r r r r r r r r r r r r r r! r" r# r$ r% r& r' r( r) r* r+ r, r- r. r/ r0 r1 r2 r3 r4 r5 rp rq Úkwargsr— ©Ú __class__ri rj r< g s & ÿþýüûúùø ÷
|
|||
|
|
ö õ ô
|
|||
|
|
óòñðïîíìëêéèçæå ä ã â á à!ß"Þ#Ý$Ü%Û&Ú'Ù(Ø)×*Ö+Õ,Ô-Ó.Ò/Ñ0Ð1Ï2Î3Í4Ì5Ë6Ê7É8È9Ç:Æ;Å<Ä=Ã>Â?Á@ÀA¿B¾C½D¼E»FºG¹H¸I·J¶KµL´M³N²O±P°Q¯R®ST¬U«VªW©X¨Y§Z¦[¥\¤]£^¢_¡` aŸbžc�dœe›fšg™h˜i—j–k•l”m“n’o‘p�q�rŽs�tŒu‹vŠw‰xˆy‡z†{…|„}ƒ~‚� �ÿ �þ �ý �ü �û �ú �ù �ø �÷
|
|||
|
|
�ö �õ �ô
|
|||
|
|
�ó �ò �ñ �ð �ï �î �í �ì �ë �ê �é �è �ç �æ �å �ä �ã �â �á
|
|||
|
|
zUnslothRLOOConfig.__init__) NNFFFrr FrN rN NNrs rs r rt ru rv rw rx ry rz r{ rM r| r} r r~ r TNr€ FrS Fr€ r� NTFFFFFFr‚ r‚ FFFFrƒ r„ FFNrM NNFr… FNr NrM NNTNFNNFr… r NNNNr† r‡ NFFrˆ NNNNTFTFFNNr‰ NNFNFNFTr„ NNNr… TFNrŠ r‹ FNNFFNNFFFNFTNrS NrŒ r� rŽ NNr� Nr� NNNNNNNr‘ r� rN Fr’ r“ rs Fr” FFTNrM )
|
|||
|
|
Ú__name__Ú
|
|||
|
|
__module__Ú __qualname__Ú__doc__rB rp r r Ú__annotations__rq Úintr< Ú
|
|||
|
|
__classcell__ri ri r? rj rl 3 sX
|
|||
|
|
)þþ �Þrl c s. e Zd ZddgZ d#dedeeeee e
|
|||
|
|
f de j de j d ee j e
|
|||
|
|
ee gee f f d
|
|||
|
|
ed ee d eeeeeef f d
|
|||
|
|
eejjejjjf deee ddfdd„Zde fdd„Z de fdd„Z dd„ Z d$de fdd„Z!‡ fd d „Z" d%d ee d ee d eeee df fd!d"„Z#‡ Z$S )&Ú_UnslothRLOOTrainerÚtrlÚrlooN©NNÚconfigÚprocessing_classÚpolicyÚ
|
|||
|
|
ref_policyÚ reward_modelÚ
|
|||
|
|
train_datasetÚ
|
|||
|
|
data_collatorÚ eval_datasetÚ
|
|||
|
|
optimizersÚ callbacksÚreturnc C s( ||u rt dƒ‚|| _|} || _|| _|d u r t| jƒ}d | jj_d | jj_|| _|| _ || _
|
|||
|
|
t |ƒ| _ || _
|
|||
|
|
|| _| \| _| _d | _| jd u rQt| j| j ƒ| _t| jd�} | | _| j| _| j| j | j | _ t| j| j ƒ| _ t| j | j ƒ| _ t | j | jdƒ| _ t | j | jdƒ| _!t" #| j| j ¡| _$t%j&tt' '¡ ƒ| j(d�}
|
|||
|
|
t)|
|
|||
|
|
dƒ *¡ }| j+› d| j,› d|› �| _-| j,| j.d | _/| j0dkrÎt1d | j$| j0 ƒ| _2t | j | j3d
|
|||
|
|
ƒ| _4|||fD ] }t5|t6j7ƒrèt8|ƒ qÜ| j9rö| j9d krö| jj| _:|| _;| j<| j$d � t=t>| jj?ƒ }|
|
|||
|
|
d u �r|n||
|
|||
|
|
| _@tA| j@| j;| j| j| jƒ| _B| C| jjD�r+tEntF¡ tGƒ | _HtI| J¡ | K¡ d
|
|||
|
|
d„ | jBj@| jHg D ƒd�| _Ld| _Md | _NtO| jjLdd ƒd u| _PtO| jjLdd ƒd u| _Qd | _R| jjS�rq| T¡ | jjU�rtVjW| jjXdd� d | _YtZ| j;dƒ�r�| j; [| j\¡ t]| j
|
|||
|
|
| j4d| j
|
|||
|
|
dd�| _^t% _| j,¡ | `| j;| j| j^¡\| _;| _| _^t% _| j/¡ t]| j| ja| j
|
|||
|
|
dd�| _b| `| jb¡| _b| jP�röt5| j t6j7ƒ�rätc| j | j| jd| jeƒ| _ tc| j| j| jd| jeƒ| _| j;| _fd S | j g| jj(¡| _t5| j t6j7ƒ�r| j g| jj(¡| _ d S d S )Nz `policy` and `ref_policy` cannot be the same object. If you want `ref_policy` to be the same as `policy`, you must mass a copy of it, or `None` if you use peft.)r£ z5`batch_size` must be a multiple of `num_mini_batches`z;`local_batch_size` must be a multiple of `num_mini_batches`©Údevicer Ú__i£† rS z/`local_batch_size` must be a multiple of rloo_kÚeos)Únum_training_stepsc S s g | ] }t |tƒr|‘qS ri )Ú
|
|||
|
|
isinstancer )Ú.0Úcbri ri rj Ú
|
|||
|
|
<listcomp>( s
|
|||
|
|
|
|||
|
|
ÿÿz0_UnslothRLOOTrainer.__init__.<locals>.<listcomp>)Úis_local_process_zeroÚis_world_process_zeroÚstateful_callbacksÚdeepspeed_pluginÚ fsdp_pluginT)Úexist_okÚadd_model_tags)r' ÚshuffleÚ
|
|||
|
|
collate_fnÚ drop_last)r' rh ri )hÚ
|
|||
|
|
ValueErrorÚargsrM rN r Úgeneration_configÚ eos_token_idÚ pad_token_idrO rP rQ ÚlenÚtrain_dataset_lenrR rS Ú optimizerÚ lr_schedulerÚoptimizer_cls_and_kwargsr rF r r r£ Ú acceleratorÚ
|
|||
|
|
num_processesr# rŸ r r& r% r' r' r) r( r3 Úceilr$ r= Útensorr< rX r# Úitemr* rÄ rÙ Ú
|
|||
|
|
process_indexÚ
|
|||
|
|
local_seedr r9 Úsample_generations_freqr0 Úlocal_dataloader_batch_sizer\ r4 ÚModuler% r r ÚmodelÚ create_optimizer_and_schedulerr r. rí rU r Úcallback_handlerÚ add_callbackrÚ r r r! Úcontrolr r` ra ÚstateÚ current_flosÚhp_search_backendÚgetattrÚis_deepspeed_enabledÚis_fsdp_enabledr÷ rõ Ú init_hf_repoÚ should_saver6 Úmakedirsr˜ Ú backup_modelÚhasattrrf Ú
|
|||
|
|
_tag_namesr Ú
|
|||
|
|
dataloaderÚ manual_seedÚpreparer Úeval_dataloaderr8 rÉ rÈ ræ rX )r= rL rM rN rO rP rQ rR rS rT rU rk rt Ú time_tensorÚtime_intÚmoduleÚdefault_callbacksri ri rj r< Á sä ÿ
|
|||
|
|
ÿ
|
|||
|
|
|
|||
|
|
|
|||
|
|
ÿ
|
|||
|
|
ÿ
|
|||
|
|
ÿ
|
|||
|
|
ÿ
|
|||
|
|
|
|||
|
|
ÿ €
|
|||
|
|
ÿÿÿý
|
|||
|
|
|
|||
|
|
û üÿÿ ÿz _UnslothRLOOTrainer.__init__c C ó | j S ©N©r� ©r= ri ri rj Úget_train_dataloaderc ó z(_UnslothRLOOTrainer.get_train_dataloaderc C r— r˜ )r’ rš ri ri rj Úget_eval_dataloaderf rœ z'_UnslothRLOOTrainer.get_eval_dataloaderc ^
|
|||
|
|
s | j }| j}| j}| j}| j| _| j}| j}| j}| j‰ |j }‡ fdd„} t
|
|||
|
|
| ƒ ƒ}
|
|||
|
|
t |j |j
|
|||
|
|
d dddd�} | d¡ t ¡ } |j|j|jf}
|
|||
|
|
tj|
|
|||
|
|
|d �}tj|
|
|||
|
|
|d �}tj|
|
|||
|
|
|d �}tj|
|
|||
|
|
|d �}tj|
|
|||
|
|
|d �}tj|
|
|||
|
|
|d �}| ¡ d
|
|||
|
|
| j_d
|
|||
|
|
| j_|j|j d | j_|j| j | j_ |j d ur¬|j d k r§t | jj|j ¡| j_ n|j | j_ |j!d urÈ|j!d k rÃt | jj|j! ¡| j_!n|j!| j_!|j"d urä|j"d k rßt | jj|j" ¡| j_"n|j"| j_"| j# $|| j| j%¡| _%t&d |jd ƒD �]ê}| j jd |j' 7 _t(|
|
|||
|
|
ƒ}t )¡ ��! |d
|
|||
|
|
*|¡}| +|j,d ¡}|j-d }g }g }g }g }g } g } t.| j| j| j j/d��} t0| ||j1|j2| ƒ\} } W d ƒ n 1 �sQw Y t&d
|
|||
|
|
|j-d
|
|||
|
|
|j1ƒD ]º}!||!|!|j1 … }"| |!|!|j1 … }#|#d d …|d …f }$| |!|!|j1 … }%t3|%|$ƒ}&~%t4ƒ t5||#|j2ƒ}'|'j6d d …|d d…f }(|(|j
|
|||
|
|
d }(t3|(|$ƒ})~'~(t4ƒ |$}*|j7d u�rÅt8|j7|j2|$ƒ}*t 9|"|*fd ¡}+t:|*|j2kƒd },t;|t<j=ƒ�rét>||+|j2|ƒ\}-}.}-ntj?||j@|+dd�ƒtjAd� *|¡}.| B|$¡ | B|*¡ | B|&¡ | B|)¡ | B|,¡ | B|.¡ �q`t 9|d
|
|||
|
|
¡}t 9|d
|
|||
|
|
¡}t 9|d
|
|||
|
|
¡}t 9|d
|
|||
|
|
¡}t 9| d
|
|||
|
|
¡} t 9| d
|
|||
|
|
¡} ~&~)~.t4ƒ tC D¡ tjE||jFkdd�}/|jGd u�rd| |/ | j jG8 < tjH|j-d |j d � +|j-d
|
|||
|
|
d ¡}0|0| Id ¡k}1t J||1tK¡}t J||1tK¡}|| }2|jL�r©| | M¡ | N¡ d } t O| |jP |jP¡} |jQ�rë|jR |2 }3|1 Sd ¡d |1 T¡ U¡ jVd dd� }4t W|2¡}5| Xdd ¡ *|2jY¡}6|5jZd |4|6d� |3 [d ¡}7|5|3 }8|8 [d ¡}9n|2 [d ¡}:|jR |: }7|7| }9|9 X|j,d¡}9|9 [d
|
|||
|
|
¡|9 |j,d };|9|; }<|< \¡ }<|j]�r%|<|< M¡ |< N¡ d }<t4ƒ W d ƒ n 1 �s3w Y t&|jƒD �][}=t^j_ `|ja¡}>d
|
|||
|
|
}?t&d
|
|||
|
|
|ja|jbƒD �]E}@|@|jb }A|>|@|A… }Bd
|
|||
|
|
}Ct&d
|
|||
|
|
|jb|jcƒD �]}D| d|¡�þ |D|jc }E|B|D|E… }F|<|F }G||F }H| |F }I||F }Jt5||I|j2ƒ}K|Kj6d d …|d d…f }%|%|j
|
|||
|
|
d }%t3|%|Hƒ}Lt J|L|1|F tK¡}L|L|J e¡ }M|L [d ¡}L|J [d ¡}J|L|J }Nt e|N¡}O|G |O }P|G t O|Od|jf d|jf ¡ }Qt g|P|Q¡}R|R M¡ }S|S}T| h|T¡ | i¡ | j¡ t )¡ �^ |Q|Pk A¡ M¡ }Utj<jkjl|%dtjmd� *|%jY¡}Vtjn|%dd�tj[|V|% dd� }Wd|Nd M¡ }X|X||=|?|Cf< |U||=|?|Cf< |S||=|?|Cf< |W M¡ ||=|?|Cf< |M M¡ ||=|?|Cf< W d ƒ n 1 �scw Y W d ƒ n 1 �ssw Y |Cd 7 }C�qi|?d 7 }?~K~%~L~N~O~P~Q~S~T~U~V~W~X~G~H~I~Jt4ƒ �qQ�q=t )¡ �ê |2 [d ¡ M¡ }Y| [d ¡ M¡ }Z|7 M¡ }[to| jjt ¡ | ƒ}\i }]|\|]d< | j p|Y¡ M¡ q¡ |]d< | j p|Z¡ M¡ q¡ |]d< | j p|[¡ M¡ q¡ |]d< | j p|9¡ M¡ q¡ |]d < | j p| M¡ ¡ M¡ q¡ |]d < | j p|¡ M¡ q¡ |]d < | j p|¡ M¡ q¡ |]d < | j p|¡ M¡ q¡ |]d < | j p|¡ M¡ q¡ |]d!< | j p|¡ M¡ q¡ |]d"< | j p|¡ M¡ q¡ |]d#< | j p|¡ r¡ q¡ |]d$< ||jFk [¡ q¡ |]d%< | js t¡ d
|
|||
|
|
|]d&< | jj|]d'< | jj|j,| j | j_u| v|]¡ W d ƒ n 1 �sŒw Y ~2~Y~Z~ | js i¡ | j jd 7 _| j# w|| j| j%¡| _%| j%jx�rÅ| jy|d d(� | j# z| j | j| j%¡| _%t4ƒ tC D¡ |j{d
|
|||
|
|
k�râ|d | j| d
|
|||
|
|
k�râ| j}dd)� q÷| j# ~|| j| j%¡| _%| j%jx�r | jy|d d d*� | j# z| j | j| j%¡| _%d S d S )+Nc 3 s � ˆ E d H qr˜ ri ri r™ ri rj Úrepeat_generatoru s €
|
|||
|
|
ÿz3_UnslothRLOOTrainer.train.<locals>.repeat_generatorr• r† rz T©Úmax_new_tokensr Útop_kÚtop_pÚ do_samplez===training policy===rW r rs rS Ú input_ids©Úgather_deepspeed3_paramsrM ©Úskip_special_tokens©ÚdtyperR ry )rP Úkeepdim)rP rQ Úsrc)rP rª g à?Úepsz objective/klzobjective/entropyzobjective/non_score_rewardzobjective/rlhf_rewardzobjective/scoreszpolicy/approxkl_avgzpolicy/clipfrac_avgzloss/policy_avgzval/clipfrac_avgzpolicy/entropy_avgz val/ratioz
|
|||
|
|
val/ratio_varzval/num_eos_tokensÚlrÚepisode)Útrial)Úsampling)r° Úmetrics)rk rt rq r~ Ú
|
|||
|
|
model_wrappedrO rP rM r� rX Úiterr r r Úprintr< r, r r£ r= ÚzerosÚtrainrƒ Ú global_stepr¯ r$ r® r rp r r¸ r3 rv rÕ r» r€ Úon_train_beginr‚ Úranger' ÚnextÚno_gradrX Úrepeatr0 rV r? r5 r" r rn r: r& r) r` r r> Úcatr( r\ r4 r} r/ rw Ú batch_decodeÚfloatr^ r+ ÚcollectÚanyrm r! Úaranger[ Ú masked_fillr r1 ÚmeanÚstdÚclampr2 r4 r. ÚsizeÚlongÚfliplrÚargmaxÚ
|
|||
|
|
zeros_likerU rª Úscatter_ÚsumÚflattenr3 r5 ÚrandomÚ permutationr& r( rŸ Ú
|
|||
|
|
accumulateÚexpr/ r9 ÚbackwardÚstepÚ zero_gradr ÚsoftmaxrY r] rF Úgather_for_metricsrx Úvarrr Ú get_last_lrÚepochÚlogÚ on_step_endrŠ Ú_save_checkpointÚon_saver r{ Úgenerate_completionsÚ on_train_end)^r= rk rt rq r~ rO rP rM rX rž Úiter_dataloaderrl Ú
|
|||
|
|
start_timeÚ stats_shapeÚapproxkl_statsÚpg_clipfrac_statsÚ
|
|||
|
|
pg_loss_statsÚvf_clipfrac_statsÚ
|
|||
|
|
entropy_statsÚ ratio_statsÚupdateÚdataÚqueriesÚcontext_lengthÚ responsesÚpostprocessed_responsesÚlogprobsÚ ref_logprobsÚscoresÚsequence_lengthsÚunwrapped_modelÚquery_responsesÚlogitssÚiÚqueryÚquery_responseÚresponser` ÚlogprobÚ
|
|||
|
|
ref_outputÚ
|
|||
|
|
ref_logitsÚ ref_logprobÚpostprocessed_responseÚ postprocessed_query_responseÚsequence_lengthÚ_ÚscoreÚcontain_eos_tokenÚ
|
|||
|
|
response_idxsÚ padding_maskÚklÚ kl_rewardÚ eos_indicesÚ last_rewardÚ
|
|||
|
|
scores_shapedÚnon_score_rewardÚrewardÚ rlhf_rewardÚ sequence_klÚbaselineÚ
|
|||
|
|
advantagesÚ
|
|||
|
|
ppo_epoch_idxÚb_indsÚ
|
|||
|
|
minibatch_idxÚmini_batch_startÚmini_batch_endÚmini_batch_indsÚgradient_accumulation_idxÚmicro_batch_startÚmicro_batch_endÚmicro_batch_indsÚ mb_advantageÚ mb_responsesÚmb_query_responsesÚ mb_logprobsÚoutputÚ new_logprobsÚ new_ratioÚ
|
|||
|
|
logprobs_diffÚratioÚ pg_lossesÚ
|
|||
|
|
pg_losses2Ú pg_loss_maxÚpg_lossÚlossÚ pg_clipfracÚ prob_distÚentropyÚapproxklÚmean_klÚ mean_entropyÚmean_non_score_rewardr r² ri r™ rj r· i sö
|
|||
|
|
û
|
|||
|
|
|
|||
|
|
|
|||
|
|
|
|||
|
|
|
|||
|
|
|
|||
|
|
|
|||
|
|
|
|||
|
|
|
|||
|
|
|
|||
|
|
|
|||
|
|
ÿ
|
|||
|
|
û ý
|
|||
|
|
|
|||
|
|
|
|||
|
|
ÿ
|
|||
|
|
ÿ ÿüû
|
|||
|
|
|
|||
|
|
|
|||
|
|
|
|||
|
|
|
|||
|
|
$ $
|
|||
|
|
|
|||
|
|
|
|||
|
|
� ö
|
|||
|
|
|
|||
|
|
|
|||
|
|
|
|||
|
|
|
|||
|
|
ÿ
|
|||
|
|
|
|||
|
|
|
|||
|
|
|
|||
|
|
|
|||
|
|
|
|||
|
|
ÿ õ€ Ö 6
|
|||
|
|
º
|
|||
|
|
Iÿ ç
|
|||
|
|
|
|||
|
|
€
|
|||
|
|
þz_UnslothRLOOTrainer.trainFr± c
|
|||
|
|
C s8 | j }| j}t| j jddddd�}ttƒ}t| j| j| j j d��±}| j
|
|||
|
|
D ]¥}|d }t ¡ �� |j
|
|||
|
|
d } t|||j
|
|||
|
|
d |j|ƒ\}
|
|||
|
|
} |
|
|||
|
|
d d …| d …f } | }
|
|||
|
|
|jd ur[t|j|j| ƒ}
|
|||
|
|
|d
|