365 lines
69 KiB
Plaintext
365 lines
69 KiB
Plaintext
|
§
|
|||
|
|
4$�h: ã ó~ — d Z ddlmZ ddlZddlmZ ddlmZ ddlmZm Z m
|
|||
|
|
Z
|
|||
|
|
m Z m Z m
|
|||
|
|
Z
|
|||
|
|
mZmZ ddlmZmZmZmZmZmZmZmZmZmZmZmZmZmZm Z m Z m Z m
|
|||
|
|
Z
|
|||
|
|
m Z m Z m!Z!m"Z"m#Z#m$Z$m%Z%m&Z&m Z m'Z'm(Z(m)Z)m*Z*m+Z+m,Z,m-Z-m.Z.m/Z/m0Z0m1Z1m2Z2m3Z3m4Z4mZm5Z5m6Z6m7Z7m8Z8mZm9Z9m:Z:m;Z;m<Z<m=Z=mZm/Z/m5Z5mZmZm
|
|||
|
|
Z
|
|||
|
|
m Z m!Z!m%Z%m0Z0m5Z5mZ ddl5Z5ddlT ddl>m?Z?m@Z@ dd lAmBZB ddlZddlCZDdd
|
|||
|
|
lEmFZF ddlmZ dd lGmHZHmIZJ d d
|
|||
|
|
d d
|
|||
|
|
d
|
|||
|
|
dœZK ejL d d eK¬¦ « d„ ¦ « ZMd„ ZNe? G d„ de ¦ « ¦ « ZO G d„ de%¦ « ZP G d„ deP¦ « ZQdS )z8
|
|||
|
|
2025.8.4
|
|||
|
|
2025.8.5
|
|||
|
|
4.55.1
|
|||
|
|
0.21.0
|
|||
|
|
__UNSLOTH_VERSIONING__
|
|||
|
|
é )ÚTensorN)Ú
|
|||
|
|
functional)ÚAnyÚListÚOptionalÚTupleÚUnionÚDictÚSetÚCallable)@r ÚAutoModelForCausalLMÚBaseImageProcessorÚBasePairwiseJudger ÚDPODataCollatorWithPaddingÚ DataCollatorÚ
|
|||
|
|
DataLoaderÚDatasetÚEvalPredictionÚFÚFeatureExtractionMixinÚGenerationConfigÚIterableDatasetÚOnlineDPOConfigÚOnlineDPOTrainerÚOptimizerNamesr ÚPathÚ PeftModelÚPreTrainedModelÚPreTrainedTokenizerBaseÚProcessorMixinÚSIMPLE_CHAT_TEMPLATEÚTrainerÚTrainerCallbackr Úapply_chat_templateÚcreate_reference_modelÚdatasetsÚdisable_dropout_in_modelÚ empty_cacheÚgenerate_model_cardÚget_comet_experiment_urlÚ
|
|||
|
|
get_rewardÚis_conversationalÚis_peft_availableÚis_wandb_availableÚjinja2ÚloggingÚmaybe_apply_chat_templateÚnnÚosÚprepare_deepspeedÚ seed_workerÚtextwrapÚtorchÚtruncate_rightÚunwrap_model_for_generationÚversionÚwarningsÚwrapsr r, r3 r7 r r r r r" r- r3 r7 )Ú*)Ú dataclassÚfield)ÚVersion)Ú nullcontext)ÚDataCollatorForSeq2SeqÚ DataCollatorForLanguageModelingTF)Úepilogue_fusionÚ max_autotuneÚ
|
|||
|
|
shape_paddingz
|
|||
|
|
trace.enabledztriton.cudagraphs)ÚdynamicÚ fullgraphÚoptionsc ó’ — t j | d| j d ¦ « dd¬¦ « }t j | d¦ « dd¬¦ « }g }t ||¦ « D ]‘\ }}| t j ¦ « }t j |d| d¦ « ¬¦ « d¦ « }t j
|
|||
|
|
|d¬¦ « }||z
|
|||
|
|
} | | ¦ « Œ’ t j |¦ « }| | j d | j d f¦ « }|S )Néÿÿÿÿé r )ÚchunksÚdim)rN Úindex©rN é )
|
|||
|
|
r7 ÚchunkÚreshapeÚshapeÚzipÚtoÚfloat32ÚgatherÚ unsqueezeÚsqueezeÚ logsumexpÚappendÚconcat)
|
|||
|
|
ÚlogitsrO Úchunked_logitsÚ
|
|||
|
|
chunked_indexÚall_per_token_logpsÚ chunk_logitsÚ chunk_indexÚselected_logitsÚlogsumexp_valuesÚper_token_logpss
|
|||
|
|
úc/workspace/Fine-tuning/DS-LLM-TEMPLATE-FINETUNING/unsloth_compiled_cache/UnslothOnlineDPOTrainer.pyÚ chunked_selective_log_softmaxrh " s5 € õ ”[ § ¢ °°F´LÀÔ4DÑ!EÔ!EÐPQÐYZÐ[Ñ[Ô[€NÝ”[ § ¢ ¨rÑ!2Ô!2¸QÀaÐHÑHÔH€MØ Ðå%(¨ ¸ Ñ%GÔ%Gð 4ð 4Ñ!ˆ �kØ#—’¥u¤}Ñ5Ô5ˆ Ý œ, |¸2À{×G\ÒG\Ð]_ÑG`ÔG`ÐaÑaÔa×iÒiÐjlÑmÔmˆÝ œ?¨<¸rÐBÑBÔBÐØ)Ð,<Ñ<ˆØ×"Ò" ?Ñ3Ô3Ð3Ð3ØÝ œ,Ð':Ñ;Ô;ÐØ-×5Ò5°v´|ÀA´ÈÌ ÐUVÌÐ6XÑYÔYÐØ Ð ó c ó0 — ddl m} |di | ¤Ž}| |_ |S )Nr )ÚSamplingParams© )Úvllmrk Ú _set_kwargs)Úkwargsrk Úsampling_paramss rg ÚvLLMSamplingParamsrq 3 s7 € Ø#Ð#Ð#Ð#Ð#Ð#Ø$�nÐ.Ð. vÐ.Ð.€OØ"(€OÔ Ø Ðri c ó® ‡ — e Zd ZU dZ edddi¬¦ « Zee ed< edddi¬¦ « Z ee
|
|||
|
|
ed < d/ˆ fd.„ Z ˆ xZ S )0ÚUnslothOnlineDPOConfigu¥
|
|||
|
|
|
|||
|
|
Configuration class for the [`OnlineDPOTrainer`].
|
|||
|
|
|
|||
|
|
This class includes only the parameters that are specific to Online DPO training. For a full list of training
|
|||
|
|
arguments, please refer to the [`~transformers.TrainingArguments`] documentation. Note that default values in this
|
|||
|
|
class may differ from those in [`~transformers.TrainingArguments`].
|
|||
|
|
|
|||
|
|
Using [`~transformers.HfArgumentParser`] we can turn this class into
|
|||
|
|
[argparse](https://docs.python.org/3/library/argparse#module-argparse) arguments that can be specified on the
|
|||
|
|
command line.
|
|||
|
|
|
|||
|
|
Parameters:
|
|||
|
|
reward_model_path (`str` or `None`, *optional*, defaults to `None`):
|
|||
|
|
Path to the reward model. Either `judge` or `reward_model_path` must be set, but not both.
|
|||
|
|
judge (`str` or `None`, *optional*, defaults to `None`):
|
|||
|
|
Name of the judge to use. Either `judge` or `reward_model_path` must be set, but not both.
|
|||
|
|
max_new_tokens (`int`, *optional*, defaults to `64`):
|
|||
|
|
Maximum number of tokens to generate per completion.
|
|||
|
|
max_length (`int`, *optional*, defaults to `256`):
|
|||
|
|
Maximum total length of the sequence (prompt + completion) used to compute log probabilities. If the
|
|||
|
|
sequence exceeds this limit, the leftmost tokens will be truncated to preserve as much of the completion as
|
|||
|
|
possible.
|
|||
|
|
temperature (`float`, *optional*, defaults to `0.9`):
|
|||
|
|
Temperature for sampling. The higher the temperature, the more random the completions.
|
|||
|
|
missing_eos_penalty (`float` or `None`, *optional*, defaults to `None`):
|
|||
|
|
Penalty applied to the score when the model fails to generate an EOS token. This is useful to encourage to
|
|||
|
|
generate completions shorter than the maximum length (`max_new_tokens`). The penalty must be a positive
|
|||
|
|
value.
|
|||
|
|
beta (`float` or `list[float]`, *optional*, defaults to `0.1`):
|
|||
|
|
Parameter controlling the deviation from the reference model. Higher β means less deviation from the
|
|||
|
|
reference model. For the IPO loss (`loss_type="ipo"`), β is the regularization parameter denoted by τ in
|
|||
|
|
the [paper](https://huggingface.co/papers/2310.12036). If a list of floats is provided then the β is
|
|||
|
|
selected for each new epoch and the last β is used for the rest of the epochs.
|
|||
|
|
loss_type (`str`, *optional*, defaults to `"sigmoid"`):
|
|||
|
|
Type of loss to use. Possible values are:
|
|||
|
|
|
|||
|
|
- `"sigmoid"`: sigmoid loss from the original [DPO](https://huggingface.co/papers/2305.18290) paper.
|
|||
|
|
- `"ipo"`: IPO loss from the [IPO](https://huggingface.co/papers/2310.12036) paper.
|
|||
|
|
|
|||
|
|
dataset_num_proc (`int` or `None`, *optional*, defaults to `None`):
|
|||
|
|
Number of processes to use for processing the dataset.
|
|||
|
|
disable_dropout (`bool`, *optional*, defaults to `True`):
|
|||
|
|
Whether to disable dropout in the model and reference model.
|
|||
|
|
use_vllm (`bool`, *optional*, defaults to `False`):
|
|||
|
|
Whether to use vLLM for generating completions. Requires vLLM to be installed (`pip install vllm`).
|
|||
|
|
vllm_model_impl (`str`, *optional*, defaults to `"vllm"`):
|
|||
|
|
Model implementation to use for vLLM. Must be one of `"transformers"` or `"vllm"`. `"transformers"`: Use
|
|||
|
|
the `transformers` backend for model implementation. `"vllm"`: Use the `vllm` library for model
|
|||
|
|
implementation.
|
|||
|
|
gpu_memory_utilization (`float`, *optional*, defaults to `0.55`):
|
|||
|
|
The vLLM memory utilization. The default value is 0.55.
|
|||
|
|
ds3_gather_for_generation (`bool`, *optional*, defaults to `True`):
|
|||
|
|
This setting applies to DeepSpeed ZeRO-3. If enabled, the policy model weights are gathered for generation,
|
|||
|
|
improving generation speed. However, disabling this option allows training models that exceed the VRAM
|
|||
|
|
capacity of a single GPU, albeit at the cost of slower generation.
|
|||
|
|
model_init_kwargs (`dict[str, Any]` or `None`, *optional*, defaults to `None`):
|
|||
|
|
Keyword arguments to pass to `AutoModelForCausalLM.from_pretrained` when instantiating the model from a
|
|||
|
|
string.
|
|||
|
|
|
|||
|
|
NÚhelpzvLLM SamplingParams)ÚdefaultÚmetadataÚvllm_sampling_paramsrK z8Chunk size to reduce memory usage. -1 is most efficient.Úunsloth_num_chunksFÚnorL é r éú ç-C ëâ6
|
|||
|
|
?ç{®Gáz„?çÍÌÌÌÌÌì?ç+‡ÙÎ÷ï?ç:Œ0âŽyE>ç ð?ç @Úlinearçš™™™™™¹?ÚpassiveÚwarningTÚstepsrQ éô éO
|
|||
|
|
ÚO1ÚautoÚ ç Ú
|
|||
|
|
adamw_8bitÚlengthÚ
|
|||
|
|
every_saveÚlasté é@ é Úsigmoidrm çš™™™™™á?c‘ óÌ •— |dk rt d|› d�¦ « ‚|dk rt d|› d�¦ « ‚|€|#dk r
|
|||
|
|
|$dk rd}d }#|ˆ€!d
|
|||
|
|
d lm}’ t |’¦ « d z d ¦ « }ˆ|…d
|
|||
|
|
k rt d
|
|||
|
|
¦ « ‚|…dk rt d¦ « ‚ t
|
|||
|
|
¦ « j dži d|“d|“d|“d|“d|“d|“d|“d|“d| “d|
|
|||
|
|
“d| “d| “d |
|
|||
|
|
“d |“d |“d |“d |“d!|“d"|“d#|“d$|“d%|“d&|“d'|“d(|“d)|“d*|“d+| “d,| “d-| “d.| “d/| “d0|!“d1|"“d2|#“d3|$“d4|%“d5|&“d6|'“d7|(“d8|)“d9|*“d:|+“d;|,“d<|-“d=|.“d>|/“d?|0“d@|1“dA|2“dB|3“dC|4“dD|5“dE|6“dF|7“dG|8“dH|9“dI|:“dJ|;“dK|<“dL|=“dM|>“dN|?“dO|@“dP|A“dQ|B“dR|C“dS|D“dT|E“dU|F“dV|G“dW|H“dX|I“dY|J“dZ|K“d[|L“d\|M“d]|N“d^|O“d_|P“d`|Q“da|R“db|S“dc|T“dd|U“de|V“df|W“dg|X“dh|Y“di|Z“dj|[“dk|\“dl|]“dm|^“dn|_“do|`“dp|a“dq|b“dr|c“ds|d“dt|e“du|f“dv|g“dw|h“dx|i“dy|j“dz|k“d{|l“d||m“d}|n“d~|o“d|p“d€|q“d�|r“d‚|s“dƒ|t“d„|u“d…|v“d†|w“d‡|x“dˆ|y“d‰|z“dŠ|{“d‹||“dŒ|}“d�|~“dŽ|“d�|€“d�|�“d‘|‚“d’|ƒ“d“|„“d”|…“d•|†“d–|‡“d—|ˆ“d˜|‰“d™|Š“dš|‹“d›|Œ“dœ|�“d�|Ž“|‘¤Ž |�| _ |�| _ d S )ŸNgH¯¼šò×z>z Unsloth: Your learning rate of `zi` is too small and less than 1e-7! Consider increasing it, otherwise gradient updates will be close to 0!rQ za` is way too larger > 1! Consider decreasing it to 1e-1, otherwise gradient updates will explode!r‡ rˆ Ú unsloth_training_checkpointsry r )Ú cpu_countrz zUUnsloth: Please set a positive non-zero temperature since your results will be wrong.é
|
|||
|
|
zgUnsloth: Please set a positive non-zero temperature less than 10, since sampling will be quite erratic.Ú
|
|||
|
|
output_dirÚoverwrite_output_dirÚdo_trainÚdo_evalÚ
|
|||
|
|
do_predictÚ
|
|||
|
|
eval_strategyÚprediction_loss_onlyÚper_device_train_batch_sizeÚper_device_eval_batch_sizeÚper_gpu_train_batch_sizeÚper_gpu_eval_batch_sizeÚgradient_accumulation_stepsÚeval_accumulation_stepsÚ
|
|||
|
|
eval_delayÚtorch_empty_cache_stepsÚ
|
|||
|
|
learning_rateÚ weight_decayÚ
|
|||
|
|
adam_beta1Ú
|
|||
|
|
adam_beta2Ú adam_epsilonÚ
|
|||
|
|
max_grad_normÚnum_train_epochsÚ max_stepsÚlr_scheduler_typeÚ warmup_ratioÚ warmup_stepsÚ log_levelÚlog_level_replicaÚlog_on_each_nodeÚ logging_dirÚlogging_strategyÚlogging_first_stepÚ
|
|||
|
|
logging_stepsÚlogging_nan_inf_filterÚ
|
|||
|
|
save_strategyÚ
|
|||
|
|
save_stepsÚsave_total_limitÚsave_safetensorsÚsave_on_each_nodeÚsave_only_modelÚ'restore_callback_states_from_checkpointÚno_cudaÚuse_cpuÚuse_mps_deviceÚseedÚ data_seedÚ
|
|||
|
|
jit_mode_evalÚuse_ipexÚbf16Úfp16Úfp16_opt_levelÚhalf_precision_backendÚbf16_full_evalÚfp16_full_evalÚtf32Ú
|
|||
|
|
local_rankÚ ddp_backendÚ
|
|||
|
|
tpu_num_coresÚtpu_metrics_debugÚdebugÚdataloader_drop_lastÚ
|
|||
|
|
eval_stepsÚdataloader_num_workersÚdataloader_prefetch_factorÚ
|
|||
|
|
past_indexÚrun_nameÚ disable_tqdmÚremove_unused_columnsÚ label_namesÚload_best_model_at_endÚmetric_for_best_modelÚgreater_is_betterÚignore_data_skipÚfsdpÚfsdp_min_num_paramsÚ fsdp_configÚ"fsdp_transformer_layer_cls_to_wrapÚaccelerator_configÚ deepspeedÚlabel_smoothing_factorÚoptimÚ
|
|||
|
|
optim_argsÚ adafactorÚgroup_by_lengthÚlength_column_nameÚ report_toÚddp_find_unused_parametersÚddp_bucket_cap_mbÚddp_broadcast_buffersÚdataloader_pin_memoryÚ dataloader_persistent_workersÚskip_memory_metricsÚuse_legacy_prediction_loopÚ push_to_hubÚresume_from_checkpointÚ hub_model_idÚ hub_strategyÚ hub_tokenÚhub_private_repoÚhub_always_pushÚ hub_revisionÚgradient_checkpointingÚ gradient_checkpointing_kwargsÚinclude_inputs_for_metricsÚeval_do_concat_batchesÚ fp16_backendÚpush_to_hub_model_idÚpush_to_hub_organizationÚpush_to_hub_tokenÚ
|
|||
|
|
mp_parametersÚauto_find_batch_sizeÚfull_determinismÚ torchdynamoÚ ray_scopeÚ ddp_timeoutÚ
|
|||
|
|
torch_compileÚtorch_compile_backendÚtorch_compile_modeÚinclude_tokens_per_secondÚ include_num_input_tokens_seenÚneftune_noise_alphaÚoptim_target_modulesÚbatch_eval_metricsÚ
|
|||
|
|
eval_on_startÚuse_liger_kernelÚliger_kernel_configÚeval_use_gather_objectÚ average_tokens_across_devicesÚreward_model_pathÚjudgeÚmax_new_tokensÚ
|
|||
|
|
max_lengthÚ temperatureÚmissing_eos_penaltyÚ loss_typeÚdataset_num_procÚdisable_dropoutÚuse_vllmÚvllm_model_implÚgpu_memory_utilizationÚds3_gather_for_generationÚmodel_init_kwargsrl )
|
|||
|
|
ÚFloatingPointErrorÚ
|
|||
|
|
OverflowErrorÚmultiprocessingr™ ÚminÚ MathErrorÚsuperÚ__init__rw rx )”Úselfr› rœ r� rž rŸ r r¡ r¢ r£ r¤ r¥ r¦ r§ r¨ r© rª r« r¬ r r® r¯ r° r± r² r³ r´ rµ r¶ r· r¸ r¹ rº r» r¼ r½ r¾ r¿ rÀ rÁ r rà rÄ rÅ rÆ rÇ rÈ rÉ rÊ rË rÌ rÍ rÎ rÏ rÐ rÑ rÒ rÓ rÔ rÕ rÖ r× rØ rÙ rÚ rÛ rÜ rÝ rÞ rß rà rá râ rã rä rå ræ rç rè ré rê rë rì rí rî rï rð rñ rò ró rô rõ rö r÷ rø rù rú rû rü rý rþ rÿ r r r r r r r r r r r
|
|||
|
|
r r r
|
|||
|
|
r r r r r r r r r r r r r r r r r r r r! r" r# r$ r% r& r' r( rw rx ro r™ Ú __class__s” €rg r/ z UnslothOnlineDPOConfig.__init__ s¿ ø€ ðh ˜4Ò Ð Õ'9ð ;VÐ]jð ;Vð ;Vð ;Vñ (Wô (Wð "WØ ˜1Ò Ð ¥Mð 3FÐUbð 3Fð 3Fð 3Fñ %Gô %Gð GØ Ð -°7Ò":Ð":¸zÈSÒ?PÐ?PØ7ˆJØ ˆMØ Ð #Ø 1Ð 1Ð 1Ð 1Ð 1Ð 1Ý " 9 9¡;¤;¨q¡=°!Ñ 4Ô 4Ð Ø ˜!Ò Ð ÝÐ sÑtÔtÐ tØ
|
|||
|
|
˜BÒ
|
|||
|
|
Ð
|
|||
|
|
Ýð Fñ Gô Gð
|
|||
|
|
Gð �‰ŒÔð N <ð N <ð N <Ø#˜ðN <à#7Ð#7ðN <ð �xðN <ð �gð N <ð
|
|||
|
|
$˜ð N <ð *˜Mð
|
|||
|
|
N <ð $8Ð#7ðN <ð +FÐ*EðN <ð *DÐ)CðN <ð (@Ð'?ðN <ð '>Ð&=ðN <ð +FÐ*EðN <ð '>Ð&=ðN <ð $˜ð N <ð '>Ð&=ð N <ð *˜Mð!N <ð" (˜<ð#N <ð$ $˜ð%N <ð& $˜ð'N <ð( (˜<ð)N <ð* *˜Mð+N <ð, 0Ð /ð-N <ð. "˜ ð/N <ð0 !2Ð 1ð1N <ð2 (˜<ð3N <ð4 (˜<ð5N <ð6 "˜ ð7N <ð8 !2Ð 1ð9N <ð: 0Ð /ð;N <ð< &˜+ð=N <ð> 0Ð /ð?N <ð@ "4Ð!3ðAN <ðB *˜MðCN <ðD &<Ð%;ðEN <ðF *˜MðGN <ðH $˜ðIN <ðJ 0Ð /ðKN <ðL 0Ð /ðMN <ðN !2Ð 1ðON <ðP .˜oðQN <ðR 7^Ð6]ðSN <ðT �gðUN <ðV �gðWN <ðX ,˜^ðYN <ðZ �4ð[N <ð\ "˜ ð]N <ð^ *˜Mð_N <ð` �xðaN <ðb �4ðcN <ðd �4ðeN <ðf ,˜^ðgN <ðh &<Ð%;ðiN <ðj ,˜^ðkN <ðl ,˜^ðmN <ðn �4ðoN <ðp $˜ðqN <ðr &˜+ðsN <ðt *˜MðuN <ðv !2Ð 1ðwN <ðx �EðyN <ðz $8Ð#7ð{N <ð| $˜ð}N <ð~ &<Ð%;ðN <ð@ *DÐ)CðAN <ðB $˜ðCN <ðD �xðEN <ðF (˜<ðGN <ðH %:Ð$9ðIN <ðJ &˜+ðKN <ðL &<Ð%;ðMN <ðN %:Ð$9ðON <ðP !2Ð 1ðQN <ðR 0Ð /ðSN <ðT �4ðUN <ðV #6Ð"5ðWN <ðX &˜+ðYN <ðZ 2TÐ1Sð[N <ð\ "4Ð!3ð]N <ð^ "˜ ð_N <ð` &<Ð%;ðaN <ðb �EðcN <ðd $˜ðeN <ðf "˜ ðgN <ðh .˜oðiN <ðj "4Ð!3ðkN <ðl "˜ ðmN <ðn *DÐ)CðoN <ðp !2Ð 1ðqN <ðr %:Ð$9ðsN <ðt %:Ð$9ðuN <ðv -JÐ,IðwN <ðx #6Ð"5ðyN <ðz *DÐ)Cð{N <ð| &˜+ð}N <ð~ &<Ð%;ðN <ð@ (˜<ðAN <ðB (˜<ðCN <ðD "˜ ðEN <ðF 0Ð /ðGN <ðH .˜oðIN <ðJ (˜<ðKN <ðL &<Ð%;ðMN <ðN -JÐ,IðON <ðP *DÐ)CðQN <ðR &<Ð%;ðSN <ðT (˜<ðUN <ðV $8Ð#7ðWN <ðX (@Ð'?ðYN <ðZ !2Ð 1ð[N <ð\ *˜Mð]N <ð^ $8Ð#7ð_N <ð` 0Ð /ðaN <ðb &˜+ðcN <ðd "˜ ðeN <ðf &˜+ðgN <ðh *˜MðiN <ðj %:Ð$9ðkN <ðl "4Ð!3ðmN <ðn )BÐ(AðoN <ðp -JÐ,IðqN <ðr #6Ð"5ðsN <ðt $8Ð#7ðuN <ðv "4Ð!3ðwN <ðx *˜MðyN <ðz 0Ð /ð{N <ð| #6Ð"5ð}N <ð~ &<Ð%;ðN <ð@ -JÐ,IðAN <ðB !2Ð 1ðCN <ðD �EðEN <ðF ,˜^ðGN <ðH $˜ðIN <ðJ &˜+ðKN <ðL #6Ð"5ðMN <ðN "˜ ðON <ðP 0Ð /ðQN <ðR .˜oðSN <ðT �xðUN <ðV .˜oðWN <ðX &<Ð%;ðYN <ðZ )BÐ(Að[N <ð\ !2Ð 1°Fð]N <ð N <ð N <ð^ %9ˆÔ!Ø"4ˆÔ Ð Ð ri )�NNFFFry FrL rL NNrz rz r r{ r| r} r~ r r€ r� r‚ rK rƒ r„ r r… r† TNr‡ FrQ Fr‡ rˆ NTFFFFFFr‰ r‰ FFFFrŠ r‹ FFNrK NNFrŒ FNr NrK NNTNFNNFrŒ r NNNNr� rŽ NFFr� NNNNTFTFFNNr� NNFNFNFTr‹ NNNrŒ TFNr‘ r’ FNNFFNNFFFNFTNNr“ r” r~ Nr• NTFrm r– TNNrK )
|
|||
|
|
Ú__name__Ú
|
|||
|
|
__module__Ú __qualname__Ú__doc__r? rw r r Ú__annotations__rx Úintr/ Ú
|
|||
|
|
__classcell__©r1 s @rg rs rs 8 s{ ø€ € € € € € ð<ð <ðz +0¨%ØØÐ 1Ð2ð+ñ +ô +И( 3œ-ð ð ñ ð */¨ØØÐ VÐWð*ñ *ô *И #œ ð ð ñ ð Ø #ØØØØ Ø $Ø&'Ø%&Ø#'Ø"&Ø&'Ø"#ØØ"%Ø ØØØØ ØØ ØØ $ØØØ Ø %Ø ØØ"Ø "ØØ!&Ø ØØ Ø Ø !Ø Ø27ØØØ ØØØ ØØØØ Ø!'Ø Ø ØØØØ Ø !ØØ $ØØ!"Ø%)ØØØØ $ØØ!&Ø $Ø Ø ØØ ØØ-1Ø !ØØ!$Ø ØØØ Ø %ØØ%)Ø Ø $Ø $Ø(-Ø "Ø%*ØØ!%ØØ#ØØ Ø ØØ!&Ø(,Ø%*Ø!%Ø Ø #Ø#'Ø ØØ #Ø ØØØØ Ø $Ø !Ø$)Ø(-Ø "Ø #Ø "Ø Ø Ø "Ø!&Ø(,Ø ØØØØØ "Ø Ø Ø ØØ Ø!%Ø$(Ø Ø #Ø ðcr5ð r5ð r5ð r5ð r5ð r5ð r5ð r5ð r5ð r5ri rs c " óö ‡ — e Zd ZdZddgZ d*deeej e f deeej df deeej df d e
|
|||
|
|
e d
|
|||
|
|
e
|
|||
|
|
e d e
|
|||
|
|
e
|
|||
|
|
d e
|
|||
|
|
eeed
|
|||
|
|
f de
|
|||
|
|
eeee ef d
|
|||
|
|
f de
|
|||
|
|
eeeeef de
|
|||
|
|
e de
|
|||
|
|
e de
|
|||
|
|
eegef de
|
|||
|
|
ee deej j ej j j f de
|
|||
|
|
eej ej gej f ddf ˆ fd„
|
|||
|
|
Z e!d„ ¦ « Z"e#de$dedee e%f fd„¦ « Z& e'e(j) ¦ « de*fd „¦ « Z) e'e(j+ ¦ « d+de
|
|||
|
|
ee ef de*fd „¦ « Z+d „ Z,d „ Z-d „ Z. d+dej d!ee eej e%f f d"e
|
|||
|
|
e/ dej fd#„Z0 d+d$„Z1ˆ fd%„Z2 d,d&e
|
|||
|
|
e d'e
|
|||
|
|
e d(ee ee df fd)„Z3ˆ xZ4S )-Ú_UnslothOnlineDPOTrainerrŒ Útrlz
|
|||
|
|
online-dpoN©NNÚmodelÚ ref_modelÚ reward_modelr ÚargsÚ
|
|||
|
|
data_collatorÚ
|
|||
|
|
train_datasetzdatasets.DatasetÚ eval_datasetÚprocessing_classÚreward_processing_classÚ peft_configÚcompute_metricsÚ callbacksÚ
|
|||
|
|
optimizersÚ preprocess_logits_for_metricsÚreturnc óÈ •— t |d¦ « r,t |d¦ « r t |dd¦ « dk rd|_ ||u rt d¦ « ‚|| _ |� |� t j dt ¦ « d }n|€|€t d¦ « ‚|| _ |
|
|||
|
|
| _ || _
|
|||
|
|
|j �|�t d¦ « ‚|€t d ¦ « ‚| €t d
|
|||
|
|
¦ « ‚|j pi }t |t ¦ « r�|}| d ¦ « }t |t j ¦ « s|d k s|€nCt |t ¦ « rt t |¦ « }||d < nt d
|
|||
|
|
|› d�¦ « ‚t% j |fi |¤Ž}n|j �t d¦ « ‚|j j | _ |j r*t7 |¦ « | j �t7 | j ¦ « |€ d | _ n || _ | j ¦ « | j �| j ¦ « |€t= | j ¬¦ « }|j | _ g g g g g g g g g g g dœ | _! | j � g | j! d< g | j! d< g | j! d< |j r`|j" | _# d| _$ tK d!d|j& |j' ddddœt t |dtQ ¦ « ¦ « d i ¦ « ¤Ž| _) n-tU |j& |j' ddd|j+ rdnd¬ ¦ « | _) d|j, d < t[ ¦ « . |||||| | |
|
|||
|
|
||¬ ¦
|
|||
|
|
«
|
|||
|
|
t | j/ d ¦ « r | j/ 0 | j1 ¦ « |j2 | _3 | j4 rh| j �+tk | j |j6 |j7 |j8 ¦ « | _ | j �-tk | j |j6 |j7 |j8 ¦ « | _ d S d S | j �)| j 9 | j: j; ¦ « | _ | j �+| j 9 | j: j; ¦ « | _ d S d S )"NÚ vllm_enginer$ FTzš`model` and `ref_model` cannot be the same object. If you want `ref_model` to be the same as `model`, either omit the `ref_model` argument or pass `None`.z€Both `reward_model` and `judge` are provided. Please choose provide only one of them. Ignoring `judge` and using `reward_model`.z2Either `reward_model` or `judge` must be provided.z@`missing_eos_penalty` is not supported when `judge` is provided.z`args` must be provided.z$`processing_class` must be provided.Ú torch_dtyper‹ zŽInvalid `torch_dtype` passed to `OnlineDPOConfig`. Expected either 'auto' or a string representing a `torch.dtype` (e.g., 'float32'), but got ú.z¦You passed `model_init_kwargs` to the `OnlineDPOConfig`, but your model is already instantiated. This argument can only be used when the `model` argument is a string.zfPEFT is not available and passed `peft_config`. Please install PEFT with `pip install peft` to use it.)Ú pad_token_id) ú objective/klúobjective/entropyúobjective/non_score_rewardúrewards/chosenúrewards/rejectedúrewards/accuraciesúrewards/marginsú logps/chosenúlogps/rejectedúval/contain_eos_tokenÚbetaúobjective/rlhf_rewardúobjective/scores_marginúobjective/scoresr rz é2 r� )ÚnÚ
|
|||
|
|
max_tokensr Útop_kÚtop_pÚ
|
|||
|
|
detokenizerw rn )r r rc rd Ú do_sampleÚ use_cacheÚestimate_tokens)
|
|||
|
|
r> rA rB rC rD rE rH rI rJ rK Úadd_model_tagsrl )<ÚhasattrÚgetattrr$ Ú
|
|||
|
|
ValueErrorr? r; ÚwarnÚ UserWarningr@ rF r r r( Ú
|
|||
|
|
isinstanceÚstrÚgetr7 Údtyper
|
|||
|
|
Úfrom_pretrainedÚconfigÚis_encoder_decoderr- Ú ImportErrorr Úmerge_and_unloadr# r' r% Úevalr rQ r ÚstatsrN ÚllmÚ_last_loaded_steprk r r rq Úgeneration_configr r Úwarnings_issuedr. r/ r> ri Ú
|
|||
|
|
_tag_namesr\ Ú_betaÚis_deepspeed_enabledr4 r¢ rÌ rË rV Ú acceleratorÚdevice)r0 r> r? r@ r rA rB rC rD rE rF rG rH rI rJ rK r( Úmodel_idrO r1 s €rg r/ z!_UnslothOnlineDPOTrainer.__init__¹ sT ø€ õ* �5˜-Ñ (Ô (ð %W°T¸:Ñ-FÔ-Fð %ݘ˜j¨%Ñ0Ô0°EÒ9Ð9Ø $�”
|
|||
|
|
Ø ˜Ð Ð Ý ðXñô ð
|
|||
|
|
ð
|
|||
|
|
#ˆŒà Ð #¨Ð(9Ý ŒMð=åñ
|
|||
|
|
ô
|
|||
|
|
ð
|
|||
|
|
ð
|
|||
|
|
ˆEˆEØ
|
|||
|
|
Ð
|
|||
|
|
! e mÝ Ð QÑRÔRÐ Rà (ˆÔØ'>ˆÔ$؈Œ
|
|||
|
|
à Ô #Ð /°EÐ4EÝ Ð _Ñ`Ô`Ð `à ˆ<Ý Ð 7Ñ8Ô8Ð 8ð Ð #Ý Ð CÑDÔDÐ Dà Ô 2Ð 8°bÐÝ �e�SÑ !Ô !ð Ø ˆHð ,×/Ò/°
|
|||
|
|
Ñ>Ô>ˆKݘ+¥u¤{Ñ3Ô3ð
|
|||
|
|
°{ÀfÒ7LÐ7LÐP[ÐPcØݘKÑ-Ô-ð
|
|||
|
|
Ý %¥e¨[Ñ 9Ô 9� Ø3>Ð! -Ñ0Ð0å ð^ØOZð^ð ^ð ^ñô ð õ
|
|||
|
|
)Ô8¸ÐWÐWÐEVÐWÐWˆEˆEàÔ%Ð1Ý ð\ñô ð ð #(¤,Ô"AˆÔ ð ð" Ô ð 9Ý $ UÑ +Ô +Ð +ØŒ~Ð)Ý(¨¬ Ñ8Ô8Ð8ð Ð ð
|
|||
|
|
&ð "&�”�à &ˆDŒNØ ŒN× Ò Ñ !Ô !Ð !ð Ô Ð (Ø Ô × "Ò "Ñ $Ô $Ð $ð Ð Ý 6ÐDTÔDaÐ bÑ bÔ bˆMà œ/ˆŒð Ø!#Ø*,Ø Ø "Ø"$Ø !Ø Ø Ø%'Øð
|
|||
|
|
ð
|
|||
|
|
ˆŒ
|
|||
|
|
ð Ô Ð (Ø24ˆDŒJÐ.Ñ /Ø46ˆDŒJÐ0Ñ 1Ø-/ˆDŒJÐ)Ñ *à Œ=ð Ø Ô(ˆDŒHÀ1¨$Ô*@Õ_mð `ØØ Ô.Ø Ô ,ØØØ ð
|
|||
|
|
`ð `õ �' $Ð(>Õ@RÑ@TÔ@TÑUÔUÐWdÐfhÑiÔið`ð `ÀdÔF\ÐF\õ &6Ø #Ô 2Ø Ô ,ØØØ Ø#'Ô#>ÐH˜%˜%ÀDð
|
|||
|
|
&ñ &ô &ˆDÔ "ð 48ˆÔ Ð /Ñ0å
|
|||
|
|
‰Œ×ÒØØØ'Ø'Ø%Ø -Ø +Ø Ø!Ø*Gð ñ
|
|||
|
|
ô
|
|||
|
|
ð
|
|||
|
|
õ �4”:Ð /Ñ 0Ô 0ð 7Ø ŒJ× %Ò % d¤oÑ 6Ô 6Ð 6à”YˆŒ
|
|||
|
|
ð Ô $ð
|
|||
|
|
RØÔ Ð,Ý$5ØÔ% tÔ'GÈÌÐTXÔT]ñ%ô %�Ô!ð Œ~Ð)Ý!2Ø”N DÔ$DÀdÄiÐQUÔQZñ"ô "�”��ð *Ð)ð
|
|||
|
|
Œ~Ð)Ø!%¤ ×!2Ò!2°4Ô3CÔ3JÑ!KÔ!K�”ØÔ Ð,Ø$(Ô$5×$8Ò$8¸Ô9IÔ9PÑ$QÔ$Q�Ô!Ð!Ð!ð -Ð,ri c óÀ — t | j t ¦ « r>| j j }|t | j ¦ « k r
|
|||
|
|
| j | n | j d S | j S )NrK )ro r ÚlistÚstateÚepochÚlen)r0 r‡ s rg r\ z _UnslothOnlineDPOTrainer.betaŠ sS € å �d”j¥$Ñ 'Ô 'ð Ø”JÔ$ˆEØ(-µ°D´J±´Ò(?Ð(?�4”:˜eÔ$Ð$ÀTÄZÐPRÄ^Ð Sà”:Ð ri ru Ú tokenizerc óL — |sp || d d¬¦ « }|j �Ut |d ¦ « }|dk s|j |d d k r#|j g|d z |d< dg|d z |d< n || d d ¬¦ « }d
|
|||
|
|
„ | ¦ « D ¦ « }|S ) z2Tokenize a single row from a DPO specific dataset.ÚpromptF)Úadd_special_tokensNÚ input_idsr rQ Úattention_maskTc ó — i | ] \ }}d |› �|“Œ S )Úprompt_rl )Ú.0ÚkeyÚvalues rg ú
|
|||
|
|
<dictcomp>z9_UnslothOnlineDPOTrainer.tokenize_row.<locals>.<dictcomp>Ÿ s$ € ÐHÐHÐH©J¨C°� ˜3� � %ÐHÐHÐHri )Ú bos_token_idrˆ Úitems)Úfeatureru r‰ ÚbatchÚprompt_len_input_idss rg Ú tokenize_rowz%_UnslothOnlineDPOTrainer.tokenize_row’ sÒ € ð "ð JØ �I˜g hÔ /ÀEÐJÑJÔJˆEàÔ%Ð1Ý'*¨5°Ô+=Ñ'>Ô'>Ð$Ø'¨1Ò,Ð,° Ô0FÈ%ÐP[ÔJ\Ð]^ÔJ_Ò0_Ð0_Ø*3Ô*@Ð)AÀEÈ+ÔDVÑ)V�E˜+Ñ&Ø/0¨c°EÐ:JÔ4KÑ.K�EÐ*Ñ+øà �I˜g hÔ /ÀDÐIÑIÔIˆEØHÐH¸%¿+º+¹-¼-ÐHÑHÔHˆØˆ ri c ó° — | j €t d¦ « ‚| j }| j }| j || j j | j j | j j dœ}t |t j
|
|||
|
|
j j ¦ « s?|
|
|||
|
|
¦ « |d< | j j |d< t |d< | j j |d< | j t' |fi |¤Ž¦ « S )Nz+Trainer: training requires a train_dataset.©Ú
|
|||
|
|
batch_sizeÚ
|
|||
|
|
collate_fnÚ num_workersÚ
|
|||
|
|
pin_memoryÚpersistent_workersÚsamplerÚ drop_lastÚworker_init_fnÚprefetch_factor)rC rl rB Ú_train_batch_sizerA rÙ rô rõ ro r7 ÚutilsÚdatar Ú_get_train_samplerr× r5 rÚ r� Úpreparer )r0 rC rB Údataloader_paramss rg Úget_train_dataloaderz-_UnslothOnlineDPOTrainer.get_train_dataloader£ sá € à Ô Ð %Ý Ð JÑKÔKÐ Kà Ô*ˆ
|
|||
|
|
Ø Ô*ˆ
|
|||
|
|
à Ô0Ø'Ø œ9Ô;Ø œ)Ô9Ø"&¤)Ô"Ið
|
|||
|
|
ð
|
|||
|
|
Ðõ ˜-¬Ô)9Ô)IÑJÔJð XØ+/×+BÒ+BÑ+DÔ+DÐ ˜iÑ (Ø-1¬YÔ-KÐ ˜kÑ *Ý2=Ð Ð .Ñ /Ø37´9Ô3WÐ Ð /Ñ 0àÔ ×'Ò'
|
|||
|
|
°=Ð(VÐ(VÐDUÐ(VÐ(VÑWÔWÐWri c ó" — |€| j €t d¦ « ‚t |t ¦ « r|nd}t | d¦ « r:|| j v r1| j j r%| j | j | ¦ « S t |t ¦ « r
|
|||
|
|
| j | n
|
|||
|
|
|�|n| j }| j
|
|||
|
|
}| j j || j j | j j
|
|||
|
|
| j j dœ}t |t j j j ¦ « s6| |¦ « |d< | j j |d< | j j |d< t+ |fi |¤Ž}| j j r$t | d¦ « r || j |< n ||i| _ | j |¦ « S )Nz-Trainer: evaluation requires an eval_dataset.rx Ú_eval_dataloadersrœ r¢ r£ r¥ )rD rl ro rp rj r® rA rõ r� rª rB Úeval_batch_sizerÙ rô r7 r§ r¨ r Ú_get_eval_samplerr× rÚ r )r0 rD Údataloader_keyrB r« Úeval_dataloaders rg Úget_eval_dataloaderz,_UnslothOnlineDPOTrainer.get_eval_dataloader» sÅ € à Ð DÔ$5Ð$=Ý Ð LÑMÔMÐ Mõ *4°LÅ#Ñ)FÔ)FÐR˜ ˜ ÈFˆå �DÐ-Ñ .Ô .ð Tà $Ô"8Ð8Ð8Ø” Ô7ð 9ð Ô#×+Ò+¨DÔ,BÀ>Ô,RÑSÔSÐ Sõ ˜,Ñ,Ô,ð
|
|||
|
|
#ˆDÔ ˜lÔ +Ð +ð Ð'ð � àÔ"ð ð Ô*ˆ
|
|||
|
|
ð œ)Ô3Ø'Ø œ9Ô;Ø œ)Ô9Ø"&¤)Ô"Ið
|
|||
|
|
ð
|
|||
|
|
Ðõ ˜,¬ Ô(8Ô(HÑIÔIð XØ+/×+AÒ+AÀ,Ñ+OÔ+OÐ ˜iÑ (Ø-1¬YÔ-KÐ ˜kÑ *Ø37´9Ô3WÐ Ð /Ñ 0õ % \ÐGÐGÐ5FÐGÐGˆØ Œ9Ô 2ð KÝ�tÐ 0Ñ1Ô1ð
|
|||
|
|
KØ9H�Ô& ~Ñ6Ð6à*8¸/Ð)J�Ô&àÔ ×'Ò'¨Ñ8Ô8Ð8ri c ó¸ ‡‡‡ ‡
|
|||
|
|
‡ — | j j Š| j j Š t d|d i¦ « r>| j || j d| j dd¬¦ « ¬¦ « Š
|
|||
|
|
n=| j || j d| j dd¬¦ « ¬¦ « Š
|
|||
|
|
ˆ
|
|||
|
|
fd„t d ¦ « D ¦ « }ˆ
|
|||
|
|
fd
|
|||
|
|
„t d ¦ « D ¦ « }t d „ |D ¦ « ¦ « Šˆfd „|D ¦ « }ˆˆ fd
|
|||
|
|
„|D ¦ « }| j j Š ˆ fd„|D ¦ « }ˆˆ fd„|D ¦ « }ˆ ˆ fd„|D ¦ « }t j || j j ¬¦ « }t j || j j ¬¦ « }t j || j j ¬¦ « }t j || j j ¬¦ « }||||fS )Nr‹ r FÚ online_dpo_trainer_lora_modelT)Ú load_tensors)Úuse_tqdmÚ lora_requestc óX •— g | ]&}‰D ]!}t |j | j ¦ « ‘Œ"Œ'S rl )r… ÚoutputsÚ token_ids)r‘ ÚiÚoutputrº s €rg ú
|
|||
|
|
<listcomp>z;_UnslothOnlineDPOTrainer._generate_vllm.<locals>.<listcomp>ú s: ø€ ÐdÐdÐdÀÐ\cÐdÐdÐRX�$˜vœ~¨aÔ 0Ô :Ñ;Ô;ÐdÐdÐdÐdri rz c óB •— g | ]}‰D ]}t |j ¦ « ‘ŒŒ S rl )r… Úprompt_token_ids)r‘ Ú_r½ rº s €rg r¾ z;_UnslothOnlineDPOTrainer._generate_vllm.<locals>.<listcomp>û s2 ø€ Ð\Ð\Ð\¸ÐT[Ð\Ð\È&•d˜6Ô2Ñ3Ô3Ð\Ð\Ð\Ð\ri c 3 ó4 K — | ]}t |¦ « V — Œd S ©N©rˆ )r‘ Úidss rg ú <genexpr>z:_UnslothOnlineDPOTrainer._generate_vllm.<locals>.<genexpr>þ s( è è € Ð ?Ð ?¨S¥ C¡¤Ð ?Ð ?Ð ?Ð ?Ð ?Ð ?ri c ód •— g | ],}d g‰t |¦ « z
|
|||
|
|
z dgt |¦ « z z ‘Œ-S )r rQ rÄ )r‘ rÅ Úmax_prompt_lengths €rg r¾ z;_UnslothOnlineDPOTrainer._generate_vllm.<locals>.<listcomp>ÿ s> ø€ ÐcÐcÐcÐQT˜�sÐ /µ#°c±(´(Ñ :Ñ;¸q¸cÅCÈÁHÄH¹nÑLÐcÐcÐcri c óB •— g | ]}‰g‰t |¦ « z
|
|||
|
|
z |z ‘Œ S rl rÄ )r‘ rÅ rÈ rQ s €€rg r¾ z;_UnslothOnlineDPOTrainer._generate_vllm.<locals>.<listcomp> s2 ø€ ÐbÐbÐbÐPS�|�nÐ(9½CÀ¹H¼HÑ(DÑEÈÑKÐbÐbÐbri c ód •— g | ],}d gt |¦ « z dg‰t |¦ « z
|
|||
|
|
z z ‘Œ-S )rQ r rÄ )r‘ rÅ rb s €rg r¾ z;_UnslothOnlineDPOTrainer._generate_vllm.<locals>.<listcomp> s< ø€ ÐdÐdÐdÈc˜A˜3¥ S¡¤™>¨Q¨C°:ÅÀCÁÄÑ3HÑ,IÑIÐdÐdÐdri c ó^ •— g | ])}|d ‰k rt |¦ « ‰k r|‰gz n|‘Œ*S )rK rÄ )r‘ rÅ Ú eos_token_idrb s €€rg r¾ z;_UnslothOnlineDPOTrainer._generate_vllm.<locals>.<listcomp> sT ø€ ð
|
|||
|
|
ð
|
|||
|
|
ð
|
|||
|
|
àð %(¨¤G¨|Ò$;Ð$;ÅÀCÁÄÈ:Ò@UÐ@UˆC�<�.Ñ Ð Ð[^ð
|
|||
|
|
ð
|
|||
|
|
ð
|
|||
|
|
ri c óB •— g | ]}|‰g‰t |¦ « z
|
|||
|
|
z z ‘Œ S rl rÄ )r‘ rÅ rb rQ s €€rg r¾ z;_UnslothOnlineDPOTrainer._generate_vllm.<locals>.<listcomp> s0 ø€ ÐcÐcÐcÈS˜# °*½sÀ3¹x¼xÑ2GÑ HÑHÐcÐcÐcri ©r‚ )rE rÌ rQ r, rz Úchatr| r> Ú load_loraÚgenerateÚrangeÚmaxrb r7 Útensorr� r‚ ) r0 r> ÚpromptsÚcompletion_idsÚ
|
|||
|
|
prompt_idsÚ prompt_maskÚcompletion_maskrÌ rÈ rb rº rQ s @@@@@rg Ú_generate_vllmz'_UnslothOnlineDPOTrainer._generate_vllmë s^ øøøøø€ ØÔ,Ô9ˆ ØÔ,Ô9ˆ ð
|
|||
|
|
à å ˜h¨°¬
|
|||
|
|
Ð 3Ñ 4Ô 4ð tØ”h—m’m G¨TÔ-CÈeÐdhÔdn×dxÒdxð zYð jnÐdxñ eoô eo�mñ pô pˆGˆGà”h×'Ò'¨°Ô1GÐRWÐhlÔhr×h|Òh|ð ~]ð nrÐh|ñ isô isÐ'ñ tô tˆGàdÐdÐdÐdÅUÈ1ÁXÄXÐdÑdÔdˆØ\Ð\Ð\Ð\½UÀ1¹X¼XÐ\Ñ\Ô\ˆ
|
|||
|
|
õ Ð ?Ð ?°JÐ ?Ñ ?Ô ?Ñ ?Ô ?ÐØcÐcÐcÐcÐXbÐcÑcÔcˆ ØbÐbÐbÐbÐbÐWaÐbÑbÔbˆ
|
|||
|
|
ØÔ+Ô6ˆ
|
|||
|
|
ØdÐdÐdÐdÐUcÐdÑdÔdˆð
|
|||
|
|
ð
|
|||
|
|
ð
|
|||
|
|
ð
|
|||
|
|
ð
|
|||
|
|
à%ð
|
|||
|
|
ñ
|
|||
|
|
ô
|
|||
|
|
ˆð dÐcÐcÐcÐcÐTbÐcÑcÔcˆõ ”\ *°TÔ5EÔ5LÐMÑMÔMˆ
|
|||
|
|
Ý”l ;°tÔ7GÔ7NÐOÑOÔOˆ Ý œ n¸TÔ=MÔ=TÐUÑUÔUˆÝ œ, ¸tÔ?OÔ?VÐWÑWÔWˆà˜;¨¸ÐGÐGri c ój ‡ — ‰ j j }‰ j j }d„ |D ¦ « }ˆ fd„|D ¦ « }ˆ fd„|D ¦ « }‰ |¦ « }‰ |¦ « }|d dd¦ « }|d dd¦ « }t
|
|||
|
|
|‰ j ‰ j j ¬¦ « 5 }|
|
|||
|
|
||‰ j ¬ ¦ « } d d d ¦ « n # 1 swxY w Y | d d …| d¦ « d …f }
|
|||
|
|
t |
|
|||
|
|
||¦ « \ }
|
|||
|
|
} |||
|
|||
|
|
| fS )
|
|||
|
|
Nc ó — g | ]}d |i‘ŒS ©r‹ rl )r‘ r‹ s rg r¾ z6_UnslothOnlineDPOTrainer._generate.<locals>.<listcomp> s € Ð;Ð;Ð;¨�8˜VÐ$Ð;Ð;Ð;ri c ó: •— g | ]}t |‰j ¦ « ‘ŒS rl )r1 rE ©r‘ Úxr0 s €rg r¾ z6_UnslothOnlineDPOTrainer._generate.<locals>.<listcomp> s' ø€ ÐVÐVÐVÈ!Õ+¨A¨tÔ/DÑEÔEÐVÐVÐVri c óR •— g | ]#}‰ |‰j ‰j ¦ « ‘Œ$S rl )rš ru rE rß s €rg r¾ z6_UnslothOnlineDPOTrainer._generate.<locals>.<listcomp> s2 ø€ ÐgÐgÐgÐ[\�$×#Ò# A tÔ'>ÀÔ@UÑVÔVÐgÐgÐgri Úprompt_input_idsrz rQ Úprompt_attention_mask)Úgather_deepspeed3_params)r� rŽ r| )rE rÌ rQ rB Ú_prepare_inputsÚrepeatr9 r� rA r' rÑ r| Úsizer8 ) r0 r> rÕ rÌ rQ Úinputsr× rØ Úunwrapped_modelr½ rÖ rÙ s ` rg Ú _generatez"_UnslothOnlineDPOTrainer._generate s© ø€ ØÔ,Ô9ˆ ØÔ,Ô9ˆ ð <Ð;°7Ð;Ñ;Ô;ˆØVÐVÐVÐVÈvÐVÑVÔVˆØgÐgÐgÐgÐ`fÐgÑgÔgˆØ×#Ò# FÑ+Ô+ˆð ×%Ò% fÑ-Ô-ˆØÐ .Ô/×6Ò6°q¸!Ñ<Ô<ˆ
|
|||
|
|
Ø Ð 4Ô5×<Ò<¸QÀÑBÔBˆ Ý
|
|||
|
|
(Ø �4Ô#¸d¼iÔ>að
|
|||
|
|
ñ
|
|||
|
|
ô
|
|||
|
|
ð à
|
|||
|
|
Ø$×-Ò-Ø$Ø *Ø"&Ô"8ð .ñ ô ˆFð ð ð ñ ô ð ð ð ð ð ð øøøð ð ð ð ð :§?¢?°1Ñ#5Ô#5Ð#7Ð#7Ð 7Ô8ˆÝ*8¸ È ÐWcÑ*dÔ*dÑ'ˆ˜à˜;¨¸ÐGÐGs à C/Ã/C3Ã6C3c óF — t | d¦ « | d¦ « z | j z
|
|||
|
|
d¦ « }|d d …|d …f }|d d …|d …f }t j ||fd¬¦ « }t j ||fd¬¦ « } |||¬¦ « } | d¦ « }
|
|||
|
|
|
|
|||
|
|
dk r|
|
|||
|
|
dz
|
|||
|
|
nd} | j d d …| d…f } t j | d¬¦ « | d¦ « d¬¦ « d¦ « }
|
|||
|
|
|
|
|||
|
|
S )NrQ r rP )rŽ rK rz )
|
|||
|
|
rÓ rç r r7 Úcatr^ Útake_along_dimÚ log_softmaxrY rZ )r0 r> r× rØ rÖ rÙ Únum_tokens_to_truncateÚprompt_completion_idsÚprompt_completion_maskr½ Ú
|
|||
|
|
prompt_lenÚ start_idxr^ Úlogprobss rg Ú_forwardz!_UnslothOnlineDPOTrainer._forward. sM € å!$ Z§_¢_°QÑ%7Ô%7¸.×:MÒ:MÈaÑ:PÔ:PÑ%PÐSWÔSbÑ%bÐdeÑ!fÔ!fÐ ð Ð#9Ð#:Ð#:Ð :Ô;ˆ
|
|||
|
|
Ø! ! ! !Ð%;Ð%<Ð%<Ð"<Ô=ˆ õ !&¤ ¨:°~Ð*FÈAÐ NÑ NÔ NÐ Ý!&¤¨K¸ Ð+IÈqÐ!QÑ!QÔ!QÐ ð �Ð,Ð=SÐTÑTÔTˆð —_’_ QÑ'Ô'ˆ
|
|||
|
|
Ø&0°1¢n n�J ‘N�N¸!ˆ Ø” ˜q˜q˜q )¨B ,˜Ô/ˆõ Ô'¨×(:Ò(:¸rÐ(:Ñ(BÔ(BÀN×D\ÒD\Ð]_ÑD`ÔD`ÐfgÐhÑhÔh×pÒpÐqsÑtÔtˆØˆri rè Únum_items_in_batchc óö ‡ ‡=— | ¦ « |d }t |¦ « }‰ j j r ‰ ||¦ « \ }}}} n‰ ||¦ « \ }}}} t
|
|||
|
|
j |‰ j j k d¬¦ « }
|
|||
|
|
‰
|
|||
|
|
||||| ¦ « } t
|
|||
|
|
j ¦ « 5 ‰ j � ‰
|
|||
|
|
‰ j |||| ¦ « } nO‰ j
|
|||
|
|
¦ « 5 ‰
|
|||
|
|
‰ j
|
|||
|
|
|||| ¦ « } d d d ¦ « n # 1 swxY w Y d d d ¦ « n # 1 swxY w Y | j }
|
|||
|
|
‰ j |d¬¦ « }t# d|d i¦ « r d„ |D ¦ « }‰ j �Èt# d|d i¦ « rIt' j ¦ « }| t, ¦ « Š=ˆ=fd„|D ¦ « }ˆ=fd „|D ¦ « }‰ j |t/ t1 |d |… ||d … ¦ « ¦ « ¦ « }t
|
|||
|
|
j d
|
|||
|
|
„ |D ¦ « |
|
|||
|
|
¬ ¦ « }�nzd |z }t# d|d i¦ « r@d
|
|||
|
|
„ t1 ||¦ « D ¦ « }ˆ fd„|D ¦ « }d„ |D ¦ « }d„ |D ¦ « }‰ |ddd¬¦ « d |
|
|||
|
|
¦ « }|j d }‰ |ddd¬¦ « d |
|
|||
|
|
¦ « }t
|
|||
|
|
j ||fd¬¦ « }t
|
|||
|
|
j ¦ « 5 t? ‰ j |‰ j j! |¦ « \ }}}‰ j j" �||
|
|||
|
|
xx ‰ j j" z cc< d d d ¦ « n # 1 swxY w Y | # |¦ « \ }}||k }t
|
|||
|
|
j$ ||
|
|||
|
|
¬ ¦ « }|| |z z } |||z z } t
|
|||
|
|
j | | fd¬¦ « } | | } | | } | % ¦ « }!|!| }"| |" z & d¦ « }#| |" z & d¦ « }$t
|
|||
|
|
j# |#|¦ « \ }%}&t
|
|||
|
|
j# |$|¦ « \ }'}(|%|&z
|
|||
|
|
})|'|(z
|
|||
|
|
}*|)|*z
|
|||
|
|
}+‰ j j' dk r tQ j) ‰ j* |+z ¦ « },n;‰ j j' dk r|+dd ‰ j* z z z
|
|||
|
|
d z },ntW d‰ j' › �¦ « ‚|, , ¦ « }-‰ j �í|| || z
|
|||
|
|
}.‰ j- d . ‰ j/ 0 |. , ¦ « ¦ « , ¦ « 1 ¦ « ¦ « ‰ j- d . ‰ j/ 0 | , ¦ « ¦ « , ¦ « 1 ¦ « ¦ « ‰ j- d . |
|
|||
|
|
2 ¦ « , ¦ « 1 ¦ « ¦ « ‰ j- d . ‰ j/ 0 |%¦ « , ¦ « 1 ¦ « ¦ « ‰ j- d . ‰ j/ 0 |&¦ « , ¦ « 1 ¦ « ¦ « | | z
|
|||
|
|
}/|/ & d¦ « , ¦ « }0‰ j- d . ‰ j/ 0 |0¦ « , ¦ « 1 ¦ « ¦ « ‰ j* |/z & d¦ « }1|1 , ¦ « }2‰ j- d . ‰ j/ 0 |2¦ « , ¦ « 1 ¦ « ¦ « ‰ j �a||1z }3‰ j- d! . ‰ j/ 0 |3¦ « , ¦ « 1 ¦ « ¦ « | & d¦ « , ¦ « }4‰ j- d" . ‰ j/ 0 |4¦ « , ¦ « 1 ¦ « ¦ « ‰ j* |%|'z
|
|||
|
|
z }5‰ j/ 0 |5¦ « }6‰ j- d# . |6 , ¦ « 1 ¦ « ¦ « ‰ j* |&|(z
|
|||
|
|
z }7‰ j/ 0 |7¦ « }8‰ j- d$ . |8 , ¦ « 1 ¦ « ¦ « |6|8z
|
|||
|
|
}9‰ j- d% . |9 , ¦ « 1 ¦ « ¦ « |9dk }:‰ j- d& . |: 2 ¦ « , ¦ « 1 ¦ « ¦ « ‰ j- d' . ‰ j* ¦ « ‰ j j3 �+‰ j4 j5 ‰ j j3 z dk rtm ¦ « i };‰ j j7 tp j9 tp j: fv r‰ ; ¦ « |;d(< ‰ j j< dk r|- , ¦ « }-‰ j= rMt| ? |-‰ j@ ¦ « 5 }<|< A ¦ « d d d ¦ « n # 1 swxY w Y n ‰ j/ jA |-fi |;¤Ž |- B ¦ « ‰ j jC z S ))Nr‹ rK rP T)Úskip_special_tokensr c ó — g | ]}d |dœg‘Œ S )Ú assistant)ÚroleÚcontentrl )r‘ Ú
|
|||
|
|
completions rg r¾ z:_UnslothOnlineDPOTrainer.training_step.<locals>.<listcomp>a s# € ÐhÐhÐhÈj [¸ZÐ HÐ HÐIÐhÐhÐhri c ó< •— g | ]}‰ |¬ ¦ « ‘ŒS ©)Úmessages©Úrender)r‘ r‹ Útemplates €rg r¾ z:_UnslothOnlineDPOTrainer.training_step.<locals>.<listcomp>l s' ø€ ÐRÐRÐRÀ˜8Ÿ?š?°F˜?Ñ;Ô;ÐRÐRÐRri c ó< •— g | ]}‰ |¬ ¦ « ‘ŒS rÿ r )r‘ rý r s €rg r¾ z:_UnslothOnlineDPOTrainer.training_step.<locals>.<listcomp>m s' ø€ Ð bÐ bÐ bÈ
|
|||
|
|
˜xŸš¸
|
|||
|
|
˜Ñ CÔ CÐ bÐ bÐ bri c ó — g | ]}|d k ‘Œ S )r rl )r‘ Úranks rg r¾ z:_UnslothOnlineDPOTrainer.training_step.<locals>.<listcomp>v s € Ð QÐ QÐ Q¨t ¨¢Ð QÐ QÐ Qri rÎ rz c ó — g | ]
|
|||
|
|
\ }}||d œ‘Œ S ))r‹ rý rl )r‘ ÚpÚcs rg r¾ z:_UnslothOnlineDPOTrainer.training_step.<locals>.<listcomp>| s$ € ÐaÐaÐa¹t¸qÀ! q¸Ð :Ð :ÐaÐaÐari c ó: •— g | ]}t |‰j ¦ « ‘ŒS rl )r$ rF )r‘ Úexampler0 s €rg r¾ z:_UnslothOnlineDPOTrainer.training_step.<locals>.<listcomp>} s( ø€ ÐoÐoÐoÐ[bÕ /°¸Ô9UÑ VÔ VÐoÐoÐori c ó — g | ]
|
|||
|
|
}|d ‘Œ S rÝ rl ©r‘ r s rg r¾ z:_UnslothOnlineDPOTrainer.training_step.<locals>.<listcomp>~ s € ÐEÐEÐE°˜7 8Ô,ÐEÐEÐEri c ó — g | ]
|
|||
|
|
}|d ‘Œ S )rý rl r
|
|||
|
|
s rg r¾ z:_UnslothOnlineDPOTrainer.training_step.<locals>.<listcomp> s € Ð MÐ MÐ M¸˜w |Ô 4Ð MÐ MÐ Mri ÚptÚleft)ÚpaddingÚreturn_tensorsÚ padding_sider� rQ Úrightr• Úipozinvalid loss type r^ r_ r[ rY rZ rR rT r] rS rU rV rX rW r\ rª )DÚtrainrˆ rA r$ rÚ rê r7 ÚanyrE rÌ rõ Úno_gradr? r> Údisable_adapterr‚ Ú batch_decoder, r r/ Ú EnvironmentÚ from_stringr! r… rU rÔ rF rV rT rì Úinference_moder+ r@ rQ r ÚsplitÚarangeÚboolÚsumr! r Ú
|
|||
|
|
logsigmoidr\ ÚNotImplementedErrorÚmeanry r\ r� Úgather_for_metricsÚitemÚfloatr© r† Ú global_stepr( rë r ÚLOMOÚADALOMOÚ_get_learning_rateÚn_gpuÚuse_apexÚampÚ
|
|||
|
|
scale_lossÚ optimizerÚbackwardÚdetachr¦ )>r0 r> rè rö rÕ r� r× rØ rÖ rÙ Úcontain_eos_tokenrô Ú ref_logprobsr‚ Ú completionsÚ environmentÚranks_of_first_completionÚmaskÚexamplesÚ prompts_idsÚcontext_lengthÚcompletions_idsrð rÁ ÚscoresÚ
|
|||
|
|
first_halfÚ second_halfÚ batch_rangeÚchosen_indicesÚrejected_indicesÚ
|
|||
|
|
cr_indicesÚ cr_logprobsÚcr_ref_logprobsÚ padding_maskÚcr_padding_maskÚcr_logprobs_sumÚcr_ref_logprobs_sumÚchosen_logprobs_sumÚrejected_logprobs_sumÚchosen_ref_logprobs_sumÚrejected_ref_logprobs_sumÚ pi_logratiosÚ
|
|||
|
|
ref_logratiosr^ ÚlossesÚlossÚ
|
|||
|
|
scores_marginÚklÚmean_klÚnon_score_rewardÚmean_non_score_rewardÚ rlhf_rewardÚ mean_entropyÚchosen_rewardsÚgathered_chosen_rewardsÚrejected_rewardsÚgathered_rejected_rewardsÚmarginÚaccuracyro Ú scaled_lossr s> ` @rg Ú
|
|||
|
|
training_stepz&_UnslothOnlineDPOTrainer.training_stepF s™ øø€ ð � Š ‰
|
|||
|
|
Œ
|
|||
|
|
ˆ
|
|||
|
|
à˜Ô"ˆÝ˜‘\”\ˆ
|
|||
|
|
à Œ9Ô ð fØGK×GZÒGZÐ[`ÐbiÑGjÔGjÑ DˆJ˜ ^°_°_àGKÇ~Â~ÐV[Ð]dÑGeÔGeÑ DˆJ˜ ^°_å !œI n¸Ô8MÔ8ZÒ&ZÐ`bÐ cÑ cÔ cÐà—=’= ¨
|
|||
|
|
°KÀ ÐQ`ÑaÔaˆÝ
|
|||
|
|
Œ]‰_Œ_ð wð wØŒ~Ð)Ø #Ÿ}š}¨T¬^¸ZÈÐVdÐfuÑ vÔ v� � à”Z×/Ò/Ñ1Ô1ð wð wØ#'§=¢=°´¸ZÈÐVdÐfuÑ#vÔ#v�Lðwð wð wñ wô wð wð wð wð wð wð wøøøð wð wð wð wð wð wð wñ wô wð wð wð wð wð wð wøøøð wð wð wð wð ” ˆØÔ+×8Ò8¸ Ð]aÐ8ÑbÔbˆ Ý ˜h¨°¬
|
|||
|
|
Ð 3Ñ 4Ô 4ð iØhÐhÐ\gÐhÑhÔhˆKð Œ:Ð !õ
|
|||
|
|
! (¨G°A¬JÐ!7Ñ8Ô8ð
|
|||
|
|
cÝ $Ô 0Ñ 2Ô 2� Ø&×2Ò2Õ3GÑHÔH�ØRÐRÐRÐRÈ'ÐRÑRÔR�Ø bÐ bÐ bÐ bÐVaÐ bÑ bÔ b� à(,¬
|
|||
|
|
×(8Ò(8Ø��c +¨k¨z¨kÔ":¸KÈ
|
|||
|
|
È È Ô<TÑ UÔ UÑVÔVñ)ô )Ð %õ ”<Ð QÐ QÐ7PÐ QÑ QÔ QÐZ`ÐaÑaÔaˆD‰Dð ˜'‘kˆGÝ (¨G°A¬JÐ!7Ñ8Ô8ð
|
|||
|
|
NØaÐaÅsÈ7ÐT_ÑG`ÔG`ÐaÑaÔa�ØoÐoÐoÐoÐfnÐoÑoÔo�ØEÐE¸HÐEÑEÔE�Ø MÐ MÀHÐ MÑ MÔ M� ð ×6Ò6Ø °dÈð 7ñ ô àôç š2˜f™:œ:ð
|
|||
|
|
ð )Ô .¨qÔ 1ˆNð #× :Ò :Ø T¸$ÈWð ;ñ ô àô ç š2˜f™:œ:ð
|
|||
|
|
õ
|
|||
|
|
%*¤I¨{¸OÐ.LÐRSÐ$TÑ$TÔ$TÐ !ÝÔ%Ñ'Ô'ð
|
|||
|
|
Pð
|
|||
|
|
PÝ )ØÔ%Ð'<¸dÔ>ZÔ>gÐiwñ ô ‘ ��6˜1ð ”9Ô0Ð<ØÐ -Ð-Ð.Ð.Ô.°$´)Ô2OÑOÐ.Ð.Ñ.ð
|
|||
|
|
Pð
|
|||
|
|
Pð
|
|||
|
|
Pñ
|
|||
|
|
Pô
|
|||
|
|
Pð
|
|||
|
|
Pð
|
|||
|
|
Pð
|
|||
|
|
Pð
|
|||
|
|
Pð
|
|||
|
|
Pð
|
|||
|
|
Pøøøð
|
|||
|
|
Pð
|
|||
|
|
Pð
|
|||
|
|
Pð
|
|||
|
|
Pð '-§l¢l°:Ñ&>Ô&>Ñ #ˆJ˜ ð Ò,ˆDå”l :°fÐ=Ñ=Ô=ˆ Ø$¨¨°
|
|||
|
|
Ñ(:Ñ;ˆØ&¨$°Ñ*;Ñ<Ðõ ”Y Ð0@Ð AÀqÐIÑIÔIˆ
|
|||
|
|
Ø ˜zÔ*ˆ Ø& zÔ2ˆð (×,Ò,Ñ.Ô.Ð.ˆ Ø& zÔ2ˆà&¨/Ð)9Ñ9×>Ò>¸qÑAÔAˆØ .°/Ð1AÑ A× FÒ FÀqÑ IÔ IÐõ 6;´[À ÐR\Ñ5]Ô5]Ñ2ÐÐ 2Ý=B¼[ÐI\Ð^hÑ=iÔ=iÑ:Ð Ð!:Ø*Ð-BÑBˆ Ø/Ð2KÑKˆ
|
|||
|
|
à
|
|||
|
|
Ñ-ˆà Œ9Ô )Ò +Ð +Ý”l 4¤9¨vÑ#5Ñ6Ô6Ð6ˆFˆFØ
|