324 lines
40 KiB
Plaintext
324 lines
40 KiB
Plaintext
|
o
|
|||
|
|
;—°h� ã @ s. d Z ddlmZ ddlZddlmZ ddlmZ ddlmZm Z m
|
|||
|
|
Z
|
|||
|
|
m Z m Z m
|
|||
|
|
Z
|
|||
|
|
mZmZ ddlmZmZmZmZmZmZmZmZmZmZmZmZmZmZm Z m Z m Z m
|
|||
|
|
Z
|
|||
|
|
m Z m Z m!Z!m"Z"m#Z#m$Z$m%Z%m&Z&m Z m'Z'm(Z(m)Z)m*Z*m+Z+m,Z,m-Z-m.Z.m/Z/m0Z0m1Z1m2Z2m3Z3m4Z4mZm5Z5m6Z6m7Z7m8Z8mZm9Z9m:Z:m;Z;m<Z<m=Z=mZm/Z/m5Z5mZmZm
|
|||
|
|
Z
|
|||
|
|
m Z m!Z!m%Z%m0Z0m5Z5mZ ddl5Z5ddlT ddl>m?Z?m@Z@ dd lAmBZB ddlZddlCZDdd
|
|||
|
|
lEmFZF ddlmZ dd lGmHZHmIZJ d d
|
|||
|
|
d d
|
|||
|
|
d
|
|||
|
|
dœZKejLd d eKd�dd„ ƒZMdd„ ZNe?G dd„ de ƒƒZO G dd„ de%ƒZPG dd„ dePƒZQdS )z9
|
|||
|
|
2025.8.9
|
|||
|
|
2025.8.10
|
|||
|
|
4.55.4
|
|||
|
|
0.21.0
|
|||
|
|
__UNSLOTH_VERSIONING__
|
|||
|
|
é )ÚTensorN)Ú
|
|||
|
|
functional)ÚAnyÚListÚOptionalÚTupleÚUnionÚDictÚSetÚCallable)@r ÚAutoModelForCausalLMÚBaseImageProcessorÚBasePairwiseJudger ÚDPODataCollatorWithPaddingÚ DataCollatorÚ
|
|||
|
|
DataLoaderÚDatasetÚEvalPredictionÚFÚFeatureExtractionMixinÚGenerationConfigÚIterableDatasetÚOnlineDPOConfigÚOnlineDPOTrainerÚOptimizerNamesr ÚPathÚ PeftModelÚPreTrainedModelÚPreTrainedTokenizerBaseÚProcessorMixinÚSIMPLE_CHAT_TEMPLATEÚTrainerÚTrainerCallbackr Úapply_chat_templateÚcreate_reference_modelÚdatasetsÚdisable_dropout_in_modelÚ empty_cacheÚgenerate_model_cardÚget_comet_experiment_urlÚ
|
|||
|
|
get_rewardÚis_conversationalÚis_peft_availableÚis_wandb_availableÚjinja2ÚloggingÚmaybe_apply_chat_templateÚnnÚosÚprepare_deepspeedÚ seed_workerÚtextwrapÚtorchÚtruncate_rightÚunwrap_model_for_generationÚversionÚwarningsÚwrapsr r+ r2 r6 r r r r r! r, r2 r6 )Ú*)Ú dataclassÚfield)ÚVersion)Ú nullcontext)ÚDataCollatorForSeq2SeqÚ DataCollatorForLanguageModelingTF)Úepilogue_fusionÚ max_autotuneÚ
|
|||
|
|
shape_paddingz
|
|||
|
|
trace.enabledztriton.cudagraphs)ÚdynamicÚ fullgraphÚoptionsc
|
|||
|
|
C s¾ t j| d| jd ¡ddd�}t j| d¡ddd�}g }t||ƒD ](\}}| t j¡}t j|d| d¡d� d¡}t j
|
|||
|
|
|dd�}|| } | | ¡ q! t |¡}| | jd | jd f¡}|S )Néÿÿÿÿé r )ÚchunksÚdim)rL Úindex©rL é )
|
|||
|
|
r6 ÚchunkÚreshapeÚshapeÚzipÚtoÚfloat32ÚgatherÚ unsqueezeÚsqueezeÚ logsumexpÚappendÚconcat)
|
|||
|
|
ÚlogitsrM Úchunked_logitsÚ
|
|||
|
|
chunked_indexÚall_per_token_logpsÚ chunk_logitsÚ chunk_indexÚselected_logitsÚlogsumexp_valuesÚper_token_logps© re úW/workspace/DS-LLM-TEMPLATE-FINETUNING/unsloth_compiled_cache/UnslothOnlineDPOTrainer.pyÚ chunked_selective_log_softmax" s
|
|||
|
|
rg c K s$ ddl m} |di | ¤Ž}| |_|S )Nr )ÚSamplingParamsre )Úvllmrh Ú _set_kwargs)Úkwargsrh Úsampling_paramsre re rf ÚvLLMSamplingParams3 s rm c sš e Zd ZU dZedddid�Zee ed< edddid�Z ee
|
|||
|
|
ed < eddd
|
|||
|
|
id�Z ee
|
|||
|
|
ed <
|
|||
|
|
! ! " # $ $ % & ' ( # $ ) * + , - . / d2‡ fd0d1„ Z ‡ Z
|
|||
|
|
S )3ÚUnslothOnlineDPOConfigu¥
|
|||
|
|
|
|||
|
|
Configuration class for the [`OnlineDPOTrainer`].
|
|||
|
|
|
|||
|
|
This class includes only the parameters that are specific to Online DPO training. For a full list of training
|
|||
|
|
arguments, please refer to the [`~transformers.TrainingArguments`] documentation. Note that default values in this
|
|||
|
|
class may differ from those in [`~transformers.TrainingArguments`].
|
|||
|
|
|
|||
|
|
Using [`~transformers.HfArgumentParser`] we can turn this class into
|
|||
|
|
[argparse](https://docs.python.org/3/library/argparse#module-argparse) arguments that can be specified on the
|
|||
|
|
command line.
|
|||
|
|
|
|||
|
|
Parameters:
|
|||
|
|
reward_model_path (`str` or `None`, *optional*, defaults to `None`):
|
|||
|
|
Path to the reward model. Either `judge` or `reward_model_path` must be set, but not both.
|
|||
|
|
judge (`str` or `None`, *optional*, defaults to `None`):
|
|||
|
|
Name of the judge to use. Either `judge` or `reward_model_path` must be set, but not both.
|
|||
|
|
max_new_tokens (`int`, *optional*, defaults to `64`):
|
|||
|
|
Maximum number of tokens to generate per completion.
|
|||
|
|
max_length (`int`, *optional*, defaults to `256`):
|
|||
|
|
Maximum total length of the sequence (prompt + completion) used to compute log probabilities. If the
|
|||
|
|
sequence exceeds this limit, the leftmost tokens will be truncated to preserve as much of the completion as
|
|||
|
|
possible.
|
|||
|
|
temperature (`float`, *optional*, defaults to `0.9`):
|
|||
|
|
Temperature for sampling. The higher the temperature, the more random the completions.
|
|||
|
|
missing_eos_penalty (`float` or `None`, *optional*, defaults to `None`):
|
|||
|
|
Penalty applied to the score when the model fails to generate an EOS token. This is useful to encourage to
|
|||
|
|
generate completions shorter than the maximum length (`max_new_tokens`). The penalty must be a positive
|
|||
|
|
value.
|
|||
|
|
beta (`float` or `list[float]`, *optional*, defaults to `0.1`):
|
|||
|
|
Parameter controlling the deviation from the reference model. Higher β means less deviation from the
|
|||
|
|
reference model. For the IPO loss (`loss_type="ipo"`), β is the regularization parameter denoted by τ in
|
|||
|
|
the [paper](https://huggingface.co/papers/2310.12036). If a list of floats is provided then the β is
|
|||
|
|
selected for each new epoch and the last β is used for the rest of the epochs.
|
|||
|
|
loss_type (`str`, *optional*, defaults to `"sigmoid"`):
|
|||
|
|
Type of loss to use. Possible values are:
|
|||
|
|
|
|||
|
|
- `"sigmoid"`: sigmoid loss from the original [DPO](https://huggingface.co/papers/2305.18290) paper.
|
|||
|
|
- `"ipo"`: IPO loss from the [IPO](https://huggingface.co/papers/2310.12036) paper.
|
|||
|
|
|
|||
|
|
dataset_num_proc (`int` or `None`, *optional*, defaults to `None`):
|
|||
|
|
Number of processes to use for processing the dataset.
|
|||
|
|
disable_dropout (`bool`, *optional*, defaults to `True`):
|
|||
|
|
Whether to disable dropout in the model and reference model.
|
|||
|
|
use_vllm (`bool`, *optional*, defaults to `False`):
|
|||
|
|
Whether to use vLLM for generating completions. Requires vLLM to be installed (`pip install vllm`).
|
|||
|
|
vllm_model_impl (`str`, *optional*, defaults to `"vllm"`):
|
|||
|
|
Model implementation to use for vLLM. Must be one of `"transformers"` or `"vllm"`. `"transformers"`: Use
|
|||
|
|
the `transformers` backend for model implementation. `"vllm"`: Use the `vllm` library for model
|
|||
|
|
implementation.
|
|||
|
|
gpu_memory_utilization (`float`, *optional*, defaults to `0.55`):
|
|||
|
|
The vLLM memory utilization. The default value is 0.55.
|
|||
|
|
ds3_gather_for_generation (`bool`, *optional*, defaults to `True`):
|
|||
|
|
This setting applies to DeepSpeed ZeRO-3. If enabled, the policy model weights are gathered for generation,
|
|||
|
|
improving generation speed. However, disabling this option allows training models that exceed the VRAM
|
|||
|
|
capacity of a single GPU, albeit at the cost of slower generation.
|
|||
|
|
model_init_kwargs (`dict[str, Any]` or `None`, *optional*, defaults to `None`):
|
|||
|
|
Keyword arguments to pass to `AutoModelForCausalLM.from_pretrained` when instantiating the model from a
|
|||
|
|
string.
|
|||
|
|
|
|||
|
|
NÚhelpzvLLM SamplingParams)ÚdefaultÚmetadataÚvllm_sampling_paramsrI z8Chunk size to reduce memory usage. -1 is most efficient.Úunsloth_num_chunksz'Maximum sequence length to truncate to.Úmax_seq_lengthFÚnorJ é r éú ç-C ëâ6
|
|||
|
|
?ç{®Gáz„?çÍÌÌÌÌÌì?ç+‡ÙÎ÷ï?ç:Œ0âŽyE>ç ð?ç @Úlinearçš™™™™™¹?ÚpassiveÚwarningTÚstepsrO éô éO
|
|||
|
|
ÚO1ÚautoÚ ç Ú
|
|||
|
|
adamw_8bitÚlengthÚ
|
|||
|
|
every_saveÚlasté é@ é Úsigmoidri çš™™™™™á?c’ ” s |dk r t d|› d�ƒ‚|dkrtd|› d�ƒ‚|d u r(|#dkr(|$dkr(d}d }#|ˆd u r:d
|
|||
|
|
d lm}“ t|“ƒ d d
|
|||
|
|
ƒ}ˆ|…d
|
|||
|
|
krBtdƒ‚|…dkrJtdƒ‚tƒ jdŸi d|“d|“d|“d|“d|“d|“d|“d|“d| “d|
|
|||
|
|
“d| “d | “d |
|
|||
|
|
“d |“d |“d |“d!|“d"|“d#|“d$|“d%|“d&|“d'|“d(|“d)|“d*|“d+|“d,| “d-| “d.| “d/| “d0| “d1|!“d2|"“d3|#“d4|$“d5|%“d6|&“d7|'“d8|(“d9|)“d:|*“d;|+“d<|,“d=|-“d>|.“d?|/“d@|0“dA|1“dB|2“dC|3“dD|4“dE|5“dF|6“dG|7“dH|8“dI|9“dJ|:“dK|;“dL|<“dM|=“dN|>“dO|?“dP|@“dQ|A“dR|B“dS|C“dT|D“dU|E“dV|F“dW|G“dX|H“dY|I“dZ|J“d[|K“d\|L“d]|M“d^|N“d_|O“d`|P“da|Q“db|R“dc|S“dd|T“de|U“df|V“dg|W“dh|X“di|Y“dj|Z“dk|[“dl|\“dm|]“dn|^“do|_“dp|`“dq|a“dr|b“ds|c“dt|d“du|e“dv|f“dw|g“dx|h“dy|i“dz|j“d{|k“d||l“d}|m“d~|n“d|o“d€|p“d�|q“d‚|r“dƒ|s“d„|t“d…|u“d†|v“d‡|w“dˆ|x“d‰|y“dŠ|z“d‹|{“dŒ||“d�|}“dŽ|~“d�|“d�|€“d‘|�“d’|‚“d“|ƒ“d”|„“d•|…“d–|†“d—|‡“d˜|ˆ“d™|‰“dš|Š“d›|‹“dœ|Œ“d�|�“dž|Ž“|’¤Ž |�| _|�| _ |‘| _
|
|||
|
|
d S ) NgH¯¼šò×z>z Unsloth: Your learning rate of `zi` is too small and less than 1e-7! Consider increasing it, otherwise gradient updates will be close to 0!rO za` is way too larger > 1! Consider decreasing it to 1e-1, otherwise gradient updates will explode!rƒ r„ Ú unsloth_training_checkpointsru r )Ú cpu_countrJ rv zUUnsloth: Please set a positive non-zero temperature since your results will be wrong.é
|
|||
|
|
zgUnsloth: Please set a positive non-zero temperature less than 10, since sampling will be quite erratic.Ú
|
|||
|
|
output_dirÚoverwrite_output_dirÚdo_trainÚdo_evalÚ
|
|||
|
|
do_predictÚ
|
|||
|
|
eval_strategyÚprediction_loss_onlyÚper_device_train_batch_sizeÚper_device_eval_batch_sizeÚper_gpu_train_batch_sizeÚper_gpu_eval_batch_sizeÚgradient_accumulation_stepsÚeval_accumulation_stepsÚ
|
|||
|
|
eval_delayÚtorch_empty_cache_stepsÚ
|
|||
|
|
learning_rateÚ weight_decayÚ
|
|||
|
|
adam_beta1Ú
|
|||
|
|
adam_beta2Ú adam_epsilonÚ
|
|||
|
|
max_grad_normÚnum_train_epochsÚ max_stepsÚlr_scheduler_typeÚ warmup_ratioÚ warmup_stepsÚ log_levelÚlog_level_replicaÚlog_on_each_nodeÚ logging_dirÚlogging_strategyÚlogging_first_stepÚ
|
|||
|
|
logging_stepsÚlogging_nan_inf_filterÚ
|
|||
|
|
save_strategyÚ
|
|||
|
|
save_stepsÚsave_total_limitÚsave_safetensorsÚsave_on_each_nodeÚsave_only_modelÚ'restore_callback_states_from_checkpointÚno_cudaÚuse_cpuÚuse_mps_deviceÚseedÚ data_seedÚ
|
|||
|
|
jit_mode_evalÚuse_ipexÚbf16Úfp16Úfp16_opt_levelÚhalf_precision_backendÚbf16_full_evalÚfp16_full_evalÚtf32Ú
|
|||
|
|
local_rankÚ ddp_backendÚ
|
|||
|
|
tpu_num_coresÚtpu_metrics_debugÚdebugÚdataloader_drop_lastÚ
|
|||
|
|
eval_stepsÚdataloader_num_workersÚdataloader_prefetch_factorÚ
|
|||
|
|
past_indexÚrun_nameÚ disable_tqdmÚremove_unused_columnsÚ label_namesÚload_best_model_at_endÚmetric_for_best_modelÚgreater_is_betterÚignore_data_skipÚfsdpÚfsdp_min_num_paramsÚ fsdp_configÚ"fsdp_transformer_layer_cls_to_wrapÚaccelerator_configÚ deepspeedÚlabel_smoothing_factorÚoptimÚ
|
|||
|
|
optim_argsÚ adafactorÚgroup_by_lengthÚlength_column_nameÚ report_toÚddp_find_unused_parametersÚddp_bucket_cap_mbÚddp_broadcast_buffersÚdataloader_pin_memoryÚ dataloader_persistent_workersÚskip_memory_metricsÚuse_legacy_prediction_loopÚ push_to_hubÚresume_from_checkpointÚ hub_model_idÚ hub_strategyÚ hub_tokenÚhub_private_repoÚhub_always_pushÚ hub_revisionÚgradient_checkpointingÚ gradient_checkpointing_kwargsÚinclude_inputs_for_metricsÚeval_do_concat_batchesÚ fp16_backendÚpush_to_hub_model_idÚpush_to_hub_organizationÚpush_to_hub_tokenÚ
|
|||
|
|
mp_parametersÚauto_find_batch_sizeÚfull_determinismÚ torchdynamoÚ ray_scopeÚ ddp_timeoutÚ
|
|||
|
|
torch_compileÚtorch_compile_backendÚtorch_compile_modeÚinclude_tokens_per_secondÚ include_num_input_tokens_seenÚneftune_noise_alphaÚoptim_target_modulesÚbatch_eval_metricsÚ
|
|||
|
|
eval_on_startÚuse_liger_kernelÚliger_kernel_configÚeval_use_gather_objectÚ average_tokens_across_devicesÚreward_model_pathÚjudgeÚmax_new_tokensÚ
|
|||
|
|
max_lengthÚ temperatureÚmissing_eos_penaltyÚ loss_typeÚdataset_num_procÚdisable_dropoutÚuse_vllmÚvllm_model_implÚgpu_memory_utilizationÚds3_gather_for_generationÚmodel_init_kwargsre ) ÚFloatingPointErrorÚ
|
|||
|
|
OverflowErrorÚmultiprocessingr” ÚmaxÚ MathErrorÚsuperÚ__init__rr rs rt )”Úselfr– r— r˜ r™ rš r› rœ r� rž rŸ r r¡ r¢ r£ r¤ r¥ r¦ r§ r¨ r© rª r« r¬ r r® r¯ r° r± r² r³ r´ rµ r¶ r· r¸ r¹ rº r» r¼ r½ r¾ r¿ rÀ rÁ r rà rÄ rÅ rÆ rÇ rÈ rÉ rÊ rË rÌ rÍ rÎ rÏ rÐ rÑ rÒ rÓ rÔ rÕ rÖ r× rØ rÙ rÚ rÛ rÜ rÝ rÞ rß rà rá râ rã rä rå ræ rç rè ré rê rë rì rí rî rï rð rñ rò ró rô rõ rö r÷ rø rù rú rû rü rý rþ rÿ r r r r r r r r r r r
|
|||
|
|
r r r
|
|||
|
|
r r r r r r r r r r r r r r r r r r r r! r" r# rr rs rt rk r” ©Ú __class__re rf r* ƒ s˜ ÿþýüûúùø ÷
|
|||
|
|
ö õ ô
|
|||
|
|
óòñðïîíìëêéèçæå ä ã â á à!ß"Þ#Ý$Ü%Û&Ú'Ù(Ø)×*Ö+Õ,Ô-Ó.Ò/Ñ0Ð1Ï2Î3Í4Ì5Ë6Ê7É8È9Ç:Æ;Å<Ä=Ã>Â?Á@ÀA¿B¾C½D¼E»FºG¹H¸I·J¶KµL´M³N²O±P°Q¯R®ST¬U«VªW©X¨Y§Z¦[¥\¤]£^¢_¡` aŸbžc�dœe›fšg™h˜i—j–k•l”m“n’o‘p�q�rŽs�tŒu‹vŠw‰xˆy‡z†{…|„}ƒ~‚� �ÿ �þ �ý �ü �û �ú �ù �ø �÷
|
|||
|
|
�ö �õ �ô
|
|||
|
|
�ó �ò �ñ
|
|||
|
|
z UnslothOnlineDPOConfig.__init__)‘NNFFFru FrJ rJ NNrv rv r rw rx ry rz r{ r| r} r~ rI r r€ r r� r‚ TNrƒ FrO Frƒ r„ NTFFFFFFr… r… FFFFr† r‡ FFNrI NNFrˆ FNr NrI NNTNFNNFrˆ r NNNNr‰ rŠ NFFr‹ NNNNTFTFFNNrŒ NNFNFNFTr‡ NNNrˆ TFNr� rŽ FNNFFNNFFFNFTNNr� r� rz Nr‘ NTFri r’ TNNrI N)Ú__name__Ú
|
|||
|
|
__module__Ú __qualname__Ú__doc__r> rr r r Ú__annotations__rs Úintrt r* Ú
|
|||
|
|
__classcell__re re r, rf rn 8 sB
|
|||
|
|
=þþþ �írn c " s> e Zd ZdZddgZ d6deeeje f deeejdf deeejdf d e
|
|||
|
|
e d
|
|||
|
|
e
|
|||
|
|
e d e
|
|||
|
|
e
|
|||
|
|
d e
|
|||
|
|
eeed
|
|||
|
|
f de
|
|||
|
|
eeee ef d
|
|||
|
|
f de
|
|||
|
|
eeeeef de
|
|||
|
|
e de
|
|||
|
|
e de
|
|||
|
|
eegef de
|
|||
|
|
ee deejj ejj j f de
|
|||
|
|
eej ej gej f ddf ‡ fdd„
|
|||
|
|
Z e!dd„ ƒZ"e#de$d edee e%f fd d „ƒZ&e'e(j)ƒde*fd d „ƒZ)e'e(j+ƒd7de
|
|||
|
|
ee ef de*fd!d"„ƒZ+d#d$„ Z,d%d&„ Z-d'd(„ Z. d7dejd)ee eej e%f f d*e
|
|||
|
|
e/ dej fd+d,„Z0 d7d-d.„Z1‡ fd/d0„Z2 d8d1e
|
|||
|
|
e d2e
|
|||
|
|
e d3ee ee df fd4d5„Z3‡ Z4S )9Ú_UnslothOnlineDPOTrainerrˆ Útrlz
|
|||
|
|
online-dpoN©NNÚmodelÚ ref_modelÚ reward_modelr ÚargsÚ
|
|||
|
|
data_collatorÚ
|
|||
|
|
train_datasetzdatasets.DatasetÚ eval_datasetÚprocessing_classÚreward_processing_classÚ peft_configÚcompute_metricsÚ callbacksÚ
|
|||
|
|
optimizersÚ preprocess_logits_for_metricsÚreturnc s~ t |dƒrt |dƒrt|ddƒdkrd|_||u r tdƒ‚|| _|d ur1|d ur1t dt¡ d }n |d u r=|d u r=tdƒ‚|| _|
|
|||
|
|
| _ || _
|
|||
|
|
|j d urS|d urStdƒ‚|d u r[td ƒ‚| d u rctd
|
|||
|
|
ƒ‚|j pgi }t
|
|||
|
|
|tƒr¤|}| d ¡}t
|
|||
|
|
|tjƒs‚|d ks‚|d u rƒnt
|
|||
|
|
|tƒr’tt|ƒ}||d < ntd
|
|||
|
|
|› d�ƒ‚tj|fi |¤Ž}n |j d urtdƒ‚|jj| _ |jrÄt|ƒ | jd urÄt| jƒ |d u rÍ d | _n|| _| j ¡ | jd urß| j ¡ |d u rét | j d�}|j | _ g g g g g g g g g g g dœ | _!| jd u�rg | j!d< g | j!d< g | j!d< |j�r7|j"| _#d| _$t%d!d|j&|j'ddddœtt|dt(ƒ ƒd i ƒ¤Ž| _)nt*|j&|j'ddd|j+�rEdndd �| _)d|j,d < t-ƒ j.|||||| | |
|
|||
|
|
||d �
|
|||
|
|
t | j/d ƒ�rm| j/ 0| j1¡ |j2| _3| j4�r�| jd u�r‡t5| j|j6|j7|j8ƒ| _| jd u�r›t5| j|j6|j7|j8ƒ| _d S d S | jd u�r¬| j 9| j:j;¡| _| jd u�r½| j 9| j:j;¡| _d S d S )"NÚ vllm_enginer FTzš`model` and `ref_model` cannot be the same object. If you want `ref_model` to be the same as `model`, either omit the `ref_model` argument or pass `None`.z€Both `reward_model` and `judge` are provided. Please choose provide only one of them. Ignoring `judge` and using `reward_model`.z2Either `reward_model` or `judge` must be provided.z@`missing_eos_penalty` is not supported when `judge` is provided.z`args` must be provided.z$`processing_class` must be provided.Ú torch_dtyper‡ zŽInvalid `torch_dtype` passed to `OnlineDPOConfig`. Expected either 'auto' or a string representing a `torch.dtype` (e.g., 'float32'), but got Ú.z¦You passed `model_init_kwargs` to the `OnlineDPOConfig`, but your model is already instantiated. This argument can only be used when the `model` argument is a string.zfPEFT is not available and passed `peft_config`. Please install PEFT with `pip install peft` to use it.)Ú pad_token_id) ú objective/klúobjective/entropyúobjective/non_score_rewardúrewards/chosenúrewards/rejectedúrewards/accuraciesúrewards/marginsú logps/chosenúlogps/rejectedúval/contain_eos_tokenÚbetaúobjective/rlhf_rewardúobjective/scores_marginúobjective/scoresr rv é2 r} )ÚnÚ
|
|||
|
|
max_tokensr Útop_kÚtop_pÚ
|
|||
|
|
detokenizerr rj )r r r\ r] Ú do_sampleÚ use_cacheÚestimate_tokens)
|
|||
|
|
r8 r; r< r= r> r? rB rC rD rE Úadd_model_tagsre )<ÚhasattrÚgetattrr Ú
|
|||
|
|
ValueErrorr9 r: ÚwarnÚ UserWarningr: r@ r r r# Ú
|
|||
|
|
isinstanceÚstrÚgetr6 Údtyper Úfrom_pretrainedÚconfigÚis_encoder_decoderr, Ú ImportErrorr Úmerge_and_unloadr r& r$ Úevalr rJ r ÚstatsrG ÚllmÚ_last_loaded_steprh r r rm Úgeneration_configr rû Úwarnings_issuedr) r* r8 rb Ú
|
|||
|
|
_tag_namesrU Ú_betaÚis_deepspeed_enabledr3 r� rÇ rÆ rT Ú acceleratorÚdevice)r+ r8 r9 r: r r; r< r= r> r? r@ rA rB rC rD rE r# Úmodel_idrH r, re rf r* ¿ sü ÿý
|
|||
|
|
|
|||
|
|
|
|||
|
|
|
|||
|
|
|
|||
|
|
|
|||
|
|
ÿÿ
|
|||
|
|
ÿ
|
|||
|
|
|
|||
|
|
|
|||
|
|
|
|||
|
|
|
|||
|
|
|
|||
|
|
õ
|
|||
|
|
|
|||
|
|
|
|||
|
|
ú
|
|||
|
|
ù
|
|||
|
|
ú
|
|||
|
|
ö ÿ
|
|||
|
|
ÿÿ ÿz!_UnslothOnlineDPOTrainer.__init__c C s< t | jtƒr| jj}|t| jƒk r| j| S | jd S | jS )NrI )rh rx ÚlistÚstateÚepochÚlen)r+ r re re rf rU � s "z _UnslothOnlineDPOTrainer.betarn Ú tokenizerc C s’ |s6|| d dd�}|j dur5t|d ƒ}|dks"|j |d d kr5|j g|d |d< dg|d |d< n|| d d d�}d
|
|||
|
|
d „ | ¡ D ƒ}|S ) z2Tokenize a single row from a DPO specific dataset.ÚpromptF)Úadd_special_tokensNÚ input_idsr rO Úattention_maskTc S s i | ]
|
|||
|
|
\}}d |› �|“qS )Úprompt_re )Ú.0ÚkeyÚvaluere re rf Ú
|
|||
|
|
<dictcomp>¥ s z9_UnslothOnlineDPOTrainer.tokenize_row.<locals>.<dictcomp>)Ú bos_token_idr€ Úitems)Úfeaturern r� ÚbatchÚprompt_len_input_idsre re rf Ú tokenize_row˜ s
|
|||
|
|
€z%_UnslothOnlineDPOTrainer.tokenize_rowc C s� | j d u r tdƒ‚| j }| j}| j|| jj| jj| jjdœ}t|t j
|
|||
|
|
j j ƒs<|
|
|||
|
|
¡ |d< | jj|d< t|d< | jj|d< | j t|fi |¤Ž¡S )Nz+Trainer: training requires a train_dataset.©Ú
|
|||
|
|
batch_sizeÚ
|
|||
|
|
collate_fnÚ num_workersÚ
|
|||
|
|
pin_memoryÚpersistent_workersÚsamplerÚ drop_lastÚworker_init_fnÚprefetch_factor)r= re r< Ú_train_batch_sizer; rÔ rï rð rh r6 ÚutilsÚdatar Ú_get_train_samplerrÒ r4 rÕ rz Úpreparer )r+ r= r< Údataloader_paramsre re rf Úget_train_dataloader© s
|
|||
|
|
û z-_UnslothOnlineDPOTrainer.get_train_dataloaderc C s |d u r
|
|||
|
|
| j d u r
|
|||
|
|
tdƒ‚t|tƒr|nd}t| dƒr-|| jv r-| jjr-| j | j| ¡S t|tƒr7| j | n|d ur=|n| j }| j
|
|||
|
|
}| jj || jj | jj
|
|||
|
|
| jjdœ}t|tjjjƒsn| |¡|d< | jj|d< | jj|d< t|fi |¤Ž}| jjrŠt| dƒr…|| j|< n||i| _| j |¡S )Nz-Trainer: evaluation requires an eval_dataset.rq Ú_eval_dataloadersr‘ r— r˜ rš )r> re rh ri rc r¢ r; rð rz rŸ r< Úeval_batch_sizerÔ rï r6 rœ r� r Ú_get_eval_samplerrÒ rÕ r )r+ r> Údataloader_keyr< r Úeval_dataloaderre re rf Úget_eval_dataloaderÁ s@ ÿ
|
|||
|
|
ÿ ÿÿûû
|
|||
|
|
|
|||
|
|
z,_UnslothOnlineDPOTrainer.get_eval_dataloaderc sd | j j‰ | j j‰ td|d iƒr$| jj|| jd| jjddd�d�‰n| jj || jd| jjddd�d�‰‡fdd „t
|
|||
|
|
d
|
|||
|
|
ƒD ƒ}‡fd d „t
|
|||
|
|
d
|
|||
|
|
ƒD ƒ}t d d
|
|||
|
|
„ |D ƒƒ‰‡fdd „|D ƒ}‡‡fdd „|D ƒ}| jj ‰‡fdd „|D ƒ}‡ ‡fdd „|D ƒ}‡‡fdd „|D ƒ}t
|
|||
|
|
j|| jjd�}t
|
|||
|
|
j|| jjd�}t
|
|||
|
|
j|| jjd�}t
|
|||
|
|
j|| jjd�}||||fS )Nr‚ r FZ online_dpo_trainer_lora_modelT)Ú load_tensors)Úuse_tqdmÚ lora_requestc s& g | ]}ˆ D ]
|
|||
|
|
}t |j| jƒ‘qqS re )r} ÚoutputsÚ token_ids)r‡ ÚiÚoutput©r« re rf Ú
|
|||
|
|
<listcomp> s & z;_UnslothOnlineDPOTrainer._generate_vllm.<locals>.<listcomp>rv c s g | ] }ˆ D ]}t |jƒ‘qqS re )r} Úprompt_token_ids)r‡ Ú_r® r¯ re rf r° s c s s � | ]}t |ƒV qd S ©N©r€ ©r‡ Úidsre re rf Ú <genexpr> s € z:_UnslothOnlineDPOTrainer._generate_vllm.<locals>.<genexpr>c s, g | ]}d gˆ t |ƒ dgt |ƒ ‘qS )r rO r´ rµ )Úmax_prompt_lengthre rf r° ó , c s" g | ]
|
|||
|
|
}ˆgˆ t |ƒ | ‘qS re r´ rµ )r¸ rJ re rf r° ó " c s, g | ]}d gt |ƒ dgˆ t |ƒ ‘qS )rO r r´ rµ )r[ re rf r° r¹ c s2 g | ]}|d ˆ krt |ƒˆk r|ˆ g n|‘qS )rI r´ rµ )Ú eos_token_idr[ re rf r° s $ÿÿc s" g | ]
|
|||
|
|
}|ˆgˆ t |ƒ ‘qS re r´ rµ )r[ rJ re rf r°
|
|||
|
|
rº ©r{ )r? r» rJ r+ rs Úchatru r8 Ú load_loraÚgenerateÚranger' r[ r6 Útensorrz r{ )r+ r8 ÚpromptsÚcompletion_idsÚ
|
|||
|
|
prompt_idsÚ prompt_maskÚcompletion_maskre )r» r¸ r[ r« rJ rf Ú_generate_vllmñ s. $" þ z'_UnslothOnlineDPOTrainer._generate_vllmc sò ˆ j j}ˆ j j}dd„ |D ƒ}‡ fdd„|D ƒ}‡ fdd„|D ƒ}ˆ |¡}ˆ |¡}|d dd¡}|d dd¡}t|ˆ jˆ jj d ��}|j
|
|||
|
|
||ˆ j d
|
|||
|
|
�} W d ƒ n1 sYw Y | d d …| d¡d …f }
|
|||
|
|
t
|
|||
|
|
|
|
|||
|
|
||ƒ\}
|
|||
|
|
} |||
|
|||
|
|
| fS ) Nc S s g | ]}d |i‘qS ©r‚ re ©r‡ r‚ re re rf r° ó z6_UnslothOnlineDPOTrainer._generate.<locals>.<listcomp>c ó g | ]}t |ˆ jƒ‘qS re )r0 r? ©r‡ Úx©r+ re rf r° ó c s g | ] }ˆ |ˆ jˆ j¡‘qS re )r� rn r? rÌ rÎ re rf r° s Úprompt_input_idsrv rO Úprompt_attention_mask)Úgather_deepspeed3_params)r„ r… ru )r? r» rJ r< Ú_prepare_inputsÚrepeatr8 rz r; r" r¿ ru Úsizer7 ) r+ r8 r r» rJ ÚinputsrÄ rÅ Úunwrapped_modelr® rà rÆ re rÎ rf Ú _generate s,
|
|||
|
|
|
|||
|
|
ÿý ý z"_UnslothOnlineDPOTrainer._generatec C sÔ t | d¡| d¡ | j dƒ}|d d …|d …f }|d d …|d …f }tj||fdd�}tj||fdd�}|||d�} | d¡}
|
|||
|
|
|
|
|||
|
|
dkrI|
|
|||
|
|
d nd} | jd d …| d…f } tj| jdd�| d¡dd� d¡}
|
|||
|
|
|
|
|||
|
|
S )NrO r rN )r… rI rv )
|
|||
|
|
r' rÕ r r6 Úcatr\ Útake_along_dimÚ log_softmaxrW rX )r+ r8 rÄ rÅ rà rÆ Únum_tokens_to_truncateÚprompt_completion_idsÚprompt_completion_maskr® Ú
|
|||
|
|
prompt_lenÚ start_idxr\ Úlogprobsre re rf Ú_forward4 s
|
|||
|
|
$z!_UnslothOnlineDPOTrainer._forwardrÖ Únum_items_in_batchc = s | ¡ |d }t|ƒ}ˆ jjrˆ ||¡\}}}} n
|
|||
|
|
ˆ ||¡\}}}} tj|ˆ jj kdd�}
|
|||
|
|
ˆ
|
|||
|
|
||||| ¡} t ¡ �7 ˆ j d urNˆ
|
|||
|
|
ˆ j |||| ¡} n ˆ j
|
|||
|
|
¡ � ˆ
|
|||
|
|
ˆ j
|
|||
|
|
|||| ¡} W d ƒ n1 shw Y W d ƒ n1 sww Y | j}
|
|||
|
|
ˆ jj|dd�}td|d iƒr–dd„ |D ƒ}ˆ jd urßtd|d iƒr¾t ¡ }| t¡‰‡fd d„|D ƒ}‡fd
|
|||
|
|
d„|D ƒ}ˆ j |tt|d |… ||d … ƒƒ¡}tjd d„ |D ƒ|
|
|||
|
|
d �}n—d
|
|||
|
|
| }td|d iƒ�r
|
|||
|
|
dd„ t||ƒD ƒ}‡ fdd„|D ƒ}dd„ |D ƒ}dd„ |D ƒ}ˆ j|dddd�d |
|
|||
|
|
¡}|j d }ˆ j|dddd�d |
|
|||
|
|
¡}tj ||fdd�}t ¡ �' t ˆ j |ˆ jj!|ƒ\}}}ˆ jj"d u�r[||
|
|||
|
|
ˆ jj"8 < W d ƒ n 1 �sfw Y | #|¡\}}||k}tj$||
|
|||
|
|
d �}|| | } ||| } tj | | fdd�} | | } | | } | %¡ }!|!| }"| |" &d¡}#| |" &d¡}$t #|#|¡\}%}&t #|$|¡\}'}(|%|& })|'|( }*|)|* }+ˆ jj'dk�rát( )ˆ j*|+ ¡ },nˆ jj'dk�rô|+dd
|
|||
|
|
ˆ j* d
|
|||
|
|
},nt+dˆ j'› �ƒ‚|, ,¡ }-ˆ j d u�r2|| || }.ˆ j-d .ˆ j/ 0|. ,¡ ¡ ,¡ 1¡ ¡ ˆ j-d .ˆ j/ 0| ,¡ ¡ ,¡ 1¡ ¡ ˆ j-d .|
|
|||
|
|
2¡ ,¡ 1¡ ¡ ˆ j-d .ˆ j/ 0|%¡ ,¡ 1¡ ¡ ˆ j-d .ˆ j/ 0|&¡ ,¡ 1¡ ¡ | | }/|/ &d¡ ,¡ }0ˆ j-d .ˆ j/ 0|0¡ ,¡ 1¡ ¡ ˆ j* |/ &d¡}1|1 ,¡ }2ˆ j-d! .ˆ j/ 0|2¡ ,¡ 1¡ ¡ ˆ j d u�r²||1 }3ˆ j-d" .ˆ j/ 0|3¡ ,¡ 1¡ ¡ | &d¡ ,¡ }4ˆ j-d# .ˆ j/ 0|4¡ ,¡ 1¡ ¡ ˆ j*|%|' }5ˆ j/ 0|5¡}6ˆ j-d$ .|6 ,¡ 1¡ ¡ ˆ j*|&|( }7ˆ j/ 0|7¡}8ˆ j-d% .|8 ,¡ 1¡ ¡ |6|8 }9ˆ j-d& .|9 ,¡ 1¡ ¡ |9dk}:ˆ j-d' .|: 2¡ ,¡ 1¡ ¡ ˆ j-d( .ˆ j*¡ ˆ jj3d u�r<ˆ j4j5ˆ jj3 dk�r<t6ƒ i };ˆ jj7t8j9t8j:fv �rOˆ ;¡ |;d)< ˆ jj<dk�rZ|- ,¡ }-ˆ j=�r{t> ?|-ˆ j@¡� }<|< A¡ W d ƒ n 1 �suw Y n
|
|||
|
|
ˆ j/jA|-fi |;¤Ž |- B¡ ˆ jjC S )*Nr‚ rI rN T)Úskip_special_tokensr c S s g | ]}d |dœg‘qS )Ú assistant)ÚroleÚcontentre ©r‡ Ú
|
|||
|
|
completionre re rf r° g rÏ z:_UnslothOnlineDPOTrainer.training_step.<locals>.<listcomp>c ó g | ]}ˆ j |d �‘qS ©)Úmessages©ÚrenderrÉ ©Útemplatere rf r° r rÏ c rê rë rí rè rï re rf r° s rÏ c S s g | ]}|d k‘qS )r re )r‡ Úrankre re rf r° | rÊ r¼ rv c S s g | ] \}}||d œ‘qS ))r‚ ré re )r‡ ÚpÚcre re rf r° ‚ s c rË re )r# r@ ©r‡ ÚexamplerÎ re rf r° ƒ rÏ c S ó g | ]}|d ‘qS rÈ re rô re re rf r° „ rÊ c S rö )ré re rô re re rf r° … rÊ ÚptÚleft)ÚpaddingÚreturn_tensorsÚ padding_sider„ rO Úrightr‘ Úipozinvalid loss type rW rX rT rR rS rK rM rV rL rN rO rQ rP rU r¥ )DÚtrainr€ r; r rÇ rØ r6 Úanyr? r» râ Úno_gradr9 r8 Údisable_adapterr{ Ú batch_decoder+ r r. Ú EnvironmentÚ from_stringr r} rS rÁ r@ rT rR rÙ Úinference_moder* r: rJ r ÚsplitÚarangeÚboolÚsumr r Ú
|
|||
|
|
logsigmoidrU ÚNotImplementedErrorÚmeanrr rZ rz Úgather_for_metricsÚitemÚfloatr¤ r~ Ú global_stepr' ræ r ÚLOMOÚADALOMOÚ_get_learning_rateÚn_gpuÚuse_apexÚampÚ
|
|||
|
|
scale_lossÚ optimizerÚbackwardÚdetachr¡ )=r+ r8 rÖ rã r r’ rÄ rÅ rà rÆ Úcontain_eos_tokenrá Ú ref_logprobsr{ Ú completionsÚ environmentÚranks_of_first_completionÚmaskÚexamplesÚ prompts_idsÚcontext_lengthÚcompletions_idsrÝ r² ÚscoresÚ
|
|||
|
|
first_halfÚ second_halfÚ batch_rangeÚchosen_indicesÚrejected_indicesÚ
|
|||
|
|
cr_indicesÚ cr_logprobsÚcr_ref_logprobsÚ padding_maskÚcr_padding_maskÚcr_logprobs_sumÚcr_ref_logprobs_sumÚchosen_logprobs_sumÚrejected_logprobs_sumÚchosen_ref_logprobs_sumÚrejected_ref_logprobs_sumÚ pi_logratiosÚ
|
|||
|
|
ref_logratiosr\ ÚlossesÚlossÚ
|
|||
|
|
scores_marginÚklÚmean_klÚnon_score_rewardÚmean_non_score_rewardÚ rlhf_rewardÚ mean_entropyÚchosen_rewardsÚgathered_chosen_rewardsÚrejected_rewardsÚgathered_rejected_rewardsÚmarginÚaccuracyrk Ú scaled_lossre )r+ rð rf Ú
|
|||
|
|
training_stepL s
|
|||
|
|
|
|||
|
|
ÿ€ ü
|
|||
|
|
|
|||
|
|
ÿÿþþ
|
|||
|
|
ÿþþ
|
|||
|
|
|
|||
|
|
ÿ€ ø
|
|||
|
|
|
|||
|
|
ÿ$
|
|||
|
|
ÿ
|
|||
|
|
ÿ€z&_UnslothOnlineDPOTrainer.training_stepc C sj | j jr~| jj| jkr~i } | |¡ ¡ ¡ }
|
|||
|
|
||8 }t|
|
|||
|
|
| jj| j dƒ| d< |d ur<t |t
|
|||
|
|
j ƒr8| ¡ ¡ n|| d< |d urE|| d< n|
|
|||
|
|
¡ | d< | j ¡ D ]\} } t| ƒt| ƒ | | < qPdd„ | jD ƒ| _| j|
|
|||
|
|
7 _| jj| _| ¡ | | |¡ d }
|
|||
|
|
| j jr›| ||¡}
|
|||
|
|
| j|
|
|||
|
|
|d�}| jjdkr›|| j _| j jr³| ||¡ | j | j| j| j ¡| _ d S d S ) NrJ r9 Ú grad_normr¥ c S s i | ]}|g “qS re re )r‡ rˆ re re rf rŠ s zE_UnslothOnlineDPOTrainer._maybe_log_save_evaluate.<locals>.<dictcomp>)ÚmetricsÚtrialÚbest) ÚcontrolÚ
|
|||
|
|
should_logr~ r Ú_globalstep_last_loggedÚ_nested_gatherr r Úroundrh r6 r r r rr rŒ r r€ Ú_total_loss_scalarÚ
|
|||
|
|
store_flosÚlogÚshould_evaluateÚ _evaluateÚ_determine_best_metricr; r¸ Ú should_saveÚ_save_checkpointÚcallback_handlerÚon_save)r+ Útr_lossrI r8 rK r Úignore_keys_for_evalÚ
|
|||
|
|
start_timer¥ ÚlogsÚtr_loss_scalarrˆ ÚvalrJ Úis_new_best_metricre re rf Ú_maybe_log_save_evaluate s6
|
|||
|
|
|
|||
|
|
þz1_UnslothOnlineDPOTrainer._maybe_log_save_evaluatec sL | j jd u rt| j jƒj}n | j j d¡d }| j|d� tƒ ||¡ d S )Nú/rI )Ú
|
|||
|
|
model_name) r; rõ r r– Únamer Úcreate_model_cardr) rY )r+ r8 rK re r, re rf rY , s
|
|||
|
|
z)_UnslothOnlineDPOTrainer._save_checkpointre Ú dataset_nameÚtagsc
|
|||
|
|
C sê | ¡ sdS t| jjdƒr tj | jjj¡s | jjj}nd}|du r&tƒ }n
|
|||
|
|
t |t
|
|||
|
|
ƒr/|h}nt|ƒ}t| jjdƒr?| d¡ | | j
|
|||
|
|
¡ t d¡}t||| j||tƒ r]tjdur]tjjndtƒ d|ddd � }| tj | jjd
|
|||
|
|
¡¡ dS ) aî
|
|||
|
|
Creates a draft of a model card using the information available to the `Trainer`.
|
|||
|
|
|
|||
|
|
Args:
|
|||
|
|
model_name (`str` or `None`, *optional*, defaults to `None`):
|
|||
|
|
Name of the model.
|
|||
|
|
dataset_name (`str` or `None`, *optional*, defaults to `None`):
|
|||
|
|
Name of the dataset used for training.
|
|||
|
|
tags (`str`, `list[str]` or `None`, *optional*, defaults to `None`):
|
|||
|
|
Tags to be associated with the model card.
|
|||
|
|
NÚ
|
|||
|
|
_name_or_pathÚunsloth_versionÚunslotha» @article{guo2024direct,
|
|||
|
|
title = {{Direct Language Model Alignment from Online AI Feedback}},
|
|||
|
|
author = {Shangmin Guo and Biao Zhang and Tianlin Liu and Tianqi Liu and Misha Khalman and Felipe Llinares and Alexandre Ram{'{e}} and Thomas Mesnard and Yao Zhao and Bilal Piot and Johan Ferret and Mathieu Blondel},
|
|||
|
|
year = 2024,
|
|||
|
|
eprint = {arXiv:2402.04792}
|
|||
|
|
}z
|
|||
|
|
Online DPOz7Direct Language Model Alignment from Online AI Feedbackz
|
|||
|
|
2402.04792) Ú
|