206 lines
29 KiB
Plaintext
206 lines
29 KiB
Plaintext
|
o
|
|||
|
|
:—°hDº ã @ s¾ d Z ddlmZ ddlZddlmZ ddlmZ ddlmZm Z m
|
|||
|
|
Z
|
|||
|
|
m Z m Z m
|
|||
|
|
Z
|
|||
|
|
mZmZ ddlmZmZmZmZmZmZmZmZmZmZmZmZmZmZm
|
|||
|
|
Z
|
|||
|
|
m Z m Z m Z m Z m Z m!Z!m Z m"Z"m#Z#m$Z$m%Z%m&Z&m'Z'm(Z(m)Z)m*Z*mZm+Z+m,Z,m-Z-mZm.Z.m/Z/ ddl+Z+ddlT ddl0m1Z1m2Z2 dd l3m4Z4 ddlZddl5Z6dd
|
|||
|
|
l7m8Z8 ddlmZ dd l9m:Z:m;Z< d d
|
|||
|
|
d d
|
|||
|
|
d
|
|||
|
|
dœZ=ej>d d e=d�dd„ ƒZ?e1G dd„ deƒƒZ@ G dd„ deƒZAG dd„ deAƒZBdS )z9
|
|||
|
|
2025.8.9
|
|||
|
|
2025.8.10
|
|||
|
|
4.55.4
|
|||
|
|
0.21.0
|
|||
|
|
__UNSLOTH_VERSIONING__
|
|||
|
|
é )ÚTensorN)Ú
|
|||
|
|
functional)ÚAnyÚListÚOptionalÚTupleÚUnionÚDictÚSetÚCallable)&r ÚBaseImageProcessorÚBasePairwiseJudger ÚDatasetÚEvalPredictionÚFÚFeatureExtractionMixinÚGeometricMixtureWrapperÚIterableDatasetÚ NashMDConfigÚ
|
|||
|
|
NashMDTrainerÚOnlineDPOTrainerÚOptimizerNamesr Ú PeftModelÚPreTrainedModelÚPreTrainedTokenizerBaseÚProcessorMixinÚSIMPLE_CHAT_TEMPLATEÚTrainerCallbackr Ú empty_cacheÚgenerate_model_cardÚget_comet_experiment_urlÚ
|
|||
|
|
get_rewardÚis_conversationalÚis_peft_availableÚis_wandb_availableÚjinja2Úmaybe_apply_chat_templateÚnnÚosÚselective_log_softmaxÚtextwrapÚtorchÚtruncate_rightÚunwrap_model_for_generation)Ú*)Ú dataclassÚfield)ÚVersion)Ú nullcontext)ÚDataCollatorForSeq2SeqÚ DataCollatorForLanguageModelingTF)Úepilogue_fusionÚ max_autotuneÚ
|
|||
|
|
shape_paddingz
|
|||
|
|
trace.enabledztriton.cudagraphs)ÚdynamicÚ fullgraphÚoptionsc
|
|||
|
|
C s¾ t j| d| jd ¡ddd�}t j| d¡ddd�}g }t||ƒD ](\}}| t j¡}t j|d| d¡d� d¡}t j
|
|||
|
|
|dd�}|| } | | ¡ q! t |¡}| | jd | jd f¡}|S )Néÿÿÿÿé r )ÚchunksÚdim)r> Úindex©r> é )
|
|||
|
|
r+ ÚchunkÚreshapeÚshapeÚzipÚtoÚfloat32ÚgatherÚ unsqueezeÚsqueezeÚ logsumexpÚappendÚconcat)
|
|||
|
|
Úlogitsr? Úchunked_logitsÚ
|
|||
|
|
chunked_indexÚall_per_token_logpsÚ chunk_logitsÚ chunk_indexÚselected_logitsÚlogsumexp_valuesÚper_token_logps© rW úT/workspace/DS-LLM-TEMPLATE-FINETUNING/unsloth_compiled_cache/UnslothNashMDTrainer.pyÚ chunked_selective_log_softmax" s
|
|||
|
|
rY c sš e Zd ZU dZedddid�Zee ed< edddid�Z ee
|
|||
|
|
ed < eddd
|
|||
|
|
id�Z ee
|
|||
|
|
ed <
|
|||
|
|
! ! " # $ $ % & ' ( # $ ) * + , - . / d2‡ fd0d1„ Z ‡ Z
|
|||
|
|
S )3ÚUnslothNashMDConfigaö
|
|||
|
|
|
|||
|
|
Configuration class for the [`NashMDTrainer`].
|
|||
|
|
|
|||
|
|
Subclass of [`OnlineDPOConfig`] we can use all its arguments and add the following:
|
|||
|
|
|
|||
|
|
Parameters:
|
|||
|
|
mixture_coef (`float` or `list[float]`, *optional*, defaults to `0.5`):
|
|||
|
|
Logit mixture coefficient for the model and reference model. If a list of floats is provided then the
|
|||
|
|
mixture coefficient is selected for each new epoch and the last coefficient is used for the rest of the
|
|||
|
|
epochs.
|
|||
|
|
|
|||
|
|
NÚhelpzvLLM SamplingParams)ÚdefaultÚmetadataÚvllm_sampling_paramsr; z8Chunk size to reduce memory usage. -1 is most efficient.Úunsloth_num_chunksz'Maximum sequence length to truncate to.Úmax_seq_lengthFÚnor< é r éú ç-C ëâ6
|
|||
|
|
?ç{®Gáz„?çÍÌÌÌÌÌì?ç+‡ÙÎ÷ï?ç:Œ0âŽyE>ç ð?ç @Úlinearçš™™™™™¹?ÚpassiveÚwarningTÚstepsrA éô éO
|
|||
|
|
ÚO1ÚautoÚ ç Ú
|
|||
|
|
adamw_8bitÚlengthÚ
|
|||
|
|
every_saveÚlasté é@ é ÚsigmoidÚvllmçš™™™™™á?c’ ” s |dk r t d|› d�ƒ‚|dkrtd|› d�ƒ‚|d u r(|#dkr(|$dkr(d}d }#|ˆd u r:d
|
|||
|
|
d lm}“ t|“ƒ d d
|
|||
|
|
ƒ}ˆ|…d
|
|||
|
|
krBtdƒ‚|…dkrJtdƒ‚tƒ jdŸi d|“d|“d|“d|“d|“d|“d|“d|“d| “d|
|
|||
|
|
“d| “d | “d |
|
|||
|
|
“d |“d |“d |“d!|“d"|“d#|“d$|“d%|“d&|“d'|“d(|“d)|“d*|“d+|“d,| “d-| “d.| “d/| “d0| “d1|!“d2|"“d3|#“d4|$“d5|%“d6|&“d7|'“d8|(“d9|)“d:|*“d;|+“d<|,“d=|-“d>|.“d?|/“d@|0“dA|1“dB|2“dC|3“dD|4“dE|5“dF|6“dG|7“dH|8“dI|9“dJ|:“dK|;“dL|<“dM|=“dN|>“dO|?“dP|@“dQ|A“dR|B“dS|C“dT|D“dU|E“dV|F“dW|G“dX|H“dY|I“dZ|J“d[|K“d\|L“d]|M“d^|N“d_|O“d`|P“da|Q“db|R“dc|S“dd|T“de|U“df|V“dg|W“dh|X“di|Y“dj|Z“dk|[“dl|\“dm|]“dn|^“do|_“dp|`“dq|a“dr|b“ds|c“dt|d“du|e“dv|f“dw|g“dx|h“dy|i“dz|j“d{|k“d||l“d}|m“d~|n“d|o“d€|p“d�|q“d‚|r“dƒ|s“d„|t“d…|u“d†|v“d‡|w“dˆ|x“d‰|y“dŠ|z“d‹|{“dŒ||“d�|}“dŽ|~“d�|“d�|€“d‘|�“d’|‚“d“|ƒ“d”|„“d•|…“d–|†“d—|‡“d˜|ˆ“d™|‰“dš|Š“d›|‹“dœ|Œ“d�|�“dž|Ž“|’¤Ž |�| _|�| _ |‘| _
|
|||
|
|
d S ) NgH¯¼šò×z>z Unsloth: Your learning rate of `zi` is too small and less than 1e-7! Consider increasing it, otherwise gradient updates will be close to 0!rA za` is way too larger > 1! Consider decreasing it to 1e-1, otherwise gradient updates will explode!ro rp Ú unsloth_training_checkpointsra r )Ú cpu_countr< rb zUUnsloth: Please set a positive non-zero temperature since your results will be wrong.é
|
|||
|
|
zgUnsloth: Please set a positive non-zero temperature less than 10, since sampling will be quite erratic.Ú
|
|||
|
|
output_dirÚoverwrite_output_dirÚdo_trainÚdo_evalÚ
|
|||
|
|
do_predictÚ
|
|||
|
|
eval_strategyÚprediction_loss_onlyÚper_device_train_batch_sizeÚper_device_eval_batch_sizeÚper_gpu_train_batch_sizeÚper_gpu_eval_batch_sizeÚgradient_accumulation_stepsÚeval_accumulation_stepsÚ
|
|||
|
|
eval_delayÚtorch_empty_cache_stepsÚ
|
|||
|
|
learning_rateÚ weight_decayÚ
|
|||
|
|
adam_beta1Ú
|
|||
|
|
adam_beta2Ú adam_epsilonÚ
|
|||
|
|
max_grad_normÚnum_train_epochsÚ max_stepsÚlr_scheduler_typeÚ warmup_ratioÚ warmup_stepsÚ log_levelÚlog_level_replicaÚlog_on_each_nodeÚ logging_dirÚlogging_strategyÚlogging_first_stepÚ
|
|||
|
|
logging_stepsÚlogging_nan_inf_filterÚ
|
|||
|
|
save_strategyÚ
|
|||
|
|
save_stepsÚsave_total_limitÚsave_safetensorsÚsave_on_each_nodeÚsave_only_modelÚ'restore_callback_states_from_checkpointÚno_cudaÚuse_cpuÚuse_mps_deviceÚseedÚ data_seedÚ
|
|||
|
|
jit_mode_evalÚuse_ipexÚbf16Úfp16Úfp16_opt_levelÚhalf_precision_backendÚbf16_full_evalÚfp16_full_evalÚtf32Ú
|
|||
|
|
local_rankÚ ddp_backendÚ
|
|||
|
|
tpu_num_coresÚtpu_metrics_debugÚdebugÚdataloader_drop_lastÚ
|
|||
|
|
eval_stepsÚdataloader_num_workersÚdataloader_prefetch_factorÚ
|
|||
|
|
past_indexÚrun_nameÚ disable_tqdmÚremove_unused_columnsÚ label_namesÚload_best_model_at_endÚmetric_for_best_modelÚgreater_is_betterÚignore_data_skipÚfsdpÚfsdp_min_num_paramsÚ fsdp_configÚ"fsdp_transformer_layer_cls_to_wrapÚaccelerator_configÚ deepspeedÚlabel_smoothing_factorÚoptimÚ
|
|||
|
|
optim_argsÚ adafactorÚgroup_by_lengthÚlength_column_nameÚ report_toÚddp_find_unused_parametersÚddp_bucket_cap_mbÚddp_broadcast_buffersÚdataloader_pin_memoryÚ dataloader_persistent_workersÚskip_memory_metricsÚuse_legacy_prediction_loopÚ push_to_hubÚresume_from_checkpointÚ hub_model_idÚ hub_strategyÚ hub_tokenÚhub_private_repoÚhub_always_pushÚ hub_revisionÚgradient_checkpointingÚ gradient_checkpointing_kwargsÚinclude_inputs_for_metricsÚeval_do_concat_batchesÚ fp16_backendÚpush_to_hub_model_idÚpush_to_hub_organizationÚpush_to_hub_tokenÚ
|
|||
|
|
mp_parametersÚauto_find_batch_sizeÚfull_determinismÚ torchdynamoÚ ray_scopeÚ ddp_timeoutÚ
|
|||
|
|
torch_compileÚtorch_compile_backendÚtorch_compile_modeÚinclude_tokens_per_secondÚ include_num_input_tokens_seenÚneftune_noise_alphaÚoptim_target_modulesÚbatch_eval_metricsÚ
|
|||
|
|
eval_on_startÚuse_liger_kernelÚliger_kernel_configÚeval_use_gather_objectÚ average_tokens_across_devicesÚreward_model_pathÚjudgeÚmax_new_tokensÚ
|
|||
|
|
max_lengthÚ temperatureÚmissing_eos_penaltyÚ loss_typeÚdataset_num_procÚdisable_dropoutÚuse_vllmÚvllm_model_implÚgpu_memory_utilizationÚds3_gather_for_generationÚmodel_init_kwargsrW ) ÚFloatingPointErrorÚ
|
|||
|
|
OverflowErrorÚmultiprocessingr� ÚmaxÚ MathErrorÚsuperÚ__init__r^ r_ r` )”Úselfrƒ r„ r… r† r‡ rˆ r‰ rŠ r‹ rŒ r� rŽ r� r� r‘ r’ r“ r” r• r– r— r˜ r™ rš r› rœ r� rž rŸ r r¡ r¢ r£ r¤ r¥ r¦ r§ r¨ r© rª r« r¬ r r® r¯ r° r± r² r³ r´ rµ r¶ r· r¸ r¹ rº r» r¼ r½ r¾ r¿ rÀ rÁ r rà rÄ rÅ rÆ rÇ rÈ rÉ rÊ rË rÌ rÍ rÎ rÏ rÐ rÑ rÒ rÓ rÔ rÕ rÖ r× rØ rÙ rÚ rÛ rÜ rÝ rÞ rß rà rá râ rã rä rå ræ rç rè ré rê rë rì rí rî rï rð rñ rò ró rô rõ rö r÷ rø rù rú rû rü rý rþ rÿ r r r r r r r r r r r
|
|||
|
|
r r r
|
|||
|
|
r r r r^ r_ r` Úkwargsr� ©Ú __class__rW rX r N s˜ ÿþýüûúùø ÷
|
|||
|
|
ö õ ô
|
|||
|
|
óòñðïîíìëêéèçæå ä ã â á à!ß"Þ#Ý$Ü%Û&Ú'Ù(Ø)×*Ö+Õ,Ô-Ó.Ò/Ñ0Ð1Ï2Î3Í4Ì5Ë6Ê7É8È9Ç:Æ;Å<Ä=Ã>Â?Á@ÀA¿B¾C½D¼E»FºG¹H¸I·J¶KµL´M³N²O±P°Q¯R®ST¬U«VªW©X¨Y§Z¦[¥\¤]£^¢_¡` aŸbžc�dœe›fšg™h˜i—j–k•l”m“n’o‘p�q�rŽs�tŒu‹vŠw‰xˆy‡z†{…|„}ƒ~‚� �ÿ �þ �ý �ü �û �ú �ù �ø �÷
|
|||
|
|
�ö �õ �ô
|
|||
|
|
�ó �ò �ñ
|
|||
|
|
z UnslothNashMDConfig.__init__)‘NNFFFra Fr< r< NNrb rb r rc rd re rf rg rh ri rj r; rk rl r rm rn TNro FrA Fro rp NTFFFFFFrq rq FFFFrr rs FFNr; NNFrt FNr Nr; NNTNFNNFrt r NNNNru rv NFFrw NNNNTFTFFNNrx NNFNFNFTrs NNNrt TFNry rz FNNFFNNFFFNFTNNr{ r| rf Nr} NTFr~ r TNNr; N)Ú__name__Ú
|
|||
|
|
__module__Ú __qualname__Ú__doc__r0 r^ r r Ú__annotations__r_ Úintr` r Ú
|
|||
|
|
__classcell__rW rW r rX rZ 3 sB
|
|||
|
|
|
|||
|
|
þþþ �írZ c sØ e Zd ZdZddgZ d0deeejf deeejf deeejdf d e e
|
|||
|
|
d
|
|||
|
|
e e d e e d e ee
|
|||
|
|
ef d
|
|||
|
|
e ee
|
|||
|
|
eee
|
|||
|
|
f f de eeeeef de e de e egef de ee deejjejj j f de e ej ej gej f ddf ‡ fdd„
|
|||
|
|
Z e dd„ ƒZ!dd„ Z"dd „ Z#d d „ Z$d d „ Z%d!d"„ Z&d#d$„ Z' dd%d&„Z( d1dejd'eeeej e)f f d(e e* dej fd)d*„Z+ d2d+e e d,e e d-eeee df fd.d/„Z,‡ Z-S )3Ú_UnslothNashMDTrainerrt Útrlznash-mdN©NNÚmodelÚ ref_modelÚ reward_modelr ÚargsÚ
|
|||
|
|
data_collatorÚ
|
|||
|
|
train_datasetÚ eval_datasetÚprocessing_classÚ peft_configÚcompute_metricsÚ callbacksÚ
|
|||
|
|
optimizersÚ preprocess_logits_for_metricsÚreturnc sz t ƒ j||||||||| | |
|
|||
|
|
| | |
|
|||
|
|
|d� | jj| _g g g g g g g g g g g g dœ | _| jd ur;g | jd< g | jd< d S d S )N)r& r' r( r r) r* r+ r, r- Úreward_processing_classr. r/ r0 r1 r2 ) úloss/klúobjective/entropyú
|
|||
|
|
loss/scoreúrewards/probabilitiesúrewards/accuraciesúrewards/marginsú logps/chosenúlogps/rejectedúval/model_contain_eos_tokenúval/ref_contain_eos_tokenÚbetaÚ mixture_coefúrewards/chosenúrewards/rejected)r r r) r@ Ú
|
|||
|
|
_mixture_coefÚstatsr( )r r& r' r( r r) r* r+ r, r- r. r/ r0 r1 r2 r rW rX r Š sF ñ
|
|||
|
|
ò
|
|||
|
|
|
|||
|
|
þz _UnslothNashMDTrainer.__init__c C s< t | jtƒr| jj}|t| jƒk r| j| S | jd S | jS )Nr; )Ú
|
|||
|
|
isinstancerC ÚlistÚstateÚepochÚlen)r rH rW rW rX r@ Æ s "z"_UnslothNashMDTrainer.mixture_coefc C sô t || jƒ�}|j|d |d | jd�}W d ƒ n1 s w Y | j |¡}| jd u r>tƒ r;t|tƒr;| ¡ }n
|
|||
|
|
|}n| j | j¡}t
|
|||
|
|
¡ �% t ||| j| j
|
|||
|
|
| jjd�}|j|d |d | jd�}W d ƒ ||fS 1 sqw Y ||fS )NÚ input_idsÚattention_mask)rJ rK Úgeneration_config)r& r' rL r@ Údevice)r- Ú acceleratorÚgeneraterL Ú unwrap_modelr' r# rE r Úget_base_modelr+ Úno_gradr r@ rM ) r r& ÚpromptsÚ unwrapped_policy_for_gen_ctxÚ model_outputÚpolicy_model_for_gmwÚref_model_for_gmwÚ
|
|||
|
|
mixture_modelÚmixture_outputrW rW rX Ú_generate_completionsÎ s< ý ÿ
|
|||
|
|
|
|||
|
|
|
|||
|
|
ûý
|
|||
|
|
÷ñz+_UnslothNashMDTrainer._generate_completionsc C sÖ |d j d }|d d …|d …f }t|| jj| jjƒ\}}tj|d |fdd�tj|d |fdd�|d dœ}|d d …|d …f }t|| jj| jjƒ\}} tj|d |fdd�tj|d | fdd�|d dœ}
|
|||
|
|
||
|
|||
|
|
fS )NrJ rA r@ rK Úraw©rJ rK r[ )rD r, r- Ú eos_token_idÚ pad_token_idr+ Úcat) r rU rY rS Úcontext_lengthÚmodel_completion_idsÚmodel_completion_maskÚ
|
|||
|
|
model_dataÚmixture_completion_idsÚmixture_completion_maskÚ mixture_datarW rW rX Ú_process_completionsþ s$ ÿýÿýz*_UnslothNashMDTrainer._process_completionsc C sØ t ¡ �& t| j|d | jj|ƒ\}}}t| j|d | jj|ƒ\}}}W d ƒ n1 s-w Y | jjd urht j|d | jj kdd�}t j|d | jj kdd�}|| | jj8 < || | jj8 < ||fS )NrJ r; r@ )
|
|||
|
|
r+ rR r! r( r- r^ r) r Úanyr] ) r rc rf r` Ú_Ú model_scoresÚmixture_scoresÚmodel_contain_eosÚmixture_contain_eosrW rW rX Ú_compute_rewards s
|
|||
|
|
|
|||
|
|
ÿ ÿ ü z&_UnslothNashMDTrainer._compute_rewardsc s |d }| j j|d d d …|d …f dd�}dd„ |D ƒ}| j j|d d d …|d …f dd�}dd„ |D ƒ}td|d iƒrpd
|
|||
|
|
d„ |D ƒ}t ¡ }| t¡‰ ‡ fd d„|D ƒ}‡ fd d„|D ƒ}d
|
|||
|
|
d„ |D ƒ}‡ fdd„|D ƒ}| jj|tt ||ƒƒdd�}t
|
|||
|
|
j ||d j d�S )Nr[ rJ T)Úskip_special_tokensc S ó g | ]}| ¡ ‘qS rW ©Ústrip©Ú.0Ú
|
|||
|
|
completionrW rW rX Ú
|
|||
|
|
<listcomp>0 ó z8_UnslothNashMDTrainer._compute_judge.<locals>.<listcomp>c S rp rW rq rs rW rW rX rv 5 rw Úpromptr c S ó g | ]}d |dœg‘qS ©Ú assistant)ÚroleÚcontentrW rs rW rW rX rv 7 ó ÿc ó g | ]}ˆ j |d �‘qS ©)Úmessages©Úrender)rt Úmessage©ÚtemplaterW rX rv < ó c r r€ r‚ rs r… rW rX rv = r‡ c S ry rz rW rs rW rW rX rv ? r~ c r r€ r‚ rs r… rW rX rv B r~ )Ú
|
|||
|
|
return_scores)rM )
|
|||
|
|
r- Ú batch_decoder" r% Ú EnvironmentÚ from_stringr r rF rE r+ ÚtensorrM ) r rc rf r` rS Úmodel_data_completionsÚmixture_data_completionsÚ environmentÚ probabilityrW r… rX Ú_compute_judge+ s: ÿÿÿ
|
|||
|
|
ÿ
|
|||
|
|
ÿ ýz$_UnslothNashMDTrainer._compute_judgec sÄ ‡ fdd„}|||ƒ}t ¡ �- | jd u r/| ¡ �
|
|||
|
|
|||ƒ}W d ƒ n1 s)w Y n|| j|ƒ}W d ƒ n1 s?w Y |d d d …ˆ d …f dk}| |d¡}| |d¡}||fS )Nc sP | |d |d d�}|j d d …ˆ d d…f }t||d d d …ˆ d …f ƒ}|S )NrJ rK )rK rA r; )rN r) )ÚmÚdataÚoutputrN Útoken_logprobs©r` rW rX Úcompute_logprobs_for_dataN s zJ_UnslothNashMDTrainer._compute_logprobs.<locals>.compute_logprobs_for_datarK r ru )r+ rR r' Údisable_adapterÚ masked_fill)r r& rc r` r— Úmodel_logprobs_model_dataÚref_logprobs_model_dataÚmodel_padding_maskrW r– rX Ú_compute_logprobsM s
|
|||
|
|
|
|||
|
|
|
|||
|
|
|
|||
|
|
ÿ€ € û z'_UnslothNashMDTrainer._compute_logprobsc C sv |d | d¡ }t ¡ � || }| d¡}W d ƒ n1 s!w Y || d¡}| j| | }| ¡ ||fS )Ng à?rA )Úsumr+ rR r? Úmean) r rš r› r� ÚscoreÚ log_ratioÚ
|
|||
|
|
kl_div_logÚ kl_div_lossÚlossrW rW rX Ú_compute_lossesf s
|
|||
|
|
þz%_UnslothNashMDTrainer._compute_lossesc sº ‡ fdd„} ˆ j d | |ƒ¡ ˆ j d | |ƒ¡ | d¡} | d¡}
|
|||
|
|
ˆ j d | | ƒ¡ ˆ j d | |
|
|||
|
|
ƒ¡ ˆ jd urQˆ j d | | ƒ¡ ˆ j d | |
|
|||
|
|
ƒ¡ ˆ j d
|
|||
|
|
| |ƒ¡ | d¡ }ˆ j d | |ƒ¡ | |
|
|||
|
|
}ˆ j d | |ƒ¡ |d
|
|||
|
|
k ¡ }ˆ j d | |ƒ¡ |d d d …|d …f ˆ jjkjdd�}|d d d …|d …f ˆ jjkjdd�}ˆ j d | | ¡ ƒ¡ ˆ j d | | ¡ ƒ¡ ˆ j d ˆ j¡ ˆ j d ˆ j ¡ d S )Nc s ˆ j | ¡ ¡ ¡ S ©N)rN Úgather_for_metricsrŸ Úitem)rŒ ©r rW rX Ú gather_meanˆ s z:_UnslothNashMDTrainer._log_statistics.<locals>.gather_meanr7 r5 rA r; r< rA rB r8 r6 r: r r9 rJ r@ r= r> r? r@ )
|
|||
|
|
rD rL rž r( Úfloatr- r] rh r? r@ )r rc rf rš r› r� r Úkl_divr` rj rk rª Ú model_logprobs_model_data_sumÚref_logprobs_model_data_sumÚentropy_model_dataÚmarginÚaccuracyÚ model_eosÚ mixture_eosrW r© rX Ú_log_statisticsz s.
|
|||
|
|
|
|||
|
|
|
|||
|
|
((z%_UnslothNashMDTrainer._log_statisticsÚinputsÚnum_items_in_batchc s | ¡ tttˆ ¡ ƒƒƒ}ˆ d }‡ fdd„t|ƒD ƒ‰ ‡fdd„ˆ D ƒ‰ ‡fdd„ˆ D ƒ‰ ˆ ˆ ¡‰ ˆ ˆ ¡‰ ˆ d jd }ˆ d ˆ d |d œ}Š ˆ ||¡\}}ˆ
|
|||
|
|
|||¡\} }
|
|||
|
|
ˆj d urrˆ | |
|
|||
|
|
|¡\} } t
|
|||
|
|
| | ¡}
|
|||
|
|
n d
|
|||
|
|
\} } ˆ | |
|
|||
|
|
|¡}
|
|||
|
|
ˆ || |¡\}}ˆ |||
|
|||
|
|
¡\}}}ˆ | |
|
|||
|
|
| ¡ ||
|
|||
|
|
| ¡ | ¡ || | ¡
|
|||
|
|
ˆjjd ur·ˆjjˆjj d kr·tƒ i }ˆjjtjtj fv rɈ ¡ |d < ˆjj dkrÓ| ¡ }ˆj ròt! "|ˆj#¡� }| $¡ W d ƒ n1 sìw Y n
|
|||
|
|
ˆj%j$|fi |¤Ž | ¡ ˆjj& S )
|
|||
|
|
Nrx c s" g | ]
|
|||
|
|
‰ ‡ fd d„ˆ ¡ D ƒ‘qS )c s i | ] \}}||ˆ “qS rW rW )rt ÚkÚv©ÚirW rX Ú
|
|||
|
|
<dictcomp>½ s zB_UnslothNashMDTrainer.training_step.<locals>.<listcomp>.<dictcomp>)Úitems)rt )rµ r¹ rX rv ½ ó " z7_UnslothNashMDTrainer.training_step.<locals>.<listcomp>c s g | ]}t |ˆ jƒ‘qS rW )r& r- ©rt Úxr© rW rX rv ¾ r‡ c s" g | ]
|
|||
|
|
}ˆ |ˆ jjjˆ j¡‘qS rW )Ú tokenize_rowr& ÚconfigÚis_encoder_decoderr- r¾ r© rW rX rv ¿ r½ Úprompt_input_idsrA Úprompt_attention_maskr\ r% r r’ )'ÚtrainrI ÚnextÚiterÚvaluesÚranger* Ú_prepare_inputsrD rZ rg r( rn r r} r‘ r� r¥ r´ Údetachr) r‘ rG Ú global_stepr rÓ r ÚLOMOÚADALOMOÚ_get_learning_rateÚn_gpurŸ Úuse_apexÚampÚ
|
|||
|
|
scale_lossÚ optimizerÚbackwardrN rŽ )r r& rµ r¶ Ú
|
|||
|
|
batch_sizerS r` rU rY rc rf rj rk r� rš r› r¤ r r¬ r Ú scaled_lossrW )rµ r rX Ú
|
|||
|
|
training_stepµ sd
|
|||
|
|
|
|||
|
|
ý
|
|||
|
|
ö
|
|||
|
|
ÿ€z#_UnslothNashMDTrainer.training_stepÚ
|
|||
|
|
model_nameÚ dataset_nameÚtagsc
|
|||
|
|
C sê | ¡ sdS t| jjdƒr tj | jjj¡s | jjj}nd}|du r&tƒ }n
|
|||
|
|
t |t
|
|||
|
|
ƒr/|h}nt|ƒ}t| jjdƒr?| d¡ | | j
|
|||
|
|
¡ t d¡}t||| j||tƒ r]tjdur]tjjndtƒ d|ddd � }| tj | jjd
|
|||
|
|
¡¡ dS ) aî
|
|||
|
|
Creates a draft of a model card using the information available to the `Trainer`.
|
|||
|
|
|
|||
|
|
Args:
|
|||
|
|
model_name (`str` or `None`, *optional*, defaults to `None`):
|
|||
|
|
Name of the model.
|
|||
|
|
dataset_name (`str` or `None`, *optional*, defaults to `None`):
|
|||
|
|
Name of the dataset used for training.
|
|||
|
|
tags (`str`, `list[str]` or `None`, *optional*, defaults to `None`):
|
|||
|
|
Tags to be associated with the model card.
|
|||
|
|
NÚ
|
|||
|
|
_name_or_pathÚunsloth_versionÚunslotha @inproceedings{munos2024nash,
|
|||
|
|
title = {{Nash Learning from Human Feedback}},
|
|||
|
|
author = {R{'{e}}mi Munos and Michal Valko and Daniele Calandriello and Mohammad Gheshlaghi Azar and Mark Rowland and Zhaohan Daniel Guo and Yunhao Tang and Matthieu Geist and Thomas Mesnard and C{\^{o}}me Fiegel and Andrea Michi and Marco Selvi and Sertan Girgin and Nikola Momchev and Olivier Bachem and Daniel J. Mankowitz and Doina Precup and Bilal Piot},
|
|||
|
|
year = 2024,
|
|||
|
|
booktitle = {Forty-first International Conference on Machine Learning, {ICML} 2024, Vienna, Austria, July 21-27, 2024},
|
|||
|
|
publisher = {OpenReview.net},
|
|||
|
|
url = {https://openreview.net/forum?id=Y5AmNYiyCQ}
|
|||
|
|
}zNash-MDz!Nash Learning from Human Feedbackz
|
|||
|
|
2312.00886) Ú
|
|||
|
|
base_modelrÙ râ rÚ rÛ Ú wandb_urlÚ comet_urlÚ trainer_nameÚtrainer_citationÚ paper_titleÚpaper_idz README.md)Úis_world_process_zeroÚhasattrr& rÁ r( ÚpathÚisdirrÜ ÚsetrE ÚstrÚaddÚupdateÚ
|
|||
|
|
_tag_namesr* Údedentr râ r$ ÚwandbÚrunÚurlr ÚsaveÚjoinr) rƒ )r rÙ rÚ rÛ rß ÚcitationÚ
|
|||
|
|
model_cardrW rW rX Úcreate_model_card s8
|
|||
|
|
|
|||
|
|
|
|||
|
|
|
|||
|
|
õ z'_UnslothNashMDTrainer.create_model_card)NNNNNNNNNNNNr% Nr¦ )NNN).r r r r rî r r r' ÚModuler r
|
|||
|
|
r r r r Údictrë r r r r r rF r Útupler+ rÓ Ú OptimizerÚ lr_schedulerÚLambdaLRr r Úpropertyr@ rZ rg rn r‘ r� r¥ r´ r r! rØ r÷ r" rW rW r rX r# … s¢ ï þ ýüûúùø ÷
|
|||
|
|
ÿö
|