434 lines
46 KiB
Plaintext
434 lines
46 KiB
Plaintext
|
o
|
|||
|
|
6—°h=3 ã @ s d Z ddlmZ ddlZddlmZ ddlmZ ddlmZm Z m
|
|||
|
|
Z
|
|||
|
|
m Z m Z m
|
|||
|
|
Z
|
|||
|
|
mZmZ ddlmZmZmZmZmZmZmZmZmZmZmZmZmZmZm
|
|||
|
|
Z
|
|||
|
|
m Z m Z m Z m Z m Z m!Z!m"Z"m#Z#m Z m$Z$m%Z%m&Z&m'Z'm(Z(m)Z)m*Z*m+Z+m,Z,m-Z-m.Z.m/Z/m0Z0m1Z1m2Z2mZm3Z3m4Z4m5Z5m6Z6m7Z7m8Z8m9Z9m:Z:m;Z;m<Z<mZm=Z=mZm
|
|||
|
|
Z
|
|||
|
|
m Z m Z m"Z"m-Z-m5Z5mZ ddl5Z5ddlT ddl>m?Z?m@Z@ dd lAmBZB ddlZddlCZ3dd
|
|||
|
|
lDm4Z4 ddlmZ dd lEmFZFmGZH d d
|
|||
|
|
d d
|
|||
|
|
d
|
|||
|
|
dœZIejJd d eId�dd„ ƒZKe?G dd„ deƒƒZL G dd„ de"ƒZMG dd„ deMƒZNdS )z9
|
|||
|
|
2025.8.9
|
|||
|
|
2025.8.10
|
|||
|
|
4.55.4
|
|||
|
|
0.21.0
|
|||
|
|
__UNSLOTH_VERSIONING__
|
|||
|
|
é )ÚTensorN)Ú
|
|||
|
|
functional)ÚAnyÚListÚOptionalÚTupleÚUnionÚDictÚSetÚCallable)<r ÚAutoModelForCausalLMÚBaseImageProcessorÚ CPOConfigÚ
|
|||
|
|
CPOTrainerr ÚDPODataCollatorWithPaddingÚ DataCollatorÚ
|
|||
|
|
DataLoaderÚDatasetÚEvalLoopOutputÚFÚFeatureExtractionMixinÚLiteralr Ú PartialStateÚPathÚ PeftModelÚPreTrainedModelÚPreTrainedTokenizerBaseÚProcessorMixinÚTrainerÚTrainerCallbackr Úadd_bos_token_if_neededÚadd_eos_token_if_neededÚautocastÚ defaultdictÚdisable_dropout_in_modelÚgenerate_model_cardÚget_comet_experiment_urlÚinspectÚis_comet_availableÚis_peft_availableÚis_torch_fx_proxyÚis_wandb_availableÚ log_table_to_comet_experimentÚmaybe_apply_chat_templateÚmaybe_extract_promptÚnnÚnpÚ nullcontextÚosÚ
|
|||
|
|
pad_to_lengthÚpdÚpeft_module_casting_to_bf16Ú prepare_model_for_kbit_trainingÚrandomÚselective_log_softmaxÚtextwrapÚtorchÚwarningsr r r r r r) r2 r: )Ú*)Ú dataclassÚfield)ÚVersion)r1 )ÚDataCollatorForSeq2SeqÚ DataCollatorForLanguageModelingTF)Úepilogue_fusionÚ max_autotuneÚ
|
|||
|
|
shape_paddingz
|
|||
|
|
trace.enabledztriton.cudagraphs)ÚdynamicÚ fullgraphÚoptionsc
|
|||
|
|
C s¾ t j| d| jd ¡ddd�}t j| d¡ddd�}g }t||ƒD ](\}}| t j¡}t j|d| d¡d� d¡}t j
|
|||
|
|
|dd�}|| } | | ¡ q! t |¡}| | jd | jd f¡}|S )Néÿÿÿÿé r )ÚchunksÚdim)rK Úindex©rK é )
|
|||
|
|
r: ÚchunkÚreshapeÚshapeÚzipÚtoÚfloat32ÚgatherÚ unsqueezeÚsqueezeÚ logsumexpÚappendÚconcat)
|
|||
|
|
ÚlogitsrL Úchunked_logitsÚ
|
|||
|
|
chunked_indexÚall_per_token_logpsÚ chunk_logitsÚ chunk_indexÚselected_logitsÚlogsumexp_valuesÚper_token_logps© rd úQ/workspace/DS-LLM-TEMPLATE-FINETUNING/unsloth_compiled_cache/UnslothCPOTrainer.pyÚ chunked_selective_log_softmax" s
|
|||
|
|
rf c sž e Zd ZU dZedddid�Zee ed< edddid�Z ee
|
|||
|
|
ed < eddd
|
|||
|
|
id�Z ee
|
|||
|
|
ed <
|
|||
|
|
! ! " # $ $ % & ' ( # $ ) * + , % - . / 0 d3‡ fd1d2„ Z ‡ Z
|
|||
|
|
S )4ÚUnslothCPOConfiguy
|
|||
|
|
|
|||
|
|
Configuration class for the [`CPOTrainer`].
|
|||
|
|
|
|||
|
|
This class includes only the parameters that are specific to CPO training. For a full list of training arguments,
|
|||
|
|
please refer to the [`~transformers.TrainingArguments`] documentation. Note that default values in this class may
|
|||
|
|
differ from those in [`~transformers.TrainingArguments`].
|
|||
|
|
|
|||
|
|
Using [`~transformers.HfArgumentParser`] we can turn this class into
|
|||
|
|
[argparse](https://docs.python.org/3/library/argparse#module-argparse) arguments that can be specified on the
|
|||
|
|
command line.
|
|||
|
|
|
|||
|
|
Parameters:
|
|||
|
|
max_length (`int` or `None`, *optional*, defaults to `1024`):
|
|||
|
|
Maximum length of the sequences (prompt + completion) in the batch. This argument is required if you want
|
|||
|
|
to use the default data collator.
|
|||
|
|
max_prompt_length (`int` or `None`, *optional*, defaults to `512`):
|
|||
|
|
Maximum length of the prompt. This argument is required if you want to use the default data collator.
|
|||
|
|
max_completion_length (`int` or `None`, *optional*, defaults to `None`):
|
|||
|
|
Maximum length of the completion. This argument is required if you want to use the default data collator
|
|||
|
|
and your model is an encoder-decoder.
|
|||
|
|
beta (`float`, *optional*, defaults to `0.1`):
|
|||
|
|
Parameter controlling the deviation from the reference model. Higher β means less deviation from the
|
|||
|
|
reference model. For the IPO loss (`loss_type="ipo"`), β is the regularization parameter denoted by τ in
|
|||
|
|
the [paper](https://huggingface.co/papers/2310.12036).
|
|||
|
|
label_smoothing (`float`, *optional*, defaults to `0.0`):
|
|||
|
|
Label smoothing factor. This argument is required if you want to use the default data collator.
|
|||
|
|
loss_type (`str`, *optional*, defaults to `"sigmoid"`):
|
|||
|
|
Type of loss to use. Possible values are:
|
|||
|
|
|
|||
|
|
- `"sigmoid"`: sigmoid loss from the original [DPO](https://huggingface.co/papers/2305.18290) paper.
|
|||
|
|
- `"hinge"`: hinge loss on the normalized likelihood from the
|
|||
|
|
[SLiC](https://huggingface.co/papers/2305.10425) paper.
|
|||
|
|
- `"ipo"`: IPO loss from the [IPO](https://huggingface.co/papers/2310.12036) paper.
|
|||
|
|
- `"simpo"`: SimPO loss from the [SimPO](https://huggingface.co/papers/2405.14734) paper.
|
|||
|
|
|
|||
|
|
disable_dropout (`bool`, *optional*, defaults to `True`):
|
|||
|
|
Whether to disable dropout in the model.
|
|||
|
|
cpo_alpha (`float`, *optional*, defaults to `1.0`):
|
|||
|
|
Weight of the BC regularizer in CPO training.
|
|||
|
|
simpo_gamma (`float`, *optional*, defaults to `0.5`):
|
|||
|
|
Target reward margin for the SimPO loss, used only when the `loss_type="simpo"`.
|
|||
|
|
label_pad_token_id (`int`, *optional*, defaults to `-100`):
|
|||
|
|
Label pad token id. This argument is required if you want to use the default data collator.
|
|||
|
|
padding_value (`int` or `None`, *optional*, defaults to `None`):
|
|||
|
|
Padding value to use. If `None`, the padding value of the tokenizer is used.
|
|||
|
|
truncation_mode (`str`,*optional*, defaults to `"keep_end"`):
|
|||
|
|
Truncation mode to use when the prompt is too long. Possible values are `"keep_end"` or `"keep_start"`.
|
|||
|
|
This argument is required if you want to use the default data collator.
|
|||
|
|
generate_during_eval (`bool`, *optional*, defaults to `False`):
|
|||
|
|
If `True`, generates and logs completions from the model to W&B or Comet during evaluation.
|
|||
|
|
is_encoder_decoder (`bool` or `None`, *optional*, defaults to `None`):
|
|||
|
|
When using the `model_init` argument (callable) to instantiate the model instead of the `model` argument,
|
|||
|
|
you need to specify if the model returned by the callable is an encoder-decoder model.
|
|||
|
|
model_init_kwargs (`dict[str, Any]` or `None`, *optional*, defaults to `None`):
|
|||
|
|
Keyword arguments to pass to `AutoModelForCausalLM.from_pretrained` when instantiating the model from a
|
|||
|
|
string.
|
|||
|
|
dataset_num_proc (`int` or `None`, *optional*, defaults to `None`):
|
|||
|
|
Number of processes to use for processing the dataset.
|
|||
|
|
|
|||
|
|
NÚhelpzvLLM SamplingParams)ÚdefaultÚmetadataÚvllm_sampling_paramsrH z8Chunk size to reduce memory usage. -1 is most efficient.Úunsloth_num_chunksz'Maximum sequence length to truncate to.Úmax_seq_lengthFÚnorI é r éú ç-C ëâ6
|
|||
|
|
?ç{®Gáz„?çÍÌÌÌÌÌì?ç+‡ÙÎ÷ï?ç:Œ0âŽyE>ç ð?ç @Úlinearçš™™™™™¹?ÚpassiveÚwarningTÚstepsrN éô éO
|
|||
|
|
ÚO1ÚautoÚ ç Ú
|
|||
|
|
adamw_8bitÚlengthÚ
|
|||
|
|
every_saveÚlasté é é Úsigmoidç à?éœÿÿÿÚkeep_endc” – sü |dk r t d|› d�ƒ‚|dkrtd|› d�ƒ‚|d u r(|#dkr(|$dkr(d}d }#|�d u r:d
|
|||
|
|
d lm}• t|•ƒ d d
|
|||
|
|
ƒ}�tƒ jdži d|“d|“d|“d|“d|“d|“d|“d|“d| “d|
|
|||
|
|
“d| “d| “d|
|
|||
|
|
“d|“d |“d |“d |“d |“d |“d!|“d"|“d#|“d$|“d%|“d&|“d'|“d(|“d)| “d*| “d+| “d,| “d-| “d.|!“d/|"“d0|#“d1|$“d2|%“d3|&“d4|'“d5|(“d6|)“d7|*“d8|+“d9|,“d:|-“d;|.“d<|/“d=|0“d>|1“d?|2“d@|3“dA|4“dB|5“dC|6“dD|7“dE|8“dF|9“dG|:“dH|;“dI|<“dJ|=“dK|>“dL|?“dM|@“dN|A“dO|B“dP|C“dQ|D“dR|E“dS|F“dT|G“dU|H“dV|I“dW|J“dX|K“dY|L“dZ|M“d[|N“d\|O“d]|P“d^|Q“d_|R“d`|S“da|T“db|U“dc|V“dd|W“de|X“df|Y“dg|Z“dh|[“di|\“dj|]“dk|^“dl|_“dm|`“dn|a“do|b“dp|c“dq|d“dr|e“ds|f“dt|g“du|h“dv|i“dw|j“dx|k“dy|l“dz|m“d{|n“d||o“d}|p“d~|q“d|r“d€|s“d�|t“d‚|u“dƒ|v“d„|w“d…|x“d†|y“d‡|z“dˆ|{“d‰||“dŠ|}“d‹|~“dŒ|“d�|€“dŽ|�“d�|‚“d�|ƒ“d‘|„“d’|…“d“|†“d”|‡“d•|ˆ“d–|‰“d—|Š“d˜|‹“d™|Œ“dš|�“d›|Ž“dœ|�“d�|�“|”¤Ž |‘| _|’| _|“| _ d S )ŸNgH¯¼šò×z>z Unsloth: Your learning rate of `zi` is too small and less than 1e-7! Consider increasing it, otherwise gradient updates will be close to 0!rN za` is way too larger > 1! Consider decreasing it to 1e-1, otherwise gradient updates will explode!r| r} Ú unsloth_training_checkpointsrn r )Ú cpu_countrI ro Ú
|
|||
|
|
output_dirÚoverwrite_output_dirÚdo_trainÚdo_evalÚ
|
|||
|
|
do_predictÚ
|
|||
|
|
eval_strategyÚprediction_loss_onlyÚper_device_train_batch_sizeÚper_device_eval_batch_sizeÚper_gpu_train_batch_sizeÚper_gpu_eval_batch_sizeÚgradient_accumulation_stepsÚeval_accumulation_stepsÚ
|
|||
|
|
eval_delayÚtorch_empty_cache_stepsÚ
|
|||
|
|
learning_rateÚ weight_decayÚ
|
|||
|
|
adam_beta1Ú
|
|||
|
|
adam_beta2Ú adam_epsilonÚ
|
|||
|
|
max_grad_normÚnum_train_epochsÚ max_stepsÚlr_scheduler_typeÚ warmup_ratioÚ warmup_stepsÚ log_levelÚlog_level_replicaÚlog_on_each_nodeÚ logging_dirÚlogging_strategyÚlogging_first_stepÚ
|
|||
|
|
logging_stepsÚlogging_nan_inf_filterÚ
|
|||
|
|
save_strategyÚ
|
|||
|
|
save_stepsÚsave_total_limitÚsave_safetensorsÚsave_on_each_nodeÚsave_only_modelÚ'restore_callback_states_from_checkpointÚno_cudaÚuse_cpuÚuse_mps_deviceÚseedÚ data_seedÚ
|
|||
|
|
jit_mode_evalÚuse_ipexÚbf16Úfp16Úfp16_opt_levelÚhalf_precision_backendÚbf16_full_evalÚfp16_full_evalÚtf32Ú
|
|||
|
|
local_rankÚ ddp_backendÚ
|
|||
|
|
tpu_num_coresÚtpu_metrics_debugÚdebugÚdataloader_drop_lastÚ
|
|||
|
|
eval_stepsÚdataloader_num_workersÚdataloader_prefetch_factorÚ
|
|||
|
|
past_indexÚrun_nameÚ disable_tqdmÚremove_unused_columnsÚ label_namesÚload_best_model_at_endÚmetric_for_best_modelÚgreater_is_betterÚignore_data_skipÚfsdpÚfsdp_min_num_paramsÚ fsdp_configÚ"fsdp_transformer_layer_cls_to_wrapÚaccelerator_configÚ deepspeedÚlabel_smoothing_factorÚoptimÚ
|
|||
|
|
optim_argsÚ adafactorÚgroup_by_lengthÚlength_column_nameÚ report_toÚddp_find_unused_parametersÚddp_bucket_cap_mbÚddp_broadcast_buffersÚdataloader_pin_memoryÚ dataloader_persistent_workersÚskip_memory_metricsÚuse_legacy_prediction_loopÚ push_to_hubÚresume_from_checkpointÚ hub_model_idÚ hub_strategyÚ hub_tokenÚhub_private_repoÚhub_always_pushÚ hub_revisionÚgradient_checkpointingÚ gradient_checkpointing_kwargsÚinclude_inputs_for_metricsÚeval_do_concat_batchesÚ fp16_backendÚpush_to_hub_model_idÚpush_to_hub_organizationÚpush_to_hub_tokenÚ
|
|||
|
|
mp_parametersÚauto_find_batch_sizeÚfull_determinismÚ torchdynamoÚ ray_scopeÚ ddp_timeoutÚ
|
|||
|
|
torch_compileÚtorch_compile_backendÚtorch_compile_modeÚinclude_tokens_per_secondÚ include_num_input_tokens_seenÚneftune_noise_alphaÚoptim_target_modulesÚbatch_eval_metricsÚ
|
|||
|
|
eval_on_startÚuse_liger_kernelÚliger_kernel_configÚeval_use_gather_objectÚ average_tokens_across_devicesÚ
|
|||
|
|
max_lengthÚmax_prompt_lengthÚmax_completion_lengthÚbetaÚlabel_smoothingÚ loss_typeÚdisable_dropoutÚ cpo_alphaÚ simpo_gammaÚlabel_pad_token_idÚ
|
|||
|
|
padding_valueÚtruncation_modeÚgenerate_during_evalÚis_encoder_decoderÚmodel_init_kwargsÚdataset_num_procrd )
|
|||
|
|
ÚFloatingPointErrorÚ
|
|||
|
|
OverflowErrorÚmultiprocessingr� ÚmaxÚsuperÚ__init__rk rl rm )–Úselfr� r‘ r’ r“ r” r• r– r— r˜ r™ rš r› rœ r� rž rŸ r r¡ r¢ r£ r¤ r¥ r¦ r§ r¨ r© rª r« r¬ r r® r¯ r° r± r² r³ r´ rµ r¶ r· r¸ r¹ rº r» r¼ r½ r¾ r¿ rÀ rÁ r rà rÄ rÅ rÆ rÇ rÈ rÉ rÊ rË rÌ rÍ rÎ rÏ rÐ rÑ rÒ rÓ rÔ rÕ rÖ r× rØ rÙ rÚ rÛ rÜ rÝ rÞ rß rà rá râ rã rä rå ræ rç rè ré rê rë rì rí rî rï rð rñ rò ró rô rõ rö r÷ rø rù rú rû rü rý rþ rÿ r r r r r r r r r r r
|
|||
|
|
r r r
|
|||
|
|
r r r r r r r r r r r r r r r r r r rk rl rm Úkwargsr� ©Ú __class__rd re r% ~ s ÿþýüûúùø ÷
|
|||
|
|
ö õ ô
|
|||
|
|
óòñðïîíìëêéèçæå ä ã â á à!ß"Þ#Ý$Ü%Û&Ú'Ù(Ø)×*Ö+Õ,Ô-Ó.Ò/Ñ0Ð1Ï2Î3Í4Ì5Ë6Ê7É8È9Ç:Æ;Å<Ä=Ã>Â?Á@ÀA¿B¾C½D¼E»FºG¹H¸I·J¶KµL´M³N²O±P°Q¯R®ST¬U«VªW©X¨Y§Z¦[¥\¤]£^¢_¡` aŸbžc�dœe›fšg™h˜i—j–k•l”m“n’o‘p�q�rŽs�tŒu‹vŠw‰xˆy‡z†{…|„}ƒ~‚� �ÿ �þ �ý �ü �û �ú �ù �ø �÷
|
|||
|
|
�ö �õ �ô
|
|||
|
|
�ó �ò �ñ �ð �ï
|
|||
|
|
zUnslothCPOConfig.__init__)“NNFFFrn FrI rI NNro ro r rp rq rr rs rt ru rv rw rH rx ry r rz r{ TNr| FrN Fr| r} NTFFFFFFr~ r~ FFFFr r€ FFNrH NNFr� FNr NrH NNTNFNNFr� r NNNNr‚ rƒ NFFr„ NNNNTFTFFNNr… NNFNFNFTr€ NNNr� TFNr† r‡ FNNFFNNFFFNFTrˆ r‰ Nry r‚ rŠ Trv r‹ rŒ Nr� FNNNNrH N)Ú__name__Ú
|
|||
|
|
__module__Ú __qualname__Ú__doc__r> rk r r Ú__annotations__rl Úintrm r% Ú
|
|||
|
|
__classcell__rd rd r( re rg 3 sF
|
|||
|
|
=þþþ �ërg c sÈ e Zd ZdZddgZ dRdeeeej e
|
|||
|
|
f dee dee d ee
|
|||
|
|
d
|
|||
|
|
eee
|
|||
|
|
ee
|
|||
|
|
e
|
|||
|
|
f f d eeeeeef d eeg ef d
|
|||
|
|
eee deejjejjjf deeej ej gej f dee deee gef f‡ fdd„
|
|||
|
|
Z dd„ Z dSdeeeej f defdd„Z e! dTd ee
|
|||
|
|
eeej"f f d e#d e$d e$d eej% dee
|
|||
|
|
ej"f f d!d"„ƒZ&d#ej'd$ej'deej'ej'ej'f fd%d&„Z(e! dUd'ej'd(ej"d)e#d e$d e#dej'f d*d+„ƒZ)dej d ee
|
|||
|
|
eeej"f f deej'ej'ej'ej'f fd,d-„Z* .dVd ee
|
|||
|
|
eeej"f f d/e+d0 fd1d2„Z, dWdeeej f d3ee
|
|||
|
|
eej e-f f deej eej ee
|
|||
|
|
ej f f f fd4d5„Z.d ee
|
|||
|
|
ej"f de
|
|||
|
|
fd6d7„Z/ dSdeeej f d3ee
|
|||
|
|
eej e-f f d8e#d9eee
|
|||
|
|
fd:d;„Z0dVd<ee
|
|||
|
|
e1f d/e+d0 ddfd=d>„Z2 ?dXd@e3dAe
|
|||
|
|
d8ee# d9eee
|
|||
|
|
dBe
|
|||
|
|
de f ‡ fdCdD„
|
|||
|
|
Z4dSdEee
|
|||
|
|
e1f dFee1 ddf‡ fdGdH„
|
|||
|
|
Z5dIdJ„ Z6‡ fdKdL„Z7 dYdMee
|
|||
|
|
dNee
|
|||
|
|
dOee
|
|||
|
|
ee
|
|||
|
|
df fdPdQ„Z8‡ Z9S )ZÚ_UnslothCPOTrainerr� ÚtrlÚcpoN©NNÚmodelÚargsÚ
|
|||
|
|
data_collatorÚ
|
|||
|
|
train_datasetÚ eval_datasetÚprocessing_classÚ
|
|||
|
|
model_initÚ callbacksÚ
|
|||
|
|
optimizersÚ preprocess_logits_for_metricsÚ peft_configÚcompute_metricsc
|
|||
|
|
|
|||
|
|
sF |j d u ri }
|
|||
|
|
n9t|tƒstdƒ‚|j }
|
|||
|
|
|
|
|||
|
|
d¡}|d urAt|tƒr+|dkr+tt|ƒ}|dkr=t|tjƒs=td|› d�ƒ‚||
|
|||
|
|
d< t|tƒrOtj |fi |
|
|||
|
|
¤Ž}d| _
|
|||
|
|
t ƒ s]| d ur]tdƒ‚t ƒ rÈ| d urÈt|t ƒrm|
|
|||
|
|
¡ }t|ddƒsyt|d dƒr�t|d
|
|||
|
|
ƒo‡d
|
|||
|
|
tt t¡jƒv }d |ji}|r”|j|d
|
|||
|
|
< t|fi |¤Ž}n|jrµt|d ƒrª| ¡ n d
|
|||
|
|
d„ }| ¡ |¡ |}|jrÇt|d dƒrÇt|ƒ d| _
|
|||
|
|
n|jràt|d ƒrÕ| ¡ n dd„ }| ¡ |¡ |jrít ƒ sít ƒ sítdƒ‚|d ur÷|j j | _ n|j d u �rtdƒ‚|j | _ | j �r|j j | _ |j j!| _!|d u �r tdƒ‚|j"d u �r+t# $dt%¡ d}n|j"}|j&d u �r=t# $dt%¡ d}n|j&}||k �sPtd|› d|› d�ƒ‚|j'd u �rc| j �rct# $dt%¡ d}n|j'}|d u �r†t(|j!|j)| j d �}|j*�r‚d|_*t# $d t%¡ d| _+nd| _+|j,�r‘t-|ƒ || _"|j| _|j)| _)|j.d u�r¥|j.n|j!| _.|| _&|j/| _/|| _'|| _0|j1d v �rÍ|j2d k�rÍt# $d |j1› d!�t%¡ |j1d"k�r×td#ƒ‚|j3| _3|j2| _2|j1| _1|j4| _4t|j d$dƒ| _5t|j d%d&ƒ| _6| j5�r| j6d&k�rt# $d't%¡ |j1d(k�r|j7| _7t8d)d*„ ƒ| _9d|j:d+< t;ƒ <¡ �J |j=t>|j?d,�}|j=t@d-|i|j?d.�}|d u�rN|j=t>|j?d,�}|j=t@d-|i|j?d.�}|j=| jA|j?d,�}|d u�re|j=| jA|j?d,�}W d ƒ n 1 �spw Y tBƒ jC|||||||| || |
|
|||
|
|
d/� d| _Dt| jEd0ƒ�r—| jE F| jG¡ t| d1ƒ�s¡tHd2ƒ‚d S )3NzRYou passed model_kwargs to the CPOTrainer. But your model is already instantiated.Ú torch_dtyper€ znInvalid `torch_dtype` passed to the CPOConfig. Expected a string with either `torch.dtype` or 'auto', but got Ú.FzvPEFT is not installed and you passed a `peft_config` in the trainer's kwargs, please install it to use the PEFT modelsÚis_loaded_in_8bitÚis_loaded_in_4bitrö Úuse_gradient_checkpointingÚenable_input_require_gradsc S ó | d¡ d S ©NT©Úrequires_grad_©ÚmoduleÚinputÚoutputrd rd re Úmake_inputs_require_gradþ ó z=_UnslothCPOTrainer.__init__.<locals>.make_inputs_require_gradTc S rG rH rI rK rd rd re rO rP z‚`generate_during_eval=True` requires Weights and Biases or Comet to be installed. Please install `wandb` or `comet-ml` to resolve.zMWhen no model is provided, you need to pass the parameter is_encoder_decoder.z=processing_class must be specified to tokenize a CPO dataset.z�`max_length` is not set in the CPOConfig's init it will default to `512` by default, but you should do it yourself in the future.r‰ zˆ`max_prompt_length` is not set in the CPOConfig's init it will default to `128` by default, but you should do it yourself in the future.é€ zmax_prompt_length (z+) should be strictly less than max_length (z).z¼When using an encoder decoder architecture, you should set `max_completion_length` in the CPOConfig's init it will default to `128` by default, but you should do it yourself in the future.)Ú pad_token_idr r z²When using DPODataCollatorWithPadding, you should set `remove_unused_columns=False` in your TrainingArguments we have set it for you, but you should do it yourself in the future.)ÚhingeÚipor zYou are using the z™ loss type that does not support label smoothing. The `label_smoothing` parameter will be ignored. Set `label_smoothing` to `0.0` to remove this warning.Úkto_pairzKSupport for kto_pair has been removed in CPOTrainer. Please use KTOTrainer.Úoutput_router_logitsÚrouter_aux_loss_coefr‚ a- You set `output_router_logits` to `True` in the model config, but `router_aux_loss_coef` is set to `0.0`, meaning the auxiliary loss will not be used. Either set `router_aux_loss_coef` to a value greater than `0.0`, or set `output_router_logits` to `False` if you don't want to use the auxiliary loss.Úsimpoc S s t tƒS ©N)r# Úlistrd rd rd re Ú<lambda>‰ s z-_UnslothCPOTrainer.__init__.<locals>.<lambda>Úestimate_tokens)Únum_procÚ tokenizer)Ú fn_kwargsr] ) r5 r6 r7 r8 r9 r: r; r@ r< r= r> Úadd_model_tagsÚ acceleratorzXYour `Trainer` does not have an `accelerator` object. Consider upgrading `transformers`.)Ir Ú
|
|||
|
|
isinstanceÚstrÚ
|
|||
|
|
ValueErrorÚgetÚgetattrr: Údtyper Úfrom_pretrainedÚ _peft_has_been_casted_to_bf16r) r Úmerge_and_unloadÚhasattrrZ r' Ú signaturer6 Ú
|
|||
|
|
parametersrõ rö rF Úget_input_embeddingsÚregister_forward_hookrÀ r5 r r+ r( Úconfigr Údecoder_start_token_idrR r r; ÚwarnÚ UserWarningr r r r rÓ Úuse_dpo_data_collatorr r$ r r r: r r r r Úaux_loss_enabledÚ
|
|||
|
|
aux_loss_coefr r# Ú_stored_metricsÚwarnings_issuedr Úmain_process_firstÚmapr. r r- Ú tokenize_rowr$ r% Úmodel_accepts_loss_kwargsr5 r` Ú
|
|||
|
|
_tag_namesÚAttributeError)r& r5 r6 r7 r8 r9 r: r; r< r= r> r? r@ r rA Ú_support_gc_kwargsÚprepare_model_kwargsrO r r r r( rd re r% ¹ s\
|
|||
|
|
|
|||
|
|
|
|||
|
|
|
|||
|
|
|
|||
|
|
ÿ
|
|||
|
|
ÿ
|
|||
|
|
ÿ
|
|||
|
|
ÿþ
|
|||
|
|
|
|||
|
|
|
|||
|
|
|
|||
|
|
€
|
|||
|
|
|
|||
|
|
ÿ
|
|||
|
|
|
|||
|
|
|
|||
|
|
ý ý
|
|||
|
|
ÿý
|
|||
|
|
ýý ý û
|
|||
|
|
ÿ
|
|||
|
|
ý
|
|||
|
|
€ ïõ ÿÿz_UnslothCPOTrainer.__init__c C s | j || dd�}| j |dd�d }|d t|ƒd… }|d t|ƒd… }t ||g¡}t |d ¡}t|ƒt|ƒkr@tdƒ‚t|ƒ} ||d d| … krR| d8 } |d d| … }|d d| … }
|
|||
|
|
t|ƒt|
|
|||
|
|
ƒkrntdƒ‚|d | d… }|d | d… }t||
|
|||
|
|
||d �S )
|
|||
|
|
a
|
|||
|
|
Llama tokenizer does satisfy `enc(a + b) = enc(a) + enc(b)`. It does ensure `enc(a + b) = enc(a) + enc(a +
|
|||
|
|
b)[len(enc(a)):]`. Reference:
|
|||
|
|
https://github.com/EleutherAI/lm-evaluation-harness/pull/531#issuecomment-1595586257
|
|||
|
|
F©Úadd_special_tokensÚ input_idsNÚattention_maskzBPrompt input ids and answer input ids should have the same length.rN z@Prompt input ids and attention mask should have the same length.)Úprompt_input_idsÚprompt_attention_maskrƒ r„ )r: Úlenr0 Ú concatenateÚarrayrd Údict) r& ÚpromptÚanswerÚfull_tokenizedr… Úanswer_input_idsÚanswer_attention_maskÚfull_concat_input_idsÚfull_input_idsÚ response_token_ids_start_idxr† rd rd re Úbuild_tokenized_answerÅ s. üz)_UnslothCPOTrainer.build_tokenized_answerÚreturnc s i }|d }|d }|d }| j �s±t|tƒs tdt|ƒ› �ƒ‚| j|dd�}dd„ | ¡ D ƒ}t|tƒs>td t|ƒ› �ƒ‚| ||¡‰ t|tƒsRtd
|
|||
|
|
t|ƒ› �ƒ‚| ||¡‰t|d ƒ}tˆ d ƒ} tˆd ƒ}
|
|||
|
|
t | |
|
|||
|
|
ƒ}| ¡ D ] \} } | d |… || < qst
|
|||
|
|
d
|
|||
|
|
d„ t ˆ d ˆd ƒD ƒƒ}
|
|||
|
|
t | |
|
|||
|
|
ƒ}|
|
|||
|
|
dksž|dkr¢tdƒ‚t
|
|||
|
|
| jj||| ˆ |
|
|||
|
|
ˆƒ\}‰ ‰t| jjˆ ˆƒ\‰ ‰ttˆ d ƒtˆd ƒƒ}ˆ ˆ|fD ]D}t|d ƒ| | jk�r| jdkrñdD ]
|
|||
|
|
} || d | j… || < qâqÍ| jdk�r dD ]} || | j d … || < qùqÍtd| j› �ƒ‚q͈ ˆfD ]#}t|d ƒ| | jk�r8dD ]} || d | j| j … || < �q&�q‡ fdd„dD ƒ}‡fdd„dD ƒ}|d d d … |d< | jgtˆ d ƒ |d d tˆ d ƒ…< |d d d … |d< | jgtˆd ƒ |d d tˆd ƒ…< |||dœ ¡ D ] \} }| ¡ D ]\}}|dk�r£�q˜||| › |› �< �q˜�q�|S | j|d | jd d �‰ | j|d | jd d �‰| j|d | jd d �}ˆ d |d < ˆd |d < |d |d < |d |d!< |d u�r t|d"ƒ�r |jt |d ¡d#�|d$< |jt |d ¡d#�|d%< |S )&a. Tokenize a single row from a CPO specific dataset.
|
|||
|
|
|
|||
|
|
At this stage, we don't convert to PyTorch tensors yet; we just handle the truncation in case the prompt +
|
|||
|
|
chosen or prompt + rejected responses is/are too long. First we truncate the prompt; if we're still too long,
|
|||
|
|
we truncate the chosen/rejected.
|
|||
|
|
|
|||
|
|
We also create the labels for the chosen/rejected responses, which are of length equal to the sum of the length
|
|||
|
|
of the prompt and the chosen/rejected response, with label_pad_token_id for the prompt tokens.
|
|||
|
|
r‹ ÚchosenÚrejectedz prompt should be an str but got Fr� c S s i | ]
|
|||
|
|
\}}d |› �|“qS ©Úprompt_rd ©Ú.0ÚkÚvrd rd re Ú
|
|||
|
|
<dictcomp> ó z3_UnslothCPOTrainer.tokenize_row.<locals>.<dictcomp>z chosen should be an str but got z"rejected should be an str but got r… Nc S s g | ]\}}||k‘qS rd rd )rš ÚaÚbrd rd re Ú
|
|||
|
|
<listcomp>& s z3_UnslothCPOTrainer.tokenize_row.<locals>.<listcomp>rN zdChosen and rejected prompt_input_ids might only differ on the last token due to tokenizer merge ops.rƒ Ú
|
|||
|
|
keep_start)r… r† r� zUnknown truncation mode: )rƒ r„ c ó$ i | ]}|ˆ d |› � ˆ | “qS r— rd ©rš r› )Ú
|
|||
|
|
chosen_tokensrd re r� T ó ÿc r£ r— rd r¤ )Úrejected_tokensrd re r� W r¦ Úlabels)Úchosen_Ú rejected_r� Útoken_type_idsT)Ú
|
|||
|
|
truncationr r‚ Ú
|
|||
|
|
chosen_labelsÚrejected_labelsr„ r† Ú%prepare_decoder_input_ids_from_labels)r¨ Úrejected_decoder_input_idsÚchosen_decoder_input_ids)r rb rc rd Útyper: Úitemsr“ r‡ ÚminÚsumrR Úabsr Ú bos_token_idr! Ú eos_token_idr# r r r r r rk r¯ r: Útensor)r& Úfeaturer5 Úbatchr‹ r• r– Ú
|
|||
|
|
prompt_tokensÚprompt_len_input_idsÚchosen_prompt_len_input_idsÚ rejected_prompt_len_input_idsr› rœ Únum_diff_tokensÚ num_diff_lenÚlonger_response_lengthÚ
|
|||
|
|
answer_tokensÚchosen_sequence_tokensÚrejected_sequence_tokensÚtoksÚtype_keyÚtokensrd )r¥ r§ re r{ ö sØ
|
|||
|
|
|
|||
|
|
|
|||
|
|
|
|||
|
|
|
|||
|
|
ÿ ÿ
|
|||
|
|
ù
|
|||
|
|
ÿ
|
|||
|
|
ÿ ÿø €
|
|||
|
|
ÿ
|
|||
|
|
ÿÿ
|
|||
|
|
þÿ
|
|||
|
|
þý
|
|||
|
|
ü
|
|||
|
|
ý é
|
|||
|
|
ÿ
|
|||
|
|
ÿ
|
|||
|
|
ÿ
|
|||
|
|
ÿ
|
|||
|
|
ÿz _UnslothCPOTrainer.tokenize_rowFrŒ r r» r r r Údevicec
|
|||
|
|
C sŒ i }|rt | d jd | d jd ƒ}nt | d jd | d jd ƒ}| D ]8}| d¡r]t| | tjƒr]d|v s:|r=|}n| d¡rE|}n| d ¡rLd
|
|||
|
|
}| dd ¡} t| | ||d �|| < q%| D ]E}| d
|
|||
|
|
¡r¥t| | tjƒr¥d|v su|rx|}n| d¡r€|}n| d ¡r‡d
|
|||
|
|
}| d
|
|||
|
|
d ¡} tj || t| | ||d �fd
|
|||
|
|
d�j
|
|||
|
|
|d�|| < q`|rÄ| d dd¡j
|
|||
|
|
|d�|d< | d dd¡j
|
|||
|
|
|d�|d< |S )aç Concatenate the chosen and rejected inputs into a single tensor.
|
|||
|
|
|
|||
|
|
Args:
|
|||
|
|
batch:
|
|||
|
|
A batch of data. Must contain the keys 'chosen_input_ids' and 'rejected_input_ids', which are tensors
|
|||
|
|
of shape (batch_size, sequence_length).
|
|||
|
|
is_encoder_decoder:
|
|||
|
|
Whether the model is an encoder-decoder model.
|
|||
|
|
label_pad_token_id:
|
|||
|
|
The label pad token id.
|
|||
|
|
padding_value:
|
|||
|
|
The padding value to use for the concatenated inputs_ids.
|
|||
|
|
device:
|
|||
|
|
The device for the concatenated inputs.
|
|||
|
|
|
|||
|
|
Returns:
|
|||
|
|
A dictionary containing the concatenated inputs under the key 'concatenated_input_ids'.
|
|||
|
|
r rN r® Úchosen_input_idsÚrejected_input_idsr• r¨ Ú
|
|||
|
|
_input_idsÚ_attention_maskr Ú concatenated)Ú pad_valuer– rM ©rÉ r… ro Úconcatenated_input_idsr† Úconcatenated_attention_mask) r# rQ Ú
|
|||
|
|
startswithrb r: r ÚendswithÚreplacer3 ÚcatrS Úrepeat)
|
|||
|
|
r» r r r rÉ Úconcatenated_batchr r› rÏ Úconcatenated_keyrd rd re Úconcatenated_inputs‡ sL
|
|||
|
|
|
|||
|
|
€
|
|||
|
|
|
|||
|
|
þû
|
|||
|
|
ú€ ÿz&_UnslothCPOTrainer.concatenated_inputsÚpolicy_chosen_logpsÚpolicy_rejected_logpsc C s4 || | jj¡}| jdkr3| j| j }|| }t | j| ¡ d| j t | j | ¡| j }nJ| jdkrSt | j| ¡ d| j t | j | ¡| j }n*| jdkrct
|
|||
|
|
d| j| ¡}n| jdkrt|dd| j d }n t d| j› d�ƒ‚| j| | jj¡ ¡ }| j| | jj¡ ¡ }|||fS ) aµ Compute the CPO loss for a batch of policy and reference model log probabilities.
|
|||
|
|
|
|||
|
|
Args:
|
|||
|
|
policy_chosen_logps:
|
|||
|
|
Log probabilities of the policy model for the chosen responses. Shape: (batch_size,)
|
|||
|
|
policy_rejected_logps:
|
|||
|
|
Log probabilities of the policy model for the rejected responses. Shape: (batch_size,)
|
|||
|
|
|
|||
|
|
Returns:
|
|||
|
|
A tuple of three tensors: (losses, chosen_rewards, rejected_rewards). The losses tensor contains the CPO
|
|||
|
|
loss for each example in the batch. The chosen_rewards and rejected_rewards tensors contain the rewards for
|
|||
|
|
the chosen and rejected responses, respectively.
|
|||
|
|
rX rN rŠ rS rT ro zUnknown loss type: z7. Should be one of ['sigmoid', 'hinge', 'ipo', 'simpo'])
|
|||
|
|
rS ra rÉ r r r r Ú
|
|||
|
|
logsigmoidr r: Úrelurd Údetach)r& rÛ rÜ r[ Úgamma_logratiosÚlossesÚchosen_rewardsÚrejected_rewardsrd rd re Úcpo_lossË s.
|
|||
|
|
ÿÿ
|
|||
|
|
ÿÿ
|
|||
|
|
|
|||
|
|
ÿ
|
|||
|
|
z_UnslothCPOTrainer.cpo_lossr[ r¨ Úaverage_log_probc C sš | j dd… |j krtdƒ‚|s)|dd…dd…f ¡ }| dd…dd…dd…f } ||k}d|||k< t| |ƒ}|rF|| d¡| d¡ S || d¡S )aŽ Compute the log probabilities of the given labels under the given logits.
|
|||
|
|
|
|||
|
|
Args:
|
|||
|
|
logits: Logits of the model (unnormalized). Shape: (batch_size, sequence_length, vocab_size)
|
|||
|
|
labels:
|
|||
|
|
Labels for which to compute the log probabilities. Label tokens with a value of label_pad_token_id are
|
|||
|
|
ignored. Shape: (batch_size, sequence_length)
|
|||
|
|
average_log_prob:
|
|||
|
|
If True, return the average log probability per (non-masked) token. Otherwise, return the sum of the
|
|||
|
|
log probabilities of the (non-masked) tokens.
|
|||
|
|
label_pad_token_id: The label pad token id.
|
|||
|
|
is_encoder_decoder: Whether the model is an encoder-decoder model.
|
|||
|
|
|
|||
|
|
Returns:
|
|||
|
|
A tensor of shape (batch_size,) containing the average/sum log probabilities of the given labels under the
|
|||
|
|
given logits.
|
|||
|
|
NrH zKLogits (batch and sequence length dim) and labels must have the same shape.rN r )rQ rd Úcloner8 rµ )r[ r¨ rå r r Ú loss_maskrc rd rd re Úget_batch_logps s
|
|||
|
|
z"_UnslothCPOTrainer.get_batch_logpsc sD ˆ j |ˆ jˆ jˆ jˆ jjd�}|d jd }ˆ jr"dˆ |d ¡ini }ˆ jr+d|d< ||d f|d d
|
|||
|
|
d œ|¤Ž}|j }‡ fd d
|
|||
|
|
„}|d
|
|||
|
|
¡ } ˆ j dkrYt
|
|||
|
|
d¡ ˆ jj¡}
|
|||
|
|
n
|
|||
|
|
||d|… | d|… ƒ}
|
|||
|
|
ˆ j||d ˆ jdv ˆ jˆ jd�} | d|… } | |d… }
|
|||
|
|
|d|… }||d… }ˆ jr›| |
|
|||
|
|
|||
|
|||
|
|
|jfS | |
|
|||
|
|
|||
|
|||
|
|
fS )zÆRun the given model on the given batch of inputs, concatenating the chosen and rejected inputs together.
|
|||
|
|
|
|||
|
|
We do this to avoid doing two forward passes, because it's faster for FSDP.
|
|||
|
|
)r r r rÉ r r Údecoder_input_idsÚconcatenated_labelsTrV rÑ rÒ F)r„ Ú use_cachec sr ˆ j s| dd d…d d …f ¡ } |ddd …f ¡ }t ¡ }| d| jd ¡} | d¡}| | j¡}|| |ƒ}|S )N.rH rN )r Ú
|
|||
|
|
contiguousr/ ÚCrossEntropyLossÚviewrQ rS rÉ )r[ r¨ Úloss_fctÚloss©r& rd re Úcross_entropy_lossN s
|
|||
|
|
|
|||
|
|
zC_UnslothCPOTrainer.concatenated_forward.<locals>.cross_entropy_lossr‚ N)rT rX )rå r r )rÚ r r r ra rÉ rQ Ú _shift_rightru r[ ræ r r: r¹ rS rè r Úaux_loss)r& r5 r» rØ Ú
|
|||
|
|
len_chosenÚ model_kwargsÚoutputsÚ
|
|||
|
|
all_logitsrò r¨ Únll_lossÚ all_logpsÚ chosen_logpsÚrejected_logpsÚ
|
|||
|
|
chosen_logitsÚrejected_logitsrd rñ re Úconcatenated_forward+ sX ûýÿûÿýü
|
|||
|
|
û z'_UnslothCPOTrainer.concatenated_forwardÚtrainÚ
|
|||
|
|
train_eval)r Úevalc C s¾ i }| ||¡}|dd… \}}}} }
|
|||
|
|
| jr|d } | ||¡\} }
|
|||
|
|
}| ¡ | j|
|
|||
|
|
}|
|
|||
|
|
|k ¡ }|dkr8dnd}| j |
|
|||
|
|
¡ ¡ ¡ ||› d�< | j |¡ ¡ ¡ ||› d�< | j |¡ ¡ ¡ ||› d�< | j |
|
|||
|
|
| ¡ ¡ ¡ ||› d �< | j |¡ ¡ ¡ ¡ ||› d
|
|||
|
|
�< | j |¡ ¡ ¡ ¡ ||› d �< | j | ¡ ¡ ¡ ¡ ¡ ||› d �< | j | ¡ ¡ ¡ ¡ ¡ ||› d
|
|||
|
|
�< | j |
|
|||
|
|
¡ ¡ ¡ ¡ ||› d�< | jrÛ|| j
|
|||
|
|
| 7 }||fS )zWCompute the CPO loss and other metrics for the given batch of inputs for train or test.Né r Úeval_r� zrewards/chosenzrewards/rejectedzrewards/accuracieszrewards/marginszlogps/rejectedz logps/chosenzlogits/rejectedz
|
|||
|
|
logits/chosenrù ) rÿ ru rä Úmeanr Úfloatra Úgather_for_metricsÚitemrß rv )r& r5 r» r ÚmetricsÚforward_outputrÛ rÜ Úpolicy_chosen_logitsÚpolicy_rejected_logitsÚpolicy_nll_lossrô rá râ rã rð Úreward_accuraciesÚprefixrd rd re Úget_batch_loss_metricsv sF
|
|||
|
|
ú
|
|||
|
|
þ ÿ ÿ ÿ ÿ ÿ"z)_UnslothCPOTrainer.get_batch_loss_metricsÚinputsc C sp | j r
|
|||
|
|
t| jjjƒntƒ }|� | j||dd�\}}W d ƒ n1 s$w Y | j|dd� |r6||fS |S )Nr ©r )ri r" ra rÉ r² r1 r Ú
|
|||
|
|
store_metrics)r& r5 r Úreturn_outputsÚnum_items_in_batchÚ compute_loss_context_managerrð r rd rd re Ú compute_loss¬ s ÿ ÿz _UnslothCPOTrainer.compute_lossc C s† | j r
|
|||
|
|
t| jjjƒntƒ }|� |j|d |d | jd| jj d�}W d ƒ n1 s+w Y t
|
|||
|
|
|| j| jj ƒ}| jj |dd�}|S )zRGenerate samples from the model and reference model for the given batch of inputs.r… r† T)rƒ r„ r Ú do_samplerR N)Úskip_special_tokens) ri r" ra rÉ r² r1 Úgenerater r: rR r3 Ú batch_decode)r& r5 r» Úgenerate_context_managerÚ
|
|||
|
|
policy_outputÚpolicy_output_decodedrd rd re Úgenerate_from_modelÁ s ÿû ÿ z&_UnslothCPOTrainer.generate_from_modelr– Ú ignore_keysc s ˆ d u rt |dƒrt|jdg ƒ‰ ng ‰ | jr t| jjjƒntƒ }t
|
|||
|
|
¡ �$ |� | j ||dd�\}}W d ƒ n1 s<w Y W d ƒ n1 sKw Y | j |dd� |r`|
|
|||
|
|
¡ d d fS |d |d dœ}‡ fdd „| ¡ D ƒ} t j| | jjd
|
|||
|
|
�} t j| jd | jjd
|
|||
|
|
�}
|
|||
|
|
|
|
|||
|
|
¡ | |
|
|||
|
|
fS ) Nrp Úkeys_to_ignore_at_inferencer r úeval_logits/chosenúeval_logits/rejected)r" r# c s g | ]
|
|||
|
|
\}}|ˆ vr|‘qS rd rd r™ ©r rd re r¡ ÷ rž z6_UnslothCPOTrainer.prediction_step.<locals>.<listcomp>rÐ r )rk rf rp ri r" ra rÉ r² r1 r: Úno_gradr r rß r³ r¹ ÚzerosrQ ) r& r5 r r– r Úprediction_context_managerrð r Ú logits_dictr[ r¨ rd r$ re Úprediction_stepØ s*
|
|||
|
|
ÿ ÿ€ þz"_UnslothCPOTrainer.prediction_stepr c C s* | ¡ D ]\}}| j| | |¡ qd S rY )r³ rw rY )r& r r ÚkeyÚvaluerd rd re r ý s ÿz _UnslothCPOTrainer.store_metricsr Ú
|
|||
|
|
dataloaderÚ descriptionÚmetric_key_prefixc
|
|||
|
|
sÌ | j rZt|jƒ}tjt|ƒ| jjd�}|j |¡}| |¡} |
|
|||
|
|
| ¡} | | j | ¡}
|
|||
|
|
t
|
|||
|
|
jddgdd„ t| d |
|
|||
|
|
ƒD ƒd�} d| jjv rNt d tj| d
|
|||
|
|
�i¡ d | jjv rZtd | d
|
|||
|
|
� tƒ |||||¡} | S )zÞ
|
|||
|
|
Overriding built-in evaluation loop to store metrics for each batch. Prediction/evaluation loop, shared by
|
|||
|
|
`Trainer.evaluate()` and `Trainer.predict()`.
|
|||
|
|
|
|||
|
|
Works both with or without labels.
|
|||
|
|
)r› ÚPromptÚPolicyc S s$ g | ]\}}||t |ƒd … g‘qS rY )r‡ )rš r‹ Úpolrd rd re r¡ r¦ z6_UnslothCPOTrainer.evaluation_loop.<locals>.<listcomp>r‹ )ÚcolumnsÚdataÚwandbÚgame_log)r3 Úcomet_mlz game_log.csv)ÚnameÚtable)r r‡ Údatasetr7 ÚsampleÚranger6 Úeval_batch_sizeÚselectr7 Ú_prepare_inputsr r5 r4 Ú DataFramerR rå r4 ÚlogÚTabler, r$ Úevaluation_loop)
|
|||
|
|
r& r, r- r– r r. Ú num_samplesÚrandom_indicesÚrandom_batch_datasetÚ random_batchr r8 Úinitial_outputr( rd re rB s0
|
|||
|
|
|
|||
|
|
|
|||
|
|
ÿþ þ
|
|||
|
|
ÿz"_UnslothCPOTrainer.evaluation_loopÚlogsÚ
|
|||
|
|
start_timec sT d|v rdnd}| j | ¡ D ]\}}t |¡ ¡ ¡ ||< q| j |= tƒ ||¡S )a1
|
|||
|
|
Log `logs` on the various objects watching training, including stored metrics.
|
|||
|
|
|
|||
|
|
Args:
|
|||
|
|
logs (`dict[str, float]`):
|
|||
|
|
The values to log.
|
|||
|
|
start_time (`float` or `None`, *optional*, defaults to `None`):
|
|||
|
|
Start time of the training.
|
|||
|
|
rð r r )rw r³ r: r¹ r r r$ r@ )r& rH rI r r* r r( rd re r@ 3 s
|
|||
|
|
z_UnslothCPOTrainer.logc C s´ | j d u r tdƒ‚t|ƒr+t |jd d… d | j ¡}tj||dd d…f gdd�}n| |j¡}|dd d…f ¡ |ddd …f< | j |d< | j d u rOtdƒ‚|
|
|||
|
|
|d k| j ¡ |S )
|
|||
|
|
Nz]model.config.decoder_start_token_id has to be defined. It is usually set to the pad_token_id.rH )rN .rM rN ).r z,model.config.pad_token_id has to be defined.rŒ ) rq rd r* r: ÚfullrQ rÖ Ú new_zerosræ rR Ú masked_fill_)r& rƒ Úshifted_input_idsrd rd re ró E s
|
|||
|
|
ÿ
|
|||
|
|
|
|||
|
|
z _UnslothCPOTrainer._shift_rightc sL | j jd u rt| j jƒj}n | j j d¡d }| j|d� tƒ ||¡ d S )Nú/rH )Ú
|
|||
|
|
model_name) r6 rï r r� r7 ÚsplitÚcreate_model_cardr$ Ú_save_checkpoint)r& r5 ÚtrialrO r( rd re rR ] s
|
|||
|
|
z#_UnslothCPOTrainer._save_checkpointrO Ú dataset_nameÚtagsc
|
|||
|
|
C sê | ¡ sdS t| jjdƒr tj | jjj¡s | jjj}nd}|du r&tƒ }n
|
|||
|
|
t |t
|
|||
|
|
ƒr/|h}nt|ƒ}t| jjdƒr?| d¡ | | j
|
|||
|
|
¡ t d¡}t||| j||tƒ r]tjdur]tjjndtƒ d|ddd � }| tj | jjd
|
|||
|
|
¡¡ dS ) aî
|
|||
|
|
Creates a draft of a model card using the information available to the `Trainer`.
|
|||
|
|
|
|||
|
|
Args:
|
|||
|
|
model_name (`str` or `None`, *optional*, defaults to `None`):
|