447 lines
90 KiB
Plaintext
447 lines
90 KiB
Plaintext
|
§
|
|||
|
|
3$�hÉW ã ó¤ — d Z ddlmZ ddlZddlmZ ddlmZ ddlmZm Z m
|
|||
|
|
Z
|
|||
|
|
m Z m Z m
|
|||
|
|
Z
|
|||
|
|
mZmZ ddlmZmZmZmZmZmZmZmZmZmZmZmZmZmZm Z m Z m
|
|||
|
|
Z
|
|||
|
|
m Z m Z m Z m!Z!m"Z"m#Z#m$Z$m%Z%m&Z&m'Z'm(Z(m)Z)m Z m*Z*m+Z+m,Z,m-Z-m.Z.m/Z/m0Z0m1Z1m2Z2m3Z3m4Z4m5Z5m6Z6m7Z7m8Z8m9Z9m:Z:m;Z;m<Z<m=Z=m>Z>mZm?Z?m@Z@mAZAmBZBmCZCmDZDmEZEmFZFmGZGmHZHmIZImZmJZJmKZKmZm
|
|||
|
|
Z
|
|||
|
|
m Z m!Z!m'Z'm7Z7m=Z=mAZAmZ ddlAZAddlT ddlLmMZMmNZN dd lOmPZP ddlZddlQZ?dd
|
|||
|
|
lRm@Z@ ddlmZ dd lSmTZTmUZV d d
|
|||
|
|
d d
|
|||
|
|
d
|
|||
|
|
dœZW ejX d d eW¬¦ « d„ ¦ « ZYeM G d„ de¦ « ¦ « ZZ G d„ de'¦ « Z[ G d„ de[¦ « Z\dS )z8
|
|||
|
|
2025.8.4
|
|||
|
|
2025.8.5
|
|||
|
|
4.55.1
|
|||
|
|
0.21.0
|
|||
|
|
__UNSLOTH_VERSIONING__
|
|||
|
|
é )ÚTensorN)Ú
|
|||
|
|
functional)ÚAnyÚListÚOptionalÚTupleÚUnionÚDictÚSetÚCallable)Kr ÚAutoModelForCausalLMÚ BCOConfigÚ
|
|||
|
|
BCOTrainerÚBaseImageProcessorÚCLF_NAMEr ÚDPODataCollatorWithPaddingÚ DataCollatorÚ
|
|||
|
|
DataLoaderÚDatasetÚEvalLoopOutputÚFÚFeatureExtractionMixinÚLiteralÚLogisticRegressionr Ú PartialStateÚPathÚ PeftModelÚPreTrainedModelÚPreTrainedTokenizerBaseÚProcessorMixinÚ RUNNING_NAMEÚRunningMomentsÚSequentialSamplerÚTrainerÚTrainerCallbackÚTrainingArgumentsr Ú_process_tokensÚ _tokenizeÚautocastÚcontextmanagerÚcreate_reference_modelÚ defaultdictÚdisable_dropout_in_modelÚgenerate_model_cardÚget_comet_experiment_urlÚ
|
|||
|
|
has_lengthÚinspectÚis_comet_availableÚis_joblib_availableÚis_peft_availableÚis_sklearn_availableÚis_wandb_availableÚ
|
|||
|
|
itemgetterÚjoblibÚ log_table_to_comet_experimentÚloggerÚmaybe_apply_chat_templateÚnnÚnpÚ nullcontextÚosÚ
|
|||
|
|
pad_to_lengthÚpdÚpeft_module_casting_to_bf16Úprepare_deepspeedÚ prepare_model_for_kbit_trainingÚrandomÚselective_log_softmaxÚtextwrapÚtorchÚtqdmÚwarningsr r r r r$ r4 r: r? rH )Ú*)Ú dataclassÚfield)ÚVersion)r> )ÚDataCollatorForSeq2SeqÚ DataCollatorForLanguageModelingTF)Úepilogue_fusionÚ max_autotuneÚ
|
|||
|
|
shape_paddingz
|
|||
|
|
trace.enabledztriton.cudagraphs)ÚdynamicÚ fullgraphÚoptionsc ó’ — t j | d| j d ¦ « dd¬¦ « }t j | d¦ « dd¬¦ « }g }t ||¦ « D ]‘\ }}| t j ¦ « }t j |d| d¦ « ¬¦ « d¦ « }t j
|
|||
|
|
|d¬¦ « }||z
|
|||
|
|
} | | ¦ « Œ’ t j |¦ « }| | j d | j d f¦ « }|S )Néÿÿÿÿé r )ÚchunksÚdim)r[ Úindex©r[ é )
|
|||
|
|
rH ÚchunkÚreshapeÚshapeÚzipÚtoÚfloat32ÚgatherÚ unsqueezeÚsqueezeÚ logsumexpÚappendÚconcat)
|
|||
|
|
Úlogitsr\ Úchunked_logitsÚ
|
|||
|
|
chunked_indexÚall_per_token_logpsÚ chunk_logitsÚ chunk_indexÚselected_logitsÚlogsumexp_valuesÚper_token_logpss
|
|||
|
|
ú]/workspace/Fine-tuning/DS-LLM-TEMPLATE-FINETUNING/unsloth_compiled_cache/UnslothBCOTrainer.pyÚ chunked_selective_log_softmaxru " s5 € õ ”[ § ¢ °°F´LÀÔ4DÑ!EÔ!EÐPQÐYZÐ[Ñ[Ô[€NÝ”[ § ¢ ¨rÑ!2Ô!2¸QÀaÐHÑHÔH€MØ Ðå%(¨ ¸ Ñ%GÔ%Gð 4ð 4Ñ!ˆ �kØ#—’¥u¤}Ñ5Ô5ˆ Ý œ, |¸2À{×G\ÒG\Ð]_ÑG`ÔG`ÐaÑaÔa×iÒiÐjlÑmÔmˆÝ œ?¨<¸rÐBÑBÔBÐØ)Ð,<Ñ<ˆØ×"Ò" ?Ñ3Ô3Ð3Ð3ØÝ œ,Ð':Ñ;Ô;ÐØ-×5Ò5°v´|ÀA´ÈÌ ÐUVÌÐ6XÑYÔYÐØ Ð ó c ó´ ‡ — e Zd ZU dZ edddi¬¦ « Zee ed< edddi¬¦ « Z ee
|
|||
|
|
ed < d0ˆ fd/„ Z ˆ xZ S )1ÚUnslothBCOConfiguù
|
|||
|
|
|
|||
|
|
Configuration class for the [`BCOTrainer`].
|
|||
|
|
|
|||
|
|
This class includes only the parameters that are specific to BCO training. For a full list of training arguments,
|
|||
|
|
please refer to the [`~transformers.TrainingArguments`] documentation. Note that default values in this class may
|
|||
|
|
differ from those in [`~transformers.TrainingArguments`].
|
|||
|
|
|
|||
|
|
Using [`~transformers.HfArgumentParser`] we can turn this class into
|
|||
|
|
[argparse](https://docs.python.org/3/library/argparse#module-argparse) arguments that can be specified on the
|
|||
|
|
command line.
|
|||
|
|
|
|||
|
|
Parameters:
|
|||
|
|
max_length (`int` or `None`, *optional*, defaults to `1024`):
|
|||
|
|
Maximum length of the sequences (prompt + completion) in the batch. This argument is required if you want
|
|||
|
|
to use the default data collator.
|
|||
|
|
max_prompt_length (`int` or `None`, *optional*, defaults to `512`):
|
|||
|
|
Maximum length of the prompt. This argument is required if you want to use the default data collator.
|
|||
|
|
max_completion_length (`int` or `None`, *optional*, defaults to `None`):
|
|||
|
|
Maximum length of the completion. This argument is required if you want to use the default data collator
|
|||
|
|
and your model is an encoder-decoder.
|
|||
|
|
beta (`float`, *optional*, defaults to `0.1`):
|
|||
|
|
Parameter controlling the deviation from the reference model. Higher β means less deviation from the
|
|||
|
|
reference model.
|
|||
|
|
label_pad_token_id (`int`, *optional*, defaults to `-100`):
|
|||
|
|
Label pad token id. This argument is required if you want to use the default data collator.
|
|||
|
|
padding_value (`int` or `None`, *optional*, defaults to `None`):
|
|||
|
|
Padding value to use. If `None`, the padding value of the tokenizer is used.
|
|||
|
|
truncation_mode (`str`, *optional*, defaults to `"keep_end"`):
|
|||
|
|
Truncation mode to use when the prompt is too long. Possible values are `"keep_end"` or `"keep_start"`.
|
|||
|
|
This argument is required if you want to use the default data collator.
|
|||
|
|
disable_dropout (`bool`, *optional*, defaults to `True`):
|
|||
|
|
Whether to disable dropout in the model and reference model.
|
|||
|
|
generate_during_eval (`bool`, *optional*, defaults to `False`):
|
|||
|
|
If `True`, generates and logs completions from both the model and the reference model to W&B or Comet
|
|||
|
|
during evaluation.
|
|||
|
|
is_encoder_decoder (`bool` or `None`, *optional*, defaults to `None`):
|
|||
|
|
When using the `model_init` argument (callable) to instantiate the model instead of the `model` argument,
|
|||
|
|
you need to specify if the model returned by the callable is an encoder-decoder model.
|
|||
|
|
precompute_ref_log_probs (`bool`, *optional*, defaults to `False`):
|
|||
|
|
Whether to precompute reference model log probabilities for training and evaluation datasets. This is
|
|||
|
|
useful when training without the reference model to reduce the total GPU memory needed.
|
|||
|
|
model_init_kwargs (`dict[str, Any]` or `None`, *optional*, defaults to `None`):
|
|||
|
|
Keyword arguments to pass to `AutoModelForCausalLM.from_pretrained` when instantiating the model from a
|
|||
|
|
string.
|
|||
|
|
ref_model_init_kwargs (`dict[str, Any]` or `None`, *optional*, defaults to `None`):
|
|||
|
|
Keyword arguments to pass to `AutoModelForCausalLM.from_pretrained` when instantiating the reference model
|
|||
|
|
from a string.
|
|||
|
|
dataset_num_proc (`int` or `None`, *optional*, defaults to `None`):
|
|||
|
|
Number of processes to use for processing the dataset.
|
|||
|
|
prompt_sample_size (`int`, *optional*, defaults to `1024`):
|
|||
|
|
Number of prompts that are fed to density ratio classifier.
|
|||
|
|
min_density_ratio (`float`, *optional*, defaults to `0.5`):
|
|||
|
|
Minimum value of the density ratio. The estimated density ratio is clamped to this value.
|
|||
|
|
max_density_ratio (`float`, *optional*, defaults to `10.0`):
|
|||
|
|
Maximum value of the density ratio. The estimated density ratio is clamped to this value.
|
|||
|
|
|
|||
|
|
NÚhelpzvLLM SamplingParams)ÚdefaultÚmetadataÚvllm_sampling_paramsrX z8Chunk size to reduce memory usage. -1 is most efficient.Úunsloth_num_chunksFÚnorY é r éú ç-C ëâ6
|
|||
|
|
?ç{®Gáz„?çÍÌÌÌÌÌì?ç+‡ÙÎ÷ï?ç:Œ0âŽyE>ç ð?ç @Úlinearçš™™™™™¹?ÚpassiveÚwarningTÚstepsr^ éô éO
|
|||
|
|
ÚO1ÚautoÚ ç Ú
|
|||
|
|
adamw_8bitÚlengthÚ
|
|||
|
|
every_saveÚlasté é é éœÿÿÿÚkeep_endç à?ç $@c” óŠ •— |dk rt d|› d�¦ « ‚|dk rt d|› d�¦ « ‚|€|#dk r
|
|||
|
|
|$dk rd}d }#|Ž€!d
|
|||
|
|
d lm}• t |•¦ « d z d ¦ « }Ž t ¦ « j dži d
|
|||
|
|
|“d|“d|“d|“d|“d|“d|“d|“d| “d|
|
|||
|
|
“d| “d| “d|
|
|||
|
|
“d|“d|“d |“d |“d |“d |“d |“d!|“d"|“d#|“d$|“d%|“d&|“d'|“d(| “d)| “d*| “d+| “d,| “d-|!“d.|"“d/|#“d0|$“d1|%“d2|&“d3|'“d4|(“d5|)“d6|*“d7|+“d8|,“d9|-“d:|.“d;|/“d<|0“d=|1“d>|2“d?|3“d@|4“dA|5“dB|6“dC|7“dD|8“dE|9“dF|:“dG|;“dH|<“dI|=“dJ|>“dK|?“dL|@“dM|A“dN|B“dO|C“dP|D“dQ|E“dR|F“dS|G“dT|H“dU|I“dV|J“dW|K“dX|L“dY|M“dZ|N“d[|O“d\|P“d]|Q“d^|R“d_|S“d`|T“da|U“db|V“dc|W“dd|X“de|Y“df|Z“dg|[“dh|\“di|]“dj|^“dk|_“dl|`“dm|a“dn|b“do|c“dp|d“dq|e“dr|f“ds|g“dt|h“du|i“dv|j“dw|k“dx|l“dy|m“dz|n“d{|o“d||p“d}|q“d~|r“d|s“d€|t“d�|u“d‚|v“dƒ|w“d„|x“d…|y“d†|z“d‡|{“dˆ||“d‰|}“dŠ|~“d‹|“dŒ|€“d�|�“dŽ|‚“d�|ƒ“d�|„“d‘|…“d’|†“d“|‡“d”|ˆ“d•|‰“d–|Š“d—|‹“d˜|Œ“d™|�“dš|Ž“d›|�“dœ|�“d�|‘“|”¤Ž |’| _ |“| _ d S )ŸNgH¯¼šò×z>z Unsloth: Your learning rate of `zi` is too small and less than 1e-7! Consider increasing it, otherwise gradient updates will be close to 0!r^ za` is way too larger > 1! Consider decreasing it to 1e-1, otherwise gradient updates will explode!rŒ r� Ú unsloth_training_checkpointsr~ r )Ú cpu_countr Ú
|
|||
|
|
output_dirÚoverwrite_output_dirÚdo_trainÚdo_evalÚ
|
|||
|
|
do_predictÚ
|
|||
|
|
eval_strategyÚprediction_loss_onlyÚper_device_train_batch_sizeÚper_device_eval_batch_sizeÚper_gpu_train_batch_sizeÚper_gpu_eval_batch_sizeÚgradient_accumulation_stepsÚeval_accumulation_stepsÚ
|
|||
|
|
eval_delayÚtorch_empty_cache_stepsÚ
|
|||
|
|
learning_rateÚ weight_decayÚ
|
|||
|
|
adam_beta1Ú
|
|||
|
|
adam_beta2Ú adam_epsilonÚ
|
|||
|
|
max_grad_normÚnum_train_epochsÚ max_stepsÚlr_scheduler_typeÚ warmup_ratioÚ warmup_stepsÚ log_levelÚlog_level_replicaÚlog_on_each_nodeÚ logging_dirÚlogging_strategyÚlogging_first_stepÚ
|
|||
|
|
logging_stepsÚlogging_nan_inf_filterÚ
|
|||
|
|
save_strategyÚ
|
|||
|
|
save_stepsÚsave_total_limitÚsave_safetensorsÚsave_on_each_nodeÚsave_only_modelÚ'restore_callback_states_from_checkpointÚno_cudaÚuse_cpuÚuse_mps_deviceÚseedÚ data_seedÚ
|
|||
|
|
jit_mode_evalÚuse_ipexÚbf16Úfp16Úfp16_opt_levelÚhalf_precision_backendÚbf16_full_evalÚfp16_full_evalÚtf32Ú
|
|||
|
|
local_rankÚ ddp_backendÚ
|
|||
|
|
tpu_num_coresÚtpu_metrics_debugÚdebugÚdataloader_drop_lastÚ
|
|||
|
|
eval_stepsÚdataloader_num_workersÚdataloader_prefetch_factorÚ
|
|||
|
|
past_indexÚrun_nameÚ disable_tqdmÚremove_unused_columnsÚ label_namesÚload_best_model_at_endÚmetric_for_best_modelÚgreater_is_betterÚignore_data_skipÚfsdpÚfsdp_min_num_paramsÚ fsdp_configÚ"fsdp_transformer_layer_cls_to_wrapÚaccelerator_configÚ deepspeedÚlabel_smoothing_factorÚoptimÚ
|
|||
|
|
optim_argsÚ adafactorÚgroup_by_lengthÚlength_column_nameÚ report_toÚddp_find_unused_parametersÚddp_bucket_cap_mbÚddp_broadcast_buffersÚdataloader_pin_memoryÚ dataloader_persistent_workersÚskip_memory_metricsÚuse_legacy_prediction_loopÚ push_to_hubÚresume_from_checkpointÚ hub_model_idÚ hub_strategyÚ hub_tokenÚhub_private_repoÚhub_always_pushÚ hub_revisionÚgradient_checkpointingÚ gradient_checkpointing_kwargsÚinclude_inputs_for_metricsÚeval_do_concat_batchesÚ fp16_backendÚpush_to_hub_model_idÚpush_to_hub_organizationÚpush_to_hub_tokenÚ
|
|||
|
|
mp_parametersÚauto_find_batch_sizeÚfull_determinismÚ torchdynamoÚ ray_scopeÚ ddp_timeoutÚ
|
|||
|
|
torch_compileÚtorch_compile_backendÚtorch_compile_modeÚinclude_tokens_per_secondÚ include_num_input_tokens_seenÚneftune_noise_alphaÚoptim_target_modulesÚbatch_eval_metricsÚ
|
|||
|
|
eval_on_startÚuse_liger_kernelÚliger_kernel_configÚeval_use_gather_objectÚ average_tokens_across_devicesÚ
|
|||
|
|
max_lengthÚmax_prompt_lengthÚmax_completion_lengthÚbetaÚlabel_pad_token_idÚ
|
|||
|
|
padding_valueÚtruncation_modeÚdisable_dropoutÚgenerate_during_evalÚis_encoder_decoderÚprecompute_ref_log_probsÚmodel_init_kwargsÚref_model_init_kwargsÚdataset_num_procÚprompt_sample_sizeÚmin_density_ratioÚmax_density_ratio© ) ÚFloatingPointErrorÚ
|
|||
|
|
OverflowErrorÚmultiprocessingr ÚminÚsuperÚ__init__r| r} )—Úselfr¡ r¢ r£ r¤ r¥ r¦ r§ r¨ r© rª r« r¬ r r® r¯ r° r± r² r³ r´ rµ r¶ r· r¸ r¹ rº r» r¼ r½ r¾ r¿ rÀ rÁ r rà rÄ rÅ rÆ rÇ rÈ rÉ rÊ rË rÌ rÍ rÎ rÏ rÐ rÑ rÒ rÓ rÔ rÕ rÖ r× rØ rÙ rÚ rÛ rÜ rÝ rÞ rß rà rá râ rã rä rå ræ rç rè ré rê rë rì rí rî rï rð rñ rò ró rô rõ rö r÷ rø rù rú rû rü rý rþ rÿ r r r r r r r r r r r
|
|||
|
|
r r r
|
|||
|
|
r r r r r r r r r r r r r r r r r r r r! r" r# r$ r% r& r' r( r) r* r+ r, r- r. r/ r0 r1 r| r} Úkwargsr Ú __class__s— €rt r8 zUnslothBCOConfig.__init__w s° ø€ ðn ˜4Ò Ð Õ'9ð ;VÐ]jð ;Vð ;Vð ;Vñ (Wô (Wð "WØ ˜1Ò Ð ¥Mð 3FÐUbð 3Fð 3Fð 3Fñ %Gô %Gð GØ Ð -°7Ò":Ð":¸zÈSÒ?PÐ?PØ7ˆJØ ˆMØ Ð #Ø 1Ð 1Ð 1Ð 1Ð 1Ð 1Ý " 9 9¡;¤;¨q¡=°!Ñ 4Ô 4Ð à�‰ŒÔð Q <ð Q <ð Q <Ø#˜ðQ <à#7Ð#7ðQ <ð �xðQ <ð �gð Q <ð
|
|||
|
|
$˜ð Q <ð *˜Mð
|
|||
|
|
Q <ð $8Ð#7ðQ <ð +FÐ*EðQ <ð *DÐ)CðQ <ð (@Ð'?ðQ <ð '>Ð&=ðQ <ð +FÐ*EðQ <ð '>Ð&=ðQ <ð $˜ð Q <ð '>Ð&=ð Q <ð *˜Mð!Q <ð" (˜<ð#Q <ð$ $˜ð%Q <ð& $˜ð'Q <ð( (˜<ð)Q <ð* *˜Mð+Q <ð, 0Ð /ð-Q <ð. "˜ ð/Q <ð0 !2Ð 1ð1Q <ð2 (˜<ð3Q <ð4 (˜<ð5Q <ð6 "˜ ð7Q <ð8 !2Ð 1ð9Q <ð: 0Ð /ð;Q <ð< &˜+ð=Q <ð> 0Ð /ð?Q <ð@ "4Ð!3ðAQ <ðB *˜MðCQ <ðD &<Ð%;ðEQ <ðF *˜MðGQ <ðH $˜ðIQ <ðJ 0Ð /ðKQ <ðL 0Ð /ðMQ <ðN !2Ð 1ðOQ <ðP .˜oðQQ <ðR 7^Ð6]ðSQ <ðT �gðUQ <ðV �gðWQ <ðX ,˜^ðYQ <ðZ �4ð[Q <ð\ "˜ ð]Q <ð^ *˜Mð_Q <ð` �xðaQ <ðb �4ðcQ <ðd �4ðeQ <ðf ,˜^ðgQ <ðh &<Ð%;ðiQ <ðj ,˜^ðkQ <ðl ,˜^ðmQ <ðn �4ðoQ <ðp $˜ðqQ <ðr &˜+ðsQ <ðt *˜MðuQ <ðv !2Ð 1ðwQ <ðx �EðyQ <ðz $8Ð#7ð{Q <ð| $˜ð}Q <ð~ &<Ð%;ðQ <ð@ *DÐ)CðAQ <ðB $˜ðCQ <ðD �xðEQ <ðF (˜<ðGQ <ðH %:Ð$9ðIQ <ðJ &˜+ðKQ <ðL &<Ð%;ðMQ <ðN %:Ð$9ðOQ <ðP !2Ð 1ðQQ <ðR 0Ð /ðSQ <ðT �4ðUQ <ðV #6Ð"5ðWQ <ðX &˜+ðYQ <ðZ 2TÐ1Sð[Q <ð\ "4Ð!3ð]Q <ð^ "˜ ð_Q <ð` &<Ð%;ðaQ <ðb �EðcQ <ðd $˜ðeQ <ðf "˜ ðgQ <ðh .˜oðiQ <ðj "4Ð!3ðkQ <ðl "˜ ðmQ <ðn *DÐ)CðoQ <ðp !2Ð 1ðqQ <ðr %:Ð$9ðsQ <ðt %:Ð$9ðuQ <ðv -JÐ,IðwQ <ðx #6Ð"5ðyQ <ðz *DÐ)Cð{Q <ð| &˜+ð}Q <ð~ &<Ð%;ðQ <ð@ (˜<ðAQ <ðB (˜<ðCQ <ðD "˜ ðEQ <ðF 0Ð /ðGQ <ðH .˜oðIQ <ðJ (˜<ðKQ <ðL &<Ð%;ðMQ <ðN -JÐ,IðOQ <ðP *DÐ)CðQQ <ðR &<Ð%;ðSQ <ðT (˜<ðUQ <ðV $8Ð#7ðWQ <ðX (@Ð'?ðYQ <ðZ !2Ð 1ð[Q <ð\ *˜Mð]Q <ð^ $8Ð#7ð_Q <ð` 0Ð /ðaQ <ðb &˜+ðcQ <ðd "˜ ðeQ <ðf &˜+ðgQ <ðh *˜MðiQ <ðj %:Ð$9ðkQ <ðl "4Ð!3ðmQ <ðn )BÐ(AðoQ <ðp -JÐ,IðqQ <ðr #6Ð"5ðsQ <ðt $8Ð#7ðuQ <ðv "4Ð!3ðwQ <ðx *˜MðyQ <ðz 0Ð /ð{Q <ð| #6Ð"5ð}Q <ð~ &<Ð%;ðQ <ð@ -JÐ,IðAQ <ðB $˜ðCQ <ðD !2Ð 1ðEQ <ðF %:Ð$9ðGQ <ðH �4ðIQ <ðJ "4Ð!3ðKQ <ðL *˜MðMQ <ðN .˜oðOQ <ðP .˜oðQQ <ðR $8Ð#7ðSQ <ðT "4Ð!3ðUQ <ðV (@Ð'?ðWQ <ðX !2Ð 1ðYQ <ðZ %:Ð$9ð[Q <ð\ 0Ð /ð]Q <ð^ "4Ð!3ð_Q <ð` !2Ð 1ðaQ <ðb !2Ð 1°FðcQ <ð Q <ð Q <ðd %9ˆÔ!Ø"4ˆÔ Ð Ð rv )“NNFFFr~ FrY rY NNr r r r€ r� r‚ rƒ r„ r… r† r‡ rX rˆ r‰ r rŠ r‹ TNrŒ Fr^ FrŒ r� NTFFFFFFrŽ rŽ FFFFr� r� FFNrX NNFr‘ FNr NrX NNTNFNNFr‘ r NNNNr’ r“ NFFr” NNNNTFTFFNNr• NNFNFNFTr� NNNr‘ TFNr– r— FNNFFNNFFFNFTr˜ r™ Nr‰ rš Nr› TFNFNNNr˜ rœ r� NrX )
|
|||
|
|
Ú__name__Ú
|
|||
|
|
__module__Ú __qualname__Ú__doc__rM r| r r Ú__annotations__r} Úintr8 Ú
|
|||
|
|
__classcell__©r; s @rt rx rx 3 s„ ø€ € € € € € ð9ð 9ðt +0¨%ØØÐ 1Ð2ð+ñ +ô +И( 3œ-ð ð ñ ð */¨ØØÐ VÐWð*ñ *ô *И #œ ð ð ñ ð Ø #ØØØØ Ø $Ø&'Ø%&Ø#'Ø"&Ø&'Ø"#ØØ"%Ø ØØØØ ØØ ØØ $ØØØ Ø %Ø ØØ"Ø "ØØ!&Ø ØØ Ø Ø !Ø Ø27ØØØ ØØØ ØØØØ Ø!'Ø Ø ØØØØ Ø !ØØ $ØØ!"Ø%)ØØØØ $ØØ!&Ø $Ø Ø ØØ ØØ-1Ø !ØØ!$Ø ØØØ Ø %ØØ%)Ø Ø $Ø $Ø(-Ø "Ø%*ØØ!%ØØ#ØØ Ø ØØ!&Ø(,Ø%*Ø!%Ø Ø #Ø#'Ø ØØ #Ø ØØØØ Ø $Ø !Ø$)Ø(-Ø "Ø #Ø "Ø Ø Ø "Ø!&Ø(,ØØ Ø $ØØ !Ø Ø$Ø Ø $Ø !Ø#(Ø Ø $Ø Ø !Ø Ø Ø #Ø ðis5ð s5ð s5ð s5ð s5ð s5ð s5ð s5ð s5ð s5rv rx c $ óR ‡ — e Zd ZdZddgZ dYdeeej e f de
|
|||
|
|
eeej e f de d e
|
|||
|
|
e d
|
|||
|
|
e
|
|||
|
|
ee e
|
|||
|
|
e e f f d e
|
|||
|
|
eeeeef d e
|
|||
|
|
e d
|
|||
|
|
e
|
|||
|
|
eg ef de
|
|||
|
|
ee deej j ej j j f de
|
|||
|
|
eej ej gej f de
|
|||
|
|
e
|
|||
|
|
de
|
|||
|
|
ee ge
|
|||
|
|
f de
|
|||
|
|
e de
|
|||
|
|
e de
|
|||
|
|
e de
|
|||
|
|
e f"ˆ fd„
|
|||
|
|
Z e d„ ¦ « Z dej! dej! fd„Z"d ej# d ej# dej! fd „Z$d e
|
|||
|
|
e eeej# f f deej! ej! f fd „Z%dZd"e d#e&dej! fd$„Z'ˆ fd%„Z(ˆ fd&„Z)e*d'„ ¦ « Z+de,fˆ fd(„ Z-d[d
|
|||
|
|
e
|
|||
|
|
e de,fˆ fd)„
|
|||
|
|
Z.d*e
|
|||
|
|
de
|
|||
|
|
fd+„Z/e0 d\d.ej! d/ej# d0e1d1e&d2e1dej! f d3„¦ « Z2dej d e
|
|||
|
|
e eeej# f f deej! ej! ej! ej! f fd4„Z3d5ej! dej! fd6„Z4 d]d8ej! d9ej! d:ej! d;ej! d<e
|
|||
|
|
ej! d5e
|
|||
|
|
ej! d=e1deej! ej! ej! ej! f fd>„Z5 d]d e
|
|||
|
|
e eeej# f f d=e1fd?„Z6 d^deeej f d@e
|
|||
|
|
e eej e7f f deej eej e
|
|||
|
|
e ej f f f fdA„Z8d_dCe
|
|||
|
|
e e9f dDe:dE ddfdF„Z;d[d"e
|
|||
|
|
e de
|
|||
|
|
ej< j= j> fdG„Z?d e
|
|||
|
|
e ej# f dee e f fdH„Z@ d[deeej f d@e
|
|||
|
|
e eej e7f f dIe1dJe
|
|||
|
|
ee fdK„ZA d`dMe,dNe dIe
|
|||
|
|
e1 dJe
|
|||
|
|
ee dOe de f ˆ fdP„
|
|||
|
|
ZBd[dQe
|
|||
|
|
e e9f dRe
|
|||
|
|
e9 ddfˆ fdS„
|
|||
|
|
ZCˆ fdT„ZD dadUe
|
|||
|
|
e dVe
|
|||
|
|
e dWee ee df fdX„ZEˆ xZFS )bÚ_UnslothBCOTrainerr‘ ÚtrlÚbcoN©NNÚmodelÚ ref_modelÚargsÚ
|
|||
|
|
train_datasetÚ eval_datasetÚprocessing_classÚ
|
|||
|
|
data_collatorÚ
|
|||
|
|
model_initÚ callbacksÚ
|
|||
|
|
optimizersÚ preprocess_logits_for_metricsÚ peft_configÚcompute_metricsÚmodel_adapter_nameÚref_adapter_nameÚembedding_funcÚembedding_tokenizerc
|
|||
|
|
ó •— |�+t ¦ « rt ¦ « st d¦ « ‚t |¦ « t u rt d¦ « ‚t
|
|||
|
|
|t ¦ « s|�||u rt d¦ « ‚|j €i }nªt
|
|||
|
|
|t ¦ « st d¦ « ‚|j }| d¦ « }|�ht
|
|||
|
|
|t ¦ « r|dk rt t |¦ « }|dk r-t
|
|||
|
|
|t j ¦ « st d|› d�¦ « ‚||d< |j
|
|||
|
|
€i }nªt
|
|||
|
|
|t ¦ « st d ¦ « ‚|j
|
|||
|
|
}| d¦ « }|�ht
|
|||
|
|
|t ¦ « r|dk rt t |¦ « }|dk r-t
|
|||
|
|
|t j ¦ « st d|› d�¦ « ‚||d< t
|
|||
|
|
|t ¦ « rt j |fi |¤Ž}t
|
|||
|
|
|t ¦ « rt j |fi |¤Ž}d
|
|||
|
|
| _ t# ¦ « s| �t d ¦ « ‚t# ¦ « �r5| ��2t
|
|||
|
|
|t$ ¦ « r| ¦ « }t |d d
|
|||
|
|
¦ « st |d
|
|||
|
|
d
|
|||
|
|
¦ « r`t) |d¦ « o,dt+ t- j t0 ¦ « j ¦ « v }d|j i}|r
|
|||
|
|
|j |d< t1 |fi |¤Ž}nV|j rOt) |d¦ « r| ¦ « n*d„ }| ¦ « |¦ « |}|j r't |d
|
|||
|
|
d
|
|||
|
|
¦ « rtA |¦ « d| _ nV|j rOt) |d¦ « r| ¦ « n*d„ }| ¦ « |¦ « |j! r+tE ¦ « s tG ¦ « st d¦ « ‚|�|j$ j% | _% n"|j% €t d¦ « ‚|j% | _% t# ¦ « ot
|
|||
|
|
|t$ ¦ « | _& || _' || _( |r|| _) n*| j& s|j* rd | _) ntW |¦ « | _) |€t d¦ « ‚|j, € t[ j. dt^ ¦ « d}|j, �|j, }|j0 € t[ j. dt^ ¦ « d}|j0 �|j0 }d }|j1 €#| j% r t[ j. dt^ ¦ « d}|j1 �| j% r|j1 }|€Qte |j3 |j4 | j% ¬ ¦ « }|j5 r!d
|
|||
|
|
|_5 t[ j. d t^ ¦ « d| _6 nd
|
|||
|
|
| _6 |j7 r*tq |¦ « | j) �tq | j) ¦ « || _, |j! | _! |j4 | _4 |j9 �|j9 n|j3 | _9 || _0 |j: | _: || _1 |j* | _* d
|
|||
|
|
| _; d
|
|||
|
|
| _< t{ d „ ¦ « | _> |j? | _? t |j$ d d
|
|||
|
|
¦ « | _@ t |j$ d d!¦ « | _A | j@ r%| jA d!k rt[ j. d"t^ ¦ « || _B || _C d|jD d#< t‹ ¦ « F ¦ « 5 | G t� d$|i|jI ¬%¦ « }|�$| G t� d$|i|jI ¬%¦ « }| G t” d|| jC d&œ|jI d'¬(¦ « }d)| j% || j, | j: | j4 | j0 | j1 d*œ}| G t– ||jI d+¬,¦ « }|�x| G t” || jC d&œd|jI d-¬.¦ « }d)| j% || j, | j: | j4 | j0 | j1 d*œ}| G t– ||jI d/¬,¦ « }| L d0„ |jI d1¬2¦ « } | L d3„ |jI d4¬2¦ « } d d d ¦ « n # 1 swxY w Y t› ¦ « N ||||||||
|
|||
|
|
| |
|
|||
|
|
| ¬5¦ « d
|
|||
|
|
| _O t) | jP d6¦ « r | jP Q | jR ¦ « t) | d7¦ « st§ d8¦ « ‚| jT r0| jU jV jW jX d9k r| j* rt d:¦ « ‚| j) € | j& s| j* st d;¦ « ‚nM| jT r t³ | j) | jU ¦ « | _) n&| jU Z | j) d¬<¦ « | _) t· | jU ¬=¦ « | _\ | jB �|j] rd S | ^ | | j_ j` ¬>¦ « } | ^ | | j_ j` ¬>¦ « } t ja | | fd?¬@¦ « } t ja t jb | d d …d?f ¦ « t jc | d d …d?f ¦ « fd?¬@¦ « }!tÉ dA¬B¦ « e | f ¦ « g ¦ « h ¦ « |! f ¦ « h ¦ « ¦ « | _i | ji j | f ¦ « g ¦ « h ¦ « t jb | d d …d?f ¦ « f ¦ « h ¦ « ¦ « }"| ji j | f ¦ « g ¦ « h ¦ « t jc | d d …d?f ¦ « f ¦ « h ¦ « ¦ « }#t× jl dC|"› dD|#› �¦ « d S )ENz}BCOTrainer with UDM requires the scikit-learn and joblib libraries. Please install it with `pip install scikit-learn joblib`.z3Please use `BCOConfig` instead `TrainingArguments`.zœ`model` and `ref_model` cannot be the same object. If you want `ref_model` to be the same as `model`, you must mass a copy of it, or `None` if you use peft.zRYou passed model_kwargs to the BCOTrainer. But your model is already instantiated.Ú torch_dtyper� znInvalid `torch_dtype` passed to the BCOConfig. Expected a string with either `torch.dtype` or 'auto', but got ú.zZYou passed ref_model_kwargs to the BCOTrainer. But your ref_model is already instantiated.FzŽPEFT is not installed and you passed a `peft_config` in the trainer's kwargs, please install it with `pip install peft` to use the PEFT modelsÚis_loaded_in_8bitÚis_loaded_in_4bitr Úuse_gradient_checkpointingÚenable_input_require_gradsc ó0 — | d¦ « d S ©NT©Úrequires_grad_©ÚmoduleÚinputÚoutputs rt Úmake_inputs_require_gradz=_UnslothBCOTrainer.__init__.<locals>.make_inputs_require_grad s € Ø ×-Ò-¨dÑ3Ô3Ð3Ð3Ð3rv Tc ó0 — | d¦ « d S rb rc re s rt ri z=_UnslothBCOTrainer.__init__.<locals>.make_inputs_require_grad5 s € Ø×)Ò)¨$Ñ/Ô/Ð/Ð/Ð/rv z‚`generate_during_eval=True` requires Weights and Biases or Comet to be installed. Please install `wandb` or `comet-ml` to resolve.zMWhen no model is provided, you need to pass the parameter is_encoder_decoder.zdmax_length or a processing_class must be specified when using the default DPODataCollatorWithPaddingz§When using DPODataCollatorWithPadding, you should set `max_length` in the `BCOConfig`. It will be set to `512` by default, but you should do it yourself in the future.r™ z®When using DPODataCollatorWithPadding, you should set `max_prompt_length` in the `BCOConfig`. It will be set to `128` by default, but you should do it yourself in the future.é€ zÜWhen using DPODataCollatorWithPadding with an encoder decoder architecture, you should set `max_completion_length` in the BCOTrainer's init it will be set to `128` by default, but you should do it yourself in the fu
|
|||
|
|
€ ˜!˜Gœ*€ rv z Filtering desirable examples)rv rx c ó — | d S r{ r2 r} s rt rp z-_UnslothBCOTrainer.__init__.<locals>.<lambda> s € ˜a œj˜.€ rv z Filtering undesirable examples) rI rK rO rL rM rN rP rU rQ rR rS Úadd_model_tagsÚ acceleratorzXYour `Trainer` does not have an `accelerator` object. Consider upgrading `transformers`.é zrYou cannot use `precompute_ref_log_probs=True` with Deepspeed ZeRO-3. Please set `precompute_ref_log_probs=False`.z]No reference model and model is not a Peft model. Try setting `precompute_ref_log_probs=True`)Úevaluation_mode)r� )Ú sample_sizer r] Úbalanced)Ú class_weightz(UDM classifier training scores: chosen: z , rejected: )mr5 r3 Ú ImportErrorÚtyper& Ú
|
|||
|
|
ValueErrorÚ
|
|||
|
|
isinstanceÚstrr, ÚgetÚgetattrrH Údtyper- r
|
|||
|
|
Úfrom_pretrainedÚ _peft_has_been_casted_to_bf16r4 r Úmerge_and_unloadÚhasattrro r1 Ú signaturerD Ú
|
|||
|
|
parametersr r r` Úget_input_embeddingsÚregister_forward_hookrÑ rB r) r6 r2 Úconfigr* Ú
|
|||
|
|
is_peft_modelrV rW rJ r+ r+ r! rJ ÚwarnÚ UserWarningr" r# r rl r% rä Úuse_dpo_data_collatorr( r- r& r' Ú _precomputed_train_ref_log_probsÚ _precomputed_eval_ref_log_probsr, Ú_stored_metricsr$ Úaux_loss_enabledÚ
|
|||
|
|
aux_loss_coefrX rY Úwarnings_issuedr Úmain_process_firstÚmapr; r. r( r' Úfilterr7 r8 Úmodel_accepts_loss_kwargsrI r€ Ú
|
|||
|
|
_tag_namesÚAttributeErrorÚis_deepspeed_enabledr� ÚstateÚdeepspeed_pluginÚ
|
|||
|
|
zero_stagerC Ú
|
|||
|
|
prepare_modelr" Úrunningrÿ Ú _get_sample_prompt_embeddingsrK r/ ÚcatÚ ones_likeÚ
|
|||
|
|
zeros_liker ÚfitÚcpuÚfloatÚnumpyÚclfÚscorer: Úinfo)%r9 rI rJ rK rL rM rN rO rP rQ rR rS rT rU rV rW rX rY r, r[ r- Ú_support_gc_kwargsÚprepare_model_kwargsri r! r" r# ru Ú desirableÚ undesirableÚchosen_embeddingsÚrejected_embeddingsÚ
|
|||
|
|
embeddingsÚlabelsÚ chosen_meanÚ
|
|||
|
|
rejected_meanr; s% €rt r8 z_UnslothBCOTrainer.__init__² sŽ ø€ ð, Ð %Õ/CÑ/EÔ/EÐ %ÕJ]ÑJ_ÔJ_Ð %Ý ð Pñô ð
|
|||
|
|
õ �‰:Œ:Õ*Ð *Ð *Ý Ð RÑSÔSÐ Så˜%¥Ñ%Ô%ð ¨%Ð*;À ÈUÐ@RÐ@RÝ ðZñô ð
|
|||
|
|
ð
|
|||
|
|
Ô !Ð )Ø "РРݘE¥3Ñ'Ô'ð
|
|||
|
|
?Ý Ð qÑrÔrÐ rà $Ô 6Ð Ø+×/Ò/°
|
|||
|
|
Ñ>Ô>ˆKØÐ&å ˜k3Ñ/Ô/ð >°KÀ6Ò4IÐ4IÝ")%°Ñ"=Ô"=�KØ &Ò(Ð(µ¸KÍÌÑ1UÔ1UÐ(Ý$ð Xð JUð Xð Xð Xñô ð ð 4?Ð! -Ñ0à Ô %Ð -Ø$&Ð !Ð !ݘI¥sÑ+Ô+ð CÝ Ølñô ð
|
|||
|
|
ð %)Ô$>Ð !Ø/×3Ò3°MÑBÔBˆKØÐ&å ˜k3Ñ/Ô/ð >°KÀ6Ò4IÐ4IÝ")%°Ñ"=Ô"=�KØ &Ò(Ð(µ¸KÍÌÑ1UÔ1UÐ(Ý$ð Xð JUð Xð Xð Xñô ð ð 8CÐ% mÑ4å �e�SÑ !Ô !ð UÝ(Ô8¸ÐTÐTÐBSÐTÐTˆEå �i¥Ñ %Ô %ð aÝ,Ô<¸YÐ`Ð`ÐJ_Ð`Ð`ˆIð .3ˆÔ*å Ñ"Ô"ð 4 ] {Ð'>Ý ð añô ð
|
|||
|
|
õ Ñ
|
|||
|
|
Ô
|
|||
|
|
ñ 0 ] [Ñ%<å˜%¥Ñ+Ô+ð
|
|||
|
|
1Ø ×.Ò.Ñ0Ô0�å�uÐ 1°5Ñ9Ô9ð
|
|||
|
|
a½WÀUÐL_ÐafÑ=gÔ=gð
|
|||
|
|
aÝ%,ØÐ9ñ&ô &ð &à5½ÝÔ%Õ&EÑFÔFÔQñ:ô :ð ð #ð )EÀdÔFaÐ'bÐ$à%ð oØLPÔLnÐ(Ð)HÑIå7¸ÐVÐVÐAUÐVÐV��ØÔ,ð
|
|||
|
|
aå˜5Ð">Ñ?Ô?ð aØ×4Ò4Ñ6Ô6Ð6Ð6ð4ð 4ð 4ð ×.Ò.Ñ0Ô0×FÒFÐG_Ñ`Ô`Ð`ð ˆEØŒyð
|
|||
|
|
:�W UÐ,?ÀÑ GÔ Gð
|
|||
|
|
:Ý+¨EÑ2Ô2Ð2à59�Ô2øð
|
|||
|
|
Ô
|
|||
|
|
(ð ]å�uÐ :Ñ;Ô;ð
|
|||
|
|
]Ø×0Ò0Ñ2Ô2Ð2Ð2ð0ð 0ð 0ð ×*Ò*Ñ,Ô,×BÒBÐC[Ñ\Ô\Ð\à Ô $ð Õ.@Ñ.BÔ.Bð ÕFXÑFZÔFZð Ý ðDñô ð
|
|||
|
|
ð
|
|||
|
|
Ð Ø&+¤lÔ&EˆDÔ #Ð #Ø
|
|||
|
|
Ô
|
|||
|
|
$Ð
|
|||
|
|
,Ý Ð lÑmÔmÐ mà&*Ô&=ˆDÔ #å .Ñ 0Ô 0Ð QµZÀÅyÑ5QÔ5QˆÔØ"4ˆÔ Ø 0ˆÔ à ð ;Ø &ˆDŒNˆNØ
|
|||
|
|
Ô
|
|||
|
|
ð ; 4Ô#@ð ;à !ˆDŒNˆNå 3°EÑ :Ô :ˆDŒNà Ð #Ý Øvñô ð
|
|||
|
|
ð Œ?Ð "Ý ŒMðcåñ
|
|||
|
|
ô
|
|||
|
|
ð
|
|||
|
|
ð
|
|||
|
|
ˆJØ Œ?Ð &Ø œ ˆJà Ô !Ð )Ý ŒMðcåñ
|
|||
|
|
ô
|
|||
|
|
ð
|
|||
|
|
ð
|
|||
|
|
!$Ð Ø Ô !Ð -Ø $Ô 6Ð à $Ð Ø Ô %Ð -°$Ô2IÐ -Ý ŒMðdåñ
|
|||
|
|
ô
|
|||
|
|
ð
|
|||
|
|
ð
|
|||
|
|
%(Ð !Ø Ô %Ð 1°dÔ6MÐ 1Ø$(Ô$>Ð !à Ð Ý 6Ø -Ô :Ø#'Ô#:Ø#'Ô#:ð ñ ô ˆMð Ô)ð
|
|||
|
|
Ø-2�Ô*å”
|
|||
|
|
ð\å ñô ð ð *.ˆDÔ &Ð &à).ˆDÔ &ð Ô ð 9Ý $ UÑ +Ô +Ð +ØŒ~Ð)Ý(¨¬ Ñ8Ô8Ð8à$ˆŒØ$(Ô$=ˆÔ!Ø"&Ô"9ˆÔ Ø37Ô3EÐ3Q˜TÔ /Ð /ÐWgÔWtˆÔØ!2ˆÔ Ø #Ô 3ˆÔ Ø%:ˆÔ"Ø(,Ô(EˆÔ%ð 16ˆÔ-Ø/4ˆÔ,õ +Ð+DÐ+DÑ EÔ EˆÔ ð ”IˆŒ Ý '¨¬ Ð6LÈeÑ TÔ TˆÔ Ý $ U¤\Ð3IÈ3Ñ OÔ OˆÔØ Ô ð TÔ%7¸3Ò%>Ð%>Ý ŒMðõ ñ
|
|||
|
|
ô
|
|||
|
|
ð
|
|||
|
|
ð -ˆÔØ#6ˆÔ ð 48ˆÔ Ð /Ñ0å
|
|||
|
|
‰^Œ^×
|
|||
|
|
.Ò
|
|||
|
|
.Ñ
|
|||
|
|
0Ô
|
|||
|
|
0ð H ð H à )× -Ò -Ý)°kÐCSÐ5TÐ_cÔ_tð .ñ ô ˆMð Ð'Ø +× /Ò /Ý-Ø *Ð,<Ð =Ø !Ô 2ð 0ñ ô � ð *× -Ò -ÝØ Ø(8ÐQUÔQiÐjÐjØ Ô.Ø/ð .ñ ô ˆMð Ø&*Ô&=Ø -Ø "œoØ#'Ô#7Ø&*Ô&=Ø%)Ô%;Ø)-Ô)Cð ð ˆIð *× -Ò -Ý Ø#Ø Ô.Ø9ð .ñ ô ˆMð Ð'à +× /Ò /Ý Ø,<ÐUYÔUmÐ nÐ nØ Ø !Ô 2Ø2ð 0ñ ô � ð !Ø*.Ô*AØ!1Ø"&¤/Ø'+Ô';Ø*.Ô*AØ)-Ô)?Ø-1Ô-Gð ð � ð ,× /Ò /Ý#Ø 'Ø !Ô 2Ø<ð 0ñ ô � ð &×,Ò,Ø$Ð$¨tÔ/DÐKið -ñ ô ˆIð (×.Ò.Ø(Ð(°4Ô3HÐOoð /ñ ô ˆKðMH ð H ð H ñ H ô H ð H ð H ð H ð H ð H ð H øøøð H ð H ð H ð H õT ‰Œ×ÒØØØ'Ø'Ø%Ø -Ø!Ø +Ø Ø!Ø*Gð ñ
|
|||
|
|
ô
|
|||
|
|
ð
|
|||
|
|
ð" */ˆÔ&õ �4”:Ð /Ñ 0Ô 0ð 7Ø ŒJ× %Ò % d¤oÑ 6Ô 6Ð 6å�t˜]Ñ+Ô+ð Ý Øjñô ð
|
|||
|
|
ð
|
|||
|
|
Ô $ð ØÔ Ô%Ô6ÔAÀQÒFÐFÈ4ÔKhÐFÝ ð Iñô ð ð Œ>Ð !ØÔ&ð
|
|||
|
|
¨$Ô*Gð
|
|||
|
|
Ý Øsñô ð øð Ô(ð
|
|||
|
|
fÝ!2°4´>À4ÔCSÑ!TÔ!T�”�à!%Ô!1×!?Ò!?ÀÄÐ`dÐ!?Ñ!eÔ!e�”å%°$Ô2BÐCÑCÔCˆŒ à Ô Ð &¨$Ô*EÐ &Ø ˆFà × >Ò >¸yÐVZÔV_ÔVrÐ >Ñ sÔ sÐØ "× @Ò @ÀÐZ^ÔZcÔZvÐ @Ñ wÔ wÐå”YÐ 1Ð3FÐ GÈQÐOÑOÔOˆ
|
|||
|
|
Ý”Ý
|
|||
|
|
Œ_Ð .¨q¨q¨q°!¨tÔ 4Ñ
|
|||
|
|
5Ô
|
|||
|
|
5µuÔ7GÐH[Ð\]Ð\]Ð\]Ð_`Ð\`ÔHaÑ7bÔ7bÐ cÐijð
|
|||
|
|
ñ
|
|||
|
|
ô
|
|||
|
|
ˆõ &°:Ð>Ñ>Ô>×BÒBØ �NŠNÑ Ô × "Ò "Ñ $Ô $× *Ò *Ñ ,Ô ,¨f¯jªj©l¬l×.@Ò.@Ñ.BÔ.Bñ
|
|||
|
|
ô
|
|||
|
|
ˆŒð ”h—n’nØ × !Ò !Ñ #Ô #× )Ò )Ñ +Ô +× 1Ò 1Ñ 3Ô 3µU´_ÐEVÐWXÐWXÐWXÐZ[ÐW[ÔE\Ñ5]Ô5]×5aÒ5aÑ5cÔ5c×5iÒ5iÑ5kÔ5kñ
|
|||
|
|
ô
|
|||
|
|
ˆ 𠜟šØ × #Ò #Ñ %Ô %× +Ò +Ñ -Ô -× 3Ò 3Ñ 5Ô 5µuÔ7GÐH[Ð\]Ð\]Ð\]Ð_`Ð\`ÔHaÑ7bÔ7b×7fÒ7fÑ7hÔ7h×7nÒ7nÑ7pÔ7pñ
|
|||
|
|
ô
|
|||
|
|
ˆ
|
|||
|
|
õ Œ Ðg¸{ÐgÐgÐXeÐgÐgÑhÔhÐhÐhÐhs Ü-E9b2â2b6â9b6c ó& — | j d uo| j d uS rn )rX rY ©r9 s rt Ú match_underlying_distributionz0_UnslothBCOTrainer.match_underlying_distributionK s € àÔ"¨$Ð.ÐW°4Ô3KÐSWÐ3WÐWrv Úprompt_embeddingsÚreturnc ó¼ — |j }|j }| j j }| j || j j ¬¦ « }|j d }| d¬¦ « | j j k }| j |¦ « }|j d dk rt j g ||¬¦ « S | j
|
|||
|
|
| ¦ « ¦ « ¦ « ¦ « dd…df }t j |||¬¦ « }| j |d¬ ¦ « }|||z ||dz z … }|| }|S )
|
|||
|
|
zÄ
|
|||
|
|
Calculates the probability if the given prompt embedding is from desirable dataset. This function calculates
|
|||
|
|
the probability in the process and ensemble across processes.
|
|||
|
|
)Ú pad_indexr r^ r] )ÚdevicerŽ N©rŽ rÊ Úmean)Ú reduction)rŽ rÊ r� Ú
|
|||
|
|
process_indexÚpad_across_processesrY rl ra rÌ re rH Útensorr¶ Ú
|
|||
|
|
predict_probar³ r´ rµ Ú as_tensorÚreduce) r9 rÆ rŽ rÊ ÚrankÚpadded_prompt_embeddingsr„ ÚnonzeroÚprobs rt Ú_get_chosen_probz#_UnslothBCOTrainer._get_chosen_probO sa € ð
|
|||
|
|
"Ô'ˆØ"Ô)ˆØÔ Ô-ˆà#'Ô#3×#HÒ#HØ ¨Ô)AÔ)Nð $Iñ $
|
|||
|
|
ô $
|
|||
|
|
Ð ð /Ô4°QÔ7ˆ Ø*×/Ò/°AÐ/Ñ6Ô6¸$Ô:RÔ:_Ò_ˆØ Ô ,× 3Ò 3Ð4LÑ MÔ MÐð Ô " 1Ô %¨Ò *Ð *Ý”< ¨6¸Ð?Ñ?Ô?Ð ?àŒx×%Ò%Ð&7×&;Ò&;Ñ&=Ô&=×&CÒ&CÑ&EÔ&E×&KÒ&KÑ&MÔ&MÑNÔNÈqÈqÈqÐRSÈtÔTˆÝŒ˜t¨5¸Ð@Ñ@Ô@ˆØÔ ×&Ò& t°vÐ&Ñ>Ô>ˆà�K $Ñ&¨¸¸q¹Ñ)AÐAÔBˆØ�GŒ}ˆàˆ rv Ú input_idsÚattention_maskc óæ — t j || j j k | j j |¦ « }t j ¦ « 5 | ||¬¦ « }ddd¦ « n # 1 swxY w Y |S )z|
|
|||
|
|
Replaces processing_class.pad_token_id to embedding_tokenizer.pad_token_id and applies self.embedding_func
|
|||
|
|
©rÙ rÚ N)rH ÚwhererN rl rY Úno_gradrX )r9 rÙ rÚ r¿ s rt Ú_vectorize_promptz$_UnslothBCOTrainer._vectorize_promptl sº € õ ”KØ ˜Ô.Ô;Ò ;Ø Ô $Ô 1Ø ñ
|
|||
|
|
ô
|
|||
|
|
ˆ õ Œ]‰_Œ_ð ð Ø ×,Ò,Ø#Ø -ð -ñ ô ˆJð ð ð ñ ô ð ð ð ð ð ð øøøð ð ð ð ð Ðs ÁA&Á&A*Á-A*Úbatchc óL — | j sdS | |d |d ¬¦ « }t j |d t j |j ¬¦ « }t j |¦ « d }t j | ¦ « d }||df }||df }||fS ) z.Extract embeddings from frozen embedding modelrH Úembedding_input_idsÚembedding_attention_maskrÜ r| rË r .)rÅ rß rH rÐ ÚboolrÊ rÝ )r9 rà r¿ rÀ Ú
|
|||
|
|
chosen_idxÚ rejected_idxr½ r¾ s rt Ú_get_prompt_embeddingsz)_UnslothBCOTrainer._get_prompt_embeddings~ s¶ € ð
|
|||
|
|
Ô1ð Ø �:à×+Ò+ØÐ 1Ô2Ø Ð!;Ô<ð ,ñ
|
|||
|
|
ô
|
|||
|
|
ˆ
|
|||
|
|
õ
|
|||
|
|
” ˜e GœnµE´JÀzÔGXÐYÑYÔYˆÝ”[ Ñ(Ô(¨Ô+ˆ
|
|||
|
|
Ý ”{ F 7Ñ+Ô+¨AÔ.ˆ à & z°3 Ô 7ÐØ (¨ °sÐ):Ô ;Ðà!Ð#6Ð7Ð7rv r™ Údatasetr„ c óÚ — t t |¦ « |¦ « }t j t |¦ « |f¬¦ « }| |¦ « }| j j | j | j j | j j
|
|||
|
|
ddœ}| j t |fi |¤Ž¦ « }t j ¦ « 5 t j d¦ « }t# |d¬¦ « D ]g} | | d | d ¬ ¦ « }
|
|||
|
|
| j |
|
|||
|
|
¦ « }
|
|||
|
|
t j ||
|
|||
|
|
¦ « f¦ « }Œh d
|
|||
|
|
d
|
|||
|
|
d
|
|||
|
|
¦ « n # 1 swxY w Y |S ) zv
|
|||
|
|
Sample instances from dataset and get prompt embeddings. Used for density ratio classifier training.
|
|||
|
|
)ÚsizeF©Ú
|
|||
|
|
batch_sizeÚ
|
|||
|
|
collate_fnÚ num_workersÚ
|
|||
|
|
pin_memoryÚshuffler z!Building sample prompt embeddings©Úiterablerx râ rã rÜ N)r6 Úlenr= rE ÚchoiceÚselectrK r¨ rO rß rú r� Úpreparer rH rÞ ÚemptyrI rß Úgather_for_metricsr¯ r³ ) r9 rè r„ Ú n_samplesÚ rand_indicesÚembedding_datasetÚdataloader_paramsÚ data_loaderÚall_embeddingsÚ padded_batchr¿ s rt r® z0_UnslothBCOTrainer._get_sample_prompt_embeddings” s· € õ �˜G™ œ kÑ2Ô2ˆ Ý”y×'Ò'¨G© ¬ ¸I¸<Ð'ÑHÔHˆ à #ŸNšN¨<Ñ 8Ô 8Ðð œ)Ô?Ø Ô,Ø œ9Ô;Ø œ)Ô9Ø ð
|
|||
|
|
ð
|
|||
|
|
Ðð Ô&×.Ò.zÐ:KÐ/aÐ/aÐO`Ð/aÐ/aÑbÔbˆ å
|
|||
|
|
Œ]‰_Œ_ð Oð OÝ "œ[¨™^œ^ˆNÝ $¨kÐ@cÐ dÑ dÔ dð
|
|||
|
|
Oð
|
|||
|
|
O� Ø !× 3Ò 3Ø *Ð+@Ô AØ#/Ð0JÔ#Kð 4ñ ô �
|
|||
|
|
ð "Ô -× @Ò @ÀÑ LÔ L�
|
|||
|
|
Ý!&¤¨N¸J¿NºNÑ<LÔ<LÐ+MÑ!NÔ!N��ð
|
|||
|
|
|
|||
|
|
Oð Oð Oð Oñ Oô Oð Oð Oð Oð Oð Oð Oøøøð Oð Oð Oð Oð Ð s ÃBE Å E$Å'E$c ó� •— |�|n | j j }t ¦ « |¦ « | j j r…| j t j
|
|||
|
|
|t ¦ « ¦ « | j rCt j | j t j
|
|||
|
|
|t ¦ « d¬¦ « d S d S d S )NT)Úcompress)rK r¡ r7 Ú _save_optimizer_and_schedulerr� Úis_main_processr Ú save_to_jsonr? ÚpathÚjoinr! rÅ r8 Údumpr¶ r )r9 r¡ r; s €rt r z0_UnslothBCOTrainer._save_optimizer_and_scheduler´ s· ø€ Ø#-Ð#9�Z�Z¸t¼yÔ?Sˆ
|
|||
|
|
Ý
|
|||
|
|
‰Œ×-Ò-¨jÑ9Ô9Ð9à Ô Ô +ð Yà ŒL× %Ò %¥b¤g§l¢l°:½|Ñ&LÔ&LÑ MÔ MÐ MàÔ1ð
|
|||
|
|
YÝ” ˜DœH¥b¤g§l¢l°:½xÑ&HÔ&HÐSWÐXÑXÔXÐXÐXÐXð Yð Yð
|
|||
|
|
Yð
|
|||
|
|
Yrv c ó •— |€t j d|› �¦ « d S t ¦ « |¦ « t j |t ¦ « }t j |¦ « r t j
|
|||
|
|
| j |¦ « | _ | j
|
|||
|
|
r_t j |t ¦ « }t j |¦ « r t j |¦ « | _ d S d S d S )NzMissing Checkpoint )r: Ú warning_oncer7 Ú _load_optimizer_and_schedulerr? r r r! Úisfiler" Úload_from_jsonr� r rÅ r r8 Úloadr¶ )r9 Ú
|
|||
|
|
checkpointÚ running_fileÚclf_filer; s €rt r
|
|||
|
|
z0_UnslothBCOTrainer._load_optimizer_and_scheduler¿ sä ø€ Ø Ð Ý Ô Ð B°jÐ BÐ BÑ CÔ CÐ CØ ˆFå
|
|||
|
|
‰Œ×-Ò-¨jÑ9Ô9Ð9õ ”w—|’| Jµ Ñ=Ô=ˆ Ý
|
|||
|
|
Œ7�>Š>˜,Ñ 'Ô 'ð YÝ)Ô8¸Ô9IÈ<ÑXÔXˆDŒLà Ô -ð 1Ý”w—|’| JµÑ9Ô9ˆHÝŒw�~Š~˜hÑ'Ô'ð
|
|||
|
|
1Ý!œ; xÑ0Ô0�”��ð 1ð 1ð
|
|||
|
|
1ð
|
|||
|
|
1rv c # óz K — | j r8| j s1| j | j ¦ « ¦ « n
|
|||
|
|
t
|
|||
|
|
¦ « 5 | j r | j | j ¦ « dV — | j r!| j | j pd¦ « ddd¦ « dS # 1 swxY w Y dS )zWContext manager for handling null reference model (that is, peft adapter manipulation).Nrz ) r˜ rW r� Ú unwrap_modelrI Údisable_adapterr> Ú set_adapterrV rÄ s rt Únull_ref_contextz#_UnslothBCOTrainer.null_ref_contextÐ s è è € ð
|
|||
|
|
Ô!ð
|
|||
|
|
Ø*.Ô*?ð
|
|||
|
|
ˆDÔ × )Ò )¨$¬*Ñ 5Ô 5× EÒ EÑ GÔ GÐ Gå ‘ ” ð Mð Mð
|
|||
|
|
Ô$ð
|
|||
|
|
>Ø”
|
|||
|
|
×&Ò& tÔ'<Ñ=Ô=Ð=Ø ˆEˆEˆEØÔ$ð
|
|||
|
|
MØ”
|
|||
|
|
×&Ò& tÔ'>Ð'KÀ)ÑLÔLÐLð Mð Mð Mñ Mô Mð Mð Mð Mð Mð Mð Mð Møøøð Mð Mð Mð Mð Mð Ms ÁAB0Â0B4Â7B4c ó¢ •— | j �r'| j �s | j j | j | j j | j j ddœ}| j t | j
|
|||
|
|
fi |¤Ž¦ « }g }t |d¬¦ « D ]X}| |¦ « }| j
|
|||
|
|
|¦ « }| | ¦ « ¦ « ŒY| j
|
|||
|
|
dt# j |¦ « ¦ « ¦ « ¬¦ « | _
|
|||
|
|
d| _ t+ ¦ « ¦ « S )z·
|
|||
|
|
Returns the training [`~torch.utils.data.DataLoader`].
|
|||
|
|
|
|||
|
|
Subclass of transformers.src.transformers.trainer.get_train_dataloader to precompute `ref_log_probs`.
|
|||
|
|
Frë z!Train dataset reference log probsrñ Úreference_logps©ÚnameÚcolumnT)r+ rœ rK r¨ rO rß rú r� rö r rL rI Úcompute_reference_log_probsrø ri r³ Ú
|
|||
|
|
add_columnrH r¯ r´ rµ r7 Úget_train_dataloader)r9 rü rý Úreference_completion_logpsrÿ Úreference_completion_logpr; s €rt r z'_UnslothBCOTrainer.get_train_dataloaderÞ sQ ø€ ð Ô (ñ 9°Ô1Vñ 9à "œiÔ CØ "Ô 0Ø #œyÔ ?Ø "œiÔ =Ø ð !ð !Ð ð Ô*×2Ò2µ:¸dÔ>PÐ3fÐ3fÐTeÐ3fÐ3fÑgÔgˆKØ)+Ð &å $¨kÐ@cÐ dÑ dÔ dð
|
|||
|
|
Sð
|
|||
|
|
S� Ø,0×,LÒ,LÈ\Ñ,ZÔ,ZÐ)à,0Ô,<×,OÒ,OÐPiÑ,jÔ,jÐ)Ø*×1Ò1Ð2K×2OÒ2OÑ2QÔ2QÑRÔRÐRÐRà!%Ô!3×!>Ò!>Ø&u¬yÐ9SÑ/TÔ/T×/ZÒ/ZÑ/\Ô/\×/bÒ/bÑ/dÔ/dð "?ñ "ô "ˆDÔ ð 59ˆDÔ 1å‰wŒw×+Ò+Ñ-Ô-Ð-rv c óê •— |€| j €t d¦ « ‚|�|n| j }| j �r&| j �s | j j | j | j j | j j ddœ}| j
|
|||
|
|
t |fi |¤Ž¦ « }g }t |d¬¦ « D ]X}|
|
|||
|
|
|¦ « }| j |¦ « }| | ¦ « ¦ « ŒY| dt% j |¦ « ¦ « ¦ « ¬¦ « }| j �|| _ d | _ t- ¦ « |¬
|
|||
|
|
¦ « S ) aé
|
|||
|
|
Returns the evaluation [`~torch.utils.data.DataLoader`].
|
|||
|
|
|
|||
|
|
Subclass of transformers.src.transformers.trainer.get_eval_dataloader to precompute `ref_log_probs`.
|
|||
|
|
|
|||
|
|
Args:
|
|||
|
|
eval_dataset (`torch.utils.data.Dataset`, *optional*):
|
|||
|
|
If provided, will override `self.eval_dataset`. If it is a [`~datasets.Dataset`], columns not accepted
|
|||
|
|
by the `model.forward()` method are automatically removed. It must implement `__len__`.
|
|||
|
|
Nz-Trainer: evaluation requires an eval_dataset.Frë z Eval dataset reference log probsrñ r r T)rM )rM r‰ r+ r� rK r© rO rß rú r� rö r rI r rø ri r³ r rH r¯ r´ rµ r7 Úget_eval_dataloader)r9 rM rü rý r rÿ r r; s €rt r! z&_UnslothBCOTrainer.get_eval_dataloader sŒ ø€ ð Ð DÔ$5Ð$=Ý Ð LÑMÔMÐ MØ'3Ð'?�|�|ÀTÔEVˆ à Ô (ñ 8°Ô1Uñ 8à "œiÔ BØ "Ô 0Ø #œyÔ ?Ø "œiÔ =Ø ð !ð !Ð ð Ô*×2Ò2µ:¸lÐ3`Ð3`ÐN_Ð3`Ð3`ÑaÔaˆKà)+Ð &å $¨kÐ@bÐ cÑ cÔ cð
|
|||
|
|
Sð
|
|||
|
|
S� Ø,0×,LÒ,LÈ\Ñ,ZÔ,ZÐ)à,0Ô,<×,OÒ,OÐPiÑ,jÔ,jÐ)Ø*×1Ò1Ð2K×2OÒ2OÑ2QÔ2QÑRÔRÐRÐRà'×2Ò2Ø&u¬yÐ9SÑ/TÔ/T×/ZÒ/ZÑ/\Ô/\×/bÒ/bÑ/dÔ/dð 3ñ ô ˆLð
|
|||
|
|
Ô Ð,Ø$0�Ô!Ø37ˆDÔ 0å‰wŒw×*Ò*¸ Ð*ÑEÔEÐErv rÿ c óä — t j ¦ « 5 | j € | ¦ « 5 | j rD| |d |d | d¦ « |d ¬¦ « j }n(| |d |d ¬ ¦ « j }ddd¦ « n # 1 swxY w Y ns| j rD| |d |d | d¦ « |d ¬¦ « j }n(| |d |d ¬ ¦ « j }ddd¦ « n # 1 swxY w Y | ||d d
|
|||
|
|
| j | j ¬ ¦ « }|S ) zfComputes log probabilities of the reference model for a single padded batch of a BCO specific dataset.NÚprompt_input_idsÚprompt_attention_maskÚ completion_decoder_input_idsÚcompletion_labels)rÚ Údecoder_input_idsrÀ Úcompletion_input_idsÚcompletion_attention_mask)rÚ F©Úaverage_log_probr* r% )
|
|||
|
|
rH rÞ rJ r r* rI rŒ rk Úget_batch_logpsr% )r9 rÿ Úcompletion_logitsÚcompletion_logpss rt r z._UnslothBCOTrainer.compute_reference_log_probs. s0 € å
|
|||
|
|
Œ]‰_Œ_ð ð ØŒ~Ð%Ø×*Ò*Ñ,Ô,ð
|
|||
|
|
!ð
|
|||
|
|
!ØÔ.ð !Ø,0¯JªJØ (Ð);Ô <Ø+7Ð8OÔ+PØ.:×.>Ò.>Ð?]Ñ.^Ô.^Ø#/Ð0CÔ#Dð -7ñ -ô -ô
|
|||
|
|
!ð *Ð)ð -1¯JªJØ (Ð)?Ô @Ø+7Ð8SÔ+Tð -7ñ -ô -ô !ð *ð
|
|||
|
|
!ð
|
|||
|
|
!ð
|
|||
|
|
!ñ
|
|||
|
|
!ô
|
|||
|
|
!ð
|
|||
|
|
!ð
|
|||
|
|
!ð
|
|||
|
|
!ð
|
|||
|
|
!ð
|
|||
|
|
!ð
|
|||
|
|
!øøøð
|
|||
|
|
!ð
|
|||
|
|
!ð
|
|||
|
|
!ð
|
|||
|
|
!øð Ô*ð Ø(,¯ªØ$Ð%7Ô8Ø'3Ð4KÔ'LØ*6×*:Ò*:Ð;YÑ*ZÔ*ZØ +Ð,?Ô @ð )7ñ )ô )ô
|
|||
|
|
ð &Ð%ð )-¯ªØ$Ð%;Ô<È\ÐZuÔMvð )7ñ )ô )ä ð &ð7 ð ð ñ ô ð ð ð ð ð ð øøøð ð ð ð ð> ×/Ò/Ø Ø Ð,Ô -Ø "Ø #Ô 6Ø #Ô 6ð 0ñ
|
|||
|
|
ô
|
|||
|
|
Ðð Ð s6 ” D;°A4B0Â$ D;Â0B4 Â4D;Â7B4 Â8A7D;Ä;D?ÅD?Frš rk rÀ r+ r% r* c ó® — | j dd… |j k rt d¦ « ‚|s2|dd…dd…f ¦ « }| dd…dd…dd…f } n| ¦ « }||k }d|||k < t | |¦ « }|r.||z d¦ « | d¦ « z S ||z d¦ « S )a Compute the log probabilities of the given labels under the given logits.
|
|||
|
|
|
|||
|
|
Args:
|
|||
|
|
logits: Logits of the model (unnormalized). Shape: (batch_size, sequence_length, vocab_size)
|
|||
|
|
labels:
|
|||
|
|
Labels for which to compute the log probabilities. Label tokens with a value of label_pad_token_id are
|
|||
|
|
ignored. Shape: (batch_size, sequence_length)
|
|||
|
|
average_log_prob:
|
|||
|
|
If True, return the average log probability per (non-masked) token. Otherwise, return the sum of the
|
|||
|
|
log probabilities of the (non-masked) tokens.
|
|||
|
|
|
|||
|
|
Returns:
|
|||
|
|
A tensor of shape (batch_size,) containing the average/sum log probabilities of the given labels under the
|
|||
|
|
given logits.
|
|||
|
|
NrX zKLogits (batch and sequence length dim) and labels must have the same shape.r^ r )ra r‰ ÚclonerF Úsum)rk rÀ r+ r% r* Ú loss_maskrs s rt r, z"_UnslothBCOTrainer.get_batch_logpsY sø € ð. Œ<˜˜˜Ô ¤ Ò ,Ð ,Ý Ð jÑkÔkÐ kà!ð $ؘA˜A˜A˜q˜r˜r˜E”]×(Ò(Ñ*Ô*ˆFؘA˜A˜A˜s ˜s A A A˜IÔ&ˆFˆFð —\’\‘^”^ˆFàÐ 0Ò0ˆ ð 01ˆˆvÐ+Ò+Ñ,å/°¸Ñ?Ô?ˆà ð 9Ø# iÑ/×4Ò4°RÑ8Ô8¸9¿=º=ÈÑ;LÔ;LÑLÐ Là# iÑ/×4Ò4°RÑ8Ô8Ð 8rv c óz ‡— | j r ‰d ‰ d¦ « dœni }| j rd|d< |‰d fd‰d i|¤Ž}|j }| |‰d d | j | j ¬
|
|||
|
|
¦ « }|j d t ‰d ¦ « k rt d
|
|||
|
|
¦ « ‚ˆfd„t |j d ¦ « D ¦ « }ˆfd„t |j d ¦ « D ¦ « }||df } ||df }
|
|||
|
|
||df } ||df } | j r | |
|
|||
|
|
| | |j
|
|||
|
|
fS | |
|
|||
|
|
| | fS )Nr& r% )rÀ r' Trq r( rÚ r) Fr* r r| z‡There is a mismatch between the number of examples in this batch and the number of examples for which an output sequence was predicted.c ó4 •— g | ]}‰d | du ¯|‘ŒS ©r| Tr2 ©Ú.0Úirà s €rt ú
|
|||
|
|
<listcomp>z._UnslothBCOTrainer.forward.<locals>.<listcomp>© s. ø€ Ð_Ð_Ð_˜AÀUÈ7Ä^ÐTUÔEVÐZ^ÐE^ÐE^�aÐE^ÐE^ÐE^rv c ó4 •— g | ]}‰d | du ¯|‘ŒS ©r| Fr2 r6 s €rt r9 z._UnslothBCOTrainer.forward.<locals>.<listcomp>ª s. ø€ ÐbÐbÐb˜aÀuÈWÄ~ÐVWÔGXÐ\aÐGaÐGa˜ÐGaÐGaÐGarv .) r* rŒ rŸ rk r, r% ra ró r‰ ÚrangeÚaux_loss)
|
|||
|
|
r9 rI rà Ú model_kwargsÚoutputsr- r. rå ræ Ú chosen_logpsÚrejected_logpsÚ
|
|||
|
|
chosen_logitsÚrejected_logitss
|
|||
|
|
` rt Úforwardz_UnslothBCOTrainer.forward† sË ø€ ð Ô&ð
|
|||
|
|
Ø Ð 3Ô4Ø%*§Y¢YÐ/MÑ%NÔ%Nð
|
|||
|
|
ð
|
|||
|
|
ð
|
|||
|
|
ð
|
|||
|
|
ð
|
|||
|
|
ð Ô ð 8Ø37ˆLÐ/Ñ 0à�%Ø Ð(Ô )ð
|
|||
|
|
ð
|
|||
|
|
à Ð!<Ô=ð
|
|||
|
|
ð ð
|
|||
|
|
ð
|
|||
|
|
ˆð
|