306 lines
58 KiB
Plaintext
306 lines
58 KiB
Plaintext
|
§
|
|||
|
|
5$�hd÷ ã ót — d Z ddlmZ ddlZddlmZ ddlmZ ddlmZm Z m
|
|||
|
|
Z
|
|||
|
|
m Z m Z m
|
|||
|
|
Z
|
|||
|
|
mZmZ ddlmZmZmZmZmZmZmZmZmZmZmZm
|
|||
|
|
Z
|
|||
|
|
mZmZm Z m Z m Z m Z m Z m!Z!m"Z"m#Z#m$Z$m Z m%Z%m&Z&m'Z'm(Z(m)Z)m*Z*m+Z+m,Z,m-Z-m.Z.m/Z/m0Z0mZm1Z1m2Z2m3Z3m4Z4m5Z5mZm6Z6m7Z7mZmZmZmZmZm
|
|||
|
|
Z
|
|||
|
|
m Z m1Z1m2Z2m
|
|||
|
|
Z
|
|||
|
|
m Z m Z m"Z"m/Z/m1Z1m3Z3mZm1Z1 ddl1Z1ddlT ddl(m'Z'm8Z8 dd l9m:Z: ddlZddl;Z<dd
|
|||
|
|
l&m=Z= ddlmZ dd l>m?Z?mZ@ d d
|
|||
|
|
d d
|
|||
|
|
d
|
|||
|
|
dœZA ejB d d eA¬¦ « d„ ¦ « ZCe' G d„ de ¦ « ¦ « ZD G d„ de"¦ « ZE G d„ deE¦ « ZFdS )z8
|
|||
|
|
2025.8.4
|
|||
|
|
2025.8.5
|
|||
|
|
4.55.1
|
|||
|
|
0.21.0
|
|||
|
|
__UNSLOTH_VERSIONING__
|
|||
|
|
é )ÚTensorN)Ú
|
|||
|
|
functional)ÚAnyÚListÚOptionalÚTupleÚUnionÚDictÚSetÚCallable)?r ÚAutoModelForCausalLMÚ
|
|||
|
|
AutoTokenizerÚBaseImageProcessorr Ú DataCollatorÚ DataCollatorForLanguageModelingÚDatasetÚEvalPredictionÚFeatureExtractionMixinÚIterableDatasetr ÚPathÚ
|
|||
|
|
PeftConfigÚ PeftModelÚPreTrainedModelÚPreTrainedTokenizerBaseÚProcessorMixinÚ SFTConfigÚ
|
|||
|
|
SFTTrainerÚTrainerÚTrainerCallbackÚTrainingArgumentsr Úclone_chat_templateÚ
|
|||
|
|
contextlibÚ dataclassÚ dataclassesÚ defaultdictÚgenerate_model_cardÚ get_act_offloading_ctx_managerÚget_comet_experiment_urlÚget_peft_modelÚis_conversationalÚis_peft_availableÚis_wandb_availableÚnnÚosÚpadÚpeftÚpeft_module_casting_to_bf16Ú prepare_model_for_kbit_trainingÚtorchÚversionÚwarningsr r r r r r r r. r/ r r r r r+ r. r0 r3 r. )Ú*)r# Úfield)ÚVersion)Ú nullcontext©ÚDataCollatorForSeq2Seqr TF)Úepilogue_fusionÚ max_autotuneÚ
|
|||
|
|
shape_paddingz
|
|||
|
|
trace.enabledztriton.cudagraphs)ÚdynamicÚ fullgraphÚoptionsc ó’ — t j | d| j d ¦ « dd¬¦ « }t j | d¦ « dd¬¦ « }g }t ||¦ « D ]‘\ }}| t j ¦ « }t j |d| d¦ « ¬¦ « d¦ « }t j
|
|||
|
|
|d¬¦ « }||z
|
|||
|
|
} | | ¦ « Œ’ t j |¦ « }| | j d | j d f¦ « }|S )Néÿÿÿÿé r )ÚchunksÚdim)rF Úindex)rF é )
|
|||
|
|
r3 ÚchunkÚreshapeÚshapeÚzipÚtoÚfloat32ÚgatherÚ unsqueezeÚsqueezeÚ logsumexpÚappendÚconcat)
|
|||
|
|
ÚlogitsrG Úchunked_logitsÚ
|
|||
|
|
chunked_indexÚall_per_token_logpsÚ chunk_logitsÚ chunk_indexÚselected_logitsÚlogsumexp_valuesÚper_token_logpss
|
|||
|
|
ú]/workspace/Fine-tuning/DS-LLM-TEMPLATE-FINETUNING/unsloth_compiled_cache/UnslothSFTTrainer.pyÚ chunked_selective_log_softmaxr_ " s5 € õ ”[ § ¢ °°F´LÀÔ4DÑ!EÔ!EÐPQÐYZÐ[Ñ[Ô[€NÝ”[ § ¢ ¨rÑ!2Ô!2¸QÀaÐHÑHÔH€MØ Ðå%(¨ ¸ Ñ%GÔ%Gð 4ð 4Ñ!ˆ �kØ#—’¥u¤}Ñ5Ô5ˆ Ý œ, |¸2À{×G\ÒG\Ð]_ÑG`ÔG`ÐaÑaÔa×iÒiÐjlÑmÔmˆÝ œ?¨<¸rÐBÑBÔBÐØ)Ð,<Ñ<ˆØ×"Ò" ?Ñ3Ô3Ð3Ð3ØÝ œ,Ð':Ñ;Ô;ÐØ-×5Ò5°v´|ÀA´ÈÌ ÐUVÌÐ6XÑYÔYÐØ Ð ó c ó² ‡ — e Zd ZU dZ edddi¬¦ « Zee ed< edddi¬¦ « Z ee
|
|||
|
|
ed < d-ˆ fd,„ Z ˆ xZ S ).ÚUnslothSFTConfiga5
|
|||
|
|
|
|||
|
|
Configuration class for the [`SFTTrainer`].
|
|||
|
|
|
|||
|
|
This class includes only the parameters that are specific to SFT training. For a full list of training arguments,
|
|||
|
|
please refer to the [`~transformers.TrainingArguments`] documentation. Note that default values in this class may
|
|||
|
|
differ from those in [`~transformers.TrainingArguments`].
|
|||
|
|
|
|||
|
|
Using [`~transformers.HfArgumentParser`] we can turn this class into
|
|||
|
|
[argparse](https://docs.python.org/3/library/argparse#module-argparse) arguments that can be specified on the
|
|||
|
|
command line.
|
|||
|
|
|
|||
|
|
Parameters:
|
|||
|
|
> Parameters that control the model
|
|||
|
|
|
|||
|
|
model_init_kwargs (`dict[str, Any]` or `None`, *optional*, defaults to `None`):
|
|||
|
|
Keyword arguments for [`~transformers.AutoModelForCausalLM.from_pretrained`], used when the `model`
|
|||
|
|
argument of the [`SFTTrainer`] is provided as a string.
|
|||
|
|
chat_template_path (`str` or `None`, *optional*, defaults to `None`):
|
|||
|
|
If specified, sets the model's chat template. This can either be the path to a tokenizer (local directory
|
|||
|
|
or Hugging Face Hub model) or a direct path to a Jinja template file. When using a Jinja file, you must
|
|||
|
|
ensure that any special tokens referenced in the template are added to the tokenizer and that the model's
|
|||
|
|
embedding layer is resized accordingly.
|
|||
|
|
|
|||
|
|
> Parameters that control the data preprocessing
|
|||
|
|
|
|||
|
|
dataset_text_field (`str`, *optional*, defaults to `"text"`):
|
|||
|
|
Name of the column that contains text data in the dataset.
|
|||
|
|
dataset_kwargs (`dict[str, Any]` or `None`, *optional*, defaults to `None`):
|
|||
|
|
Dictionary of optional keyword arguments for the dataset preparation. The only supported key is
|
|||
|
|
`skip_prepare_dataset`.
|
|||
|
|
dataset_num_proc (`int` or `None`, *optional*, defaults to `None`):
|
|||
|
|
Number of processes to use for processing the dataset.
|
|||
|
|
eos_token (`str` or `None`, *optional*, defaults to `None`):
|
|||
|
|
Token used to indicate the end of a turn or sequence. If `None`, it defaults to
|
|||
|
|
`processing_class.eos_token`.
|
|||
|
|
pad_token (`int` or `None`, *optional*, defaults to `None`):
|
|||
|
|
Token used for padding. If `None`, it defaults to `processing_class.pad_token`, or if that is also `None`,
|
|||
|
|
it falls back to `processing_class.eos_token`.
|
|||
|
|
max_length (`int` or `None`, *optional*, defaults to `1024`):
|
|||
|
|
Maximum length of the tokenized sequence. Sequences longer than `max_length` are truncated from the right.
|
|||
|
|
If `None`, no truncation is applied. When packing is enabled, this value sets the sequence length.
|
|||
|
|
packing (`bool`, *optional*, defaults to `False`):
|
|||
|
|
Whether to group multiple sequences into fixed-length blocks to improve computational efficiency and reduce
|
|||
|
|
padding. Uses `max_length` to define sequence length.
|
|||
|
|
packing_strategy (`str`, *optional*, defaults to `"bfd"`):
|
|||
|
|
Strategy for packing sequences. Can be either `"bfd"` (best-fit decreasing, default), or `"wrapped"`.
|
|||
|
|
padding_free (`bool`, *optional*, defaults to `False`):
|
|||
|
|
Whether to perform forward passes without padding by flattening all sequences in the batch into a single
|
|||
|
|
continuous sequence. This reduces memory usage by eliminating padding overhead. Currently, this is only
|
|||
|
|
supported with the FlashAttention 2 or 3, which can efficiently handle the flattened batch structure. When
|
|||
|
|
packing is enabled with strategy `"bfd"`, padding-free is enabled, regardless of the value of this
|
|||
|
|
parameter.
|
|||
|
|
pad_to_multiple_of (`int` or `None`, *optional*, defaults to `None`):
|
|||
|
|
If set, the sequences will be padded to a multiple of this value.
|
|||
|
|
eval_packing (`bool` or `None`, *optional*, defaults to `None`):
|
|||
|
|
Whether to pack the eval dataset. If `None`, uses the same value as `packing`.
|
|||
|
|
|
|||
|
|
> Parameters that control the training
|
|||
|
|
|
|||
|
|
completion_only_loss (`bool` or `None`, *optional*, defaults to `None`):
|
|||
|
|
Whether to compute loss only on the completion part of the sequence. If set to `True`, loss is computed
|
|||
|
|
only on the completion, which is supported only for [prompt-completion](#prompt-completion) datasets. If
|
|||
|
|
`False`, loss is computed on the entire sequence. If `None` (default), the behavior depends on the dataset:
|
|||
|
|
loss is computed on the completion for [prompt-completion](#prompt-completion) datasets, and on the full
|
|||
|
|
sequence for [language modeling](#language-modeling) datasets.
|
|||
|
|
assistant_only_loss (`bool`, *optional*, defaults to `False`):
|
|||
|
|
Whether to compute loss only on the assistant part of the sequence. If set to `True`, loss is computed
|
|||
|
|
only on the assistant responses, which is supported only for [conversational](#conversational) datasets. If `False`,
|
|||
|
|
loss is computed on the entire sequence.
|
|||
|
|
activation_offloading (`bool`, *optional*, defaults to `False`):
|
|||
|
|
Whether to offload the activations to the CPU.
|
|||
|
|
|
|||
|
|
NÚhelpzvLLM SamplingParams)ÚdefaultÚmetadataÚvllm_sampling_paramsrC z8Chunk size to reduce memory usage. -1 is most efficient.Úunsloth_num_chunksFÚnorD é r éú ç-C ëâ6
|
|||
|
|
?ç{®Gáz„?çÍÌÌÌÌÌì?ç+‡ÙÎ÷ï?ç:Œ0âŽyE>ç ð?ç @Úlinearçš™™™™™¹?ÚpassiveÚwarningTÚstepsrH éô éO
|
|||
|
|
ÚO1ÚautoÚ ç Ú
|
|||
|
|
adamw_8bitÚlengthÚ
|
|||
|
|
every_saveÚlasté Útexté Úbfdc“ ó„ •— |dk rt d|› d�¦ « ‚|dk rt d|› d�¦ « ‚|€|#dk r
|
|||
|
|
|$dk rd}d }#|…€!d
|
|||
|
|
d lm}” t |”¦ « d z d ¦ « }… t ¦ « j d�i d
|
|||
|
|
|“d|“d|“d|“d|“d|“d|“d|“d| “d|
|
|||
|
|
“d| “d| “d|
|
|||
|
|
“d|“d|“d |“d |“d |“d |“d |“d!|“d"|“d#|“d$|“d%|“d&|“d'|“d(| “d)| “d*| “d+| “d,| “d-|!“d.|"“d/|#“d0|$“d1|%“d2|&“d3|'“d4|(“d5|)“d6|*“d7|+“d8|,“d9|-“d:|.“d;|/“d<|0“d=|1“d>|2“d?|3“d@|4“dA|5“dB|6“dC|7“dD|8“dE|9“dF|:“dG|;“dH|<“dI|=“dJ|>“dK|?“dL|@“dM|A“dN|B“dO|C“dP|D“dQ|E“dR|F“dS|G“dT|H“dU|I“dV|J“dW|K“dX|L“dY|M“dZ|N“d[|O“d\|P“d]|Q“d^|R“d_|S“d`|T“da|U“db|V“dc|W“dd|X“de|Y“df|Z“dg|[“dh|\“di|]“dj|^“dk|_“dl|`“dm|a“dn|b“do|c“dp|d“dq|e“dr|f“ds|g“dt|h“du|i“dv|j“dw|k“dx|l“dy|m“dz|n“d{|o“d||p“d}|q“d~|r“d|s“d€|t“d�|u“d‚|v“dƒ|w“d„|x“d…|y“d†|z“d‡|{“dˆ||“d‰|}“dŠ|~“d‹|“dŒ|€“d�|�“dŽ|‚“d�|ƒ“d�|„“d‘|…“d’|†“d“|‡“d”|ˆ“d•|‰“d–|Š“d—|‹“d˜|Œ“d™|�“dš|Ž“d›|�“dœ|�“|“¤Ž |‘| _ |’| _ d S )žNgH¯¼šò×z>z Unsloth: Your learning rate of `zi` is too small and less than 1e-7! Consider increasing it, otherwise gradient updates will be close to 0!rH za` is way too larger > 1! Consider decreasing it to 1e-1, otherwise gradient updates will explode!rv rw Ú unsloth_training_checkpointsrh r )Ú cpu_countri Ú
|
|||
|
|
output_dirÚoverwrite_output_dirÚdo_trainÚdo_evalÚ
|
|||
|
|
do_predictÚ
|
|||
|
|
eval_strategyÚprediction_loss_onlyÚper_device_train_batch_sizeÚper_device_eval_batch_sizeÚper_gpu_train_batch_sizeÚper_gpu_eval_batch_sizeÚgradient_accumulation_stepsÚeval_accumulation_stepsÚ
|
|||
|
|
eval_delayÚtorch_empty_cache_stepsÚ
|
|||
|
|
learning_rateÚ weight_decayÚ
|
|||
|
|
adam_beta1Ú
|
|||
|
|
adam_beta2Ú adam_epsilonÚ
|
|||
|
|
max_grad_normÚnum_train_epochsÚ max_stepsÚlr_scheduler_typeÚ warmup_ratioÚ warmup_stepsÚ log_levelÚlog_level_replicaÚlog_on_each_nodeÚ logging_dirÚlogging_strategyÚlogging_first_stepÚ
|
|||
|
|
logging_stepsÚlogging_nan_inf_filterÚ
|
|||
|
|
save_strategyÚ
|
|||
|
|
save_stepsÚsave_total_limitÚsave_safetensorsÚsave_on_each_nodeÚsave_only_modelÚ'restore_callback_states_from_checkpointÚno_cudaÚuse_cpuÚuse_mps_deviceÚseedÚ data_seedÚ
|
|||
|
|
jit_mode_evalÚuse_ipexÚbf16Úfp16Úfp16_opt_levelÚhalf_precision_backendÚbf16_full_evalÚfp16_full_evalÚtf32Ú
|
|||
|
|
local_rankÚ ddp_backendÚ
|
|||
|
|
tpu_num_coresÚtpu_metrics_debugÚdebugÚdataloader_drop_lastÚ
|
|||
|
|
eval_stepsÚdataloader_num_workersÚdataloader_prefetch_factorÚ
|
|||
|
|
past_indexÚrun_nameÚ disable_tqdmÚremove_unused_columnsÚ label_namesÚload_best_model_at_endÚmetric_for_best_modelÚgreater_is_betterÚignore_data_skipÚfsdpÚfsdp_min_num_paramsÚ fsdp_configÚ"fsdp_transformer_layer_cls_to_wrapÚaccelerator_configÚ deepspeedÚlabel_smoothing_factorÚoptimÚ
|
|||
|
|
optim_argsÚ adafactorÚgroup_by_lengthÚlength_column_nameÚ report_toÚddp_find_unused_parametersÚddp_bucket_cap_mbÚddp_broadcast_buffersÚdataloader_pin_memoryÚ dataloader_persistent_workersÚskip_memory_metricsÚuse_legacy_prediction_loopÚ push_to_hubÚresume_from_checkpointÚ hub_model_idÚ hub_strategyÚ hub_tokenÚhub_private_repoÚhub_always_pushÚ hub_revisionÚgradient_checkpointingÚ gradient_checkpointing_kwargsÚinclude_inputs_for_metricsÚeval_do_concat_batchesÚ fp16_backendÚpush_to_hub_model_idÚpush_to_hub_organizationÚpush_to_hub_tokenÚ
|
|||
|
|
mp_parametersÚauto_find_batch_sizeÚfull_determinismÚ torchdynamoÚ ray_scopeÚ ddp_timeoutÚ
|
|||
|
|
torch_compileÚtorch_compile_backendÚtorch_compile_modeÚinclude_tokens_per_secondÚ include_num_input_tokens_seenÚneftune_noise_alphaÚoptim_target_modulesÚbatch_eval_metricsÚ
|
|||
|
|
eval_on_startÚuse_liger_kernelÚliger_kernel_configÚeval_use_gather_objectÚ average_tokens_across_devicesÚmodel_init_kwargsÚchat_template_pathÚdataset_text_fieldÚdataset_kwargsÚdataset_num_procÚ eos_tokenÚ pad_tokenÚ
|
|||
|
|
max_lengthÚpackingÚpacking_strategyÚ padding_freeÚpad_to_multiple_ofÚ eval_packingÚcompletion_only_lossÚassistant_only_lossÚactivation_offloading© ) ÚFloatingPointErrorÚ
|
|||
|
|
OverflowErrorÚmultiprocessingr‡ ÚminÚsuperÚ__init__rf rg )–Úselfrˆ r‰ rŠ r‹ rŒ r� rŽ r� r� r‘ r’ r“ r” r• r– r— r˜ r™ rš r› rœ r� rž rŸ r r¡ r¢ r£ r¤ r¥ r¦ r§ r¨ r© rª r« r¬ r r® r¯ r° r± r² r³ r´ rµ r¶ r· r¸ r¹ rº r» r¼ r½ r¾ r¿ rÀ rÁ r rà rÄ rÅ rÆ rÇ rÈ rÉ rÊ rË rÌ rÍ rÎ rÏ rÐ rÑ rÒ rÓ rÔ rÕ rÖ r× rØ rÙ rÚ rÛ rÜ rÝ rÞ rß rà rá râ rã rä rå ræ rç rè ré rê rë rì rí rî rï rð rñ rò ró rô rõ rö r÷ rø rù rú rû rü rý rþ rÿ r r r r r r r r r r r
|
|||
|
|
r r r
|
|||
|
|
r r r r r r r r r r rf rg Úkwargsr‡ Ú __class__s– €r^ r zUnslothSFTConfig.__init__‡ s2
|
|||
|
|
ø€ ðl ˜4Ò Ð Õ'9ð ;VÐ]jð ;Vð ;Vð ;Vñ (Wô (Wð "WØ ˜1Ò Ð ¥Mð 3FÐUbð 3Fð 3Fð 3Fñ %Gô %Gð GØ Ð -°7Ò":Ð":¸zÈSÒ?PÐ?PØ7ˆJØ ˆMØ Ð #Ø 1Ð 1Ð 1Ð 1Ð 1Ð 1Ý " 9 9¡;¤;¨q¡=°!Ñ 4Ô 4Ð à�‰ŒÔð P Dð P Dð P DØ#˜ðP Dà#7Ð#7ðP Dð �xðP Dð �gð P Dð
|
|||
|
|
$˜ð P Dð *˜Mð
|
|||
|
|
P Dð $8Ð#7ðP Dð +FÐ*EðP Dð *DÐ)CðP Dð (@Ð'?ðP Dð '>Ð&=ðP Dð +FÐ*EðP Dð '>Ð&=ðP Dð $˜ð P Dð '>Ð&=ð P Dð *˜Mð!P Dð" (˜<ð#P Dð$ $˜ð%P Dð& $˜ð'P Dð( (˜<ð)P Dð* *˜Mð+P Dð, 0Ð /ð-P Dð. "˜ ð/P Dð0 !2Ð 1ð1P Dð2 (˜<ð3P Dð4 (˜<ð5P Dð6 "˜ ð7P Dð8 !2Ð 1ð9P Dð: 0Ð /ð;P Dð< &˜+ð=P Dð> 0Ð /ð?P Dð@ "4Ð!3ðAP DðB *˜MðCP DðD &<Ð%;ðEP DðF *˜MðGP DðH $˜ðIP DðJ 0Ð /ðKP DðL 0Ð /ðMP DðN !2Ð 1ðOP DðP .˜oðQP DðR 7^Ð6]ðSP DðT �gðUP DðV �gðWP DðX ,˜^ðYP DðZ �4ð[P Dð\ "˜ ð]P Dð^ *˜Mð_P Dð` �xðaP Dðb �4ðcP Dðd �4ðeP Dðf ,˜^ðgP Dðh &<Ð%;ðiP Dðj ,˜^ðkP Dðl ,˜^ðmP Dðn �4ðoP Dðp $˜ðqP Dðr &˜+ðsP Dðt *˜MðuP Dðv !2Ð 1ðwP Dðx �EðyP Dðz $8Ð#7ð{P Dð| $˜ð}P Dð~ &<Ð%;ðP Dð@ *DÐ)CðAP DðB $˜ðCP DðD �xðEP DðF (˜<ðGP DðH %:Ð$9ðIP DðJ &˜+ðKP DðL &<Ð%;ðMP DðN %:Ð$9ðOP DðP !2Ð 1ðQP DðR 0Ð /ðSP DðT �4ðUP DðV #6Ð"5ðWP DðX &˜+ðYP DðZ 2TÐ1Sð[P Dð\ "4Ð!3ð]P Dð^ "˜ ð_P Dð` &<Ð%;ðaP Dðb �EðcP Dðd $˜ðeP Dðf "˜ ðgP Dðh .˜oðiP Dðj "4Ð!3ðkP Dðl "˜ ðmP Dðn *DÐ)CðoP Dðp !2Ð 1ðqP Dðr %:Ð$9ðsP Dðt %:Ð$9ðuP Dðv -JÐ,IðwP Dðx #6Ð"5ðyP Dðz *DÐ)Cð{P Dð| &˜+ð}P Dð~ &<Ð%;ðP Dð@ (˜<ðAP DðB (˜<ðCP DðD "˜ ðEP DðF 0Ð /ðGP DðH .˜oðIP DðJ (˜<ðKP DðL &<Ð%;ðMP DðN -JÐ,IðOP DðP *DÐ)CðQP DðR &<Ð%;ðSP DðT (˜<ðUP DðV $8Ð#7ðWP DðX (@Ð'?ðYP DðZ !2Ð 1ð[P Dð\ *˜Mð]P Dð^ $8Ð#7ð_P Dð` 0Ð /ðaP Dðb &˜+ðcP Dðd "˜ ðeP Dðf &˜+ðgP Dðh *˜MðiP Dðj %:Ð$9ðkP Dðl "4Ð!3ðmP Dðn )BÐ(AðoP Dðp -JÐ,IðqP Dðr #6Ð"5ðsP Dðt $8Ð#7ðuP Dðv "4Ð!3ðwP Dðx *˜MðyP Dðz 0Ð /ð{P Dð| #6Ð"5ð}P Dð~ &<Ð%;ðP Dð@ -JÐ,IðAP DðB !2Ð 1ðCP DðD "4Ð!3ðEP DðF "4Ð!3ðGP DðH ,˜^ðIP DðJ 0Ð /ðKP DðL "˜ ðMP DðN "˜ ðOP DðP $˜ðQP DðR �gðSP DðT 0Ð /ðUP DðV (˜<ðWP DðX "4Ð!3ðYP DðZ (˜<ð[P Dð\ $8Ð#7ð]P Dð^ #6Ð"5ð_P Dð` %:Ð$9¸FðaP Dð P Dð P Dðb %9ˆÔ!Ø"4ˆÔ Ð Ð r` )’NNFFFrh FrD rD NNri ri r rj rk rl rm rn ro rp rq rC rr rs r rt ru TNrv FrH Frv rw NTFFFFFFrx rx FFFFry rz FFNrC NNFr{ FNr NrC NNTNFNNFr{ r NNNNr| r} NFFr~ NNNNTFTFFNNr NNFNFNFTrz NNNr{ TFNr€ r� FNNFFNNFFFNFTNNr‚ NNNNrƒ Fr„ FNNNFFNrC )
|
|||
|
|
Ú__name__Ú
|
|||
|
|
__module__Ú __qualname__Ú__doc__r7 rf r r Ú__annotations__rg Úintr Ú
|
|||
|
|
__classcell__©r! s @r^ rb rb 3 sƒ ø€ € € € € € ðIð IðT +0¨%ØØÐ 1Ð2ð+ñ +ô +И( 3œ-ð ð ñ ð */¨ØØÐ VÐWð*ñ *ô *И #œ ð ð ñ ð Ø #ØØØØ Ø $Ø&'Ø%&Ø#'Ø"&Ø&'Ø"#ØØ"%Ø ØØØØ ØØ ØØ $ØØØ Ø %Ø ØØ"Ø "ØØ!&Ø ØØ Ø Ø !Ø Ø27ØØØ ØØØ ØØØØ Ø!'Ø Ø ØØØØ Ø !ØØ $ØØ!"Ø%)ØØØØ $ØØ!&Ø $Ø Ø ØØ ØØ-1Ø !ØØ!$Ø ØØØ Ø %ØØ%)Ø Ø $Ø $Ø(-Ø "Ø%*ØØ!%ØØ#ØØ Ø ØØ!&Ø(,Ø%*Ø!%Ø Ø #Ø#'Ø ØØ #Ø ØØØØ Ø $Ø !Ø$)Ø(-Ø "Ø #Ø "Ø Ø Ø "Ø!&Ø(,Ø Ø !Ø #Ø Ø ØØØØØ Ø Ø !ØØ #Ø #Ø %Ø #Ø ðgq5ð q5ð q5ð q5ð q5ð q5ð q5ð q5ð q5ð q5r` rb c óØ ‡ — e Zd ZdZddgZ d+deeej e f de
|
|||
|
|
ee e f de
|
|||
|
|
e
|
|||
|
|
d e
|
|||
|
|
eeef d
|
|||
|
|
e
|
|||
|
|
eeeeef f d e
|
|||
|
|
eeeeef d e
|
|||
|
|
e d
|
|||
|
|
e
|
|||
|
|
eegef de
|
|||
|
|
ee dee
|
|||
|
|
ej j e
|
|||
|
|
ej j j f de
|
|||
|
|
ee ej j eee f f de
|
|||
|
|
eej! ej! gej! f de
|
|||
|
|
d de
|
|||
|
|
eegef f ˆ fd„
|
|||
|
|
Z"dede de fd„Z#de de de de fd„Z$de de de fd„Z%de de de fd„Z&d eeef d e'de
|
|||
|
|
eegef d edeeef f
|
|||
|
|
d „Z(d „ Z)d,ˆ fd"„ Z*ˆ fd#„Z+d-d$eee,f d%e
|
|||
|
|
e, ddfˆ fd&„
|
|||
|
|
Z-ˆ fd'„Z. d.d(e
|
|||
|
|
e d e
|
|||
|
|
e d)eeee df fd*„Z/ˆ xZ0S )/Ú_UnslothSFTTrainerr{ ÚtrlÚsftN©NNÚmodelÚargsÚ
|
|||
|
|
data_collatorÚ
|
|||
|
|
train_datasetÚ eval_datasetÚprocessing_classÚcompute_loss_funcÚcompute_metricsÚ callbacksÚ
|
|||
|
|
optimizersÚoptimizer_cls_and_kwargsÚ preprocess_logits_for_metricsÚ peft_configr Úformatting_funcc óÚ
|
|||
|
|
•‡ ‡‡‡‡— t |t ¦ « r|n |j j }‰€.| d¦ « d }t |› d�¦ « Šnit ‰t ¦ « rTt ‰t
|
|||
|
|
¦ « s?‰ ¦ « }‰j |d< | d¦ « t d)i |¤ŽŠ‰€t j |¦ « Š‰j �E‰j }‰
|
|||
|
|
|¦ « }|€ t d|› d‰j j › d�¦ « ‚|‰_ ‰j �)t |t ¦ « st' j d ¦ « t |t ¦ « r‰ |‰¦ « }‰j �£t. j ‰j ¦ « rd‰j d
|
|||
|
|
¦ « rJt7 ‰j d ¬ ¦ « 5 }| ¦ « ‰_ d d d ¦ « n # 1 swxY w Y g }n t= |‰‰j ¦ « \ }Š}ng } ‰j# p‰j$ o
|
|||
|
|
‰j% dk ‰ _# |j j& dv }‰ j# rs|�t d¦ « ‚‰j$ r ‰j% dk rt' j d¦ « |st' j d¦ « ‰j' dk r‰j$ st' j d¦ « tQ tS |¦ « ¦ « }‰j* €
|
|||
|
|
d|v ‰ _* n ‰j* ‰ _* |€o‰j+ p
|
|||
|
|
‰j+ p‰j }‰
|
|||
|
|
|¦ « }|€ t d|› d‰j j › d�¦ « ‚tY |‰ j* ‰ j# |‰j- ¬ ¦ « }‰j$ r!‰j% dk r|st' j d ¦ « ‰j. r t_ |¦ « st d ¦ « ‚‰j0 d u p‰j0 1 d d
|
|||
|
|
¦ « }|r¢‰ j* r‰rt d ¦ « ‚‰ 2 |‰‰‰j$ ‰d!¦ « }|�i‰j3 €‰j$ n‰j3 Št |th ¦ « r%ˆˆˆˆˆ fd"„| 5 ¦ « D ¦ « }n‰ 2 |‰‰‰‰d#¦ « }tm tn ¦ « tm tn ¦ « d$œ‰ _8 d%‰ _9 tu ¦ « ; |‰|||‰||| |
|
|||
|
|
| | ¬&¦ « ‰ j< j= rt} ‰ j? ¬'¦ « ‰ _@ ntƒ jB ¦ « ‰ _@ t‡ ‰ j? d(¦ « r!‰ j? D ‰ jE ¦ « d S d S )*Nú/rC z-SFTré rô z The specified `eos_token` ('zC') is not found in the vocabulary of the given `processing_class` (zX). Ensure that the `eos_token` exists in the vocabulary before using it as an EOS token.z�You passed model_init_kwargs to the `SFTConfig`, but your model is already instantiated. The `model_init_kwargs` will be ignored.)z.jinjaz.j2zutf-8)ÚencodingFÚ embed_tokensÚlm_heada- Cloning chat template added new tokens to the tokenizer, but 'lm_head' is not in PEFT's `modules_to_save`. As a result, the model may not learn to generate outputs with these new tokens, leading to degraded generation quality. To fix this, add `modules_to_save=['lm_head']` to your PEFT configuration.r„ )Úflash_attention_2z"kernels-community/vllm-flash-attn3zHPassing a custom data collator is not supported when using padding-free.Úwrappedz¯You are passing `padding_free=True` with the 'wrapped' packing strategy, which is not recommended. Please refer to the documentation to understand why this is not recommended.aç Padding-free training is enabled, but the attention implementation is not set to 'flash_attention_2'. Padding-free training flattens batches into a single sequence, and 'flash_attention_2' is the only known attention mechanism that reliably supports this. Using other implementations may lead to unexpected behavior. To ensure compatibility, set `attn_implementation='flash_attention_2'` in the model configuration, or verify that your attention mechanism can handle flattened sequences.rH zÎYou are using a per_device_train_batch_size of 1 with padding-free training. Using a batch size of 1 anihilate the benefits of padding-free training. Please consider increasing the batch size to at least 2.Úpromptz The specified `pad_token` ('z[). Ensure that the `pad_token` exists in the vocabulary before using it as a padding token.)Ú pad_token_idr r Úreturn_position_idsr a$ You are using packing, but the attention implementation is not set to 'flash_attention_2' or 'kernels-community/vllm-flash-attn3'. Packing flattens batches into a single sequence, and Flash Attention is the only known attention mechanisms that reliably support this. Using other implementations may lead to cross-contamination between batches. To avoid this, either disable packing by setting `packing=False`, or set `attn_implementation='flash_attention_2'` or `attn_implementation='kernels-community/vllm-flash-attn3'` in the model configuration.z…You set `assistant_only_loss=True`, but the dataset is not conversational. This option is only supported for conversational datasets.Úskip_prepare_datasetaE A formatting function was provided while `completion_only_loss=True`, which is incompatible. Using a formatter converts the dataset to a language modeling type, conflicting with completion-only loss. To resolve this, apply your formatting function before passing the dataset, or disable `completion_only_loss` in `SFTConfig`.Útrainc óL •— i | ] \ }}|‰ |‰‰‰‰|¦ « “Œ!S r )Ú_prepare_dataset)Ú.0ÚkeyÚdatasetr0 r< r r4 r s €€€€€r^ ú
|
|||
|
|
<dictcomp>z/_UnslothSFTTrainer.__init__.<locals>.<dictcomp>w sK ø€ ð $ð $ð $á (˜C ð ˜T× 2Ò 2°7Ð<LÈdÐT[Ð]lÐnqÑ rÔ rð$ð $ð $r` Úeval)rH rO r ) r/ r0 r1 r2 r3 r4 r5 r6 r7 r8 r9 r: )r/ Úadd_model_tagsr )FÚ
|
|||
|
|
isinstanceÚstrÚconfigÚ
|
|||
|
|
_name_or_pathÚsplitr r Úto_dictré Úpopr Úfrom_pretrainedr
|
|||
|
|
Úconvert_tokens_to_idsÚ
|
|||
|
|
ValueErrorr! r" Ú eos_token_idr r5 ÚwarnÚ_create_model_from_pathr r. ÚpathÚisfileÚendswithÚopenÚreadÚ
|
|||
|
|
chat_templater! Útrainable_token_indicesÚextendÚmodules_to_saverS r r r Ú_attn_implementationr� ÚnextÚiterr r r r r r* r ÚgetrJ r ÚdictÚitemsr% ÚlistÚ_metricsÚ_total_train_tokensr r r0 r r' r/ Ú maybe_activation_offload_contextr" r9 ÚhasattrrP Ú
|
|||
|
|
_tag_names) r r/ r0 r1 r2 r3 r4 r5 r6 r7 r8 r9 r: r; r< Úmodel_idÚ
|
|||
|
|
model_nameÚ dict_argsr
|
|||
|
|
r[ Úchat_template_fileÚ added_tokensÚuse_flash_attentionÚdataset_sampler rE Úpreprocess_datasetr r! s ` ` ` ` @€r^ r z_UnslothSFTTrainer.__init__À sí øøøøøø€ õ( ' ucÑ 2Ô 2ÐR�5�5¸¼ Ô8RˆØ ˆ<Ø!Ÿ š ¨Ñ,Ô,¨RÔ0ˆJÝ
|
|||
|
|
Ð 0Ð 0Ð 0Ñ1Ô1ˆDˆDÝ
|
|||
|
|
˜Õ /Ñ
|
|||
|
|
0Ô
|
|||
|
|
0ð *½ÀDÍ)Ñ9TÔ9Tð *Ø Ÿ š ™œˆIØ%)¤^ˆI�kÑ "Ø �MŠMÐ-Ñ .Ô .Ð .Ý Ð)Ð)˜yÐ)Ð)ˆDð Ð #Ý ,Ô <¸XÑ FÔ FÐ à Œ>Ð %Ø œˆIØ+×AÒAÀ)ÑLÔLˆLØÐ#Ý ðI°9ð Ið IØ+;Ô+EÔ+NðIð Ið Iñô ð ð
|
|||
|
|
-9Ð Ô )ð Ô !Ð -µjÀÍÑ6LÔ6LÐ -Ý ŒMð;ñ
|
|||
|
|
ô
|
|||
|
|
ð
|
|||
|
|
õ �e�SÑ !Ô !ð >Ø×0Ò0°¸Ñ=Ô=ˆEà Ô "Ð .ÝŒw�~Š~˜dÔ 5Ñ6Ô6ð
|
|||
|
|
¸4Ô;R×;[Ò;[Ð\mÑ;nÔ;nð
|
|||
|
|
ݘ$Ô1¸GÐDÑDÔDð OÐHZØ5G×5LÒ5LÑ5NÔ5NÐ$Ô2ðOð Oð Oñ Oô Oð Oð Oð Oð Oð Oð Oøøøð Oð Oð Oð Oà !� � å8KØÐ+¨TÔ-Dñ9ô 9Ñ5�Ð'¨ ¨ ð ˆLð Fð0 ð !Ô -Ð b°$´,Ð2aÀ4ÔCXÐ\aÒCaˆÔØ #œlÔ ?ð D
|
|||
|
|
ð
|
|||
|
|
Ðð Ô ð Ø Ð(Ý Ð!kÑlÔlÐlØŒ|ð
|
|||
|
|
Ô 5¸Ò BÐ BÝ”
|
|||
|
|
ðpñô ð ð 'ð
|
|||
|
|
Ý”
|
|||
|
|
ðJñô ð ð Ô/°1Ò4Ð4¸T¼\Ð4Ý”
|
|||
|
|
ð%ñô ð õ �d =Ñ 1Ô 1Ñ2Ô2ˆØ Ô $Ð ,Ø(0°NÐ(BˆDÔ %Ð %à(,Ô(AˆDÔ %à Ð ð œÐbÐ*:Ô*DÐbÐHXÔHbˆIØ+×AÒAÀ)ÑLÔLˆLØÐ#Ý ðL°9ð Lð LØ+;Ô+EÔ+NðLð Lð Lñô ð õ
|
|||
|
|
<Ø )Ø%)Ô%>Ø !Ô .à$7Ø#'Ô#:ð
|
|||
|
|
ñ ô ˆMð Œ<ð ˜DÔ 1°UÒ :Ð :ÐCVÐ :Ý ŒMðiñ
|
|||
|
|
ô
|
|||
|
|
ð
|
|||
|
|
ð Ô #ð Õ,=¸nÑ,MÔ,Mð Ý ð9ñô ð
|
|||
|
|
ð "Ô 0°DÐ 8Ð vÀÔ@S×@WÒ@WÐXnÐpuÑ@vÔ@vÐ<vÐØ ð ØÔ(ð
|
|||
|
|
¨_ð
|
|||
|
|
Ý ðQñô ð ð !× 1Ò 1Ø Ð /°°t´|À_ÐV]ñ ô ˆMð Ð'Ø*.Ô*;Ð*C˜$œ,˜,ÈÔIZ�Ý ˜lDÑ1Ô1ð ð$ð $ð $ð $ð $ð $ð $ð $à,8×,>Ò,>Ñ,@Ô,@ð$ñ $ô $�L�Lð
|
|||
|
|
$(×#8Ò#8Ø$Ð&6¸¸gÀÐX^ñ$ô $�Lõ
|
|||
|
|
#.dÑ"3Ô"3½[ÍÑ=NÔ=NÐOÐOˆŒ
|
|||
|
|
Ø#$ˆÔ õ ‰Œ×ÒØØØ'Ø'Ø%Ø -Ø /Ø +Ø Ø!Ø%=Ø*Gð ñ
|
|||
|
|
|
|||
|
|
ô
|
|||
|
|
|
|||
|
|
ð
|
|||
|
|
|
|||
|
|
ð Œ9Ô *ð MÝ4RÐY]ÔYcÐ4dÑ4dÔ4dˆDÔ 1Ð 1å4>Ô4JÑ4LÔ4LˆDÔ 1õ �4”:Ð /Ñ 0Ô 0ð 7Ø ŒJ× %Ò % d¤oÑ 6Ô 6Ð 6Ð 6Ð 6ð 7ð 7s ÇHÈHÈHÚ
|
|||
|
|
model_pathÚreturnc ó2 — |j pi }| d¦ « }t |t j ¦ « s|dk s|€nCt |t
|
|||
|
|
¦ « rt
|
|||
|
|
t |¦ « }||d< nt d|› d�¦ « ‚t j |fi |¤Ž}|S )z0Creates a model from a path or model identifier.Ú torch_dtyperz NzˆInvalid `torch_dtype` passed to `SFTConfig`. Expected either 'auto' or a string representing a `torch.dtype` (e.g., 'float32'), but got ú.)
|
|||
|
|
r rj rQ r3 ÚdtyperR ÚgetattrrZ r
|
|||
|
|
rX )r r{ r0 r r~ r/ s r^ r] z*_UnslothSFTTrainer._create_model_from_path£ sÅ € à Ô 2Ð 8°bÐà'×+Ò+¨MÑ:Ô:ˆ Ý �k¥5¤;Ñ /Ô /ð °;À&Ò3HÐ3HÈKÐL_Ø Ý
|
|||
|
|
˜ ¥SÑ
|
|||
|
|
)Ô
|
|||
|
|
)ð Ý!¥%¨Ñ5Ô5ˆKØ/:Ð ˜mÑ ,Ð ,å ðMØ>IðMð Mð Mñô ð
|
|||
|
|
õ %Ô4°ZÐUÐUÐCTÐUÐUˆØˆ r` c óî — t ¦ « st d¦ « ‚t |dd¦ « pt |dd¦ « }d}t |dd¦ « r?| ¦ « D ]*\ }}|j j dk r|j j j dv } nŒ+|r/|s-| ||¦ « }t j |d¬¦ « }n |j r|
|
|||
|
|
||¦ « }|�jt j t j ¦ « t j d ¦ « k r&t |dd¦ « r|rt% ||d¬
|
|||
|
|
¦ « }nt% ||¦ « }|j r"t |dd¦ « r|st) |¦ « |S ) z#Prepares a model for PEFT training.z9To use PeftModel, you need to install the `peft` library.Úis_loaded_in_4bitFÚis_loaded_in_8bitÚ
|
|||
|
|
Params4bit> ÚcpuÚmeta)rí Nz0.12)Úautocast_adapter_dtype)r+ Ú ImportErrorr� Únamed_parametersr! r" ÚdataÚdeviceÚtypeÚ _prepare_model_for_kbit_trainingr$ Úreplacerí Ú _enable_gradient_checkpointingr4 Úparser0 Ú __version__r) r¸ r1 )r r/ r; r0 Úis_qloraÚis_sharded_qloraÚ_Úparams r^ Ú_prepare_peft_modelz&_UnslothSFTTrainer._prepare_peft_modelº s» € å Ñ"Ô"ð [Ý Ð YÑZÔZÐ Zõ ˜5Ð"5°uÑ=Ô=ÐkÅÈÐPcÐejÑAkÔAkˆà ÐÝ �5Ð-¨uÑ 5Ô 5ð à !× 2Ò 2Ñ 4Ô 4ð
|
|||
|
|
ð
|
|||
|
|
‘��5Ø”?Ô+¨|Ò;Ð;Ø',¤zÔ'8Ô'=À Ð'PÐ$Ø�Eð <ð
|
|||
|
|
ð EÐ ,ð EØ×9Ò9¸%ÀÑFÔFˆEå Ô& tÀEÐJÑJÔJˆDˆDØ
|
|||
|
|
Ô
|
|||
|
|
(ð EØ×7Ò7¸¸tÑDÔDˆEð Ð "å”
|
|||
|
|
�dÔ .Ñ/Ô/µ7´=ÀÑ3HÔ3HÒHÐHݘEÐ#6¸Ñ>Ô>ð Ià$ð Iõ ' u¨kÐRWÐXÑXÔX��å& u¨kÑ:Ô:�ð Œ9ð /� Ð(;¸UÑCÔCð /ÐL\ð /Ý '¨Ñ .Ô .Ð .àˆ r` c ó> — |j |j pi dœ}t |fi |¤ŽS )z-Prepares a quantized model for kbit training.)Úuse_gradient_checkpointingrî )rí rî r2 )r r/ r0 Úprepare_model_kwargss r^ rŽ z3_UnslothSFTTrainer._prepare_model_for_kbit_trainingã s< € ð +/Ô*EØ-1Ô-OÐ-UÐSUð
|
|||
|
|
ð
|
|||
|
|
Ð õ
|
|||
|
|
/¨uÐMÐMÐ8LÐMÐMÐMr` c óÒ — |j pi }d|vp|d }|rOt |d¦ « r| ¦ « n*d„ }| ¦ « |¦ « |S )z-Enables gradient checkpointing for the model.Ú
|
|||
|
|
use_reentrantÚenable_input_require_gradsc ó0 — | d¦ « d S )NT)Úrequires_grad_)ÚmoduleÚinputÚoutputs r^ Úmake_inputs_require_gradzS_UnslothSFTTrainer._enable_gradient_checkpointing.<locals>.make_inputs_require_gradø s € Ø×)Ò)¨$Ñ/Ô/Ð/Ð/Ð/r` )rî rq r� Úget_input_embeddingsÚregister_forward_hook)r r/ r0 rî rœ r£ s r^ r� z1_UnslothSFTTrainer._enable_gradient_checkpointingì s• € à(,Ô(JÐ(PÈbÐ%à Ð#@Ð @Ð rÐDaÐbqÔDrð ð ð ]Ý�uÐ :Ñ;Ô;ð
|
|||
|
|
]Ø×0Ò0Ñ2Ô2Ð2Ð2ð0ð 0ð 0ð ×*Ò*Ñ,Ô,×BÒBÐC[Ñ\Ô\Ð\àˆ r` rM r Ú dataset_namec ó ‡‡‡‡‡‡‡— t |t ¦ « r|S n# Y nxY wi }t |t ¦ « }t |d¦ « } |Š| r|j Št |dd¦ « Š‰dk rt |dd¦ « Š‰dk rt | dd¦ « Š‰dk rt | dd¦ « Š‰dk rt
|
|||
|
|
d¦ « ‚t |dd¦ « Š‰dk Šd Šd
|
|||
|
|
}
|
|||
|
|
t t t |¦ « ¦ «
|
|||
|
|
¦ « ¦ « } d g} d | v r| d ¦ « dd
|
|||
|
|
l m
|
|||
|
|
}
|
|||
|
|
m} d| v rR| r(t ‰d¦ « st
|
|||
|
|
d|j › d�¦ « ‚ |
|
|||
|
|
‰¦ « | _ | d¦ « d }
|
|||
|
|
nZd | v r?| r(t ‰d¦ « st
|
|||
|
|
d|j › d�¦ « ‚ |‰d ¬¦ « | _ d }
|
|||
|
|
n‰| vrd
|
|||
|
|
Š‰€t
|
|||
|
|
d¦ « ‚ |
|
|||
|
|
�r†‰rR ‰t t |¦ « ¦ « ¦ « }t |t" ¦ « st% d¦ « ‚|d }n(t t |¦ « ¦ « ‰ d }t |dd¦ « }|dk r| rt ‰dd¦ « }|€d}d
|
|||
|
|
Št |dd ¦ « }t ‰dd ¦ « }|p|}|�*| |¦ « s||v rd Št) d¦ « ˆˆˆˆˆˆˆfd„} t |t* ¦ « st |dd¦ « |d < n|j j |d < |r d ‰› d �|d < |j |fd!d
|
|||
|
|
i|¤Ž}| r$t |d¦ « s |‰d ¬¦ « }|| _ |rt) d"¦ « |S |S )#NÚ tokenizerr r Úmax_seq_lengthÚmax_seqz1Unsloth: max_seq_length is 0! Please specify one!r
|
|||
|
|
r‚ FTÚ input_idsÚattention_maskr: Úlabelsr/ z Unsloth: z does not have .pad!)Úmlmz-Unsloth: You must specify a `formatting_func`zIUnsloth: The `formatting_func` should return a list of processed strings.rc r{ Ú bos_tokenzHUnsloth: We found double BOS tokens - we shall remove one automatically.c óJ •— ‰‰s| ‰ n
|
|||
|
|
‰| ¦ « ‰‰d‰¬¦ « S )NF)Ú
|
|||
|
|
truncationr Úreturn_token_type_idsÚadd_special_tokensr )Úexampler³ r
|
|||
|
|
Údo_formatting_funcÚ
|
|||
|
|
do_truncationr< r© r¨ s €€€€€€€r^ Ú _tokenizez6_UnslothSFTTrainer._prepare_dataset.<locals>._tokenize\ sF ø€ Ø �yØ7IÐg�GÐ .Ô/Ð/ÈÈÐ_fÑOgÔOgØ!.Ø!/Ø,1Ø);ð ñ ô ð r` r ri Únum_procÚ
|
|||
|
|
batch_sizezUnsloth: Tokenizing ["z"]ÚdescÚbatchedzPUnsloth: Hugging Face's packing is currently buggy - we're disabling it for now!)rQ ÚConstantLengthDatasetr rq r¨ r� Ú RuntimeErrorÚsetrh ri ÚkeysrS Ú transformersr; r r! r1 rm rZ Ú
|
|||
|
|
startswithÚprintr Ú _ex_iterabler¹ ÚmapÚselect_columnsÚ
|
|||
|
|
pack_examples) r rM r4 r0 r r< r¦ Ú
|
|||
|
|
map_kwargsÚuse_descÚis_vlmÚ do_tokenizeÚ column_namesÚused_column_namesr; r Ú test_textrc Ú bos_token_1Ú bos_token_2r¯ r· r1 r³ r
|
|||
|
|
rµ r¶ r© r¨ s ` @@@@@@r^ rJ z#_UnslothSFTTrainer._prepare_datasetÿ sÌ øøøøøøø€ ð ݘ'Õ#8Ñ9Ô9Ð IÀ'¸>Ð Iøð Ø ˆDøøøàˆ
|
|||
|
|
Ý ˜g¥wÑ/Ô/ˆÝÐ)¨;Ñ7Ô7ˆØ$ˆ Ø Ð9Ð /Ô 9�9õ ! |°QÑ7Ô7ˆØ ˜QÒ Ð µ¸Ð?OÐQRÑ1SÔ1S Ø ˜QÒ Ð µ¸Ð?OÐQRÑ1SÔ1S Ø ˜QÒ Ð µ¸¸yÈ!Ñ1LÔ1L Ø ˜QÒ Ð ¥lÐ3fÑ&gÔ&gÐ gÝ $ TÐ+?ÀÑ HÔ HÐØ&¨!Ò+ˆ
|
|||
|
|
Ø "Ð؈ õ �4¥ W¡
|
|||
|
|
¤
|
|||
|
|
Ñ.Ô.×3Ò3Ñ5Ô5Ñ6Ô6ˆ Ø (˜MÐØ ˜|Ð +Ð +Ø × $Ò $Ð%5Ñ 6Ô 6Ð 6ð YÐXÐXÐXÐXÐXÐXÐXØ �|Ð #Ð #àð
|
|||
|
|
a�g i°Ñ 7Ô 7ð
|
|||
|
|
aå"Ð#_Ð/?Ô/IÐ#_Ð#_Ð#_Ñ`Ô`Ð`Ø!7Ð!7¸ Ñ!BÔ!BˆDÔ Ø × $Ò $ XÑ .Ô .Ð .Ø ˆKˆKØ
|
|||
|
|
˜LÐ
|
|||
|
|
(Ð
|
|||
|
|
(àð
|
|||
|
|
a�g i°Ñ 7Ô 7ð
|
|||
|
|
aå"Ð#_Ð/?Ô/IÐ#_Ð#_Ð#_Ñ`Ô`Ð`Ø!@Ð!@ÀÐRWÐ!XÑ!XÔ!XˆDÔ Ø ˆKˆKØ
|
|||
|
|
|Ð
|
|||
|
|
3Ð
|
|||
|
|
3Ø!%Ð Ø Ð&Ý"Ð#RÑSÔSÐSØ à ñ 6 à!ð
|
|||
|
|
GØ +˜ODµ°g± ´ Ñ,?Ô,?Ñ @Ô @� Ý! )TÑ2Ô2ð Ý$Øcñô ð ð & aœL� � å ¥ g¡ ¤ Ñ /Ô /Ð0BÔ CÀAÔ F� õ $Ð$4°oÀrÑ JÔ JˆMØ Ò"Ð" vÐ"Ý '¨ °?ÀBÑ GÔ G�
|
|||
|
|
Ø Ð$Ø "�
|
|||
|
|
ð "&Ð Ý!Ð"2°KÀÑFÔFˆKÝ! )¨[¸$Ñ?Ô?ˆKØ#Ð2 {ˆIàÐ$Ø ×'Ò'¨ Ñ2Ô2ð f°iÀ=Ð6PÐ6PØ).Ð&ÝÐdÑeÔeÐeØ ð
|
|||
|
|
ð
|
|||
|
|
ð
|
|||
|
|
ð
|
|||
|
|
ð
|
|||
|
|
ð
|
|||
|
|
ð
|
|||
|
|
ð
|
|||
|
|
ð
|
|||
|
|
ð
|
|||
|
|
ð
|
|||
|
|
ð
|