107 lines
16 KiB
Plaintext
107 lines
16 KiB
Plaintext
|
Ë
|
|||
|
|
Ào“hµ: ã ób — G d „ d« Z d dedededee def
|
|||
|
|
d„Zd „ Zed
|
|||
|
|
k( r e« yy) c
|
|||
|
|
ó€ — e Zd ZdZd„ Z d dedee dee deded ef d
|
|||
|
|
„Zd ed e e
|
|||
|
|
e e ee f f fd „Z
|
|||
|
|
d
|
|||
|
|
e
|
|||
|
|
e d ed e ee
|
|||
|
|
e f fd„Zd
|
|||
|
|
e
|
|||
|
|
e d e
|
|||
|
|
e fd„Zd d
|
|||
|
|
e
|
|||
|
|
e ded e
|
|||
|
|
e fd„Zd d
|
|||
|
|
e
|
|||
|
|
e ded e
|
|||
|
|
e fd„Zd d
|
|||
|
|
e
|
|||
|
|
e ded e
|
|||
|
|
e fd„Zd d
|
|||
|
|
e
|
|||
|
|
e dedefd„Z d d ededed e ee f fd„Zd ed e ee f fd„Zy)!ÚClassificationPipelinez Main classification pipelinec ó@ — t « | _ t « | _ y ©N)Ú
|
|||
|
|
DataValidatorÚ validatorÚHuggingFaceDataLoaderÚ hf_loader)Úselfs úclassification_pipeline.pyÚ__init__z ClassificationPipeline.__init__ s € Ü&› ˆŒÜ.Ó0ˆ�ó NÚ data_sourceÚ dataset_nameÚ data_pathÚ input_fieldÚ label_fieldÚreturnc ó® — |dk( r |s t d« ‚t d||||dœ|¤Ž}|S |dk( r |s t d« ‚t d||||dœ|¤Ž}|S t d|› �« ‚) z#Create classification configurationÚ huggingfacez2Dataset name is required for Hugging Face datasets)r r r r Úcustomz)Data path is required for custom datasets)r r r r úUnsupported data source: © )Ú
|
|||
|
|
ValueErrorÚClassificationConfig)r
|
|||
|
|
r r r r r ÚkwargsÚconfigs r Ú
|
|||
|
|
create_configz$ClassificationPipeline.create_config
|
|||
|
|
s¡ € ð ˜-Ò 'Ù Ü Ð!UÓVÐVÜ)ð Ø 'Ø )Ø 'Ø 'ñ ð
|
|||
|
|
ñ ˆFð( ˆ
|
|||
|
|
ð ˜HÒ
|
|||
|
|
$Ù Ü Ð!LÓMÐMÜ)ð Ø 'Ø#Ø 'Ø 'ñ ð
|
|||
|
|
ñ ˆFð ˆ
|
|||
|
|
ô Ð 8¸¸
|
|||
|
|
Ð FÓGÐ Gr
|
|||
|
|
r c ó ‡— |j dk( r8| j j |« }| j j ||« }n_|j dk( r8| j j |« }| j j ||« }nt d|j › �« ‚|rét
|
|||
|
|
|d j « « }t d|› �« |j |v }|j |v }|r|sŸd}|d|j › d|j › d �z
|
|||
|
|
}|d
|
|||
|
|
t |« › d �z
|
|||
|
|
}|D �‡cg c] Št ˆfd „d
|
|||
|
|
D « « sŒ‰‘Œ }}|D �‡cg c] Št ˆfd„dD « « sŒ‰‘Œ } }|r |d|› d �z
|
|||
|
|
}| r |d| › d �z
|
|||
|
|
}t |« ‚| j j ||« \ }
|
|||
|
|
} |
|
|||
|
|
s?t j! d« | D ] } t j! d| › �« Œ t d« ‚| j j# ||« }
|
|||
|
|
t j% d« t j% d|
|
|||
|
|
d › �« t j% d|
|
|||
|
|
d › �« t j% d|
|
|||
|
|
d › �« ||
|
|||
|
|
fS c c}w c c}w ) zLoad and preprocess datar r r é z Available fields in dataset: z(Configured fields not found in dataset.
|
|||
|
|
zLooking for: input_field='z', label_field='z'
|
|||
|
|
zAvailable fields: Ú
|
|||
|
|
c 3 óB •K — | ] }|‰j « v –— Œ y wr ©Úlower©Ú.0ÚkeywordÚfs €r Ú <genexpr>z=ClassificationPipeline.load_and_preprocess.<locals>.<genexpr>J s( øè ø€ ð BeÙcð \cÀ'ÈQÏWÉWËYÔBVÙcùó ƒ ©ÚtextÚsentenceÚcontentÚinputÚcommentÚmessageÚtitleÚbodyc 3 óB •K — | ] }|‰j « v –— Œ y wr r" r$ s €r r( z=ClassificationPipeline.load_and_preprocess.<locals>.<genexpr>L s( øè ø€ ð CbÙ`ð ]dÀ7ÈaÏgÉgËiÔCWÙ`ùr) ©ÚlabelÚclassÚcategoryÚtargetÚemotionÚlabelsÚtagÚtypezSuggested text fields: zSuggested label fields: zData validation failed:z - zData validation failedzDataset analysis:z Total samples: Ú
|
|||
|
|
total_samplesz Unique labels: Ú
|
|||
|
|
unique_labelsz Label distribution: Úlabel_distribution)r r ÚloadÚ
|
|||
|
|
preprocessÚ
|
|||
|
|
custom_loaderr ÚsetÚkeysÚprintr r ÚlistÚanyr Ú validate_classification_dataÚloggerÚerrorÚanalyze_datasetÚinfo)r
|
|||
|
|
r ÚdataÚavailable_fieldsÚinput_field_existsÚlabel_field_existsÚ error_msgr' Ú text_fieldsÚ label_fieldsÚis_validÚerrorsrJ Úanalysiss ` r Úload_and_preprocessz*ClassificationPipeline.load_and_preprocess. s½ ø€ ð × Ñ Ò .Ø—>‘>×&Ñ& vÓ.ˆDØ—>‘>×,Ñ,¨T°6Ó:‰DØ
|
|||
|
|
×
|
|||
|
|
Ñ
|
|||
|
|
8Ò
|
|||
|
|
+Ø×%Ñ%×*Ñ*¨6Ó2ˆDØ×%Ñ%×0Ñ0°°vÓ>‰Dä Ð 8¸×9KÑ9KÐ8LÐ MÓNÐ Nñ Ü " 4¨¡7§<¡<£>Ó 2Ð Ü Ð1Ð2BÐ1CÐDÔ Eð "(×!3Ñ!3Ð7GÐ!GÐ Ø!'×!3Ñ!3Ð7GÐ!GÐ á%Ñ-?Ø G� ØÐ 9¸&×:LÑ:LÐ9MÐM]Ð^d×^pÑ^pÐ]qÐqtÐ uÑu� ØÐ 1´$Ð7GÓ2HÐ1IÈÐ LÑL� ñ +;ô fÑ*: Q¼có BeÙcóBeõ ?ešqÐ*:� ð fá+;ô cÑ+; a¼só CbÙ`óCbõ @b¢Ð+;� ð cñ Ø Ð#:¸;¸-ÀrÐ!JÑJ�IÙ Ø Ð#;¸L¸>ÈÐ!LÑL�Iä Ó+Ð+ð Ÿ>™>×FÑFÀtÈVÓTш�&ÙÜ �L‰LÐ2Ô 3Û �Ü— ‘ ˜t E 7˜^Õ,ð ä Ð 5Ó6Ð 6ð —>‘>×1Ñ1°$¸Ó?ˆä� ‰ Ð'Ô(Ü� ‰ Ð'¨° Ñ(AÐ'BÐCÔDÜ� ‰ Ð'¨° Ñ(AÐ'BÐCÔDÜ� ‰ Ð,¨XÐ6JÑ-KÐ,LÐMÔNà�Xˆ~Ð ùò9 fùò cs Ä-I9ÅI9ÅI>Å,I>rM c
|
|||
|
|
ó2 — t t |« |j z « }t t |« |j z « }|d| }||||z }|||z d }|||dœ}t j dt |« › dt |« › dt |« › �« |S )z*Split data into train/validation/test setsN)ÚtrainÚ
|
|||
|
|
validationÚtestzData splits: Train=z, Val=z, Test=)ÚintÚlenÚ train_splitÚvalidation_splitrI rL ) r
|
|||
|
|
rM r Ú
|
|||
|
|
train_sizeÚval_sizeÚ
|
|||
|
|
train_dataÚval_dataÚ test_dataÚsplitss r Ú
|
|||
|
|
split_dataz!ClassificationPipeline.split_datah s¯ € ô œ˜T› V×%7Ñ%7Ñ7Ó8ˆ
|
|||
|
|
Ü”s˜4“y 6×#:Ñ#:Ñ:Ó;ˆà˜+˜:Ð&ˆ
|
|||
|
|
ؘ
|
|||
|
|
:°Ñ#8Ð9ˆØ˜ hÑ.Ð/Ð0ˆ ð Ø"Ø ñ
|
|||
|
|
ˆô � ‰ Ð)¬#¨j«/Ð):¸&ÄÀXà ÀÈwÔWZÐ[dÓWeÐVfÐgÔhàˆ
|
|||
|
|
r
|
|||
|
|
c óL — g }|D ] }|j |d |d dœ« Œ |S )z=Convert classification data to standard classification formatr. r5 )r+ r5 ©Úappend)r
|
|||
|
|
rM Úclassification_dataÚitems r Ú convert_to_classification_formatz7ClassificationPipeline.convert_to_classification_format} s; € à ÐãˆDØ × &Ñ &Ø ˜W™
|
|||
|
|
Ø ˜g™ ñ(õ
|
|||
|
|
ð ð #Ð"r
|
|||
|
|
Útask_descriptionc ó` — g }|D ]& }|j ||d t |d « dœ« Œ( |S )z1Convert classification data to instruction formatr. r5 )Ú instructionr. Úoutput©ri Ústr)r
|
|||
|
|
rM rm Úinstruction_datark s r Ú convert_to_instruction_formatz4ClassificationPipeline.convert_to_instruction_format‰ sC € à ÐãˆDØ × #Ñ #Ø /Ø ˜g™ Ü ˜d 7™mÓ,ñ%õ
|
|||
|
|
ð ð Ð r
|
|||
|
|
c ón — g }|D ]- }d|› d|d › �dœdd|d › �dœg}|j d|i« Œ/ |S ) z2Convert classification data to conversation formatÚuserú
|
|||
|
|
|
|||
|
|
r. )Úroler- Ú assistantzThe classification is: r5 Ú
|
|||
|
|
conversationsrh )r
|
|||
|
|
rM rm Úconversation_datark rz s r Ú convert_to_conversation_formatz5ClassificationPipeline.convert_to_conversation_format– sp € à ÐãˆDð #Ø"2Ð!3°4¸¸W¹
|
|||
|
|
°Ð Gñð
|
|||
|
|
(Ø!8¸¸g¹ ¸Ð Hñð ˆMð
|
|||
|
|
× $Ñ $Ø ð&õ
|
|||
|
|
ð ð !Ð r
|
|||
|
|
Úquestion_templatec óh — g }|D ]* }|j |› d|d › �t |d « dœ« Œ, |S )z)Convert classification data to Q&A formatrw r. r5 )ÚquestionÚanswerrq )r
|
|||
|
|
rM r} Úqa_datark s r Úconvert_to_qa_formatz+ClassificationPipeline.convert_to_qa_format¬ sH € àˆãˆDØ �N‰NØ 0Ð 1°°d¸7±m°_Ð EÜ ˜d 7™mÓ,ñ õ
|
|||
|
|
ð ð ˆr
|
|||
|
|
Ú output_pathÚformatc ó. — t |« }|j j dd¬« |dk( rIt |dd¬« 5 }|D ]+ }|j t
|
|||
|
|
j
|
|||
|
|
|d¬« d z « Œ- d
|
|||
|
|
d
|
|||
|
|
d
|
|||
|
|
« nc|d k( r1t |dd¬« 5 }t
|
|||
|
|
j ||dd ¬
|
|||
|
|
« d
|
|||
|
|
d
|
|||
|
|
d
|
|||
|
|
« n-|dk( r(t j |« }|j |d¬« t j dt |« › d|› �« y
|
|||
|
|
# 1 sw Y Œ.xY w# 1 sw Y Œ:xY w)zSave processed data to fileT©ÚparentsÚexist_okÚjsonlÚwzutf-8)ÚencodingF)Ú ensure_asciir NÚjsoné )rŒ ÚindentÚcsv)ÚindexzSaved z samples to )ÚPathÚparentÚmkdirÚopenÚwriter� ÚdumpsÚdumpÚpdÚ DataFrameÚto_csvrI rL r] )r
|
|||
|
|
rM rƒ r„ Ú output_filer' rk Údfs r Ú save_dataz ClassificationPipeline.save_data¸ só € ä˜;Ó'ˆ Ø×Ñ× Ñ ¨¸Ð Ô=à �WÒ Ü�k 3°Õ9¸QÛ �DØ—G‘GœDŸJ™J t¸%˜JÓ @À4Ñ GÕHñ !÷ :Ð9ð �vÒ
|
|||
|
|
Ü�k 3°Õ9¸QÜ— ‘ ˜$ °¸a� Ô@÷ :Ð9à
|