Files
DS-LLM-TEMPLATE-FINETUNING/utils/config/__pycache__/config_manager.cpython-310.pyc
T

48 lines
4.7 KiB
Plaintext
Raw Normal View History

2025-08-28 16:46:24 +00:00
o
üz°hçã@ddlZddlZddlmZddlmZmZmZddlZe  e
¡Z GdddƒZ de
dee
effdd „Zd dee
dee
effd
d ZdS)
éN)ÚPath)ÚDictÚAnyÚOptionalc @eZdZdZddZdedeeeffddZdeeefd eeefdeeeffd
d Z d eeefd
edefddZ
dd eeefd
ededefddZ ddede j
fddZddeed eeeefdeeeffddZdS)Ú
ConfigManagerzHManages configuration loading from YAML files and command-line argumentscCs
i|_dS©N)Úconfig)Úself©r
úD/workspace/DS-LLM-TEMPLATE-FINETUNING/utils/config/config_manager.pyÚ__init__ s
zConfigManager.__init__Ú config_pathÚreturncCsjt|ƒ}| ¡std|ƒt|ddd
}t |¡}Wdƒn1s&wYt d|¡|S)z!Load configuration from YAML filezConfiguration file not found: Úrzutf-8)ÚencodingNzLoaded configuration from: )rÚexistsÚFileNotFoundErrorÚopenÚyamlÚ safe_loadÚloggerÚinfo)r r
Z config_fileÚfrr
r
r Úload_yaml_configs ÿzConfigManager.load_yaml_configÚ yaml_configÚcli_argscCs4| ¡}| ¡D]\}}|dur| |||¡q|S)z4Merge YAML configuration with command-line argumentsN)ÚcopyÚitemsÚ_set_nested_value)r rrÚ
merged_configÚkeyÚvaluer
r
r Ú
merge_configss zConfigManager.merge_configsrr r!cCsH| d¡}|}|ddD]}||vri||<||}q
|||d<dS)zMSet a nested value in config using dot notation (e.g., 'training.batch_size')Ú.Néÿÿÿÿ)Úsplit)r rr r!ÚkeysÚcurrentÚkr
r
r r's

zConfigManager._set_nested_valueNÚdefaultcCs>| d¡}|}|D]}t|tƒr||vr||}q |S|S)z1Get a nested value from config using dot notationr#)r%Ú
isinstanceÚdict)r rr r)r&r'r(r
r
r Úget_config_value3s

zConfigManager.get_config_valueú Pipeline with YAML configurationÚ descriptioncCs0tj|d}|jdtdd|jdddgdd |jd
td d|jd td
d|jdgd¢dd |jdtdd|jdtdd|jdtdd|jdtdd|jdtdd|jdtdd|jdtdd|jdtd d|jd!td"d|jd#td$d|jd%gd&¢d'd |jd(gd)¢d*d |S)+z6Create argument parser that supports YAML config files)r.z--configzPath to YAML configuration file)ÚtypeÚhelpz
--data-sourceZ huggingfaceZcustomz Data source)Úchoicesr0z--dataset-namezHuggingFace dataset namez --data-pathzPath to custom data filez
--data-format)ZjsonlZcsvÚjsonz Data formatz
--input-fieldzInput field namez
--label-fieldzLabel field namez
--max-sampleszMaximum samples to processz --output-dirzOutput directoryz --model-namez
Model namez --max-lengthzMaximum sequence lengthz --num-epochszNumber of epochsz --batch-sizez
Batch sizez--learning-ratez
Learning ratez --model-pathzPath to saved modelz--device)ÚautoÚcudaÚcpuZDevicez --log-level)ÚDEBUGÚINFOZWARNINGZERRORz Log level)ÚargparseÚArgumentParserÚ add_argumentÚstrÚintÚfloat)r r.Úparserr
r
r Úcreate_argparser_with_yaml@s& z(ConfigManager.create_argparser_with_yamlcCsFi}|r | |¡}|p i}dd| ¡Dƒ}| ||¡}t d¡|S)z-Load YAML config and merge with CLI argumentscSsi|] \}}|dur||qSrr