ds_erp_ai/__pycache__/utils.cpython-311.pyc

§
F¨¿fs\ã	ó—dd
lm
Z

ddlmZ
ddlZddlmZ
ddlmZ
ddl	m
Z

ddl	mZ
ddl	mZ
dd	l
mZ
dd
lmZ
ddlmZ
ddlmZ
dd
lmZ
ddlmZ
ddlZddlZej ej ej ej e ¦
«
d¦«¦
«
¦
«

ddl!m"Z"
ddl#Z#ddl$m%Z%m&Z&m'Z'
ddl(m)Z)
ddl*Z*ddl+Z+ddl,m-Z-
ddl.Z.ddl/Z/ddl0Z0ddl1Z2ddl3m4Z4
ddl5Z5ddl6Z6ddl7m8Z8
ddl9Z9ddl:m;Z;
e;¦«
ej<d¦
«
a=ej<d¦
«
ej>d<e-ej<d¦
«
¬¦
«
Z?dZ@eddd¬¦«ZAd„ZBe"jCd¦
«

eB¦«ZDe"jCd ¦
«

dGd"„
ZEd#„ZFd$„ZGd%„ZHd&„ZId'„ZJd(„ZKdHd*„
ZLd+„ZMdId-„
ZNdJd/„
ZOdKd1„
ZPd2eQd3eRfd4„ZSd5eTd6eQfd7„ZUdLd9„
ZVeDd8fd:„
ZWd;„ZXd<eQfd=„ZYd>eTd?eTd@eTfdA„ZZd>eTd?eTd@eTfdB„Z[dC„Z\dD„Z]dMdF„
Z^dS)Né©
ÚHuggingFaceBgeEmbeddings)
ÚRecursiveCharacterTextSplitterN)
ÚInMemoryDocstore)
ÚFAISS)
ÚPyPDFLoader)
Ú
TextLoader)
ÚDocx2txtLoader)
ÚChatGroq)
ÚPromptTemplate)
ÚStrOutputParser)
Úuuid4)
ÚDocument)
Ú
TextExtractorz..)
Úlogger)ÚImageÚ	ImageDrawÚ	ImageFont)
ÚThreadPoolExecutor)
ÚGroq)
ÚAudioSegment)
Ú
VideoFileClip)
Úload_dotenvÚOPENAI_API_KEYÚGROQ_API_KEY)
Úapi_keyúwhisper-large-v3zllama3-8b-8192éd)ÚtemperatureÚ
model_nameÚ
max_tokenscó>—d
}ddi
}
ddi
}t
||
|¬¦«}|S)NzBAAI/bge-small-enÚdeviceÚcudaÚnormalize_embeddingsT)r Úmodel_kwargsÚ
encode_kwargsr)r r&r'Ú
embeddingss    úHc:\Users\timmy_3aupohg\Downloads\Manaknight Projects\ds_aiindex\utils.pyÚload_embedding_modelr*2s?€Ø$€JؘfÐ%€LØ+¨TÐ2€MÝ)Ø%°LÐP]ðñô€JðÐózLoading the embedding modelzEmbedding model loadedÚtextcó~
—t
j
d
¦
«

|dj}|dj}t	ddt
d¬¦«}| |g
¦
«
}g}t|¦
«
D]N\}}| ¦«}	||	d<|
|	d<t|j|	¬	¦«}
| 
|
¦
«

ŒO|S)
NzCreating documents from textréèé
F)Ú
chunk_sizeÚ
chunk_overlapÚlength_functionÚis_separator_regexÚpageÚ	file_type©Úpage_contentÚmetadata)rÚinfor7r8rÚlenÚcreate_documentsÚ	enumerateÚcopyrÚappend)Údocr5r,r8Ú
text_splitterÚdocsÚ	documentsÚ
iÚchunkÚdoc_metadataÚdocuments           r)r;r;CsÒ€Ý
„KÐ/Ñ0Ô0Ð0؈qŒ6Ô€DØ�1ŒvŒ€HÝ2ØØÝØ ð	ñô€Mð×)Ò)¨4¨&Ñ1Ô1€Dà€Iݘd‘O”Oð#ð#‰ˆ
ˆ5à—}’}‘”ˆØ ˆ�VÑØ$-ˆ�[Ñ!ݨÔ);ÀlÐSÑSÔSˆØ×Ò˜Ñ"Ô"Ð"Ð"ØÐr+c
óÈ—t
j
d
|›�¦
«

	t|¦
«
}
|
 ¦«}t	|¦
«
}|S#
td|›�¦
«
‚
xYw
)NzLoading text document from úError loading -- )rr9r	Úloadr;Ú
ValueError)Ú
document_pathÚtxt_docr,rAs    r)Úload_txt_documentrMZsl€Ý
„KÐ=¨mÐ=Ð=Ñ>Ô>Ð>ð>ݘ]Ñ+Ô+ˆØ�|Š|‰~Œ~ˆå Ñ%Ô%ˆØˆøð
>ÝÐ<¨]Ð<Ð<Ñ=Ô=Ð=øøøó™3A
Á
A!c
óÈ—t
j
d
|›�¦
«

	t|¦
«
}
|
 ¦«}t	|¦
«
}|S#
td|›�¦
«
‚
xYw
)NzLoading docx document from rH)rr9r
rIr;rJ)rKÚdocx_docr,rAs    r)Úload_docx_documentrQfsl€Ý
„KÐ=¨mÐ=Ð=Ñ>Ô>Ð>ð>Ý! -Ñ0Ô0ˆØ�}Š}‰Œˆå Ñ%Ô%ˆØˆøð
>ÝÐ<¨]Ð<Ð<Ñ=Ô=Ð=øøørNc
óª—t
j
d
|›�¦
«

	t|¦
«
}
|
 ¦«}|S#
t	d|›�¦
«
‚
xYw
)NzLoading pdf document from rH)rr9rÚload_and_splitrJ)rKÚpdf_docÚpagess   r)Úload_pdf_documentrVssc€Ý
„KÐ<¨]Ð<Ð<Ñ=Ô=Ð=ð>ݘmÑ,Ô,ˆØ×&Ò&Ñ(Ô(ˆØˆøð
>ÝÐ<¨]Ð<Ð<Ñ=Ô=Ð=øøøs	™$>¾Ac
óþ—| d
¦
«
rt|¦
«
S| d¦
«
rt|¦
«
S| d¦
«
rt|¦
«
St	d|›�¦
«
‚
)Nú.pdfú.txtú.docxzUnsupported document type for )ÚendswithrVrMrQrJ)
rKs
 r)Ú
load_documentr\~sˆ€Ø×Ò˜fÑ%Ô%ðK
Ý  Ñ/Ô/Ð/Ø	×	Ò	 Ñ	'Ô	'ðK
Ý  Ñ/Ô/Ð/Ø	×	Ò	 Ñ	(Ô	(ðK
Ý! -Ñ0Ô0Ð0åÐI¸-ÐIÐIÑJÔJÐJr+c
óö—t
j
d
|›�¦
«

t|d¦«5}
tj|
 ¦«¦
«
 d¦
«
cddd¦«
S#1swxYw

Y

dS)NzEncoding image Úrbzutf-8)rr9ÚopenÚbase64Ú	b64encodeÚreadÚdecode)Ú
image_pathÚ
image_files  r)Úencode_imagerfŠsÅ€Ý
„KÐ. *Ð.Ð.Ñ/Ô/Ð/Ý	
ˆj˜$Ñ	Ô	ð
C
 :ÝÔ 
§¢Ñ 1Ô 1Ñ2Ô2×9Ò9¸'ÑBÔBð
C
ð
C
ð
C
ð
C
ñ
C
ô
C
ð
C
ð
C
ð
C
ð
C
ð
C
ð
C
øøøð
C
ð
C
ð
C
ð
C
ð
C
ð
C
s¨9A.Á.A2Á5
A2c
óV
—t
j
d
|›�¦
«

t|¦
«
}
ddt›�dœ}	ddddd	œd
dd|
›�i
d
œgdœg
ddœ}t	jd||¬¦«}| ¦«dddd}n#t$r}d}Yd}~nd}~w
wxYw
|S)NzProcessing image zapplication/jsonzBearer )zContent-TypeÚ
Authorizationzgpt-4o-miniÚuserr,uWhat’s in this image?)Útyper,Ú	image_urlÚurlzdata:image/jpeg;base64,)rjrk)ÚroleÚcontenti,
)ÚmodelÚmessagesr!z*https://api.openai.com/v1/chat/completions)ÚheadersÚjsonÚchoicesrÚmessagernú$Image not good enough for processing)rr9rfrÚrequestsÚpostrrÚ	Exception)rdÚbase64_imagerqÚpayloadÚresponseÚ
es      r)Ú
process_imager}�s
€Ý
„KÐ0 JÐ0Ð0Ñ1Ô1Ð1õ  
Ñ+Ô+€Lð+Ø,¥7Ð,Ð,ðð€Gð
:à"ð#ð%+Ø$=ððð
%0à %Ð'OÀÐ'OÐ'Oð*ððð ðððð"ð'
ð
ˆõ,”=Ð!MÐW^ÐelÐmÑmÔmˆà—=’=‘?”? 9Ô-¨aÔ0°Ô;¸IÔFˆˆøÝð
:ð
:ð
:Ø9ˆˆˆˆˆˆøøøøð
:øøøð€OsµABÂ
B&ÂB!Â!B&Úimagecóˆ
—t
j
d
|›�¦
«

| d¦
«
d d¦
«
d}||
dœ}t¦«}| |¦
«
}d d„|D¦«¦
«
}|dkrt
|¦
«
}|d	krt||¬
¦«}|g
SdS)NzCreating image document from ú
\éÿÿÿÿú
.r)Úsourcer5Úc
3óvK
—|]4}
|
 ¦«s|
 
¦«s|
dk¯0|
V—

Œ5d
S)ú

N)ÚisalnumÚisspace)Ú.0r|s  r)ú	<genexpr>z(create_image_document.<locals>.<genexpr>Æs@èè€ÐNÐN˜ a§i¢i¡k¤kÐN°Q·Y²Y±[´[ÐNÀAÈÂIÀI�1ÀIÀIÀIÀIÐNÐNr+rur6)rr9ÚsplitrÚread_text_from_imageÚjoinr}r)rdr5Ú
image_namer8Útext_extractorr,r?s       r)Úcreate_image_documentr�½sÓ€Ý
„KÐ<°
Ð<Ð<Ñ=Ô=Ð=à×!Ò! $Ñ'Ô'¨Ô+×1Ò1°#Ñ6Ô6°qÔ9€Jà$°9Ð=Ð=€HÝ"‘_”_€NØ×.Ò.¨zÑ:Ô:€Dà
�7Š7ÐNÐN˜dÐNÑNÔNÑNÔN€Dðˆr‚z€zݘZÑ(Ô(ˆðÐ5Ò5Ð5å D°8Ð<Ñ<Ô<ˆàˆuˆàˆr+c
ó
—t
j
d
|›�¦
«

t|d¦«5}
tjj ||
 ¦«fd¬¦«}ddd¦«
n#1swxYw

Y

|jS)NzTranscribing audio file r^r)Úfilero)	rr9r_ÚclientÚaudioÚtranslationsÚcreaterbr,)Úfilepathr’Útranslations   r)Ú
audio_to_textr™×sº€Ý
„KÐ5¨8Ð5Ð5Ñ6Ô6Ð6Ý	
ˆh˜Ñ	Ô	ð
 Ý”lÔ/×6Ò6ؘDŸIšI™KœKÐ(Ø$ð7ñ
ô
ˆð
ð
ð
ñ
ô
ð
ð
ð
ð
ð
ð
øøøð
ð
ð
ð
ð
ÔÐs¨;A/Á/A3Á6
A3Tcó^—t
j
d
|›d�¦
«

|
dzdz}tj|¦
«
}t	|¦
«
}t
j |¦
«
 d¦
«
d}|›d�}t
j 	|¦
«
stj
|¦
«

g}||kr§tj||z¦
«
}	tt|	¦
«
¦
«
D]r}
|
|z}t||z|¦«}|||…}
|›d|›d	|
d
z›d�}|
 |d¬
¦«
| |¦
«

|rt%d|›�¦
«

ŒsnH|›d|›d�}| |d¬
¦«
| |¦
«

|rt%d|›�¦
«

||fS)NzSplitting audio file z by durationé<r.r‚rÚ_chunksú
/Ú_chunké
ú.mp3Úmp3)
Úformatz
Exporting z_chunk1.mp3)rr9rÚ	from_filer:ÚosÚpathÚbasenamer‹ÚexistsÚmakedirsÚmathÚceilÚrangeÚintÚminÚexportr>Úprint)Úaudio_file_pathÚchunk_duration_minutesÚprint_outputÚchunk_length_msr”Úaudio_duration_msÚ
base_filenameÚchunk_folderÚchunk_pathsÚ
num_chunksrCÚstart_msÚend_msrDÚchunk_filenames               r)Úsplit_audio_by_durationr¼ásõ
€Ý
„KÐE¨ÐEÐEÐEÑFÔFÐFà,¨rÑ1°DÑ8€Oõ
Ô" ?Ñ3Ô3€EݘE™
œ
Ðõ”G×$Ò$ _Ñ5Ô5×;Ò;¸CÑ@Ô@ÀÔC€MØ#Ð,Ð,Ð,€LÝ
Œ7�>Š>˜,Ñ'Ô'ð
"Ý
Œ�LÑ!Ô!Ð!à€Kà˜?Ò*Ð*å”YÐ0°?ÑBÑCÔCˆ
å•s˜:‘”Ñ'Ô'ð	5ð	5ˆAؘ?Ñ*ˆHݘ OÑ3Ð5FÑGÔGˆFؘ( 6˜/Ô*ˆEØ ,ÐMÐM¨}ÐMÐMÀAÀaÁCÐMÐMÐMˆNØ�LŠL˜°ˆLÑ6Ô6Ð6Ø×Ò˜~Ñ.Ô.Ð.Øð

5ÝÐ3 >Ð3Ð3Ñ4Ô4Ð4øð	5ð)ÐEÐE¨=ÐEÐEÐEˆØ
�Š�^¨EˆÑ2Ô2Ð2Ø×Ò˜>Ñ*Ô*Ð*Øð
	1ÝÐ/˜~Ð/Ð/Ñ0Ô0Ð0à˜Ð$Ð$r+r”c	ó8—t
j
d
|›�¦
«

t||
¦«\}}g}|D�
]Á}t|¦
«
}tj |¦
«
}tjd|¦«}	|	r8|	 	d¦
«
}
t|	 	d¦
«
¦
«
}n'tj |¦
«
d}
d}|dz
|
z}||
z}
t|
ttj|¦
«
¦
«
dz¦«}|dzdkr%t|¦
«
›d�}t|
¦
«
›d�}
n|t!|¦
«
 d¦
«
\}}t!|
¦
«
 d¦
«
\}}t|¦
«
d	z}t|¦
«
d	z}|›d
|›�}|›d
|›�}
|
|›d|
›�|dœ}t%||¬
¦«}| |¦
«

�
ŒÃt)j|¦
«

t-jd¦
«

|S)NzTranscribing audio chunks from z(.*)_chunk(\d+)\.mp3$rŸéri`êz:00r‚éú
:ú
-)rƒÚ	timestampr5r6gš™™™™™É?)rr9r¼r™r¤r¥r¦ÚreÚsearchÚgroupr¬Úsplitextr­r:rr£Ústrr‹rr>ÚshutilÚrmtreeÚtimeÚsleep)r°r±r5r¶r·rBÚ
chunk_pathÚ
transcriptr»ÚmatchrµÚchunk_indexÚ	start_minÚend_minÚactual_end_minÚ
start_min_intÚ
start_min_decÚend_min_intÚend_min_decÚ	start_secÚend_secr8rFs                       r)Útranscribe_audio_chunksrÙ	
sI€Ý
„KÐC°/ÐCÐCÑDÔDÐDå 7¸ÐI_Ñ `Ô `Ñ€L�+à€IØ!ð)#ñ)#ˆ
å" :Ñ.Ô.ˆ
õœ×)Ò)¨*Ñ5Ô5ˆÝ”	Ð2°NÑCÔCˆØð	Ø!ŸKšK¨
™NœNˆMݘeŸkšk¨!™nœnÑ-Ô-ˆKˆKõœG×,Ò,¨^Ñ<Ô<¸QÔ?ˆM؈Kð! 1‘_Ð(>Ñ>ˆ	ØÐ 6Ñ6ˆÝ˜W¥s­<Ô+AÀ/Ñ+RÔ+RÑ'SÔ'SÐW\Ñ'\Ñ^Ô^ˆð�q‰=˜AÒÐݘy™>œ>Ð.Ð.Ð.ˆIݘW™œÐ*Ð*Ð*ˆGˆGõ,/¨y©>¬>×+?Ò+?ÀÑ+DÔ+DÑ(ˆM˜=Ý'*¨7¡|¤|×'9Ò'9¸#Ñ'>Ô'>Ñ$ˆK˜å˜MÑ*Ô*¨QÑ.ˆIݘ+Ñ&Ô&¨Ñ*ˆGØ(Ð6Ð6¨9Ð6Ð6ˆIØ$Ð0Ð0 wÐ0Ð0ˆGð(Ø )Ð5Ð5¨GÐ5Ð5Ø&ððˆõ
¨¸hÐGÑGÔGˆØ×Ò˜Ñ"Ô"Ð"Ñ"õ„M�,ÑÔÐõ	„Jˆs�O„O€OàÐr+écó(—t
||
|¦«}|S)
N)
rÙ)r°r±r5rBs    r)Úcreate_audio_documentrÜE
s€Ý'¨Ð9OÐQZÑ[Ô[€IØÐr+Ú
video_pathÚ
time_intervalcó>—t
j
d
|›�¦
«

t|¦
«
}|j}| dd¦«}|j |¦
«
}tj 	tj 
|¦
«
¦
«
d}tj tj |¦
«
|›¦«}tj
|d¬¦«
tj|¦
«
}t!|dd¦
«
}t#dt%|¦
«
|
¦«D]‹}	|	}
t'|	|
zt%|¦
«
¦«}|
›d	|›�}tj ||›d
�¦«}
tj||
¬¦« |
d¬
¦« ¦«
ŒŒt/d|›d�¦
«

t1|dd¬¦«}t
j
d|›�¦
«

tj|¦
«

|S)NzPreprocessing video data from z.mp4r rT©
Úexist_okr¢ÚdurationrÁzs.png)
ÚssrŸ)
ÚvframeszSnapshots saved in r‚çà?Úvideo)r±r5zDocuments created from video )rr9rrâÚreplacer”Úwrite_audiofiler¤r¥rÆr¦r�Údirnamer¨ÚffmpegÚprobeÚfloatr«r¬r­ÚinputÚoutputÚrunr¯rÜÚremove)rÝrÞrærâÚ
audio_pathÚ
_Ú
video_nameÚsnapshot_dirrërCÚ
start_timeÚend_timeÚinterval_strÚ	frame_imgrBs               r)Úpreprocess_video_datarùJ
s€Ý
„KÐ=°Ð=Ð=Ñ>Ô>Ð>å˜*Ñ%Ô%€EðŒ~€Hð×#Ò# F¨FÑ3Ô3€JØ
Œ×#Ò# JÑ/Ô/€Aõ”×!Ò!¥"¤'×"2Ò"2°:Ñ">Ô">Ñ?Ô?À
ÔB€Jõ”7—<’<¥¤§¢°
Ñ ;Ô ;À
¸_ÑMÔM€LÝ„K� tÐ,Ñ,Ô,Ð,õ
ŒL˜Ñ$Ô$€EÝ�U˜8”_ ZÔ0Ñ1Ô1€Hõ�1•c˜(‘m”m ]Ñ
3Ô
3ð
ð
ˆ
؈
Ý�q˜=Ñ(­#¨h©-¬-Ñ8Ô8ˆð%Ð1Ð1 xÐ1Ð1ˆõ”G—L’L °,Ð/EÐ/EÐ/EÑFÔFˆ	õ
Ü
�: *ð

.ñ

.ô

.ç
ŠV�I qˆVÑ
)Ô
)ß
ŠS‰UŒUˆUˆUõ
Ð
/ Ð
/Ð
/Ð
/Ñ0Ô0Ð0õ& jÈÐX_Ð`Ñ`Ô`€IÝ
„KÐ<°
Ð<Ð<Ñ=Ô=Ð=õ„IˆjÑÔÐØÐr+Ú
document_pageÚreturnc
ó²—t
j
d
¦
«

tddg
¬¦«}
|
tzt	¦«z}| d|i
¦
«
}|S)NzSummarizing documentao
<|begin_of_text|><|start_header_id|>system<|end_header_id|>
        Create a short summary of the document based on the provided text. 
        
        Start with: This document is about...
        
        <|eot_id|><|start_header_id|>user<|end_header_id|>
        DOCUMENT: {document_page} 

        
        <|eot_id|><|start_header_id|>assistant<|end_header_id|>rú)ÚtemplateÚinput_variables)rr9rÚGROQ_LLMr
Úinvoke)rúÚinitiator_promptÚinitiator_routerrîs    r)Údoc_summarizerr
‚
sk€Ý
„KÐ'Ñ(Ô(Ð(Ý%ðC
ð)Ð)ðñôÐð(­(Ñ2µ_Ñ5FÔ5FÑFÐØ