data_ingestion/__pycache__/utils.cpython-311.pyc

§
ÅÊ»fZHãóB—ddlmZddlmZddlZddlmZddlmZddl	m
Z
ddl	mZddl	mZdd	l
mZdd
lmZddlmZddlZddlZddlmZddlZddlZddlZdd
lmZddlZddlZddlm Z ddl!Z!ddl"m#Z#e#¦«ej$d¦«a%eej$d¦«¬¦«Z&dZ'd„Z(e(¦«Z)d5d„Z*d„Z+d„Z,d„Z-d„Z.d„Z/d„Z0d6d„Z1d„Z2d7d!„Z3d8d#„Z4d9d%„Z5d&e6d'e7fd(„Z8d:d*„Z9e)d)fd+„Z:d,e6fd-„Z;d.e<d/e<d0e<fd1„Z=d.e<d/e<d0e<fd2„Z>e:¦«Z?d;d4„Z@dS)<é©ÚHuggingFaceBgeEmbeddings)ÚRecursiveCharacterTextSplitterN)ÚInMemoryDocstore)ÚFAISS)ÚPyPDFLoader)Ú
TextLoader)ÚDocx2txtLoader)Úuuid4)ÚDocument)Ú
TextExtractor)ÚGroq)ÚAudioSegment)Ú
VideoFileClip)Úload_dotenvÚOPENAI_API_KEYÚGROQ_API_KEY)Úapi_keyúwhisper-large-v3có>—d}ddi}ddi}t|||¬¦«}|S)NzBAAI/bge-small-enÚdeviceÚcudaÚnormalize_embeddingsT)Ú
model_nameÚmodel_kwargsÚ
encode_kwargsr)rrrÚ
embeddingss    úWc:\Users\timmy_3aupohg\Downloads\Manaknight Projects\ds_aiindex\data_ingestion\utils.pyÚload_embedding_modelr"s?€Ø$€JØ˜fÐ%€LØ+¨TÐ2€MÝ)Ø%°LÐP]ðñô€JðÐóÚtextcóV—|dj}|dj}tddtd¬¦«}| |g¦«}g}t|¦«D]N\}}| ¦«}	||	d<||	d<t|j|	¬¦«}
| |
¦«ŒO|S)	Nréèé
F)Ú
chunk_sizeÚ
chunk_overlapÚlength_functionÚis_separator_regexÚpageÚ	file_type©Úpage_contentÚmetadata)	r,r-rÚlenÚcreate_documentsÚ	enumerateÚcopyrÚappend)Údocr*r!r-Ú
text_splitterÚdocsÚ	documentsÚiÚchunkÚdoc_metadataÚdocuments           rr/r/1sÁ€ØˆqŒ6Ô€DØ�1ŒvŒ€HÝ2ØØÝØ ð	ñô€Mð×)Ò)¨4¨&Ñ1Ô1€Dà€IÝ˜d‘O”Oð#ð#‰ˆˆ5à—}’}‘”ˆØ ˆ�VÑØ$-ˆ�[Ñ!Ý¨Ô);ÀlÐSÑSÔSˆØ×Ò˜Ñ"Ô"Ð"Ð"ØÐr cóš—	t|¦«}| ¦«}t|¦«}|S#td|›�¦«‚xYw©NzError loading -- )r	Úloadr/Ú
ValueError)Ú
document_pathÚtxt_docr!r5s    rÚload_txt_documentrAGsS€ð>Ý˜]Ñ+Ô+ˆØ�|Š|‰~Œ~ˆå Ñ%Ô%ˆØˆøð>ÝÐ<¨]Ð<Ð<Ñ=Ô=Ð=øøøó	‚36¶A
cóš—	t|¦«}| ¦«}t|¦«}|S#td|›�¦«‚xYwr<)r
r=r/r>)r?Údocx_docr!r5s    rÚload_docx_documentrERsS€ð>Ý! -Ñ0Ô0ˆØ�}Š}‰Œˆå Ñ%Ô%ˆØˆøð>ÝÐ<¨]Ð<Ð<Ñ=Ô=Ð=øøørBcó|—	t|¦«}| ¦«}|S#td|›�¦«‚xYwr<)rÚload_and_splitr>)r?Úpdf_docÚpagess   rÚload_pdf_documentrJ^sJ€ð>Ý˜mÑ,Ô,ˆØ×&Ò&Ñ(Ô(ˆØˆøð>ÝÐ<¨]Ð<Ð<Ñ=Ô=Ð=øøøs‚$'§;cóþ—| d¦«rt|¦«S| d¦«rt|¦«S| d¦«rt|¦«St	d|›�¦«‚)Nz.pdfz.txtz.docxzUnsupported document type for )ÚendswithrJrArEr>)r?s rÚ
load_documentrMhsˆ€Ø×Ò˜fÑ%Ô%ðKÝ  Ñ/Ô/Ð/Ø	×	Ò	 Ñ	'Ô	'ðKÝ  Ñ/Ô/Ð/Ø	×	Ò	 Ñ	(Ô	(ðKÝ! -Ñ0Ô0Ð0åÐI¸-ÐIÐIÑJÔJÐJr cóÈ—t|d¦«5}tj| ¦«¦« d¦«cddd¦«S#1swxYwYdS)NÚrbzutf-8)ÚopenÚbase64Ú	b64encodeÚreadÚdecode)Ú
image_pathÚ
image_files  rÚencode_imagerWts™€ÝˆJ˜ÑÔð? ÝÔ˜JŸOšOÑ-Ô-Ñ.Ô.×5Ò5°gÑ>Ô>ð?ð?ð?ð?ñ?ô?ð?ð?ð?ð?ð?ð?øøøð?ð?ð?ð?ð?ð?s‘9AÁAÁAcó(—t|¦«}ddt›�dœ}	dddddœd	d
d|›�idœgd
œgddœ}tjd||¬¦«}| ¦«dddd}n#t
$r}d}Yd}~nd}~wwxYw|S)Nzapplication/jsonzBearer )zContent-TypeÚ
Authorizationzgpt-4o-miniÚuserr!uWhatâ€™s in this image?)Útyper!Ú	image_urlÚurlzdata:image/jpeg;base64,)r[r\)ÚroleÚcontenti,)ÚmodelÚmessagesÚ
max_tokensz*https://api.openai.com/v1/chat/completions)ÚheadersÚjsonÚchoicesrÚmessager_ú$Image not good enough for processing)rWrÚrequestsÚpostrdÚ	Exception)rUÚbase64_imagercÚpayloadÚresponseÚes      rÚ
process_imageroys€õ  
Ñ+Ô+€Lð+Ø,¥7Ð,Ð,ðð€Gð
:à"ð#ð%+Ø$=ððð
%0à %Ð'OÀÐ'OÐ'Oð*ððð ðððð"ð'
ð
ˆõ,”=Ð!MÐW^ÐelÐmÑmÔmˆà—=’=‘?”? 9Ô-¨aÔ0°Ô;¸IÔFˆˆøÝð:ð:ð:Ø9ˆˆˆˆˆˆøøøøð:øøøð€OsžAA9Á9
BÂB
Â
BÚimagecóZ—| d¦«d d¦«d}||dœ}t¦«}| |¦«}d d„|D¦«¦«}|dkrt	|¦«}|dkrt||¬	¦«}|gSdS)
Nú/éÿÿÿÿú.r)Úfilenamer*Úc3óvK—|]4}| ¦«s| ¦«s|dk¯0|V—Œ5dS)ú
N)ÚisalnumÚisspace)Ú.0rns  rú	<genexpr>z(create_image_document.<locals>.<genexpr>s@èè€ÐNÐN˜ a§i¢i¡k¤kÐN°Q·Y²Y±[´[ÐNÀAÈÂIÀI�1ÀIÀIÀIÀIÐNÐNr rgr+)Úsplitr
Úread_text_from_imageÚjoinror)rUr*Ú
image_namer-Útext_extractorr!r3s       rÚcreate_image_documentr‚¥sº€à×!Ò! #Ñ&Ô& rÔ*×0Ò0°Ñ5Ô5°aÔ8€Jà&°YÐ?Ð?€HÝ"‘_”_€NØ×.Ò.¨zÑ:Ô:€Dà
�7Š7ÐNÐN˜dÐNÑNÔNÑNÔN€Dðˆr‚z€zÝ˜ZÑ(Ô(ˆðÐ5Ò5Ð5å D°8Ð<Ñ<Ô<ˆàˆuˆàˆr cóÔ—t|d¦«5}tjj || ¦«fd¬¦«}ddd¦«n#1swxYwY|jS)NrOr)Úfiler`)rPÚclientÚaudioÚtranslationsÚcreaterSr!)Úfilepathr„Útranslations   rÚ
audio_to_textr‹¾s¡€Ý	
ˆh˜Ñ	Ô	ð
 Ý”lÔ/×6Ò6Ø˜DŸIšI™KœKÐ(Ø$ð7ñ
ô
ˆð
ð
ð
ñ
ô
ð
ð
ð
ð
ð
ð
øøøð
ð
ð
ð
ð
ÔÐs‘;AÁAÁATcó—|dzdz}tj|¦«}t|¦«}tj |¦« d¦«d}|›d�}tj |¦«stj|¦«g}||kr–||z||zdkrdndz}	t|	¦«D]r}
|
|z}t||z|¦«}|||…}
|›d|›d|
dz›d	�}|
 |d
¬¦«| |¦«|rtd|›�¦«ŒsnH|›d|›d
�}| |d
¬¦«| |¦«|rtd|›�¦«||fS)Né<r#rtrÚ_chunksérrÚ_chunkú.mp3Úmp3)Úformatz
Exporting z_chunk1.mp3)rÚ	from_filer.ÚosÚpathÚbasenamer}ÚexistsÚmakedirsÚrangeÚminÚexportr2Úprint)Úaudio_file_pathÚchunk_duration_minutesÚprint_outputÚchunk_length_msr†Úaudio_duration_msÚ
base_filenameÚchunk_folderÚchunk_pathsÚ
num_chunksr7Ústart_msÚend_msr8Úchunk_filenames               rÚsplit_audio_by_durationrªÇsä€à,¨rÑ1°DÑ8€Oõ
Ô" ?Ñ3Ô3€EÝ˜E™
œ
Ðõ”G×$Ò$ _Ñ5Ô5×;Ò;¸CÑ@Ô@ÀÔC€MØ#Ð,Ð,Ð,€LÝ
Œ7�>Š>˜,Ñ'Ô'ð"Ý
Œ�LÑ!Ô!Ð!à€Kà˜?Ò*Ð*à&¨/Ñ9ÐBSÐVeÑBeÐijÒBjÐBj¸Q¸QÐpqÑrˆ
å�zÑ"Ô"ð	5ð	5ˆAØ˜?Ñ*ˆHÝ˜ OÑ3Ð5FÑGÔGˆFØ˜( 6˜/Ô*ˆEØ ,ÐMÐM¨}ÐMÐMÀAÀaÁCÐMÐMÐMˆNØ�LŠL˜°ˆLÑ6Ô6Ð6Ø×Ò˜~Ñ.Ô.Ð.Øð
5ÝÐ3 >Ð3Ð3Ñ4Ô4Ð4øð	5ð)ÐEÐE¨=ÐEÐEÐEˆØ
�Š�^¨EˆÑ2Ô2Ð2Ø×Ò˜>Ñ*Ô*Ð*Øð	1ÝÐ/˜~Ð/Ð/Ñ0Ô0Ð0à˜Ð$Ð$r r†c	ó�—t||¦«\}}g}|D�]}t|¦«}tj |¦«}tjd|¦«}	|	r8|	 d¦«}
t|	 d¦«¦«}n'tj 	|¦«d}
d}|dz
|z}||z}
t|
ttj
|¦«¦«dz¦«}|
|›d|
›d�|dœ}t||¬	¦«}| |¦«�Œt!j|¦«|S)
Nz(.*)_chunk(\d+)\.mp3$r�éri`êú-z minutes)ruÚdurationr*r+)rªr‹r•r–r—ÚreÚsearchÚgroupÚintÚsplitextr›r.rr”rr2ÚshutilÚrmtree)ržrŸr*r¤r¥r6Ú
chunk_pathÚ
transcriptr©Úmatchr£Úchunk_indexÚ	start_minÚend_minÚactual_end_minr-r:s                 rÚtranscribe_audio_chunksr½îsa€å 7¸ÐI_Ñ `Ô `Ñ€L�+à€IØ!ð#ñ#ˆ
å" :Ñ.Ô.ˆ
õœ×)Ò)¨*Ñ5Ô5ˆÝ”	Ð2°NÑCÔCˆØð	Ø!ŸKšK¨™NœNˆMÝ˜eŸkšk¨!™nœnÑ-Ô-ˆKˆKõœG×,Ò,¨^Ñ<Ô<¸QÔ?ˆMØˆKð! 1‘_Ð(>Ñ>ˆ	ØÐ 6Ñ6ˆÝ˜W¥s<Ô+AÀ/Ñ+RÔ+RÑ'SÔ'SÐW\Ñ'\Ñ^Ô^ˆð*Ø(Ð<Ð<¨7Ð<Ð<Ð<Ø&ððˆõ
¨¸hÐGÑGÔGˆØ×Ò˜Ñ"Ô"Ð"Ñ"õ„M�,ÑÔÐàÐr écó(—t|||¦«}|S)N)r½)ržrŸr*r6s    rÚcreate_audio_documentrÀs€Ý'¨Ð9OÐQZÑ[Ô[€IØÐr Ú
video_pathÚ
time_intervalcóz—t|¦«}|j}| dd¦«}|j |¦«}t
j t
j |¦«¦«d}t
j 	t
j 
|¦«|›d�¦«}tj|d¬¦«d}tj
|¦«}	t|	dd	¦«}tdt!|¦«|¦«D]h}
|
}t
j 	|d
|dz›d�¦«}tj||¬
¦« |d¬¦« ¦«Œit)d|›d�¦«t+|d¬¦«}
|
S)Nz.mp4r‘rÚ
_snapshotsT)Úexist_oké´r“r®Ú	frame_at_r�zmin.png)Ússr�)ÚvframeszSnapshots saved in rtÚvideo)r*)rr®Úreplacer†Úwrite_audiofiler•r–r³r—rÚdirnamer™ÚffmpegÚprobeÚfloatršr²ÚinputÚoutputÚrunr�rÀ)rÁrÂrÊr®Ú
audio_pathÚ_Ú
video_nameÚsnapshot_dirÚintervalrÏr7Ú
frame_timeÚ	frame_imgr6s              rÚpreprocess_video_datarÛsŸ€õ
˜*Ñ%Ô%€EðŒ~€Hð×#Ò# F¨FÑ3Ô3€JØ
Œ×#Ò# JÑ/Ô/€Aõ”×!Ò!¥"¤'×"2Ò"2°:Ñ">Ô">Ñ?Ô?ÀÔB€Jõ”7—<’<¥¤§¢°
Ñ ;Ô ;À
Ð=VÐ=VÐ=VÑWÔW€LÝ„K� tÐ,Ñ,Ô,Ð,ð€Hõ
-											Pdf Ingestion pipeline completed
										
										
											2024-08-05 22:14:19 +01:00
+								§
-											updated utils.. working on the api
										
										
											2024-08-13 22:16:12 +01:00
+								ÅÊ»fZHãóB—ddlmZddlmZddlZddlmZddlmZddl	m
-											complete document ingestion pipeline
										
										
											2024-08-07 17:50:40 +01:00
+								Z
 								ddl	mZddl	mZdd	l
-											updated utils.. working on the api
										
										
											2024-08-13 22:16:12 +01:00
+								mZdd
-											Indexing completed.
										
										
											2024-08-13 21:30:01 +01:00
+								lmZddlmZddlZddlZddlmZddlZddlZddlZdd
 								lmZddlZddlZddlm Z ddl!Z!ddl"m#Z#e#¦«ej$d¦«a%eej$d¦«¬¦«Z&dZ'd„Z(e(¦«Z)d5d„Z*d„Z+d„Z,d„Z-d„Z.d„Z/d„Z0d6d„Z1d„Z2d7d!„Z3d8d#„Z4d9d%„Z5d&e6d'e7fd(„Z8d:d*„Z9e)d)fd+„Z:d,e6fd-„Z;d.e<d/e<d0e<fd1„Z=d.e<d/e<d0e<fd2„Z>e:¦«Z?d;d4„Z@dS)<é©ÚHuggingFaceBgeEmbeddings)ÚRecursiveCharacterTextSplitterN)ÚInMemoryDocstore)ÚFAISS)ÚPyPDFLoader)Ú
 								TextLoader)ÚDocx2txtLoader)Úuuid4)ÚDocument)Ú
 								TextExtractor)ÚGroq)ÚAudioSegment)Ú
 								VideoFileClip)Úload_dotenvÚOPENAI_API_KEYÚGROQ_API_KEY)Úapi_keyúwhisper-large-v3có>—d}ddi}ddi}t|||¬¦«}|S)NzBAAI/bge-small-enÚdeviceÚcudaÚnormalize_embeddingsT)Ú
-											complete document ingestion pipeline
										
										
											2024-08-07 17:50:40 +01:00
+								model_nameÚmodel_kwargsÚ
-											Indexing completed.
										
										
											2024-08-13 21:30:01 +01:00
+								encode_kwargsr)rrrÚ
 								embeddingss    úWc:\Users\timmy_3aupohg\Downloads\Manaknight Projects\ds_aiindex\data_ingestion\utils.pyÚload_embedding_modelr"s?€Ø$€JØ˜fÐ%€LØ+¨TÐ2€MÝ)Ø%°LÐP]ðñô€JðÐóÚtextcóV—|dj}|dj}tddtd¬¦«}| |g¦«}g}t|¦«D]N\}}| ¦«}	||	d<||	d<t|j|	¬¦«}
 								| |
 								¦«ŒO|S)	Nréèé
-											image text extraction completed
										
										
											2024-08-08 14:58:44 +01:00
+								F)Ú
-											Indexing completed.
										
										
											2024-08-13 21:30:01 +01:00
+								chunk_sizeÚ
-											image pipeline perfected. audio pipeline in progress
										
										
											2024-08-08 22:06:39 +01:00
+								chunk_overlapÚlength_functionÚis_separator_regexÚpageÚ	file_type©Úpage_contentÚmetadata)	r,r-rÚlenÚcreate_documentsÚ	enumerateÚcopyrÚappend)Údocr*r!r-Ú
-											Indexing completed.
										
										
											2024-08-13 21:30:01 +01:00
+								text_splitterÚdocsÚ	documentsÚiÚchunkÚdoc_metadataÚdocuments           rr/r/1sÁ€ØˆqŒ6Ô€DØ�1ŒvŒ€HÝ2ØØÝØ ð	ñô€Mð×)Ò)¨4¨&Ñ1Ô1€Dà€IÝ˜d‘O”Oð#ð#‰ˆˆ5à—}’}‘”ˆØ ˆ�VÑØ$-ˆ�[Ñ!Ý¨Ô);ÀlÐSÑSÔSˆØ×Ò˜Ñ"Ô"Ð"Ð"ØÐr cóš—	t|¦«}| ¦«}t|¦«}|S#td|›�¦«‚xYw©NzError loading -- )r	Úloadr/Ú
 								ValueError)Ú
-											image pipeline perfected. audio pipeline in progress
										
										
											2024-08-08 22:06:39 +01:00
+								document_pathÚtxt_docr!r5s    rÚload_txt_documentrAGsS€ð>Ý˜]Ñ+Ô+ˆØ�|Š|‰~Œ~ˆå Ñ%Ô%ˆØˆøð>ÝÐ<¨]Ð<Ð<Ñ=Ô=Ð=øøøó	‚36¶A
 								cóš—	t|¦«}| ¦«}t|¦«}|S#td|›�¦«‚xYwr<)r
 								r=r/r>)r?Údocx_docr!r5s    rÚload_docx_documentrERsS€ð>Ý! -Ñ0Ô0ˆØ�}Š}‰Œˆå Ñ%Ô%ˆØˆøð>ÝÐ<¨]Ð<Ð<Ñ=Ô=Ð=øøørBcó|—	t|¦«}| ¦«}|S#td|›�¦«‚xYwr<)rÚload_and_splitr>)r?Úpdf_docÚpagess   rÚload_pdf_documentrJ^sJ€ð>Ý˜mÑ,Ô,ˆØ×&Ò&Ñ(Ô(ˆØˆøð>ÝÐ<¨]Ð<Ð<Ñ=Ô=Ð=øøøs‚$'§;cóþ—| d¦«rt|¦«S| d¦«rt|¦«S| d¦«rt|¦«St	d|›�¦«‚)Nz.pdfz.txtz.docxzUnsupported document type for )ÚendswithrJrArEr>)r?s rÚ
 								load_documentrMhsˆ€Ø×Ò˜fÑ%Ô%ðKÝ  Ñ/Ô/Ð/Ø	×	Ò	 Ñ	'Ô	'ðKÝ  Ñ/Ô/Ð/Ø	×	Ò	 Ñ	(Ô	(ðKÝ! -Ñ0Ô0Ð0åÐI¸-ÐIÐIÑJÔJÐJr cóÈ—t|d¦«5}tj| ¦«¦« d¦«cddd¦«S#1swxYwYdS)NÚrbzutf-8)ÚopenÚbase64Ú	b64encodeÚreadÚdecode)Ú
 								image_pathÚ
 								image_files  rÚencode_imagerWts™€ÝˆJ˜ÑÔð? ÝÔ˜JŸOšOÑ-Ô-Ñ.Ô.×5Ò5°gÑ>Ô>ð?ð?ð?ð?ñ?ô?ð?ð?ð?ð?ð?ð?øøøð?ð?ð?ð?ð?ð?s‘9AÁAÁAcó(—t|¦«}ddt›�dœ}	dddddœd	d
 								d|›�idœgd
-											Indexing completed.
										
										
											2024-08-13 21:30:01 +01:00
+								œgddœ}tjd||¬¦«}| ¦«dddd}n#t
 								$r}d}Yd}~nd}~wwxYw|S)Nzapplication/jsonzBearer )zContent-TypeÚ
 								Authorizationzgpt-4o-miniÚuserr!uWhatâ€™s in this image?)Útyper!Ú	image_urlÚurlzdata:image/jpeg;base64,)r[r\)ÚroleÚcontenti,)ÚmodelÚmessagesÚ
 								max_tokensz*https://api.openai.com/v1/chat/completions)ÚheadersÚjsonÚchoicesrÚmessager_ú$Image not good enough for processing)rWrÚrequestsÚpostrdÚ	Exception)rUÚbase64_imagercÚpayloadÚresponseÚes      rÚ
-											audio preprocessing pipeline completed.
										
										
											2024-08-09 16:33:21 +01:00
+								process_imageroys€õ
 								Ñ+Ô+€Lð+Ø,¥7Ð,Ð,ðð€Gð
 								:à"ð#ð%+Ø$=ððð
 								%0à %Ð'OÀÐ'OÐ'Oð*ððð ðððð"ð'
 								ð
 								ˆõ,”=Ð!MÐW^ÐelÐmÑmÔmˆà—=’=‘?”? 9Ô-¨aÔ0°Ô;¸IÔFˆˆøÝð:ð:ð:Ø9ˆˆˆˆˆˆøøøøð:øøøð€OsžAA9Á9
 								BÂB
 								Â
 								BÚimagecóZ—| d¦«d d¦«d}||dœ}t¦«}| |¦«}d d„|D¦«¦«}|dkrt	|¦«}|dkrt||¬	¦«}|gSdS)
 								Nú/éÿÿÿÿú.r)Úfilenamer*Úc3óvK—|]4}| ¦«s| ¦«s|dk¯0|V—Œ5dS)ú
 								N)ÚisalnumÚisspace)Ú.0rns  rú	<genexpr>z(create_image_document.<locals>.<genexpr>s@èè€ÐNÐN˜ a§i¢i¡k¤kÐN°Q·Y²Y±[´[ÐNÀAÈÂIÀI�1ÀIÀIÀIÀIÐNÐNr rgr+)Úsplitr
 								Úread_text_from_imageÚjoinror)rUr*Ú
 								image_namer-Útext_extractorr!r3s       rÚcreate_image_documentr‚¥sº€à×!Ò! #Ñ&Ô& rÔ*×0Ò0°Ñ5Ô5°aÔ8€Jà&°YÐ?Ð?€HÝ"‘_”_€NØ×.Ò.¨zÑ:Ô:€Dà
 								�7Š7ÐNÐN˜dÐNÑNÔNÑNÔN€Dðˆr‚z€zÝ˜ZÑ(Ô(ˆðÐ5Ò5Ð5å D°8Ð<Ñ<Ô<ˆàˆuˆàˆr cóÔ—t|d¦«5}tjj || ¦«fd¬¦«}ddd¦«n#1swxYwY|jS)NrOr)Úfiler`)rPÚclientÚaudioÚtranslationsÚcreaterSr!)Úfilepathr„Útranslations   rÚ
 								audio_to_textr‹¾s¡€Ý
 								ˆh˜Ñ	Ô	ð
 								 Ý”lÔ/×6Ò6Ø˜DŸIšI™KœKÐ(Ø$ð7ñ
 								ô
 								ˆð
 								ð
 								ð
 								ñ
 								ô
-											Indexing completed.
										
										
											2024-08-13 21:30:01 +01:00
+								ð
 								ð
 								ð
-											audio preprocessing pipeline completed.
										
										
											2024-08-09 16:33:21 +01:00
+								ð
 								ð
 								ð
-											Indexing completed.
										
										
											2024-08-13 21:30:01 +01:00
+								øøøð
 								ð
 								ð
 								ð
 								ð
 								ÔÐs‘;AÁAÁATcó—|dzdz}tj|¦«}t|¦«}tj |¦« d¦«d}|›d�}tj |¦«stj|¦«g}||kr–||z||zdkrdndz}	t|	¦«D]r}
-											audio preprocessing pipeline completed.
										
										
											2024-08-09 16:33:21 +01:00
+								|
-											Indexing completed.
										
										
											2024-08-13 21:30:01 +01:00
+								|z}t||z|¦«}|||…}
-											audio preprocessing pipeline completed.
										
										
											2024-08-09 16:33:21 +01:00
+								|›d|›d|
-											Indexing completed.
										
										
											2024-08-13 21:30:01 +01:00
+								dz›d	�}|
 								 |d
 								¬¦«| |¦«|rtd|›�¦«ŒsnH|›d|›d
 								�}| |d
 								¬¦«| |¦«|rtd|›�¦«||fS)Né<r#rtrÚ_chunksérrÚ_chunkú.mp3Úmp3)Úformatz
 								Exporting z_chunk1.mp3)rÚ	from_filer.ÚosÚpathÚbasenamer}ÚexistsÚmakedirsÚrangeÚminÚexportr2Úprint)Úaudio_file_pathÚchunk_duration_minutesÚprint_outputÚchunk_length_msr†Úaudio_duration_msÚ
 								base_filenameÚchunk_folderÚchunk_pathsÚ
 								num_chunksr7Ústart_msÚend_msr8Úchunk_filenames               rÚsplit_audio_by_durationrªÇsä€à,¨rÑ1°DÑ8€Oõ
 								Ô" ?Ñ3Ô3€EÝ˜E™
 								œ
 								Ðõ”G×$Ò$ _Ñ5Ô5×;Ò;¸CÑ@Ô@ÀÔC€MØ#Ð,Ð,Ð,€LÝ
 								Œ7�>Š>˜,Ñ'Ô'ð"Ý
 								Œ�LÑ!Ô!Ð!à€Kà˜?Ò*Ð*à&¨/Ñ9ÐBSÐVeÑBeÐijÒBjÐBj¸Q¸QÐpqÑrˆ
 								å�zÑ"Ô"ð	5ð	5ˆAØ˜?Ñ*ˆHÝ˜ OÑ3Ð5FÑGÔGˆFØ˜( 6˜/Ô*ˆEØ ,ÐMÐM¨}ÐMÐMÀAÀaÁCÐMÐMÐMˆNØ�LŠL˜°ˆLÑ6Ô6Ð6Ø×Ò˜~Ñ.Ô.Ð.Øð
 ÝÐ3 >Ð3Ð3Ñ4Ô4Ð4øð	5ð)ÐEÐE¨=ÐEÐEÐEˆØ
 								�Š�^¨EˆÑ2Ô2Ð2Ø×Ò˜>Ñ*Ô*Ð*Øð	1ÝÐ/˜~Ð/Ð/Ñ0Ô0Ð0à˜Ð$Ð$r r†c	ó�—t||¦«\}}g}|D�]}t|¦«}tj |¦«}tjd|¦«}	|	r8|	 d¦«}
 								t|	 d¦«¦«}n'tj 	|¦«d}
 								d}|dz
 								|z}||z}
 								t|
 								ttj
 								|¦«¦«dz¦«}|
 								|›d|
 								›d�|dœ}t||¬	¦«}| |¦«�Œt!j|¦«|S)
 								Nz(.*)_chunk(\d+)\.mp3$r�éri`êú-z minutes)ruÚdurationr*r+)rªr‹r•r–r—ÚreÚsearchÚgroupÚintÚsplitextr›r.rr”rr2ÚshutilÚrmtree)ržrŸr*r¤r¥r6Ú
 								chunk_pathÚ
 								transcriptr©Úmatchr£Úchunk_indexÚ	start_minÚend_minÚactual_end_minr-r:s                 rÚtranscribe_audio_chunksr½îsa€å 7¸ÐI_Ñ `Ô `Ñ€L�+à€IØ!ð#ñ#ˆ
 								å" :Ñ.Ô.ˆ
-											image text extraction completed
										
										
											2024-08-08 14:58:44 +01:00
+								õœ×)Ò)¨*Ñ5Ô5ˆÝ”	Ð2°NÑCÔCˆØð	Ø!ŸKšK¨™NœNˆMÝ˜eŸkšk¨!™nœnÑ-Ô-ˆKˆKõœG×,Ò,¨^Ñ<Ô<¸QÔ?ˆMØˆKð! 1‘_Ð(>Ñ>ˆ	ØÐ 6Ñ6ˆÝ˜W¥s<Ô+AÀ/Ñ+RÔ+RÑ'SÔ'SÐW\Ñ'\Ñ^Ô^ˆð*Ø(Ð<Ð<¨7Ð<Ð<Ð<Ø&ððˆõ
-											audio preprocessing pipeline completed.
										
										
											2024-08-09 16:33:21 +01:00
+								¨¸hÐGÑGÔGˆØ×Ò˜Ñ"Ô"Ð"Ñ"õ„M�,ÑÔÐàÐr écó(—t|||¦«}|S)N)r½)ržrŸr*r6s    rÚcreate_audio_documentrÀs€Ý'¨Ð9OÐQZÑ[Ô[€IØÐr Ú
-											image text extraction completed
										
										
											2024-08-08 14:58:44 +01:00
+								video_pathÚ
 								time_intervalcóz—t|¦«}|j}| dd¦«}|j |¦«}t
 								j t
-											Indexing completed.
										
										
											2024-08-13 21:30:01 +01:00
+								j |¦«¦«d}t
-											audio preprocessing pipeline completed.
										
										
											2024-08-09 16:33:21 +01:00
+								j 	t
 								j
-											Indexing completed.
										
										
											2024-08-13 21:30:01 +01:00
+								|¦«|›d�¦«}tj|d¬¦«d}tj
 								|¦«}	t|	dd	¦«}tdt!|¦«|¦«D]h}
 								|
 								}t
 								j 	|d
 								|dz›d�¦«}tj||¬
-											updated utils.. working on the api
										
										
											2024-08-13 22:16:12 +01:00
+								¦« |d¬¦« ¦«Œit)d|›d�¦«t+|d¬¦«}
 								|
 								S)Nz.mp4r‘rÚ
 								_snapshotsT)Úexist_oké´r“r®Ú	frame_at_r�zmin.png)Ússr�)ÚvframeszSnapshots saved in rtÚvideo)r*)rr®Úreplacer†Úwrite_audiofiler•r–r³r—rÚdirnamer™ÚffmpegÚprobeÚfloatršr²ÚinputÚoutputÚrunr�rÀ)rÁrÂrÊr®Ú
 								audio_pathÚ_Ú
-											image pipeline perfected. audio pipeline in progress
										
										
											2024-08-08 22:06:39 +01:00
+								video_nameÚsnapshot_dirÚintervalrÏr7Ú
-											audio preprocessing pipeline completed.
										
										
											2024-08-09 16:33:21 +01:00
+								frame_timeÚ	frame_imgr6s              rÚpreprocess_video_datarÛsŸ€õ
 								˜*Ñ%Ô%€EðŒ~€Hð×#Ò# F¨FÑ3Ô3€JØ
-											Indexing completed.
										
										
											2024-08-13 21:30:01 +01:00
+								Œ×#Ò# JÑ/Ô/€Aõ”×!Ò!¥"¤'×"2Ò"2°:Ñ">Ô">Ñ?Ô?ÀÔB€Jõ”7—<’<¥¤§¢°
 								Ñ ;Ô ;À
-											updated utils.. working on the api
										
										
											2024-08-13 22:16:12 +01:00
+								Ð=VÐ=VÐ=VÑWÔW€LÝ„K� tÐ,Ñ,Ô,Ð,ð€Hõ