Files
2025-04-28 22:00:47 +01:00

897 lines
57 KiB
Plaintext

2025-04-21 20:33:27,700 - root - INFO - Processing upload for document ID: 78afc395-9b7c-4388-8d8b-aa1d02fbf75f
2025-04-21 20:33:27,705 - root - INFO - File saved to data/uploads/78afc395-9b7c-4388-8d8b-aa1d02fbf75f_2.Tender Specifications.docx
2025-04-21 20:33:27,722 - root - INFO - Processing document 78afc395-9b7c-4388-8d8b-aa1d02fbf75f with content length: 17509
2025-04-21 20:33:28,023 - httpx - INFO - HTTP Request: POST https://api.cohere.com/v1/embed "HTTP/1.1 200 OK"
2025-04-21 20:33:28,939 - root - INFO - Stored embedding for document 78afc395-9b7c-4388-8d8b-aa1d02fbf75f
2025-04-21 20:34:54,473 - httpx - INFO - HTTP Request: POST https://api.cohere.com/v1/rerank "HTTP/1.1 200 OK"
2025-04-21 20:36:26,368 - root - INFO - Document 78afc395-9b7c-4388-8d8b-aa1d02fbf75f processed successfully
2025-04-21 20:36:26,384 - root - INFO - Document 78afc395-9b7c-4388-8d8b-aa1d02fbf75f processed successfully
2025-04-21 21:10:13,818 - root - INFO - Processing upload for document ID: 1130b5e6-c350-40f9-94c2-9ad43d6fa4a4
2025-04-21 21:10:13,818 - root - INFO - File saved to data/uploads/1130b5e6-c350-40f9-94c2-9ad43d6fa4a4_1.Invitation to Tender.docx
2025-04-21 21:10:13,834 - root - INFO - Processing document 1130b5e6-c350-40f9-94c2-9ad43d6fa4a4 with content length: 13880
2025-04-21 21:10:14,176 - httpx - INFO - HTTP Request: POST https://api.cohere.com/v1/embed "HTTP/1.1 200 OK"
2025-04-21 21:10:14,994 - root - INFO - Stored embedding for document 1130b5e6-c350-40f9-94c2-9ad43d6fa4a4
2025-04-21 21:11:14,319 - httpx - INFO - HTTP Request: POST https://api.cohere.com/v1/rerank "HTTP/1.1 200 OK"
2025-04-21 21:11:14,326 - root - ERROR - Error processing document 1130b5e6-c350-40f9-94c2-9ad43d6fa4a4: 'NoneType' object has no attribute 'lower'
2025-04-21 21:11:14,326 - root - ERROR - Error processing document: 'NoneType' object has no attribute 'lower'
2025-04-21 21:11:14,326 - root - ERROR - Traceback (most recent call last):
File "c:\Users\babaw\Documents\Work\Mana Knight Digital\ds_task_scp\src\main.py", line 112, in upload_document
await document_processor.process_document(doc_id, file_path, document_type)
File "c:\Users\babaw\Documents\Work\Mana Knight Digital\ds_task_scp\src\services\document_processor.py", line 147, in process_document
"issues": self._format_issues(reranked_issues),
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "c:\Users\babaw\Documents\Work\Mana Knight Digital\ds_task_scp\src\services\document_processor.py", line 227, in _format_issues
if any(word in issue_text.lower() for word in ['critical', 'severe', 'major', 'high risk']):
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "c:\Users\babaw\Documents\Work\Mana Knight Digital\ds_task_scp\src\services\document_processor.py", line 227, in <genexpr>
if any(word in issue_text.lower() for word in ['critical', 'severe', 'major', 'high risk']):
^^^^^^^^^^^^^^^^
AttributeError: 'NoneType' object has no attribute 'lower'
2025-04-21 21:22:01,931 - root - INFO - Processing upload for document ID: aecbb62c-b7ed-4c2e-beff-fe5e292de9f1
2025-04-21 21:22:01,940 - root - INFO - File saved to data/uploads/aecbb62c-b7ed-4c2e-beff-fe5e292de9f1_4.Scope of Work.docx
2025-04-21 21:22:01,994 - root - INFO - Processing document aecbb62c-b7ed-4c2e-beff-fe5e292de9f1 with content length: 15493
2025-04-21 21:22:02,392 - httpx - INFO - HTTP Request: POST https://api.cohere.com/v1/embed "HTTP/1.1 200 OK"
2025-04-21 21:22:03,409 - root - INFO - Stored embedding for document aecbb62c-b7ed-4c2e-beff-fe5e292de9f1
2025-04-21 21:22:56,055 - httpx - INFO - HTTP Request: POST https://api.cohere.com/v1/rerank "HTTP/1.1 200 OK"
2025-04-21 21:24:51,709 - root - INFO - Document aecbb62c-b7ed-4c2e-beff-fe5e292de9f1 processed successfully
2025-04-21 21:24:51,717 - root - INFO - Document aecbb62c-b7ed-4c2e-beff-fe5e292de9f1 processed successfully
2025-04-21 21:34:15,975 - root - INFO - Processing upload for document ID: a444141c-e0c0-4ee9-a448-4555494aaede
2025-04-21 21:34:16,015 - root - INFO - File saved to data/uploads/a444141c-e0c0-4ee9-a448-4555494aaede_3.Bill of Quantities.docx
2025-04-21 21:34:16,058 - root - INFO - Processing document a444141c-e0c0-4ee9-a448-4555494aaede with content length: 13346
2025-04-21 21:34:16,557 - httpx - INFO - HTTP Request: POST https://api.cohere.com/v1/embed "HTTP/1.1 200 OK"
2025-04-21 21:34:18,925 - root - INFO - Stored embedding for document a444141c-e0c0-4ee9-a448-4555494aaede
2025-04-21 21:35:43,925 - httpx - INFO - HTTP Request: POST https://api.cohere.com/v1/rerank "HTTP/1.1 200 OK"
2025-04-21 21:36:09,310 - root - INFO - Document a444141c-e0c0-4ee9-a448-4555494aaede processed successfully
2025-04-21 21:36:09,328 - root - INFO - Document a444141c-e0c0-4ee9-a448-4555494aaede processed successfully
2025-04-21 21:38:22,988 - root - ERROR - Error retrieving analysis: list object has no element 1
2025-04-21 21:38:23,032 - root - ERROR - Traceback (most recent call last):
File "c:\Users\babaw\Documents\Work\Mana Knight Digital\ds_task_scp\src\main.py", line 139, in get_analysis
return templates.TemplateResponse(
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\babaw\Documents\Work\Mana Knight Digital\ds_task_scp\.venv\Lib\site-packages\starlette\templating.py", line 209, in TemplateResponse
return _TemplateResponse(
^^^^^^^^^^^^^^^^^^
File "C:\Users\babaw\Documents\Work\Mana Knight Digital\ds_task_scp\.venv\Lib\site-packages\starlette\templating.py", line 40, in __init__
content = template.render(context)
^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\babaw\Documents\Work\Mana Knight Digital\ds_task_scp\.venv\Lib\site-packages\jinja2\environment.py", line 1295, in render
self.environment.handle_exception()
File "C:\Users\babaw\Documents\Work\Mana Knight Digital\ds_task_scp\.venv\Lib\site-packages\jinja2\environment.py", line 942, in handle_exception
raise rewrite_traceback_stack(source=source)
File "src\templates\analysis.html", line 1, in top-level template code
{% extends "base.html" %}
File "src\templates\base.html", line 34, in top-level template code
{% block content %}{% endblock %}
^^^^^^^^^^^^^^^^^^^^^^^^^
File "src\templates\analysis.html", line 50, in block 'content'
{{ analysis.recommendations[loop.index0]|markdown|safe }}
^^^^^^^^^^^^^^^^^^^^^^^^^
File "c:\Users\babaw\Documents\Work\Mana Knight Digital\ds_task_scp\src\main.py", line 44, in markdown_filter
return markdown.markdown(text, extensions=['extra', 'nl2br'])
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\babaw\Documents\Work\Mana Knight Digital\ds_task_scp\.venv\Lib\site-packages\markdown\core.py", line 482, in markdown
return md.convert(text)
^^^^^^^^^^^^^^^^
File "C:\Users\babaw\Documents\Work\Mana Knight Digital\ds_task_scp\.venv\Lib\site-packages\markdown\core.py", line 341, in convert
if not source.strip():
^^^^^^^^^^^^
jinja2.exceptions.UndefinedError: list object has no element 1
2025-04-21 21:40:33,659 - root - INFO - Processing upload for document ID: befee86e-cc92-43cf-b935-3c8ec78ef275
2025-04-21 21:40:33,663 - root - INFO - File saved to data/uploads/befee86e-cc92-43cf-b935-3c8ec78ef275_4.Scope of Work.docx
2025-04-21 21:40:33,694 - root - INFO - Processing document befee86e-cc92-43cf-b935-3c8ec78ef275 with content length: 15493
2025-04-21 21:40:34,174 - httpx - INFO - HTTP Request: POST https://api.cohere.com/v1/embed "HTTP/1.1 200 OK"
2025-04-21 21:40:35,491 - root - INFO - Stored embedding for document befee86e-cc92-43cf-b935-3c8ec78ef275
2025-04-21 21:41:47,586 - httpx - INFO - HTTP Request: POST https://api.cohere.com/v1/rerank "HTTP/1.1 200 OK"
2025-04-21 21:42:22,869 - root - INFO - Document befee86e-cc92-43cf-b935-3c8ec78ef275 processed successfully
2025-04-21 21:42:22,887 - root - INFO - Document befee86e-cc92-43cf-b935-3c8ec78ef275 processed successfully
2025-04-21 21:53:13,705 - root - INFO - Processing upload for document ID: 3f10b972-3eb6-43c9-bfeb-777d0a33c4a3
2025-04-21 21:53:13,713 - root - INFO - File saved to data/uploads/3f10b972-3eb6-43c9-bfeb-777d0a33c4a3_8.form of tender.docx
2025-04-21 21:53:13,755 - root - INFO - Processing document 3f10b972-3eb6-43c9-bfeb-777d0a33c4a3 with content length: 16555
2025-04-21 21:53:16,168 - httpx - INFO - HTTP Request: POST https://api.cohere.com/v1/embed "HTTP/1.1 200 OK"
2025-04-21 21:53:18,290 - root - INFO - Stored embedding for document 3f10b972-3eb6-43c9-bfeb-777d0a33c4a3
2025-04-21 21:54:26,818 - httpx - INFO - HTTP Request: POST https://api.cohere.com/v1/rerank "HTTP/1.1 200 OK"
2025-04-21 21:55:06,268 - root - INFO - Document 3f10b972-3eb6-43c9-bfeb-777d0a33c4a3 processed successfully
2025-04-21 21:55:06,268 - root - INFO - Document 3f10b972-3eb6-43c9-bfeb-777d0a33c4a3 processed successfully
2025-04-21 22:11:40,110 - root - INFO - Processing upload for document ID: 2de0becf-fcfc-4793-aad0-7369180c6980
2025-04-21 22:11:40,113 - root - INFO - File saved to data/uploads/2de0becf-fcfc-4793-aad0-7369180c6980_3.Bill of Quantities.docx
2025-04-21 22:11:40,158 - root - INFO - Processing document 2de0becf-fcfc-4793-aad0-7369180c6980 with content length: 13346
2025-04-21 22:11:40,648 - httpx - INFO - HTTP Request: POST https://api.cohere.com/v1/embed "HTTP/1.1 200 OK"
2025-04-21 22:11:41,772 - root - INFO - Stored embedding for document 2de0becf-fcfc-4793-aad0-7369180c6980
2025-04-21 22:12:35,639 - httpx - INFO - HTTP Request: POST https://api.cohere.com/v1/rerank "HTTP/1.1 200 OK"
2025-04-21 22:13:08,662 - root - INFO - Document 2de0becf-fcfc-4793-aad0-7369180c6980 processed successfully
2025-04-21 22:13:08,672 - root - INFO - Document 2de0becf-fcfc-4793-aad0-7369180c6980 processed successfully
2025-04-21 22:15:10,060 - root - INFO - Processing upload for document ID: d6c97c26-b59b-4854-bf6d-6f7f58f3dc11
2025-04-21 22:15:10,060 - root - INFO - File saved to data/uploads/d6c97c26-b59b-4854-bf6d-6f7f58f3dc11_4.Scope of Work.docx
2025-04-21 22:15:10,140 - root - INFO - Processing document d6c97c26-b59b-4854-bf6d-6f7f58f3dc11 with content length: 15493
2025-04-21 22:15:10,472 - httpx - INFO - HTTP Request: POST https://api.cohere.com/v1/embed "HTTP/1.1 200 OK"
2025-04-21 22:15:11,322 - root - INFO - Stored embedding for document d6c97c26-b59b-4854-bf6d-6f7f58f3dc11
2025-04-21 22:16:10,272 - httpx - INFO - HTTP Request: POST https://api.cohere.com/v1/rerank "HTTP/1.1 200 OK"
2025-04-21 22:16:43,822 - root - INFO - Analysis for document d6c97c26-b59b-4854-bf6d-6f7f58f3dc11: {
"document_id": "d6c97c26-b59b-4854-bf6d-6f7f58f3dc11",
"summary": "Here are the key sections and requirements extracted from the document:\n\n### 1. **Content Types**\n - Defines the types of content within the document (e.g., XML files, relationships, themes).\n\n### 2. **Relationships**\n - Specifies the relationships between different parts of the document, such as how document parts are connected.\n\n### 3. **Document Content**\n - Contains the main content of the document, including text and formatting.\n\n### 4. **Document Relationships**\n - Defines the relationships specific to the main document part.\n\n### 5. **Theme**\n - Contains theme-related settings for the document, including color schemes and fonts.\n\n### 6. **Settings**\n - Includes document settings such as proofing options, compatibility settings, and other Word-specific configurations.\n\n### 7. **Numbering**\n - Defines numbering formats and styles used in the document.\n\n### 8. **Styles**\n - Specifies the styles applied throughout the document, including paragraph and character styles.\n\n### 9. **Web Settings**\n - Contains settings related to how the document is displayed in a web browser.\n\n### 10. **Font Table**\n - Lists the fonts used in the document and their properties.\n\n### 11. **Core Properties**\n - Includes metadata about the document, such as title, author, and creation date.\n\n### 12. **Application Properties**\n - Contains properties specific to the application (e.g., Microsoft Word) used to create the document.\n\n### Key Requirements:\n- The document must adhere to the defined content types and relationships.\n- Proper formatting and styling must be applied as specified in the styles and numbering sections.\n- Theme settings must be consistent throughout the document.\n- Metadata (core and application properties) must be accurately filled out.\n- Web settings should ensure proper display if the document is viewed online.\n\nThis summary provides an overview of the document's structure and key components, which are essential for maintaining consistency and functionality in the Word document.",
"issues": [
{
"issue": null,
"severity": "high",
"rank": 1
},
{
"issue": null,
"severity": "high",
"rank": 2
},
{
"issue": null,
"severity": "high",
"rank": 3
},
{
"issue": null,
"severity": "medium",
"rank": 4
},
{
"issue": null,
"severity": "medium",
"rank": 5
},
{
"issue": null,
"severity": "medium",
"rank": 6
},
{
"issue": null,
"severity": "low",
"rank": 7
},
{
"issue": null,
"severity": "low",
"rank": 8
},
{
"issue": null,
"severity": "low",
"rank": 9
},
{
"issue": null,
"severity": "low",
"rank": 10
},
{
"issue": null,
"severity": "low",
"rank": 11
},
{
"issue": null,
"severity": "low",
"rank": 12
},
{
"issue": null,
"severity": "low",
"rank": 13
},
{
"issue": null,
"severity": "low",
"rank": 14
},
{
"issue": null,
"severity": "low",
"rank": 15
},
{
"issue": null,
"severity": "low",
"rank": 16
},
{
"issue": null,
"severity": "low",
"rank": 17
},
{
"issue": null,
"severity": "low",
"rank": 18
},
{
"issue": null,
"severity": "low",
"rank": 19
},
{
"issue": null,
"severity": "low",
"rank": 20
},
{
"issue": null,
"severity": "low",
"rank": 21
},
{
"issue": null,
"severity": "low",
"rank": 22
},
{
"issue": null,
"severity": "low",
"rank": 23
},
{
"issue": null,
"severity": "low",
"rank": 24
},
{
"issue": null,
"severity": "low",
"rank": 25
},
{
"issue": null,
"severity": "low",
"rank": 26
},
{
"issue": null,
"severity": "low",
"rank": 27
},
{
"issue": null,
"severity": "low",
"rank": 28
},
{
"issue": null,
"severity": "low",
"rank": 29
},
{
"issue": null,
"severity": "low",
"rank": 30
},
{
"issue": null,
"severity": "low",
"rank": 31
},
{
"issue": null,
"severity": "low",
"rank": 32
},
{
"issue": null,
"severity": "low",
"rank": 33
},
{
"issue": null,
"severity": "low",
"rank": 34
},
{
"issue": null,
"severity": "low",
"rank": 35
},
{
"issue": null,
"severity": "low",
"rank": 36
},
{
"issue": null,
"severity": "low",
"rank": 37
},
{
"issue": null,
"severity": "low",
"rank": 38
},
{
"issue": null,
"severity": "low",
"rank": 39
},
{
"issue": null,
"severity": "low",
"rank": 40
},
{
"issue": null,
"severity": "low",
"rank": 41
},
{
"issue": null,
"severity": "low",
"rank": 42
},
{
"issue": null,
"severity": "low",
"rank": 43
},
{
"issue": null,
"severity": "low",
"rank": 44
},
{
"issue": null,
"severity": "low",
"rank": 45
},
{
"issue": null,
"severity": "low",
"rank": 46
},
{
"issue": null,
"severity": "low",
"rank": 47
},
{
"issue": null,
"severity": "low",
"rank": 48
},
{
"issue": null,
"severity": "low",
"rank": 49
},
{
"issue": null,
"severity": "low",
"rank": 50
}
],
"recommendations": [
"### **Comprehensive Compliance Recommendation for Document Issues** \n\n#### **1. Missing or Unidentified Documents (Indexes 0, 49, 1, 30, 25)** \n**Issue:** Several documents are either missing (`document=None`) or improperly indexed, leading to potential compliance risks. \n\n**Action Steps:** \n- **Conduct a Document Audit:** \n - Identify all missing documents (`index=0, 49, 1, 30, 25`) and verify if they were misplaced, incorrectly labeled, or never properly stored. \n - Cross-reference with a master document registry to confirm expected files. \n- **Implement a Document Tracking System:** \n - Use a **Document Management System (DMS)** with version control, metadata tagging, and audit trails. \n - Assign unique identifiers (e.g., `DOC-001`, `POL-2024-01`) to prevent indexing errors. \n- **Enforce Mandatory Metadata:** \n - Require fields such as: \n - Document title \n - Version number \n - Creation/modification dates \n - Author/owner \n - Compliance status (e.g., \"Reviewed,\" \"Pending Approval\") \n\n#### **2. Inconsistent Relevance Scores (Ranging from 0.23 to 0.98)** \n**Issue:** The relevance scores vary significantly, indicating potential inconsistencies in document classification, retrieval, or applicability. \n\n**Action Steps:** \n- **Standardize Relevance Scoring:** \n - Define clear criteria (e.g., regulatory impact, business criticality, frequency of use) to score documents objectively. \n - Use AI/ML tools (if available) to auto-tag relevance based on content analysis. \n- **Review Low-Scoring Documents (Indexes 25, 30):** \n - Determine if these documents are obsolete, redundant, or incorrectly tagged. \n - Archive or deprecate irrelevant files to reduce clutter. \n- **Flag High-Scoring Documents (Indexes 0, 49, 1):** \n - Prioritize review and updates for documents with high relevance to ensure compliance. \n\n#### **3. General Compliance Framework Enhancement** \nTo prevent recurrence, implement the following best practices: \n- **Automated Compliance Checks:** \n - Use compliance software (e.g., **OneTrust, LogicGate, Smarsh**) to flag missing/expired documents. \n- **Regular Training & Accountability:** \n - Train staff on proper document handling and compliance protocols. \n - Assign a **Compliance Officer** to oversee document integrity. \n- **Periodic Reassessment:** \n - Conduct quarterly audits to verify document accuracy, relevance, and accessibility. \n\n### **Final Recommendation Summary** \n1. **Locate & properly index missing documents.** \n2. **Adopt a DMS with strict metadata requirements.** \n3. **Standardize relevance scoring and purge obsolete files.** \n4. **Automate compliance monitoring and enforce accountability.** \n\nBy implementing these steps, the organization can resolve current compliance gaps and establish a robust, future-proof document management process. \n\nWould you like a tailored checklist for immediate execution?"
]
}
2025-04-21 22:16:43,889 - root - INFO - Document d6c97c26-b59b-4854-bf6d-6f7f58f3dc11 processed successfully
2025-04-21 22:16:43,903 - root - INFO - Document d6c97c26-b59b-4854-bf6d-6f7f58f3dc11 processed successfully
2025-04-21 22:16:54,195 - root - INFO - Retrieved analysis for document d6c97c26-b59b-4854-bf6d-6f7f58f3dc11: {
"document_id": "d6c97c26-b59b-4854-bf6d-6f7f58f3dc11",
"summary": "Here are the key sections and requirements extracted from the document:\n\n### 1. **Content Types**\n - Defines the types of content within the document (e.g., XML files, relationships, themes).\n\n### 2. **Relationships**\n - Specifies the relationships between different parts of the document, such as how document parts are connected.\n\n### 3. **Document Content**\n - Contains the main content of the document, including text and formatting.\n\n### 4. **Document Relationships**\n - Defines the relationships specific to the main document part.\n\n### 5. **Theme**\n - Contains theme-related settings for the document, including color schemes and fonts.\n\n### 6. **Settings**\n - Includes document settings such as proofing options, compatibility settings, and other Word-specific configurations.\n\n### 7. **Numbering**\n - Defines numbering formats and styles used in the document.\n\n### 8. **Styles**\n - Specifies the styles applied throughout the document, including paragraph and character styles.\n\n### 9. **Web Settings**\n - Contains settings related to how the document is displayed in a web browser.\n\n### 10. **Font Table**\n - Lists the fonts used in the document and their properties.\n\n### 11. **Core Properties**\n - Includes metadata about the document, such as title, author, and creation date.\n\n### 12. **Application Properties**\n - Contains properties specific to the application (e.g., Microsoft Word) used to create the document.\n\n### Key Requirements:\n- The document must adhere to the defined content types and relationships.\n- Proper formatting and styling must be applied as specified in the styles and numbering sections.\n- Theme settings must be consistent throughout the document.\n- Metadata (core and application properties) must be accurately filled out.\n- Web settings should ensure proper display if the document is viewed online.\n\nThis summary provides an overview of the document's structure and key components, which are essential for maintaining consistency and functionality in the Word document.",
"issues": [
{
"issue": null,
"severity": "high",
"rank": 1
},
{
"issue": null,
"severity": "high",
"rank": 2
},
{
"issue": null,
"severity": "high",
"rank": 3
},
{
"issue": null,
"severity": "medium",
"rank": 4
},
{
"issue": null,
"severity": "medium",
"rank": 5
},
{
"issue": null,
"severity": "medium",
"rank": 6
},
{
"issue": null,
"severity": "low",
"rank": 7
},
{
"issue": null,
"severity": "low",
"rank": 8
},
{
"issue": null,
"severity": "low",
"rank": 9
},
{
"issue": null,
"severity": "low",
"rank": 10
},
{
"issue": null,
"severity": "low",
"rank": 11
},
{
"issue": null,
"severity": "low",
"rank": 12
},
{
"issue": null,
"severity": "low",
"rank": 13
},
{
"issue": null,
"severity": "low",
"rank": 14
},
{
"issue": null,
"severity": "low",
"rank": 15
},
{
"issue": null,
"severity": "low",
"rank": 16
},
{
"issue": null,
"severity": "low",
"rank": 17
},
{
"issue": null,
"severity": "low",
"rank": 18
},
{
"issue": null,
"severity": "low",
"rank": 19
},
{
"issue": null,
"severity": "low",
"rank": 20
},
{
"issue": null,
"severity": "low",
"rank": 21
},
{
"issue": null,
"severity": "low",
"rank": 22
},
{
"issue": null,
"severity": "low",
"rank": 23
},
{
"issue": null,
"severity": "low",
"rank": 24
},
{
"issue": null,
"severity": "low",
"rank": 25
},
{
"issue": null,
"severity": "low",
"rank": 26
},
{
"issue": null,
"severity": "low",
"rank": 27
},
{
"issue": null,
"severity": "low",
"rank": 28
},
{
"issue": null,
"severity": "low",
"rank": 29
},
{
"issue": null,
"severity": "low",
"rank": 30
},
{
"issue": null,
"severity": "low",
"rank": 31
},
{
"issue": null,
"severity": "low",
"rank": 32
},
{
"issue": null,
"severity": "low",
"rank": 33
},
{
"issue": null,
"severity": "low",
"rank": 34
},
{
"issue": null,
"severity": "low",
"rank": 35
},
{
"issue": null,
"severity": "low",
"rank": 36
},
{
"issue": null,
"severity": "low",
"rank": 37
},
{
"issue": null,
"severity": "low",
"rank": 38
},
{
"issue": null,
"severity": "low",
"rank": 39
},
{
"issue": null,
"severity": "low",
"rank": 40
},
{
"issue": null,
"severity": "low",
"rank": 41
},
{
"issue": null,
"severity": "low",
"rank": 42
},
{
"issue": null,
"severity": "low",
"rank": 43
},
{
"issue": null,
"severity": "low",
"rank": 44
},
{
"issue": null,
"severity": "low",
"rank": 45
},
{
"issue": null,
"severity": "low",
"rank": 46
},
{
"issue": null,
"severity": "low",
"rank": 47
},
{
"issue": null,
"severity": "low",
"rank": 48
},
{
"issue": null,
"severity": "low",
"rank": 49
},
{
"issue": null,
"severity": "low",
"rank": 50
}
],
"recommendations": [
"### **Comprehensive Compliance Recommendation for Document Issues** \n\n#### **1. Missing or Unidentified Documents (Indexes 0, 49, 1, 30, 25)** \n**Issue:** Several documents are either missing (`document=None`) or improperly indexed, leading to potential compliance risks. \n\n**Action Steps:** \n- **Conduct a Document Audit:** \n - Identify all missing documents (`index=0, 49, 1, 30, 25`) and verify if they were misplaced, incorrectly labeled, or never properly stored. \n - Cross-reference with a master document registry to confirm expected files. \n- **Implement a Document Tracking System:** \n - Use a **Document Management System (DMS)** with version control, metadata tagging, and audit trails. \n - Assign unique identifiers (e.g., `DOC-001`, `POL-2024-01`) to prevent indexing errors. \n- **Enforce Mandatory Metadata:** \n - Require fields such as: \n - Document title \n - Version number \n - Creation/modification dates \n - Author/owner \n - Compliance status (e.g., \"Reviewed,\" \"Pending Approval\") \n\n#### **2. Inconsistent Relevance Scores (Ranging from 0.23 to 0.98)** \n**Issue:** The relevance scores vary significantly, indicating potential inconsistencies in document classification, retrieval, or applicability. \n\n**Action Steps:** \n- **Standardize Relevance Scoring:** \n - Define clear criteria (e.g., regulatory impact, business criticality, frequency of use) to score documents objectively. \n - Use AI/ML tools (if available) to auto-tag relevance based on content analysis. \n- **Review Low-Scoring Documents (Indexes 25, 30):** \n - Determine if these documents are obsolete, redundant, or incorrectly tagged. \n - Archive or deprecate irrelevant files to reduce clutter. \n- **Flag High-Scoring Documents (Indexes 0, 49, 1):** \n - Prioritize review and updates for documents with high relevance to ensure compliance. \n\n#### **3. General Compliance Framework Enhancement** \nTo prevent recurrence, implement the following best practices: \n- **Automated Compliance Checks:** \n - Use compliance software (e.g., **OneTrust, LogicGate, Smarsh**) to flag missing/expired documents. \n- **Regular Training & Accountability:** \n - Train staff on proper document handling and compliance protocols. \n - Assign a **Compliance Officer** to oversee document integrity. \n- **Periodic Reassessment:** \n - Conduct quarterly audits to verify document accuracy, relevance, and accessibility. \n\n### **Final Recommendation Summary** \n1. **Locate & properly index missing documents.** \n2. **Adopt a DMS with strict metadata requirements.** \n3. **Standardize relevance scoring and purge obsolete files.** \n4. **Automate compliance monitoring and enforce accountability.** \n\nBy implementing these steps, the organization can resolve current compliance gaps and establish a robust, future-proof document management process. \n\nWould you like a tailored checklist for immediate execution?"
]
}
2025-04-21 22:18:34,156 - root - ERROR - Error retrieving analysis: list object has no element 1
2025-04-21 22:18:34,163 - root - ERROR - Traceback (most recent call last):
File "c:\Users\babaw\Documents\Work\Mana Knight Digital\ds_task_scp\src\main.py", line 138, in get_analysis
return templates.TemplateResponse(
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\babaw\Documents\Work\Mana Knight Digital\ds_task_scp\.venv\Lib\site-packages\starlette\templating.py", line 209, in TemplateResponse
return _TemplateResponse(
^^^^^^^^^^^^^^^^^^
File "C:\Users\babaw\Documents\Work\Mana Knight Digital\ds_task_scp\.venv\Lib\site-packages\starlette\templating.py", line 40, in __init__
content = template.render(context)
^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\babaw\Documents\Work\Mana Knight Digital\ds_task_scp\.venv\Lib\site-packages\jinja2\environment.py", line 1295, in render
self.environment.handle_exception()
File "C:\Users\babaw\Documents\Work\Mana Knight Digital\ds_task_scp\.venv\Lib\site-packages\jinja2\environment.py", line 942, in handle_exception
raise rewrite_traceback_stack(source=source)
File "src\templates\analysis.html", line 1, in top-level template code
{% extends "base.html" %}
File "src\templates\base.html", line 34, in top-level template code
{% block content %}{% endblock %}
^^^^^^^^^^^^^^^^^^^^^^^^^
File "src\templates\analysis.html", line 50, in block 'content'
{{ analysis.recommendations[loop.index0]|markdown|safe }}
^^^^^^^^^^^^^^^^^^^^^^^^^
File "c:\Users\babaw\Documents\Work\Mana Knight Digital\ds_task_scp\src\main.py", line 44, in markdown_filter
return markdown.markdown(text, extensions=['extra', 'nl2br'])
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\babaw\Documents\Work\Mana Knight Digital\ds_task_scp\.venv\Lib\site-packages\markdown\core.py", line 482, in markdown
return md.convert(text)
^^^^^^^^^^^^^^^^
File "C:\Users\babaw\Documents\Work\Mana Knight Digital\ds_task_scp\.venv\Lib\site-packages\markdown\core.py", line 341, in convert
if not source.strip():
^^^^^^^^^^^^
jinja2.exceptions.UndefinedError: list object has no element 1
2025-04-21 22:18:40,245 - root - INFO - Deleted uploaded file: data/uploads/d6c97c26-b59b-4854-bf6d-6f7f58f3dc11_4.Scope of Work.docx
2025-04-21 22:18:41,105 - root - INFO - Deleted document d6c97c26-b59b-4854-bf6d-6f7f58f3dc11 from index
2025-04-21 22:18:41,105 - root - INFO - Removed document d6c97c26-b59b-4854-bf6d-6f7f58f3dc11 from vector store
2025-04-21 22:18:49,878 - root - INFO - Deleted uploaded file: data/uploads/2de0becf-fcfc-4793-aad0-7369180c6980_3.Bill of Quantities.docx
2025-04-21 22:18:50,166 - root - INFO - Deleted document 2de0becf-fcfc-4793-aad0-7369180c6980 from index
2025-04-21 22:18:50,166 - root - INFO - Removed document 2de0becf-fcfc-4793-aad0-7369180c6980 from vector store
2025-04-21 22:18:54,080 - root - INFO - Deleted uploaded file: data/uploads/3f10b972-3eb6-43c9-bfeb-777d0a33c4a3_8.form of tender.docx
2025-04-21 22:18:54,321 - root - INFO - Deleted document 3f10b972-3eb6-43c9-bfeb-777d0a33c4a3 from index
2025-04-21 22:18:54,322 - root - INFO - Removed document 3f10b972-3eb6-43c9-bfeb-777d0a33c4a3 from vector store
2025-04-21 22:18:57,881 - root - INFO - Deleted uploaded file: data/uploads/befee86e-cc92-43cf-b935-3c8ec78ef275_4.Scope of Work.docx
2025-04-21 22:18:58,105 - root - INFO - Deleted document befee86e-cc92-43cf-b935-3c8ec78ef275 from index
2025-04-21 22:18:58,105 - root - INFO - Removed document befee86e-cc92-43cf-b935-3c8ec78ef275 from vector store
2025-04-21 22:19:02,930 - root - INFO - Deleted uploaded file: data/uploads/a444141c-e0c0-4ee9-a448-4555494aaede_3.Bill of Quantities.docx
2025-04-21 22:19:03,739 - root - INFO - Deleted document a444141c-e0c0-4ee9-a448-4555494aaede from index
2025-04-21 22:19:03,748 - root - INFO - Removed document a444141c-e0c0-4ee9-a448-4555494aaede from vector store
2025-04-21 22:19:18,005 - root - INFO - Processing upload for document ID: 77063b1d-633c-421e-9591-cde2eb90a979
2025-04-21 22:19:18,008 - root - INFO - File saved to data/uploads/77063b1d-633c-421e-9591-cde2eb90a979_7.Supplier SQualification requirements.docx
2025-04-21 22:19:18,041 - root - INFO - Processing document 77063b1d-633c-421e-9591-cde2eb90a979 with content length: 15335
2025-04-21 22:19:18,695 - httpx - INFO - HTTP Request: POST https://api.cohere.com/v1/embed "HTTP/1.1 200 OK"
2025-04-21 22:19:19,239 - root - INFO - Stored embedding for document 77063b1d-633c-421e-9591-cde2eb90a979
2025-04-21 22:20:24,699 - httpx - INFO - HTTP Request: POST https://api.cohere.com/v1/rerank "HTTP/1.1 200 OK"
2025-04-21 22:22:05,759 - root - INFO - Document 77063b1d-633c-421e-9591-cde2eb90a979 processed successfully
2025-04-21 22:22:05,775 - root - INFO - Document 77063b1d-633c-421e-9591-cde2eb90a979 processed successfully
2025-04-21 22:31:21,934 - root - INFO - Processing upload for document ID: e79aeb90-799a-4d06-9efd-1d19315eebcc
2025-04-21 22:31:21,936 - root - INFO - File saved to data/uploads/e79aeb90-799a-4d06-9efd-1d19315eebcc_2.Tender Specifications.docx
2025-04-21 22:31:21,953 - root - INFO - Processing document e79aeb90-799a-4d06-9efd-1d19315eebcc with content length: 17509
2025-04-21 22:32:04,265 - httpx - INFO - HTTP Request: POST https://api.cohere.com/v1/embed "HTTP/1.1 200 OK"
2025-04-21 22:32:05,457 - root - INFO - Stored embedding for document e79aeb90-799a-4d06-9efd-1d19315eebcc
2025-04-21 22:33:04,556 - httpx - INFO - HTTP Request: POST https://api.cohere.com/v1/rerank "HTTP/1.1 200 OK"
2025-04-21 22:34:43,603 - root - INFO - Document e79aeb90-799a-4d06-9efd-1d19315eebcc processed successfully
2025-04-21 22:34:43,611 - root - INFO - Document e79aeb90-799a-4d06-9efd-1d19315eebcc processed successfully
2025-04-22 09:39:48,962 - root - INFO - Processing upload for document ID: e838ee14-75a7-483a-9a6e-46b218127dc5
2025-04-22 09:39:48,968 - root - INFO - File saved to data/uploads/e838ee14-75a7-483a-9a6e-46b218127dc5_7.Supplier SQualification requirements.docx
2025-04-22 09:39:49,014 - root - INFO - Processing document e838ee14-75a7-483a-9a6e-46b218127dc5 with content length: 15335
2025-04-22 09:39:50,452 - httpx - INFO - HTTP Request: POST https://api.cohere.com/v1/embed "HTTP/1.1 200 OK"
2025-04-22 09:39:51,219 - root - INFO - Stored embedding for document e838ee14-75a7-483a-9a6e-46b218127dc5
2025-04-22 09:41:03,267 - httpx - INFO - HTTP Request: POST https://api.cohere.com/v1/rerank "HTTP/1.1 200 OK"
2025-04-22 09:41:56,158 - root - INFO - Document e838ee14-75a7-483a-9a6e-46b218127dc5 processed successfully
2025-04-22 09:41:56,169 - root - INFO - Document e838ee14-75a7-483a-9a6e-46b218127dc5 processed successfully
2025-04-22 09:50:01,851 - root - INFO - Processing upload for document ID: eaed1290-2993-4133-9a04-6a4e5cb8b431
2025-04-22 09:50:01,854 - root - INFO - File saved to data/uploads/eaed1290-2993-4133-9a04-6a4e5cb8b431_7.Supplier SQualification requirements.docx
2025-04-22 09:50:02,083 - root - ERROR - Error reading Word document: No module named 'exceptions'
2025-04-22 09:50:02,083 - root - INFO - Processing document eaed1290-2993-4133-9a04-6a4e5cb8b431 with content length: 0
2025-04-22 09:50:02,520 - httpx - INFO - HTTP Request: POST https://api.cohere.com/v1/embed "HTTP/1.1 200 OK"
2025-04-22 09:50:03,207 - root - INFO - Stored embedding for document eaed1290-2993-4133-9a04-6a4e5cb8b431
2025-04-22 09:50:47,257 - httpx - INFO - HTTP Request: POST https://api.cohere.com/v1/rerank "HTTP/1.1 200 OK"
2025-04-22 09:51:36,358 - root - INFO - Document eaed1290-2993-4133-9a04-6a4e5cb8b431 processed successfully
2025-04-22 09:51:36,377 - root - INFO - Document eaed1290-2993-4133-9a04-6a4e5cb8b431 processed successfully
2025-04-22 09:55:01,750 - root - INFO - Processing upload for document ID: aec927db-9f43-49b9-90af-2af8ad64e793
2025-04-22 09:55:01,752 - root - INFO - File saved to data/uploads/aec927db-9f43-49b9-90af-2af8ad64e793_7.Supplier SQualification requirements.docx
2025-04-22 09:55:01,791 - root - ERROR - Error reading Word document: No module named 'exceptions'
2025-04-22 09:55:01,791 - root - INFO - Processing document aec927db-9f43-49b9-90af-2af8ad64e793 with content length: 0
2025-04-22 09:55:02,033 - httpx - INFO - HTTP Request: POST https://api.cohere.com/v1/embed "HTTP/1.1 200 OK"
2025-04-22 09:55:02,565 - root - INFO - Stored embedding for document aec927db-9f43-49b9-90af-2af8ad64e793
2025-04-22 09:55:57,175 - httpx - INFO - HTTP Request: POST https://api.cohere.com/v1/rerank "HTTP/1.1 200 OK"
2025-04-22 09:56:48,889 - root - INFO - Document aec927db-9f43-49b9-90af-2af8ad64e793 processed successfully
2025-04-22 09:56:48,898 - root - INFO - Document aec927db-9f43-49b9-90af-2af8ad64e793 processed successfully
2025-04-22 10:00:35,648 - root - INFO - Processing upload for document ID: 48480333-d451-4907-988b-f059166fd1a5
2025-04-22 10:00:35,652 - root - INFO - File saved to data/uploads/48480333-d451-4907-988b-f059166fd1a5_9.confidentiality agreement.docx
2025-04-22 10:00:36,025 - root - INFO - Processing document 48480333-d451-4907-988b-f059166fd1a5 with content length: 161
2025-04-22 10:00:36,689 - httpx - INFO - HTTP Request: POST https://api.cohere.com/v1/embed "HTTP/1.1 200 OK"
2025-04-22 10:00:37,466 - root - INFO - Stored embedding for document 48480333-d451-4907-988b-f059166fd1a5
2025-04-22 10:01:31,476 - httpx - INFO - HTTP Request: POST https://api.cohere.com/v1/rerank "HTTP/1.1 200 OK"
2025-04-22 10:02:20,575 - root - INFO - Document 48480333-d451-4907-988b-f059166fd1a5 processed successfully
2025-04-22 10:02:20,592 - root - INFO - Document 48480333-d451-4907-988b-f059166fd1a5 processed successfully
2025-04-22 10:06:04,139 - root - INFO - Deleted document f1d07dde-5de4-4bf6-a14d-b69a433aa855 from index
2025-04-22 10:06:04,142 - root - INFO - Removed document f1d07dde-5de4-4bf6-a14d-b69a433aa855 from vector store
2025-04-22 10:06:04,289 - root - ERROR - Error retrieving metadata for document f1d07dde-5de4-4bf6-a14d-b69a433aa855: Metadata not found for document f1d07dde-5de4-4bf6-a14d-b69a433aa855
2025-04-22 10:06:04,289 - root - ERROR - Error deleting document: Metadata not found for document f1d07dde-5de4-4bf6-a14d-b69a433aa855
2025-04-22 10:06:04,289 - root - ERROR - Traceback (most recent call last):
File "C:\Users\babaw\Documents\Work\Mana Knight Digital\ds_task_scp\src\main.py", line 213, in delete_document
metadata = database.get_metadata(doc_id)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\babaw\Documents\Work\Mana Knight Digital\ds_task_scp\src\services\database.py", line 114, in get_metadata
raise FileNotFoundError(f"Metadata not found for document {document_id}")
FileNotFoundError: Metadata not found for document f1d07dde-5de4-4bf6-a14d-b69a433aa855
2025-04-22 10:06:08,722 - root - INFO - Deleted document 82d9da57-6291-4ddb-b2e3-d43f467f4dd0 from index
2025-04-22 10:06:08,722 - root - INFO - Removed document 82d9da57-6291-4ddb-b2e3-d43f467f4dd0 from vector store
2025-04-22 10:06:13,324 - root - INFO - Deleted uploaded file: data/uploads/78afc395-9b7c-4388-8d8b-aa1d02fbf75f_2.Tender Specifications.docx
2025-04-22 10:06:13,533 - root - INFO - Deleted document 78afc395-9b7c-4388-8d8b-aa1d02fbf75f from index
2025-04-22 10:06:13,533 - root - INFO - Removed document 78afc395-9b7c-4388-8d8b-aa1d02fbf75f from vector store
2025-04-22 10:06:17,591 - root - INFO - Deleted uploaded file: data/uploads/aecbb62c-b7ed-4c2e-beff-fe5e292de9f1_4.Scope of Work.docx
2025-04-22 10:06:17,860 - root - INFO - Deleted document aecbb62c-b7ed-4c2e-beff-fe5e292de9f1 from index
2025-04-22 10:06:17,860 - root - INFO - Removed document aecbb62c-b7ed-4c2e-beff-fe5e292de9f1 from vector store
2025-04-22 10:06:22,432 - root - INFO - Deleted uploaded file: data/uploads/77063b1d-633c-421e-9591-cde2eb90a979_7.Supplier SQualification requirements.docx
2025-04-22 10:06:22,572 - root - INFO - Deleted document 77063b1d-633c-421e-9591-cde2eb90a979 from index
2025-04-22 10:06:22,572 - root - INFO - Removed document 77063b1d-633c-421e-9591-cde2eb90a979 from vector store
2025-04-22 10:06:26,959 - root - INFO - Deleted uploaded file: data/uploads/e79aeb90-799a-4d06-9efd-1d19315eebcc_2.Tender Specifications.docx
2025-04-22 10:06:27,114 - root - INFO - Deleted document e79aeb90-799a-4d06-9efd-1d19315eebcc from index
2025-04-22 10:06:27,119 - root - INFO - Removed document e79aeb90-799a-4d06-9efd-1d19315eebcc from vector store
2025-04-22 10:06:31,340 - root - INFO - Deleted uploaded file: data/uploads/e838ee14-75a7-483a-9a6e-46b218127dc5_7.Supplier SQualification requirements.docx
2025-04-22 10:06:31,518 - root - INFO - Deleted document e838ee14-75a7-483a-9a6e-46b218127dc5 from index
2025-04-22 10:06:31,520 - root - INFO - Removed document e838ee14-75a7-483a-9a6e-46b218127dc5 from vector store
2025-04-22 10:06:50,772 - root - INFO - Processing upload for document ID: 788f85e4-a873-407b-bfbc-0b7d6a676b9a
2025-04-22 10:06:50,779 - root - INFO - File saved to data/uploads/788f85e4-a873-407b-bfbc-0b7d6a676b9a_2.Tender Specifications.docx
2025-04-22 10:06:51,181 - root - INFO - Processing document 788f85e4-a873-407b-bfbc-0b7d6a676b9a with content length: 2363
2025-04-22 10:06:51,477 - httpx - INFO - HTTP Request: POST https://api.cohere.com/v1/embed "HTTP/1.1 200 OK"
2025-04-22 10:06:52,399 - root - INFO - Stored embedding for document 788f85e4-a873-407b-bfbc-0b7d6a676b9a
2025-04-22 10:08:26,609 - httpx - INFO - HTTP Request: POST https://api.cohere.com/v1/rerank "HTTP/1.1 200 OK"
2025-04-22 10:09:18,265 - root - INFO - Document 788f85e4-a873-407b-bfbc-0b7d6a676b9a processed successfully
2025-04-22 10:09:18,287 - root - INFO - Document 788f85e4-a873-407b-bfbc-0b7d6a676b9a processed successfully
2025-04-22 10:31:42,225 - root - INFO - Processing upload for document ID: 804d09d1-22b8-4e49-9bab-6fd7d34181b7
2025-04-22 10:31:42,227 - root - INFO - File saved to data/uploads/804d09d1-22b8-4e49-9bab-6fd7d34181b7_7.Supplier SQualification requirements.docx
2025-04-22 10:31:42,381 - root - INFO - Processing document 804d09d1-22b8-4e49-9bab-6fd7d34181b7 with content length: 229
2025-04-22 10:31:42,613 - httpx - INFO - HTTP Request: POST https://api.cohere.com/v1/embed "HTTP/1.1 200 OK"
2025-04-22 10:31:44,246 - root - INFO - Stored embedding for document 804d09d1-22b8-4e49-9bab-6fd7d34181b7
2025-04-22 10:32:27,429 - httpx - INFO - HTTP Request: POST https://api.cohere.com/v1/rerank "HTTP/1.1 200 OK"
2025-04-22 10:33:19,434 - root - INFO - Document 804d09d1-22b8-4e49-9bab-6fd7d34181b7 processed successfully
2025-04-22 10:33:19,446 - root - INFO - Document 804d09d1-22b8-4e49-9bab-6fd7d34181b7 processed successfully
2025-04-22 10:51:21,371 - root - INFO - Processing upload for document ID: 8d580b49-94bb-473e-90dd-66ee00f77048
2025-04-22 10:51:21,372 - root - INFO - File saved to data/uploads/8d580b49-94bb-473e-90dd-66ee00f77048_8.form of tender.docx
2025-04-22 10:51:21,496 - root - INFO - Processing document 8d580b49-94bb-473e-90dd-66ee00f77048 with content length: 523
2025-04-22 10:51:21,994 - httpx - INFO - HTTP Request: POST https://api.cohere.com/v1/embed "HTTP/1.1 200 OK"
2025-04-22 10:51:22,895 - root - INFO - Stored embedding for document 8d580b49-94bb-473e-90dd-66ee00f77048
2025-04-22 10:52:05,978 - httpx - INFO - HTTP Request: POST https://api.cohere.com/v1/rerank "HTTP/1.1 400 Bad Request"
2025-04-22 10:52:05,978 - root - ERROR - Error using Cohere reranker: status_code: 400, body: {'message': 'invalid request: number of total max chunks (number of documents * max chunks per doc) must be less than 10000'}
2025-04-22 10:56:57,188 - root - INFO - Processing upload for document ID: 5ab90386-7d4e-45b2-a1a6-40ad23f59428
2025-04-22 10:56:57,189 - root - INFO - File saved to data/uploads/5ab90386-7d4e-45b2-a1a6-40ad23f59428_8.form of tender.docx
2025-04-22 10:56:57,312 - root - INFO - Processing document 5ab90386-7d4e-45b2-a1a6-40ad23f59428 with content length: 523
2025-04-22 10:56:57,661 - httpx - INFO - HTTP Request: POST https://api.cohere.com/v1/embed "HTTP/1.1 200 OK"
2025-04-22 10:56:58,794 - root - INFO - Stored embedding for document 5ab90386-7d4e-45b2-a1a6-40ad23f59428
2025-04-22 10:57:57,544 - httpx - INFO - HTTP Request: POST https://api.cohere.com/v1/rerank "HTTP/1.1 400 Bad Request"
2025-04-22 10:57:57,544 - root - ERROR - Error using Cohere reranker: status_code: 400, body: {'message': 'invalid request: list of documents must not be empty'}
2025-04-22 10:57:57,563 - root - INFO - Document 5ab90386-7d4e-45b2-a1a6-40ad23f59428 processed successfully
2025-04-22 10:57:57,563 - root - INFO - Document 5ab90386-7d4e-45b2-a1a6-40ad23f59428 processed successfully
2025-04-22 11:04:23,803 - root - INFO - Processing upload for document ID: 3065e0dd-0b2e-454c-8f7e-dd0e464dbf7a
2025-04-22 11:04:23,806 - root - INFO - File saved to data/uploads/3065e0dd-0b2e-454c-8f7e-dd0e464dbf7a_3.Bill of Quantities.docx
2025-04-22 11:04:23,950 - root - INFO - Processing document 3065e0dd-0b2e-454c-8f7e-dd0e464dbf7a with content length: 2057
2025-04-22 11:04:24,294 - httpx - INFO - HTTP Request: POST https://api.cohere.com/v1/embed "HTTP/1.1 200 OK"
2025-04-22 11:04:24,927 - root - INFO - Stored embedding for document 3065e0dd-0b2e-454c-8f7e-dd0e464dbf7a
2025-04-22 11:05:28,953 - httpx - INFO - HTTP Request: POST https://api.cohere.com/v1/rerank "HTTP/1.1 400 Bad Request"
2025-04-22 11:05:28,953 - root - ERROR - Error using Cohere reranker: status_code: 400, body: {'message': 'invalid request: list of documents must not be empty'}
2025-04-22 11:05:28,969 - root - INFO - Document 3065e0dd-0b2e-454c-8f7e-dd0e464dbf7a processed successfully
2025-04-22 11:05:28,980 - root - INFO - Document 3065e0dd-0b2e-454c-8f7e-dd0e464dbf7a processed successfully
2025-04-22 11:13:54,204 - root - INFO - Processing upload for document ID: 201e4896-3d89-466b-852d-783ef3e30f83
2025-04-22 11:13:54,206 - root - INFO - File saved to data/uploads/201e4896-3d89-466b-852d-783ef3e30f83_7.Supplier SQualification requirements.docx
2025-04-22 11:13:54,311 - root - INFO - Processing document 201e4896-3d89-466b-852d-783ef3e30f83 with content length: 229
2025-04-22 11:13:54,726 - httpx - INFO - HTTP Request: POST https://api.cohere.com/v1/embed "HTTP/1.1 200 OK"
2025-04-22 11:13:55,243 - root - INFO - Stored embedding for document 201e4896-3d89-466b-852d-783ef3e30f83
2025-04-22 11:14:42,644 - httpx - INFO - HTTP Request: POST https://api.cohere.com/v1/rerank "HTTP/1.1 400 Bad Request"
2025-04-22 11:14:42,644 - root - ERROR - Error using Cohere reranker: status_code: 400, body: {'message': 'invalid request: list of documents must not be empty'}
2025-04-22 11:14:42,644 - root - INFO - Document 201e4896-3d89-466b-852d-783ef3e30f83 processed successfully
2025-04-22 11:14:42,659 - root - INFO - Document 201e4896-3d89-466b-852d-783ef3e30f83 processed successfully
2025-04-22 11:18:42,103 - root - INFO - Processing upload for document ID: bc1e71ac-6b65-4b2c-b1e8-81844b49ba5d
2025-04-22 11:18:42,104 - root - INFO - File saved to data/uploads/bc1e71ac-6b65-4b2c-b1e8-81844b49ba5d_7.Supplier SQualification requirements.docx
2025-04-22 11:18:42,225 - root - INFO - Processing document bc1e71ac-6b65-4b2c-b1e8-81844b49ba5d with content length: 229
2025-04-22 11:18:42,443 - httpx - INFO - HTTP Request: POST https://api.cohere.com/v1/embed "HTTP/1.1 200 OK"
2025-04-22 11:18:45,759 - root - INFO - Stored embedding for document bc1e71ac-6b65-4b2c-b1e8-81844b49ba5d
2025-04-22 11:19:35,113 - httpx - INFO - HTTP Request: POST https://api.cohere.com/v1/rerank "HTTP/1.1 200 OK"
2025-04-22 11:19:35,138 - root - INFO - Document bc1e71ac-6b65-4b2c-b1e8-81844b49ba5d processed successfully
2025-04-22 11:19:35,145 - root - INFO - Document bc1e71ac-6b65-4b2c-b1e8-81844b49ba5d processed successfully
2025-04-22 11:27:23,936 - root - INFO - Processing upload for document ID: a76f7f9c-59ec-4f7b-8b4e-168f2db5f92e
2025-04-22 11:27:23,938 - root - INFO - File saved to data/uploads/a76f7f9c-59ec-4f7b-8b4e-168f2db5f92e_7.Supplier SQualification requirements.docx
2025-04-22 11:27:24,051 - root - INFO - Processing document a76f7f9c-59ec-4f7b-8b4e-168f2db5f92e with content length: 229
2025-04-22 11:27:24,259 - httpx - INFO - HTTP Request: POST https://api.cohere.com/v1/embed "HTTP/1.1 200 OK"
2025-04-22 11:27:24,775 - root - INFO - Stored embedding for document a76f7f9c-59ec-4f7b-8b4e-168f2db5f92e
2025-04-22 11:28:04,075 - httpx - INFO - HTTP Request: POST https://api.cohere.com/v1/rerank "HTTP/1.1 200 OK"
2025-04-22 11:28:04,108 - root - INFO - Document a76f7f9c-59ec-4f7b-8b4e-168f2db5f92e processed successfully
2025-04-22 11:28:04,108 - root - INFO - Document a76f7f9c-59ec-4f7b-8b4e-168f2db5f92e processed successfully
2025-04-22 11:33:42,952 - root - INFO - Processing upload for document ID: 6fd270a4-e76a-4234-a4c4-a40d2ac64d56
2025-04-22 11:33:42,954 - root - INFO - File saved to data/uploads/6fd270a4-e76a-4234-a4c4-a40d2ac64d56_4.Scope of Work.docx
2025-04-22 11:33:43,183 - root - INFO - Processing document 6fd270a4-e76a-4234-a4c4-a40d2ac64d56 with content length: 282
2025-04-22 11:33:43,425 - httpx - INFO - HTTP Request: POST https://api.cohere.com/v1/embed "HTTP/1.1 200 OK"
2025-04-22 11:33:43,875 - root - INFO - Stored embedding for document 6fd270a4-e76a-4234-a4c4-a40d2ac64d56
2025-04-22 11:34:30,942 - httpx - INFO - HTTP Request: POST https://api.cohere.com/v1/rerank "HTTP/1.1 200 OK"
2025-04-22 11:34:30,960 - root - INFO - Document 6fd270a4-e76a-4234-a4c4-a40d2ac64d56 processed successfully
2025-04-22 11:34:30,960 - root - INFO - Document 6fd270a4-e76a-4234-a4c4-a40d2ac64d56 processed successfully
2025-04-22 11:43:04,651 - root - INFO - Processing upload for document ID: e92e078c-6d36-46b1-89ba-03f7387947e9
2025-04-22 11:43:04,652 - root - INFO - File saved to data/uploads/e92e078c-6d36-46b1-89ba-03f7387947e9_9.confidentiality agreement.docx
2025-04-22 11:43:04,758 - root - INFO - Processing document e92e078c-6d36-46b1-89ba-03f7387947e9 with content length: 161
2025-04-22 11:43:05,057 - httpx - INFO - HTTP Request: POST https://api.cohere.com/v1/embed "HTTP/1.1 200 OK"
2025-04-22 11:43:05,924 - root - INFO - Stored embedding for document e92e078c-6d36-46b1-89ba-03f7387947e9
2025-04-22 11:43:46,791 - httpx - INFO - HTTP Request: POST https://api.cohere.com/v1/rerank "HTTP/1.1 200 OK"
2025-04-22 11:43:46,807 - root - INFO - Document e92e078c-6d36-46b1-89ba-03f7387947e9 processed successfully
2025-04-22 11:43:46,807 - root - INFO - Document e92e078c-6d36-46b1-89ba-03f7387947e9 processed successfully
2025-04-22 11:52:38,067 - root - INFO - Processing upload for document ID: 2813c3dc-8496-4aff-b945-0b04e5c439c0
2025-04-22 11:52:38,069 - root - INFO - File saved to data/uploads/2813c3dc-8496-4aff-b945-0b04e5c439c0_7.Supplier SQualification requirements.docx
2025-04-22 11:52:38,184 - root - INFO - Processing document 2813c3dc-8496-4aff-b945-0b04e5c439c0 with content length: 229
2025-04-22 11:52:38,574 - httpx - INFO - HTTP Request: POST https://api.cohere.com/v1/embed "HTTP/1.1 200 OK"
2025-04-22 11:52:39,190 - root - INFO - Stored embedding for document 2813c3dc-8496-4aff-b945-0b04e5c439c0
2025-04-22 11:53:23,757 - root - ERROR - Error saving analysis for document 2813c3dc-8496-4aff-b945-0b04e5c439c0: table analysis has no column named issues_and_recommendations
2025-04-22 11:53:23,757 - root - ERROR - Error processing document 2813c3dc-8496-4aff-b945-0b04e5c439c0: table analysis has no column named issues_and_recommendations
2025-04-22 11:53:23,757 - root - ERROR - Error processing document: table analysis has no column named issues_and_recommendations
2025-04-22 11:53:23,783 - root - ERROR - Traceback (most recent call last):
File "C:\Users\babaw\Documents\Work\Mana Knight Digital\ds_task_scp\src\main.py", line 112, in upload_document
await document_processor.process_document(doc_id, file_path, document_type)
File "C:\Users\babaw\Documents\Work\Mana Knight Digital\ds_task_scp\src\services\document_processor.py", line 141, in process_document
self.database.save_analysis(doc_id, analysis)
File "C:\Users\babaw\Documents\Work\Mana Knight Digital\ds_task_scp\src\services\database.py", line 50, in save_analysis
cursor.execute('''
sqlite3.OperationalError: table analysis has no column named issues_and_recommendations
2025-04-22 11:55:32,835 - root - INFO - Processing upload for document ID: 9dc21524-8c93-427b-a6cc-04b7585a9545
2025-04-22 11:55:32,836 - root - INFO - File saved to data/uploads/9dc21524-8c93-427b-a6cc-04b7585a9545_8.form of tender.docx
2025-04-22 11:55:32,958 - root - INFO - Processing document 9dc21524-8c93-427b-a6cc-04b7585a9545 with content length: 523
2025-04-22 11:55:33,174 - httpx - INFO - HTTP Request: POST https://api.cohere.com/v1/embed "HTTP/1.1 200 OK"
2025-04-22 11:55:33,740 - root - INFO - Stored embedding for document 9dc21524-8c93-427b-a6cc-04b7585a9545
2025-04-22 11:56:27,580 - root - INFO - Document 9dc21524-8c93-427b-a6cc-04b7585a9545 processed successfully
2025-04-22 11:56:27,588 - root - INFO - Document 9dc21524-8c93-427b-a6cc-04b7585a9545 processed successfully
2025-04-23 22:03:47,212 - root - INFO - Processing upload for document ID: 65dd5884-72ef-4eb4-bb03-b9fad0886da4
2025-04-23 22:03:47,215 - root - INFO - File saved to data/uploads/65dd5884-72ef-4eb4-bb03-b9fad0886da4_3.Bill of Quantities.docx
2025-04-23 22:03:47,260 - root - INFO - Document 65dd5884-72ef-4eb4-bb03-b9fad0886da4 upload initiated successfully
2025-04-23 22:03:47,905 - root - INFO - Processing document 65dd5884-72ef-4eb4-bb03-b9fad0886da4 with content length: 2057
2025-04-23 22:03:48,399 - httpx - INFO - HTTP Request: POST https://api.cohere.com/v1/embed "HTTP/1.1 200 OK"
2025-04-23 22:03:49,194 - root - INFO - Stored embedding for document 65dd5884-72ef-4eb4-bb03-b9fad0886da4 in namespace ''
2025-04-23 22:04:54,737 - root - INFO - Document 65dd5884-72ef-4eb4-bb03-b9fad0886da4 processed successfully
2025-04-23 22:08:00,451 - root - INFO - Processing upload for document ID: 74bcb446-dccb-4707-afb3-dbd45abba9a2
2025-04-23 22:08:00,455 - root - INFO - File saved to data/uploads/74bcb446-dccb-4707-afb3-dbd45abba9a2_8.form of tender.docx
2025-04-23 22:08:00,522 - root - INFO - Document 74bcb446-dccb-4707-afb3-dbd45abba9a2 upload initiated successfully
2025-04-23 22:08:01,027 - root - INFO - Processing document 74bcb446-dccb-4707-afb3-dbd45abba9a2 with content length: 523
2025-04-23 22:08:01,262 - httpx - INFO - HTTP Request: POST https://api.cohere.com/v1/embed "HTTP/1.1 200 OK"
2025-04-23 22:08:01,797 - root - INFO - Stored embedding for document 74bcb446-dccb-4707-afb3-dbd45abba9a2 in namespace 'gcp-starter'
2025-04-23 22:08:46,050 - root - INFO - Document 74bcb446-dccb-4707-afb3-dbd45abba9a2 processed successfully
2025-04-28 21:04:14,200 - root - INFO - Processing upload for document ID: de1e735c-15a7-4d03-9fbf-92e8acc822fd
2025-04-28 21:04:14,200 - root - INFO - File saved to data/uploads/de1e735c-15a7-4d03-9fbf-92e8acc822fd_2.Tender Specifications.docx
2025-04-28 21:04:14,215 - root - INFO - Document de1e735c-15a7-4d03-9fbf-92e8acc822fd upload initiated successfully
2025-04-28 21:04:16,258 - tika.tika - INFO - Retrieving http://search.maven.org/remotecontent?filepath=org/apache/tika/tika-server-standard/3.1.0/tika-server-standard-3.1.0.jar to C:\Users\babaw\AppData\Local\Temp\tika-server.jar.
2025-04-28 21:04:52,162 - tika.tika - INFO - Retrieving http://search.maven.org/remotecontent?filepath=org/apache/tika/tika-server-standard/3.1.0/tika-server-standard-3.1.0.jar.md5 to C:\Users\babaw\AppData\Local\Temp\tika-server.jar.md5.
2025-04-28 21:04:53,338 - tika.tika - WARNING - Failed to see startup log message; retrying...
2025-04-28 21:04:58,353 - tika.tika - WARNING - Failed to see startup log message; retrying...
2025-04-28 21:05:03,354 - tika.tika - WARNING - Failed to see startup log message; retrying...
2025-04-28 21:05:08,367 - tika.tika - ERROR - Tika startup log message not received after 3 tries.
2025-04-28 21:05:08,367 - tika.tika - ERROR - Failed to receive startup confirmation from startServer.
2025-04-28 21:05:08,367 - root - ERROR - Error parsing document with Tika: Unable to start Tika server.
2025-04-28 21:05:08,369 - root - INFO - Processing document de1e735c-15a7-4d03-9fbf-92e8acc822fd with content length: 0
2025-04-28 21:05:19,572 - root - ERROR - Error processing document de1e735c-15a7-4d03-9fbf-92e8acc822fd: [Errno 11001] getaddrinfo failed
2025-04-28 21:53:58,710 - root - INFO - Processing upload for document ID: c7b13f74-699b-4735-b9bc-a3c2e66545a6
2025-04-28 21:53:58,715 - root - INFO - File saved to data/uploads/c7b13f74-699b-4735-b9bc-a3c2e66545a6_9.confidentiality agreement.pdf
2025-04-28 21:53:58,731 - root - INFO - Document c7b13f74-699b-4735-b9bc-a3c2e66545a6 upload initiated successfully
2025-04-28 21:53:59,507 - root - INFO - Processing document c7b13f74-699b-4735-b9bc-a3c2e66545a6 with content length: 217
2025-04-28 21:54:10,920 - root - ERROR - Error processing document c7b13f74-699b-4735-b9bc-a3c2e66545a6: [Errno 11001] getaddrinfo failed