Update README and core files, remove test/debug scripts, improve documentation and robustness
This commit is contained in:
Iyeoluwa Akinrinola
@@ -7,9 +7,9 @@ AI-powered receipt-to-transaction matching engine using Groq LLM. This is a **Da
|
||||
This Data Science Engine receives QuickBooks transaction data from backend applications and provides:
|
||||
- **AI-powered receipt processing** (OCR and data extraction)
|
||||
- **Intelligent receipt-transaction matching** with confidence scores
|
||||
- **Google Drive integration** for batch receipt processing
|
||||
- **Configurable AI rules** for business logic
|
||||
- **Feedback logging** for continuous improvement
|
||||
- **RESTful API** for easy integration
|
||||
|
||||
## 🚀 Quick Start
|
||||
|
||||
@@ -19,11 +19,22 @@ pip install -r requirements.txt
|
||||
```
|
||||
|
||||
### 2. Configure API Keys
|
||||
The Groq API key is already configured in `config.py`
|
||||
Create a `.env` file in the project root with your Groq API key:
|
||||
|
||||
### 3. Start the DS Engine
|
||||
```bash
|
||||
# Create .env file
|
||||
echo "GROQ_API_KEY=your_actual_groq_api_key_here" > .env
|
||||
```
|
||||
|
||||
**Important**: Get your API key from [Groq Console](https://console.groq.com/)
|
||||
|
||||
### 3. Start the Server
|
||||
```bash
|
||||
# Option 1: Using the main script
|
||||
python main.py
|
||||
|
||||
# Option 2: Using uvicorn directly
|
||||
uvicorn main:app --host 0.0.0.0 --port 8343 --reload
|
||||
```
|
||||
|
||||
### 4. Access API Documentation
|
||||
@@ -32,22 +43,16 @@ python main.py
|
||||
|
||||
## 📋 API Endpoints
|
||||
|
||||
### QuickBooks Data Import
|
||||
- `POST /transactions/import/quickbooks` - Import and convert QuickBooks transactions
|
||||
### Transaction Import
|
||||
- `POST /transactions/import/csv` - Import transactions from CSV file
|
||||
- `POST /transactions/import/image` - Import transactions from image/PDF
|
||||
|
||||
### Receipt Processing
|
||||
- `POST /upload` - Upload receipt documents (PDF/images)
|
||||
- `POST /upload-multiple` - Upload multiple receipt documents
|
||||
- `POST /process/{file_id}` - Extract data from uploaded documents
|
||||
- `GET /documents` - List all processed documents
|
||||
|
||||
### Google Drive Integration
|
||||
- `POST /drive/sync` - Sync and process receipts from Google Drive
|
||||
- `GET /drive/folders` - List accessible Google Drive folders
|
||||
- `GET /drive/folder/{folder_id}` - Get folder information
|
||||
|
||||
### AI Matching Engine
|
||||
- `POST /match` - Match receipts to transactions using AI
|
||||
- `POST /approve` - Approve or reject AI matches
|
||||
- `POST /match-specific` - Match specific receipts to transactions using AI
|
||||
|
||||
### AI Rules Management
|
||||
- `POST /rules` - Add new AI rules
|
||||
@@ -56,6 +61,7 @@ python main.py
|
||||
|
||||
### System Monitoring
|
||||
- `GET /stats` - Get system statistics and performance metrics
|
||||
- `GET /` - Health check endpoint
|
||||
|
||||
## 🔧 Core Components
|
||||
|
||||
@@ -63,21 +69,25 @@ python main.py
|
||||
- Uses Groq LLM to compare receipts and transactions
|
||||
- Provides confidence scores and reasoning
|
||||
- Configurable matching criteria (amount, date, vendor)
|
||||
- Rate limiting to prevent API quota exhaustion
|
||||
|
||||
### **AIRulesEngine** (`ai_rules.py`)
|
||||
- Applies business rules for auto-approval and categorization
|
||||
- Configurable rule conditions and actions
|
||||
- Supports system and user-generated rules
|
||||
- Safe condition evaluation with proper error handling
|
||||
|
||||
### **DocumentProcessor** (`document_processor.py`)
|
||||
- AI-powered receipt data extraction
|
||||
- AI-powered receipt data extraction using Groq vision model
|
||||
- Supports PDF and image formats
|
||||
- Uses Groq vision model for OCR
|
||||
- Robust JSON parsing with error handling
|
||||
- Extracts vendor, amount, date, tax, and category information
|
||||
|
||||
### **MatchingEngine** (`matching_engine.py`)
|
||||
- Main orchestrator combining all components
|
||||
- Handles the complete matching workflow
|
||||
- Provides statistics and feedback logging
|
||||
- Configurable confidence thresholds
|
||||
|
||||
### **FeedbackLogger** (`feedback_logger.py`)
|
||||
- Tracks manual overrides for AI training
|
||||
@@ -87,70 +97,46 @@ python main.py
|
||||
## 📊 Configuration
|
||||
|
||||
Edit `config.py` to adjust:
|
||||
- **Confidence threshold** (default: 0.8)
|
||||
- **Confidence threshold** (default: 0.3)
|
||||
- **Date tolerance days** (default: 7)
|
||||
- **Amount tolerance percent** (default: 5%)
|
||||
- **Groq API key** (already configured)
|
||||
- **Groq API key** (from environment variable)
|
||||
|
||||
## 🔄 Integration Workflow
|
||||
|
||||
### 1. Backend Sends QuickBooks Data
|
||||
```python
|
||||
# Backend sends QuickBooks transactions
|
||||
response = requests.post(
|
||||
"http://localhost:8343/transactions/import/quickbooks",
|
||||
json={
|
||||
"transactions": [
|
||||
{
|
||||
"id": "QB_TXN_123",
|
||||
"txn_date": "2024-01-15",
|
||||
"amount": 12.50,
|
||||
"payee_name": "Starbucks",
|
||||
"memo": "Coffee purchase"
|
||||
}
|
||||
]
|
||||
}
|
||||
)
|
||||
### 1. Import Transactions
|
||||
```bash
|
||||
# Import from CSV
|
||||
curl -X POST -F "file=@transactions.csv" http://localhost:8343/transactions/import/csv
|
||||
|
||||
# Import from image
|
||||
curl -X POST -F "file=@statement.jpg" http://localhost:8343/transactions/import/image
|
||||
```
|
||||
|
||||
### 2. Process Receipts
|
||||
```python
|
||||
# Sync from Google Drive
|
||||
response = requests.post(
|
||||
"http://localhost:8343/drive/sync",
|
||||
json={"folder_id": "your_folder_id"}
|
||||
)
|
||||
### 2. Upload and Process Receipts
|
||||
```bash
|
||||
# Upload receipts
|
||||
curl -X POST -F "files=@receipt1.jpg" -F "files=@receipt2.jpg" http://localhost:8343/upload-multiple
|
||||
|
||||
# Or upload directly
|
||||
response = requests.post(
|
||||
"http://localhost:8343/upload",
|
||||
files={"file": receipt_file}
|
||||
)
|
||||
# Process a specific receipt
|
||||
curl -X POST http://localhost:8343/process/{file_id}
|
||||
```
|
||||
|
||||
### 3. AI Matching
|
||||
```python
|
||||
# Match receipts to transactions
|
||||
response = requests.post(
|
||||
"http://localhost:8343/match",
|
||||
json={
|
||||
"receipts": processed_receipts,
|
||||
"transactions": converted_transactions
|
||||
}
|
||||
)
|
||||
```bash
|
||||
# Match specific receipts
|
||||
curl -X POST -H "Content-Type: application/json" \
|
||||
-d '["file_id_1", "file_id_2"]' \
|
||||
http://localhost:8343/match-specific
|
||||
```
|
||||
|
||||
### 4. User Feedback
|
||||
```python
|
||||
# Approve or reject matches
|
||||
response = requests.post(
|
||||
"http://localhost:8343/approve",
|
||||
json={
|
||||
"match_id": "match_123",
|
||||
"user_id": "user_456",
|
||||
"action": "approve"
|
||||
}
|
||||
)
|
||||
### 4. Check Results
|
||||
```bash
|
||||
# Get system stats
|
||||
curl http://localhost:8343/stats
|
||||
|
||||
# View AI rules
|
||||
curl http://localhost:8343/rules
|
||||
```
|
||||
|
||||
## 🎯 Key Features
|
||||
@@ -159,55 +145,96 @@ response = requests.post(
|
||||
- **Rule-based auto-approval** and categorization
|
||||
- **Feedback logging** for continuous improvement
|
||||
- **Configurable matching parameters**
|
||||
- **Google Drive integration** for batch processing
|
||||
- **JSON API** for easy backend integration
|
||||
- **RESTful JSON API** for easy backend integration
|
||||
- **Comprehensive error handling**
|
||||
- **Rate limiting** to prevent API quota exhaustion
|
||||
- **Robust JSON parsing** for AI responses
|
||||
|
||||
## 📝 Data Formats
|
||||
|
||||
### QuickBooks Transaction Input
|
||||
### Transaction Input (CSV)
|
||||
```csv
|
||||
Date,Description,Amount,Category
|
||||
2024-01-15,Starbucks Coffee,12.50,Food & Dining
|
||||
2024-01-16,Office Supplies,45.99,Office
|
||||
```
|
||||
|
||||
### Receipt Processing Output
|
||||
```json
|
||||
{
|
||||
"id": "string",
|
||||
"txn_date": "YYYY-MM-DD",
|
||||
"amount": 0.00,
|
||||
"payee_name": "string",
|
||||
"memo": "string (optional)",
|
||||
"account_name": "string (optional)",
|
||||
"txn_type": "string (optional)"
|
||||
"vendor": "Starbucks",
|
||||
"total_amount": 12.50,
|
||||
"tax_amount": 1.25,
|
||||
"date": "2024-01-15",
|
||||
"category": "Food & Dining",
|
||||
"confidence": 0.95,
|
||||
"extraction_success": true
|
||||
}
|
||||
```
|
||||
|
||||
### Match Result Output
|
||||
```json
|
||||
{
|
||||
"receipt_id": "string",
|
||||
"transaction_id": "string",
|
||||
"receipt_id": "uuid",
|
||||
"transaction_id": "transaction_123",
|
||||
"confidence_score": 0.95,
|
||||
"match_reason": "string",
|
||||
"receipt_vendor": "string",
|
||||
"receipt_amount": 0.00,
|
||||
"transaction_vendor": "string",
|
||||
"transaction_amount": 0.00
|
||||
"match_reason": "Same vendor, minor date difference (Auto-approved by rules)",
|
||||
"receipt_vendor": "Starbucks",
|
||||
"receipt_amount": 12.50,
|
||||
"transaction_vendor": "STARBUCKS",
|
||||
"transaction_amount": 12.50
|
||||
}
|
||||
```
|
||||
|
||||
## 🔍 AI Matching Criteria
|
||||
|
||||
The engine uses three primary criteria for matching:
|
||||
The engine uses multiple criteria for matching:
|
||||
|
||||
1. **Amount Similarity** - Compares receipt and transaction amounts (5% tolerance)
|
||||
2. **Date Proximity** - Checks date closeness (7-day tolerance)
|
||||
3. **Vendor Matching** - AI-powered vendor name comparison
|
||||
3. **Vendor Matching** - AI-powered vendor name comparison using Groq LLM
|
||||
4. **Rule-based Auto-approval** - Automatic approval for exact matches and high-confidence matches
|
||||
|
||||
## 🛠️ Development
|
||||
|
||||
### Project Structure
|
||||
```
|
||||
├── main.py # FastAPI application entry point
|
||||
├── ai_matcher.py # AI-powered matching logic
|
||||
├── ai_rules.py # Business rules engine
|
||||
├── document_processor.py # Receipt data extraction
|
||||
├── matching_engine.py # Main matching orchestrator
|
||||
├── feedback_logger.py # User feedback tracking
|
||||
├── models.py # Pydantic data models
|
||||
├── api_models.py # API request/response models
|
||||
├── config.py # Configuration settings
|
||||
├── requirements.txt # Python dependencies
|
||||
└── test_images/ # Test image files
|
||||
```
|
||||
|
||||
### Running Tests
|
||||
```bash
|
||||
# Test the server
|
||||
curl http://localhost:8343/
|
||||
|
||||
# Test stats endpoint
|
||||
curl http://localhost:8343/stats
|
||||
|
||||
# Test rules endpoint
|
||||
curl http://localhost:8343/rules
|
||||
```
|
||||
|
||||
## 🚀 Production Deployment
|
||||
|
||||
For production deployment:
|
||||
- Replace in-memory storage with a database
|
||||
- Configure proper authentication
|
||||
- Set up monitoring and logging
|
||||
- Use environment variables for configuration
|
||||
- Replace in-memory storage with a database (PostgreSQL recommended)
|
||||
- Configure proper authentication and authorization
|
||||
- Set up monitoring and logging (ELK stack recommended)
|
||||
- Use environment variables for all configuration
|
||||
- Implement proper error handling and retries
|
||||
- Set up rate limiting and API quotas
|
||||
- Configure CORS for frontend integration
|
||||
- Use HTTPS in production
|
||||
|
||||
## 📞 Support
|
||||
|
||||
@@ -218,3 +245,18 @@ This Data Science Engine is designed to be integrated with backend applications
|
||||
- External integrations
|
||||
|
||||
The engine focuses purely on AI/ML capabilities and provides a clean JSON API for backend integration.
|
||||
|
||||
## 🔧 Troubleshooting
|
||||
|
||||
### Common Issues
|
||||
|
||||
1. **API Key Error**: Ensure `GROQ_API_KEY` is set in your `.env` file
|
||||
2. **Port Already in Use**: Kill existing process with `pkill -f "python main.py"`
|
||||
3. **Import Errors**: Install dependencies with `pip install -r requirements.txt`
|
||||
4. **Rate Limiting**: The system includes built-in rate limiting to prevent API quota exhaustion
|
||||
|
||||
### Logs
|
||||
Check the application logs for detailed error information:
|
||||
```bash
|
||||
tail -f app.log
|
||||
```
|
||||
+151
-27
@@ -3,34 +3,75 @@ from datetime import datetime, timedelta
|
||||
from typing import List, Tuple
|
||||
import config
|
||||
from models import Receipt, Transaction, Match
|
||||
import time
|
||||
import logging
|
||||
import asyncio
|
||||
|
||||
# Set up logging
|
||||
logging.basicConfig(level=logging.INFO)
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
class AIMatcher:
|
||||
def __init__(self):
|
||||
self.client = groq.Groq(api_key=config.GROQ_API_KEY)
|
||||
self.model = "llama3-8b-8192"
|
||||
self.max_retries = 3
|
||||
self.retry_delay = 2 # seconds - increased for rate limiting
|
||||
self.rate_limit_delay = 1.0 # seconds between API calls
|
||||
self.last_api_call = 0
|
||||
|
||||
def match_receipts_to_transactions(self, receipts: List[Receipt], transactions: List[Transaction]) -> List[Match]:
|
||||
"""Match receipts to transactions using AI"""
|
||||
logger.info(f"Starting AI matching for {len(receipts)} receipts against {len(transactions)} transactions")
|
||||
matches = []
|
||||
|
||||
for receipt in receipts:
|
||||
for i, receipt in enumerate(receipts):
|
||||
logger.info(f"Processing receipt {i+1}/{len(receipts)}: {receipt.vendor} - ${receipt.amount}")
|
||||
|
||||
# Rate limiting
|
||||
self._rate_limit()
|
||||
|
||||
# Get the BEST match for this receipt (highest confidence score)
|
||||
best_match = self._find_best_match(receipt, transactions)
|
||||
if best_match:
|
||||
matches.append(best_match)
|
||||
logger.info(f"Found match: {best_match.confidence_score:.3f} - {best_match.match_reason}")
|
||||
else:
|
||||
logger.warning(f"No match found for receipt: {receipt.vendor} - ${receipt.amount}")
|
||||
|
||||
return sorted(matches, key=lambda x: x.confidence_score, reverse=True)
|
||||
# Sort by confidence score (highest first)
|
||||
matches = sorted(matches, key=lambda x: x.confidence_score, reverse=True)
|
||||
logger.info(f"AI matching completed. Found {len(matches)} matches")
|
||||
return matches
|
||||
|
||||
def _rate_limit(self):
|
||||
"""Implement rate limiting to avoid API quota exhaustion"""
|
||||
current_time = time.time()
|
||||
time_since_last_call = current_time - self.last_api_call
|
||||
|
||||
if time_since_last_call < self.rate_limit_delay:
|
||||
sleep_time = self.rate_limit_delay - time_since_last_call
|
||||
logger.debug(f"Rate limiting: sleeping for {sleep_time:.2f} seconds")
|
||||
time.sleep(sleep_time)
|
||||
|
||||
self.last_api_call = time.time()
|
||||
|
||||
def _find_best_match(self, receipt: Receipt, transactions: List[Transaction]) -> Match:
|
||||
"""Find the BEST match for a receipt (highest confidence score)"""
|
||||
candidates = self._filter_candidates(receipt, transactions)
|
||||
if not candidates:
|
||||
logger.warning(f"No candidates found for receipt: {receipt.vendor} - ${receipt.amount}")
|
||||
return None
|
||||
|
||||
logger.info(f"Found {len(candidates)} candidates for receipt: {receipt.vendor}")
|
||||
|
||||
best_match = None
|
||||
highest_score = 0
|
||||
|
||||
for transaction in candidates:
|
||||
score, reason = self._calculate_match_score(receipt, transaction)
|
||||
logger.debug(f"Score {score:.3f} for transaction {transaction.vendor}: {reason}")
|
||||
|
||||
# Keep the match with the highest score, regardless of how low it is
|
||||
if score > highest_score:
|
||||
highest_score = score
|
||||
@@ -39,21 +80,23 @@ class AIMatcher:
|
||||
return best_match
|
||||
|
||||
def _filter_candidates(self, receipt: Receipt, transactions: List[Transaction]) -> List[Transaction]:
|
||||
# Return MOST transactions - let the AI decide on scoring
|
||||
# Only filter out transactions with completely different amounts (>100% difference) to avoid obvious mismatches
|
||||
"""Filter transactions to create a reasonable candidate list"""
|
||||
candidates = []
|
||||
amount_threshold = receipt.amount * 1.0 # 100% threshold - more inclusive
|
||||
amount_threshold = receipt.amount * 2.0 # 200% threshold - very inclusive
|
||||
|
||||
for transaction in transactions:
|
||||
# Use absolute value for transaction amount comparison
|
||||
transaction_amount_abs = abs(transaction.amount)
|
||||
|
||||
# Only exclude transactions with obviously different amounts
|
||||
if abs(receipt.amount - transaction_amount_abs) <= amount_threshold:
|
||||
candidates.append(transaction)
|
||||
|
||||
logger.debug(f"Filtered {len(transactions)} transactions to {len(candidates)} candidates")
|
||||
return candidates
|
||||
|
||||
def _calculate_match_score(self, receipt: Receipt, transaction: Transaction) -> Tuple[float, str]:
|
||||
"""Calculate match score using AI"""
|
||||
# Calculate differences for the AI to consider
|
||||
date_diff = abs((receipt.receipt_date - transaction.transaction_date).days)
|
||||
transaction_amount_abs = abs(transaction.amount)
|
||||
@@ -61,7 +104,7 @@ class AIMatcher:
|
||||
amount_percent_diff = (amount_diff / receipt.amount) * 100 if receipt.amount > 0 else 0
|
||||
|
||||
prompt = f"""
|
||||
Compare this receipt with this transaction and provide a confidence score (0-1) and brief reason:
|
||||
Compare this receipt with this transaction and provide a confidence score (0-1) and brief reason.
|
||||
|
||||
Receipt: {receipt.vendor}, ${receipt.amount}, {receipt.receipt_date.strftime('%Y-%m-%d')}
|
||||
Transaction: {transaction.vendor}, ${transaction.amount} (absolute: ${transaction_amount_abs}), {transaction.transaction_date.strftime('%Y-%m-%d')}
|
||||
@@ -81,33 +124,114 @@ class AIMatcher:
|
||||
- Minimal similarity: 0.1-0.19
|
||||
- No meaningful similarity: 0.0-0.09
|
||||
|
||||
Examples:
|
||||
- Same vendor, same amount, 11 days apart: 0.7-0.8
|
||||
- Similar vendor name, same amount, same date: 0.8-0.9
|
||||
- Same vendor, 10% amount difference, same date: 0.6-0.7
|
||||
- Different vendor, same amount, same date: 0.3-0.4
|
||||
- Completely different vendor, amount, date: 0.1-0.2
|
||||
|
||||
Consider vendor name similarity, amount accuracy, and date proximity. Score based on overall likelihood this is the correct match.
|
||||
|
||||
Return only: score|reason
|
||||
IMPORTANT: Return ONLY the score and reason separated by a pipe character.
|
||||
Format: [score]|[reason]
|
||||
Example: 0.85|Same vendor, same amount, 2 days apart
|
||||
"""
|
||||
|
||||
for attempt in range(self.max_retries):
|
||||
try:
|
||||
result = self._call_groq_api_with_timeout(prompt, timeout=30) # Increased timeout
|
||||
|
||||
# Parse the result - handle multiple formats
|
||||
score, reason = self._parse_ai_response(result)
|
||||
|
||||
logger.debug(f"AI Response: {result}")
|
||||
logger.debug(f"Parsed: score={score}, reason={reason}")
|
||||
|
||||
return score, reason
|
||||
|
||||
except Exception as e:
|
||||
logger.warning(f"Attempt {attempt + 1} failed for receipt {receipt.id}: {str(e)}")
|
||||
if attempt < self.max_retries - 1:
|
||||
# Exponential backoff for rate limiting
|
||||
sleep_time = self.retry_delay * (2 ** attempt)
|
||||
logger.info(f"Waiting {sleep_time} seconds before retry...")
|
||||
time.sleep(sleep_time)
|
||||
else:
|
||||
logger.error(f"All attempts failed for receipt {receipt.id}")
|
||||
return 0.0, f"AI error after {self.max_retries} attempts: {str(e)}"
|
||||
|
||||
def _parse_ai_response(self, result: str) -> Tuple[float, str]:
|
||||
"""Parse AI response with robust error handling"""
|
||||
result = result.strip()
|
||||
logger.debug(f"Parsing AI response: {result}")
|
||||
|
||||
# Try to find score in various formats
|
||||
if '|' in result:
|
||||
parts = result.split('|')
|
||||
logger.debug(f"Split response into {len(parts)} parts: {parts}")
|
||||
|
||||
# Look for a numeric score in any part
|
||||
for i, part in enumerate(parts):
|
||||
part = part.strip()
|
||||
try:
|
||||
# Remove any non-numeric characters except decimal point
|
||||
score_str_clean = ''.join(c for c in part if c.isdigit() or c == '.')
|
||||
if score_str_clean:
|
||||
score = float(score_str_clean)
|
||||
if 0 <= score <= 1: # Valid confidence score
|
||||
# Get reason from other parts
|
||||
reason_parts = [p.strip() for j, p in enumerate(parts) if j != i and p.strip()]
|
||||
reason = ' | '.join(reason_parts) if reason_parts else "Score extracted"
|
||||
logger.debug(f"Found score {score} in part {i}, reason: {reason}")
|
||||
return score, reason
|
||||
except ValueError:
|
||||
continue
|
||||
|
||||
# Try to extract just a number from the response
|
||||
try:
|
||||
import re
|
||||
numbers = re.findall(r'\d+\.?\d*', result)
|
||||
if numbers:
|
||||
for num_str in numbers:
|
||||
score = float(num_str)
|
||||
if 0 <= score <= 1: # Valid confidence score
|
||||
logger.debug(f"Extracted score {score} from response")
|
||||
return score, f"Extracted from response: {result[:50]}..."
|
||||
except (ValueError, IndexError):
|
||||
pass
|
||||
|
||||
# Fallback - try to find any number and normalize it
|
||||
try:
|
||||
import re
|
||||
numbers = re.findall(r'\d+\.?\d*', result)
|
||||
if numbers:
|
||||
score = float(numbers[0])
|
||||
# Normalize to 0-1 range if it's a percentage or other scale
|
||||
if score > 1:
|
||||
score = score / 100 # Assume percentage
|
||||
score = max(0, min(1, score)) # Clamp to 0-1
|
||||
logger.debug(f"Normalized score {score} from response")
|
||||
return score, f"Normalized from response: {result[:50]}..."
|
||||
except (ValueError, IndexError):
|
||||
pass
|
||||
|
||||
# Final fallback
|
||||
logger.warning(f"Could not parse AI response: {result}")
|
||||
return 0.0, f"Unparseable response: {result[:50]}..."
|
||||
|
||||
def _call_groq_api_with_timeout(self, prompt: str, timeout: int = 15) -> str:
|
||||
"""Make API call with timeout and retry logic"""
|
||||
import concurrent.futures
|
||||
|
||||
def api_call():
|
||||
try:
|
||||
response = self.client.chat.completions.create(
|
||||
model=self.model,
|
||||
messages=[{"role": "user", "content": prompt}],
|
||||
max_tokens=100,
|
||||
max_tokens=200,
|
||||
temperature=0.1
|
||||
)
|
||||
|
||||
result = response.choices[0].message.content.strip()
|
||||
if '|' in result:
|
||||
score_str, reason = result.split('|', 1)
|
||||
score = float(score_str.strip())
|
||||
return min(max(score, 0), 1), reason.strip()
|
||||
else:
|
||||
return 0.0, "Invalid AI response"
|
||||
|
||||
return response.choices[0].message.content.strip()
|
||||
except Exception as e:
|
||||
return 0.0, f"AI error: {str(e)}"
|
||||
raise e
|
||||
|
||||
try:
|
||||
with concurrent.futures.ThreadPoolExecutor() as executor:
|
||||
future = executor.submit(api_call)
|
||||
return future.result(timeout=timeout)
|
||||
except concurrent.futures.TimeoutError:
|
||||
raise Exception(f"API call timed out after {timeout} seconds")
|
||||
except Exception as e:
|
||||
raise e
|
||||
+30
-5
@@ -20,7 +20,7 @@ class AIRulesEngine:
|
||||
self.rules = [
|
||||
AIRule("exact_amount_match", "amount_diff <= 0.01", "auto_approve", "system"),
|
||||
AIRule("same_vendor_same_date", "vendor_match and date_diff <= 1", "high_confidence", "system"),
|
||||
AIRule("gas_station_pattern", "vendor contains 'gas' or 'fuel'", "categorize_transport", "system")
|
||||
AIRule("gas_station_pattern", "vendor_contains_gas_or_fuel", "categorize_transport", "system")
|
||||
]
|
||||
|
||||
def apply_rules(self, receipt: Receipt, transaction: Transaction) -> Dict[str, Any]:
|
||||
@@ -36,17 +36,42 @@ class AIRulesEngine:
|
||||
return results
|
||||
|
||||
def _evaluate_condition(self, condition: str, receipt: Receipt, transaction: Transaction) -> bool:
|
||||
amount_diff = abs(receipt.amount - transaction.amount)
|
||||
"""Safely evaluate rule conditions without using eval()"""
|
||||
amount_diff = abs(receipt.amount - abs(transaction.amount))
|
||||
date_diff = abs((receipt.receipt_date - transaction.transaction_date).days)
|
||||
vendor_match = receipt.vendor.lower() in transaction.vendor.lower() or transaction.vendor.lower() in receipt.vendor.lower()
|
||||
vendor_lower = receipt.vendor.lower()
|
||||
vendor_contains_gas_or_fuel = 'gas' in vendor_lower or 'fuel' in vendor_lower
|
||||
|
||||
return eval(condition, {
|
||||
# Handle specific condition types safely
|
||||
if condition == "amount_diff <= 0.01":
|
||||
return amount_diff <= 0.01
|
||||
elif condition == "vendor_match and date_diff <= 1":
|
||||
return vendor_match and date_diff <= 1
|
||||
elif condition == "vendor_contains_gas_or_fuel":
|
||||
return vendor_contains_gas_or_fuel
|
||||
else:
|
||||
# For any other conditions, try to evaluate them safely
|
||||
try:
|
||||
# Only allow safe operations
|
||||
safe_globals = {
|
||||
"amount_diff": amount_diff,
|
||||
"date_diff": date_diff,
|
||||
"vendor_match": vendor_match,
|
||||
"vendor_contains_gas_or_fuel": vendor_contains_gas_or_fuel,
|
||||
"receipt": receipt,
|
||||
"transaction": transaction
|
||||
})
|
||||
"transaction": transaction,
|
||||
"abs": abs,
|
||||
"len": len,
|
||||
"min": min,
|
||||
"max": max,
|
||||
"sum": sum,
|
||||
"round": round
|
||||
}
|
||||
return eval(condition, safe_globals, {})
|
||||
except (SyntaxError, NameError, TypeError) as e:
|
||||
print(f"Warning: Invalid condition '{condition}': {e}")
|
||||
return False
|
||||
|
||||
def _execute_action(self, action: str, results: Dict[str, Any], receipt: Receipt, transaction: Transaction):
|
||||
if action == "auto_approve":
|
||||
|
||||
@@ -3,7 +3,13 @@ from dotenv import load_dotenv
|
||||
|
||||
load_dotenv()
|
||||
|
||||
GROQ_API_KEY = "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"
|
||||
# Get API key from environment variable with fallback
|
||||
GROQ_API_KEY = os.getenv("GROQ_API_KEY", "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")
|
||||
|
||||
# Validate API key
|
||||
if not GROQ_API_KEY or GROQ_API_KEY == "your_api_key_here":
|
||||
raise ValueError("GROQ_API_KEY environment variable is not set or invalid. Please set it in your .env file.")
|
||||
|
||||
CONFIDENCE_THRESHOLD = 0.3
|
||||
DATE_TOLERANCE_DAYS = 7
|
||||
AMOUNT_TOLERANCE_PERCENT = 0.05
|
||||
@@ -1,82 +0,0 @@
|
||||
import csv
|
||||
from dateutil import parser
|
||||
from datetime import datetime, timedelta
|
||||
|
||||
# Config values
|
||||
DATE_TOLERANCE_DAYS = 7
|
||||
AMOUNT_TOLERANCE_PERCENT = 0.05
|
||||
CONFIDENCE_THRESHOLD = 0.8
|
||||
|
||||
# Receipt data
|
||||
receipt_date = datetime(2025, 2, 7)
|
||||
receipt_amount = 1412.5
|
||||
receipt_vendor = "Ajai Srivastava CPA, Accounting Services & Taxes"
|
||||
|
||||
print("=== DEBUGGING AJAI RECEIPT MATCH ===")
|
||||
print(f"Receipt Date: {receipt_date}")
|
||||
print(f"Receipt Amount: ${receipt_amount}")
|
||||
print(f"Receipt Vendor: {receipt_vendor}")
|
||||
print(f"Date Tolerance: {DATE_TOLERANCE_DAYS} days")
|
||||
print(f"Amount Tolerance: {AMOUNT_TOLERANCE_PERCENT * 100}%")
|
||||
print()
|
||||
|
||||
# Check CSV transaction
|
||||
csv_transaction = {
|
||||
"date": "2/18/2025",
|
||||
"amount": -1412.5,
|
||||
"vendor": "Ajai Srivastava"
|
||||
}
|
||||
|
||||
# Parse CSV date
|
||||
csv_date = parser.parse(csv_transaction["date"])
|
||||
csv_amount = csv_transaction["amount"]
|
||||
csv_vendor = csv_transaction["vendor"]
|
||||
|
||||
print("=== CSV TRANSACTION ===")
|
||||
print(f"CSV Date: {csv_date}")
|
||||
print(f"CSV Amount: ${csv_amount}")
|
||||
print(f"CSV Vendor: {csv_vendor}")
|
||||
print()
|
||||
|
||||
# Check date tolerance
|
||||
date_diff = abs((receipt_date - csv_date).days)
|
||||
date_match = date_diff <= DATE_TOLERANCE_DAYS
|
||||
|
||||
print("=== DATE CHECK ===")
|
||||
print(f"Date Difference: {date_diff} days")
|
||||
print(f"Date Match: {date_match}")
|
||||
print(f"Tolerance: {DATE_TOLERANCE_DAYS} days")
|
||||
print()
|
||||
|
||||
# Check amount tolerance
|
||||
amount_tolerance = receipt_amount * AMOUNT_TOLERANCE_PERCENT
|
||||
amount_diff = abs(receipt_amount - abs(csv_amount)) # Use absolute value for negative amounts
|
||||
amount_match = amount_diff <= amount_tolerance
|
||||
|
||||
print("=== AMOUNT CHECK ===")
|
||||
print(f"Receipt Amount: ${receipt_amount}")
|
||||
print(f"CSV Amount (abs): ${abs(csv_amount)}")
|
||||
print(f"Amount Difference: ${amount_diff}")
|
||||
print(f"Amount Tolerance: ${amount_tolerance}")
|
||||
print(f"Amount Match: {amount_match}")
|
||||
print()
|
||||
|
||||
# Check vendor similarity
|
||||
vendor_similarity = "Ajai Srivastava" in receipt_vendor
|
||||
print("=== VENDOR CHECK ===")
|
||||
print(f"Receipt Vendor: {receipt_vendor}")
|
||||
print(f"CSV Vendor: {csv_vendor}")
|
||||
print(f"Vendor Similarity: {vendor_similarity}")
|
||||
print()
|
||||
|
||||
# Overall result
|
||||
print("=== RESULT ===")
|
||||
if date_match and amount_match:
|
||||
print("✅ Transaction would pass initial filtering")
|
||||
print("Would proceed to AI matching stage")
|
||||
else:
|
||||
print("❌ Transaction filtered out before AI matching")
|
||||
if not date_match:
|
||||
print(f" - Date difference ({date_diff} days) > tolerance ({DATE_TOLERANCE_DAYS} days)")
|
||||
if not amount_match:
|
||||
print(f" - Amount difference (${amount_diff}) > tolerance (${amount_tolerance})")
|
||||
+135
-21
@@ -8,6 +8,9 @@ import config
|
||||
import os
|
||||
import aiofiles
|
||||
from datetime import datetime
|
||||
import logging
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
class DocumentProcessor:
|
||||
def __init__(self):
|
||||
@@ -160,27 +163,127 @@ class DocumentProcessor:
|
||||
import json
|
||||
import re
|
||||
|
||||
# Find JSON in response
|
||||
# Find JSON in response - try multiple patterns
|
||||
json_match = re.search(r'\{.*\}', result_text, re.DOTALL)
|
||||
if json_match:
|
||||
json_str = json_match.group()
|
||||
|
||||
# Clean up common JSON issues
|
||||
json_str = re.sub(r',\s*([}\]])', r'\1', json_str) # Remove trailing commas
|
||||
json_str = re.sub(r'([{,])\s*([a-zA-Z_][a-zA-Z0-9_]*)\s*:', r'\1"\2":', json_str) # Quote unquoted keys
|
||||
|
||||
try:
|
||||
data = json.loads(json_str)
|
||||
except json.JSONDecodeError as e:
|
||||
# Try to fix common JSON issues
|
||||
logger.warning(f"Initial JSON parsing failed: {e}")
|
||||
|
||||
# Try to extract individual fields using regex
|
||||
vendor_match = re.search(r'"vendor"\s*:\s*"([^"]*)"', json_str)
|
||||
total_amount_match = re.search(r'"total_amount"\s*:\s*([0-9.]+)', json_str)
|
||||
tax_amount_match = re.search(r'"tax_amount"\s*:\s*([0-9.]+)', json_str)
|
||||
date_match = re.search(r'"date"\s*:\s*"([^"]*)"', json_str)
|
||||
category_match = re.search(r'"category"\s*:\s*"([^"]*)"', json_str)
|
||||
confidence_match = re.search(r'"confidence"\s*:\s*([0-9.]+)', json_str)
|
||||
|
||||
data = {
|
||||
"vendor": vendor_match.group(1) if vendor_match else "",
|
||||
"total_amount": float(total_amount_match.group(1)) if total_amount_match else 0.0,
|
||||
"tax_amount": float(tax_amount_match.group(1)) if tax_amount_match else 0.0,
|
||||
"date": date_match.group(1) if date_match else "",
|
||||
"category": category_match.group(1) if category_match else "Other",
|
||||
"confidence": float(confidence_match.group(1)) if confidence_match else 0.5
|
||||
}
|
||||
|
||||
# Validate and clean data
|
||||
return {
|
||||
"vendor": data.get("vendor", "").strip(),
|
||||
"vendor": str(data.get("vendor", "")).strip(),
|
||||
"total_amount": float(data.get("total_amount", 0)),
|
||||
"tax_amount": float(data.get("tax_amount", 0)),
|
||||
"date": data.get("date", ""),
|
||||
"category": data.get("category", "Other"),
|
||||
"date": str(data.get("date", "")).strip(),
|
||||
"category": str(data.get("category", "Other")).strip(),
|
||||
"confidence": float(data.get("confidence", 0.5)),
|
||||
"extraction_success": True
|
||||
}
|
||||
else:
|
||||
return {"error": "Could not parse JSON from AI response"}
|
||||
# Try to extract fields from plain text
|
||||
logger.warning("No JSON found in response, attempting text extraction")
|
||||
return self._extract_from_plain_text(result_text)
|
||||
|
||||
except Exception as e:
|
||||
return {"error": f"JSON parsing error: {str(e)}"}
|
||||
logger.error(f"JSON parsing error: {str(e)}")
|
||||
return {"error": f"JSON parsing error: {str(e)}", "extraction_success": False}
|
||||
|
||||
def _extract_from_plain_text(self, text: str) -> Dict[str, Any]:
|
||||
"""Extract receipt data from plain text when JSON parsing fails"""
|
||||
try:
|
||||
import re
|
||||
|
||||
# Extract vendor (look for common patterns)
|
||||
vendor_patterns = [
|
||||
r'(?:vendor|store|merchant|company)\s*[:\-]?\s*([A-Za-z0-9\s&.,]+)',
|
||||
r'([A-Z][A-Za-z0-9\s&.,]{3,30})', # Capitalized words
|
||||
]
|
||||
|
||||
vendor = ""
|
||||
for pattern in vendor_patterns:
|
||||
match = re.search(pattern, text, re.IGNORECASE)
|
||||
if match:
|
||||
vendor = match.group(1).strip()
|
||||
break
|
||||
|
||||
# Extract amount (look for currency patterns)
|
||||
amount_patterns = [
|
||||
r'\$?\s*([0-9,]+\.?[0-9]*)',
|
||||
r'(?:total|amount|sum)\s*[:\-]?\s*\$?\s*([0-9,]+\.?[0-9]*)',
|
||||
]
|
||||
|
||||
total_amount = 0.0
|
||||
for pattern in amount_patterns:
|
||||
match = re.search(pattern, text, re.IGNORECASE)
|
||||
if match:
|
||||
try:
|
||||
total_amount = float(match.group(1).replace(',', ''))
|
||||
break
|
||||
except ValueError:
|
||||
continue
|
||||
|
||||
# Extract date
|
||||
date_patterns = [
|
||||
r'(\d{4}-\d{2}-\d{2})',
|
||||
r'(\d{1,2}/\d{1,2}/\d{2,4})',
|
||||
r'(Jan|Feb|Mar|Apr|May|Jun|Jul|Aug|Sep|Oct|Nov|Dec)\s+\d{1,2},?\s+\d{4}',
|
||||
]
|
||||
|
||||
date = ""
|
||||
for pattern in date_patterns:
|
||||
match = re.search(pattern, text, re.IGNORECASE)
|
||||
if match:
|
||||
date = match.group(0)
|
||||
break
|
||||
|
||||
return {
|
||||
"vendor": vendor or "Unknown",
|
||||
"total_amount": total_amount,
|
||||
"tax_amount": 0.0,
|
||||
"date": date or "",
|
||||
"category": "Other",
|
||||
"confidence": 0.3, # Low confidence for text extraction
|
||||
"extraction_success": True
|
||||
}
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"Text extraction error: {str(e)}")
|
||||
return {
|
||||
"vendor": "Unknown",
|
||||
"total_amount": 0.0,
|
||||
"tax_amount": 0.0,
|
||||
"date": "",
|
||||
"category": "Other",
|
||||
"confidence": 0.1,
|
||||
"extraction_success": False,
|
||||
"error": f"Text extraction failed: {str(e)}"
|
||||
}
|
||||
|
||||
async def save_uploaded_file(self, file_content: bytes, filename: str) -> str:
|
||||
"""Save uploaded file to temporary storage"""
|
||||
@@ -287,19 +390,37 @@ class DocumentProcessor:
|
||||
import json
|
||||
import re
|
||||
|
||||
# Find JSON in response
|
||||
json_match = re.search(r'\{.*\}', result_text, re.DOTALL)
|
||||
if json_match:
|
||||
json_str = json_match.group()
|
||||
# Find the first '{' and last '}'
|
||||
start = result_text.find('{')
|
||||
end = result_text.rfind('}')
|
||||
if start == -1 or end == -1 or end <= start:
|
||||
return {
|
||||
"extraction_success": False,
|
||||
"error": "Could not find JSON object in AI response",
|
||||
"transactions": []
|
||||
}
|
||||
json_str = result_text[start:end+1]
|
||||
|
||||
# Remove trailing commas before } or ]
|
||||
json_str = re.sub(r',\s*([}\]])', r'\1', json_str)
|
||||
|
||||
try:
|
||||
data = json.loads(json_str)
|
||||
except Exception as e:
|
||||
import logging
|
||||
logging.error(f"JSON parsing error: {str(e)}")
|
||||
logging.error(f"Offending JSON string:\n{json_str}")
|
||||
return {
|
||||
"extraction_success": False,
|
||||
"error": f"JSON parsing error: {str(e)}",
|
||||
"transactions": []
|
||||
}
|
||||
|
||||
# Validate and clean data
|
||||
transactions = data.get("transactions", [])
|
||||
cleaned_transactions = []
|
||||
|
||||
for txn in transactions:
|
||||
try:
|
||||
# Clean and validate each transaction
|
||||
cleaned_txn = {
|
||||
"date": str(txn.get("date", "")).strip(),
|
||||
"amount": float(str(txn.get("amount", 0)).replace('$', '').replace(',', '')),
|
||||
@@ -308,22 +429,15 @@ class DocumentProcessor:
|
||||
}
|
||||
cleaned_transactions.append(cleaned_txn)
|
||||
except Exception as e:
|
||||
# Skip invalid transactions
|
||||
continue
|
||||
|
||||
return {
|
||||
"extraction_success": data.get("extraction_success", True),
|
||||
"transactions": cleaned_transactions,
|
||||
"total_transactions": len(cleaned_transactions)
|
||||
}
|
||||
else:
|
||||
return {
|
||||
"extraction_success": False,
|
||||
"error": "Could not parse JSON from AI response",
|
||||
"transactions": []
|
||||
}
|
||||
|
||||
except Exception as e:
|
||||
import logging
|
||||
logging.error(f"JSON parsing error (outer): {str(e)}")
|
||||
return {
|
||||
"extraction_success": False,
|
||||
"error": f"JSON parsing error: {str(e)}",
|
||||
|
||||
@@ -1,49 +0,0 @@
|
||||
import json
|
||||
import requests
|
||||
import csv
|
||||
from dateutil import parser
|
||||
|
||||
# Prepare transactions
|
||||
transactions = []
|
||||
with open("chequing statement.csv", newline="") as f:
|
||||
reader = csv.DictReader(f)
|
||||
idx = 1
|
||||
for row in reader:
|
||||
try:
|
||||
txn_id = f"{row['Account Number']}_{idx}"
|
||||
txn_date = parser.parse(row["Transaction Date"]).isoformat()
|
||||
amount = float(row["Amount"].replace(",", "").strip())
|
||||
vendor = row["Description 2"].strip()
|
||||
notes = f"{row['Account Type']} {row['Cheque Number']} {row['Description 1']}".strip()
|
||||
transactions.append({
|
||||
"id": txn_id,
|
||||
"transaction_date": txn_date,
|
||||
"amount": amount,
|
||||
"vendor": vendor,
|
||||
"notes": notes
|
||||
})
|
||||
idx += 1
|
||||
except Exception as e:
|
||||
continue
|
||||
|
||||
# Receipt data for Ajai Invoice (3).jpg
|
||||
receipt = {
|
||||
"id": "33754868-bff5-4caf-9ece-cfd63f4e52d9",
|
||||
"file_name": "Ajai Invoice (3).jpg",
|
||||
"upload_date": "2025-07-02T15:31:23.641315",
|
||||
"receipt_date": "2025-02-07T00:00:00",
|
||||
"amount": 1412.5,
|
||||
"tax": 162.5,
|
||||
"vendor": "Ajai Srivastava CPA, Accounting Services & Taxes",
|
||||
"category": "Office"
|
||||
}
|
||||
|
||||
# Build request
|
||||
data = {
|
||||
"receipts": [receipt],
|
||||
"transactions": transactions
|
||||
}
|
||||
|
||||
# Post to /match
|
||||
response = requests.post("http://localhost:8000/match", json=data)
|
||||
print(json.dumps(response.json(), indent=2))
|
||||
Reference in New Issue
Block a user