feat: Implement database ingestion for investors and companies

- Added main ingestion logic in main.py to process CSV files for investors and companies.
- Implemented data cleaning functions for names, strings, integers, and websites.
- Established relationships between investors, companies, and sectors using SQLAlchemy ORM.
- Created models for investors, companies, sectors, and their relationships in models.py.
- Set up logging for error tracking during data processing.
- Initialized database and created necessary tables.
This commit is contained in:
bolade
2025-10-07 20:01:19 +01:00
parent a9589e54f3
commit 84e3c7b72a
32 changed files with 4 additions and 33994 deletions
Binary file not shown.
Binary file not shown.
+3 -3
View File
@@ -12,9 +12,9 @@ Base = declarative_base()
# Database configuration
# Use the preprocessor's database for consistency
# Get absolute path to the preprocessor database
APP_DIR = Path(__file__).parent.parent
PREPROCESSOR_DB = APP_DIR.parent / "preprocessor" / "version_two.db"
DATABASE_URL = os.getenv("DATABASE_URL", f"sqlite:///{PREPROCESSOR_DB}")
# APP_DIR = Path(__file__).parent.parent
# PREPROCESSOR_DB = APP_DIR.parent / "preprocessor" / "version_two.db"
DATABASE_URL = os.getenv("DATABASE_URL", "sqlite:///./version_two.db")
# Create engine
engine = create_engine(DATABASE_URL, echo=False)