feat: Implement database ingestion for investors and companies
- Added main ingestion logic in main.py to process CSV files for investors and companies. - Implemented data cleaning functions for names, strings, integers, and websites. - Established relationships between investors, companies, and sectors using SQLAlchemy ORM. - Created models for investors, companies, sectors, and their relationships in models.py. - Set up logging for error tracking during data processing. - Initialized database and created necessary tables.
This commit is contained in:
Binary file not shown.
Binary file not shown.
+3
-3
@@ -12,9 +12,9 @@ Base = declarative_base()
|
||||
# Database configuration
|
||||
# Use the preprocessor's database for consistency
|
||||
# Get absolute path to the preprocessor database
|
||||
APP_DIR = Path(__file__).parent.parent
|
||||
PREPROCESSOR_DB = APP_DIR.parent / "preprocessor" / "version_two.db"
|
||||
DATABASE_URL = os.getenv("DATABASE_URL", f"sqlite:///{PREPROCESSOR_DB}")
|
||||
# APP_DIR = Path(__file__).parent.parent
|
||||
# PREPROCESSOR_DB = APP_DIR.parent / "preprocessor" / "version_two.db"
|
||||
DATABASE_URL = os.getenv("DATABASE_URL", "sqlite:///./version_two.db")
|
||||
|
||||
# Create engine
|
||||
engine = create_engine(DATABASE_URL, echo=False)
|
||||
|
||||
Reference in New Issue
Block a user