Aherobo Ovie Victor
80ee708348
- Added `extract_listings.py` for extracting stock listings from TSX, TSXV, CSE, and CBOE using Playwright. - Created `main.py` to orchestrate the entire stock intelligence system, including extraction, database import, financial scraping, news scraping, and report generation. - Developed `populate_database.py` to populate the database with existing JSON data. - Introduced `scrape_nasdaq_tsx_only.py` for focused scraping of NASDAQ and TSX stocks. - Added `setup.py` for initial setup and testing of the system. - Created `watchlist.txt` template for user-defined stock tracking. - Generated `final_test_output.txt` to log the results of the test run.
152 lines
5.7 KiB
Plaintext
152 lines
5.7 KiB
Plaintext
✅ Database tables created successfully
|
|
|
|
======================================================================
|
|
PRODUCTION-READY STOCK INTELLIGENCE SYSTEM
|
|
======================================================================
|
|
Started: 2025-11-06 03:15:22
|
|
⚠️ RUNNING IN TEST MODE
|
|
======================================================================
|
|
|
|
======================================================================
|
|
STEP 1: EXTRACTING STOCK LISTINGS
|
|
======================================================================
|
|
📂 Loading existing listings from data/listings/all_listings_combined.json
|
|
✅ Loaded 3 stocks from file
|
|
|
|
======================================================================
|
|
STEP 2: IMPORTING TO DATABASE
|
|
======================================================================
|
|
|
|
📥 Importing listings from data/listings/all_listings_combined.json...
|
|
✅ Imported 0 stocks
|
|
|
|
⚠️ Limited to 2 stocks for testing
|
|
|
|
======================================================================
|
|
STEP 3: SCRAPING FINANCIAL DATA
|
|
======================================================================
|
|
============================================================
|
|
YAHOO FINANCE SCRAPING
|
|
============================================================
|
|
|
|
🔍 Scraping SHOP.TO...
|
|
✅ SHOP.TO data scraped successfully
|
|
|
|
🔍 Scraping AAPL...
|
|
✅ AAPL data scraped successfully
|
|
|
|
============================================================
|
|
✅ Successfully scraped: 2
|
|
❌ Failed: 0
|
|
📁 Data saved to: data/financials/
|
|
============================================================
|
|
|
|
======================================================================
|
|
STEP 4: CALCULATING FINANCIAL METRICS
|
|
======================================================================
|
|
Calculating metrics for SHOP.TO...
|
|
Calculating metrics for AAPL...
|
|
✅ Calculated metrics for 2 stocks
|
|
|
|
======================================================================
|
|
STEP 5: SCRAPING NEWS & PRESS RELEASES
|
|
======================================================================
|
|
📡 Using SerpAPI for robust news collection...
|
|
======================================================================
|
|
SERPAPI NEWS & PRESS RELEASE SCRAPER
|
|
======================================================================
|
|
|
|
🔍 Fetching news & PR via SerpAPI for SHOP.TO - Shopify Inc.
|
|
Searching Google News via SerpAPI: "Shopify Inc." OR "SHOP.TO" stock earnings financial...
|
|
Found 55 articles
|
|
Searching press release sites via SerpAPI...
|
|
Found 10 press releases
|
|
✅ Saved SerpAPI data: 55 news, 10 PR
|
|
|
|
🔍 Fetching news & PR via SerpAPI for AAPL - Apple Inc.
|
|
Searching Google News via SerpAPI: "Apple Inc." OR "AAPL" stock earnings financial...
|
|
Found 55 articles
|
|
Searching press release sites via SerpAPI...
|
|
Found 10 press releases
|
|
✅ Saved SerpAPI data: 55 news, 10 PR
|
|
|
|
✅ Completed scraping 2 stocks via SerpAPI
|
|
|
|
======================================================================
|
|
STEP 6: SCRAPING SEC EDGAR FILINGS
|
|
======================================================================
|
|
|
|
🔍 Scraping SEC filings for AAPL...
|
|
Found CIK: 0000320193
|
|
Found 100 recent filings
|
|
Scraping 10-K from 2025-10-31...
|
|
Scraping 8-K from 2025-10-30...
|
|
Scraping 10-Q from 2025-08-01...
|
|
Scraping DEF 14A from 2025-01-10...
|
|
Getting ownership data...
|
|
✅ Saved SEC data to data/sec_filings/AAPL_sec_filings.json
|
|
|
|
======================================================================
|
|
STEP 7: SCRAPING SEDAR+ FILINGS
|
|
======================================================================
|
|
======================================================================
|
|
SEDAR+ SCRAPER
|
|
======================================================================
|
|
|
|
======================================================================
|
|
SCRAPING SEDAR+ FOR: SHOP.TO - Shopify Inc.
|
|
======================================================================
|
|
|
|
🔍 Searching SEDAR+ for Shopify Inc. (SHOP.TO)...
|
|
Saved search results to data/sedar_filings/SHOP.TO_sedar_search.html
|
|
✅ Found 0 potential filings
|
|
✅ Saved SEDAR+ data to data/sedar_filings/SHOP.TO_sedar_data.json
|
|
|
|
✅ Completed scraping 1 companies
|
|
|
|
======================================================================
|
|
STEP 8: GENERATING REPORTS
|
|
======================================================================
|
|
✅ Generated 23 comprehensive reports
|
|
|
|
======================================================================
|
|
STEP 9: EXPORTING TO CSV
|
|
======================================================================
|
|
|
|
======================================================================
|
|
CSV EXPORT - ALL DATA
|
|
======================================================================
|
|
|
|
📤 Exporting stock list to data/exports/stocks_export.csv...
|
|
✅ Exported 23 stocks to CSV
|
|
|
|
📤 Exporting detailed financials to data/exports/stocks_detailed.csv...
|
|
⚠️ No financial metrics found in database
|
|
|
|
📤 Exporting news summary to data/exports/news_summary.csv...
|
|
✅ Exported 0 news articles
|
|
|
|
📤 Exporting filings summary to data/exports/filings_summary.csv...
|
|
✅ Exported 0 filings
|
|
|
|
======================================================================
|
|
✅ Created 3 CSV files:
|
|
- data/exports/stocks_export.csv
|
|
- data/exports/news_summary.csv
|
|
- data/exports/filings_summary.csv
|
|
======================================================================
|
|
|
|
======================================================================
|
|
FINAL STATISTICS
|
|
======================================================================
|
|
Duration: 0:02:10.036167
|
|
Stocks processed: 3
|
|
Financials scraped: 2
|
|
Metrics calculated: 2
|
|
News articles collected: 2
|
|
Filings scraped: 2
|
|
Errors: 0
|
|
======================================================================
|
|
|
|
✅ PIPELINE COMPLETED SUCCESSFULLY!
|