94 lines
2.9 KiB
Markdown
94 lines
2.9 KiB
Markdown
# DS Task AI News
|
|
|
|
## Project Overview
|
|
|
|
DS Task AI News is an AI-powered news retrieval system that gathers news articles from various online sources, stores them in a vector database, and enables users to discover relevant articles based on their interests. The system uses advanced AI techniques to find and recommend related news articles dynamically.
|
|
|
|
## Features
|
|
|
|
* **News Aggregation** : Fetches news using RSS feeds from various online portals.
|
|
* **Vector Database Storage** : Stores news articles in a vector database for efficient similarity searches.
|
|
* **AI-powered Recommendations** : Uses Cohere embeddings and re-ranking to provide relevant news recommendations.
|
|
* **LLM-powered Analysis** : Utilizes Groq for AI-driven insights and processing.
|
|
|
|
## Tech Stack
|
|
|
|
* **LLM** : Groq
|
|
* **Search** : RSS Feeds for news aggregation
|
|
* **Embeddings & Re-Ranking** : Cohere
|
|
* **Vector Database** : (e.g., Pinecone, Weaviate, or FAISS)
|
|
* **Backend** : FastAPI
|
|
|
|
## File Structure
|
|
|
|
```
|
|
DS_Task_AI_News/
|
|
│-- backend/
|
|
│ │-- main.py # FastAPI backend
|
|
│ │-- news_fetcher.py # Fetches news using RSS feeds
|
|
│ │-- vector_store.py # Handles vector database operations
|
|
│ │-- embeddings.py # Generates embeddings using Cohere
|
|
│ │-- recommender.py # Fetches related news articles
|
|
│ │-- config.py # Configuration settings
|
|
│ │-- requirements.txt # Dependencies
|
|
│
|
|
│-- data/
|
|
│ │-- raw_news/ # Stores raw news articles before processing
|
|
│ │-- processed_news/ # Stores cleaned and processed articles
|
|
│
|
|
│-- docs/
|
|
│ │-- README.md # Documentation for new developers
|
|
│ │-- API_Documentation.md # API details
|
|
│
|
|
│-- .env # Environment variables
|
|
│-- .gitignore # Git ignore file
|
|
│-- LICENSE # License information
|
|
```
|
|
|
|
## Setup & Installation
|
|
|
|
### 1. Clone the Repository
|
|
|
|
```bash
|
|
git clone http://23.29.118.76:3000/Test/ds_task_ai_news
|
|
cd ds-task-ai-news
|
|
```
|
|
|
|
### 2. Set Up the Backend
|
|
|
|
```bash
|
|
cd backend
|
|
pip install -r requirements.txt
|
|
python main.py
|
|
```
|
|
|
|
## Fetching News Using RSS Feeds
|
|
|
|
* News is aggregated from RSS feeds of different news sources.
|
|
* The `news_fetcher.py` script pulls data from RSS feeds, extracts relevant information, and stores it in the database.
|
|
|
|
### **Example RSS Fetching Code (Python)**
|
|
|
|
```python
|
|
import feedparser
|
|
|
|
def fetch_rss_news(feed_url):
|
|
feed = feedparser.parse(feed_url)
|
|
articles = []
|
|
for entry in feed.entries:
|
|
articles.append({
|
|
"title": entry.title,
|
|
"content": entry.summary,
|
|
"date": entry.published,
|
|
"slug": entry.title.lower().replace(" ", "-"),
|
|
"categories": ["Technology", "AI and Innovation"],
|
|
"tags": ["AI", "Technology", "Innovation"]
|
|
})
|
|
return articles
|
|
```
|
|
|
|
## API Endpoints
|
|
|
|
* `GET /fetch-news`: Fetches news from RSS feeds.
|
|
* `GET /recommend-news?article_id=xyz`: Retrieves similar news based on the selected article.
|