module4_backend_project/README.md

# Ticket Scaling and Multi-Event Ticketing Microservice Challenge

## Overview

This challenge tests your ability to build a highly scalable, multi-event ticket purchasing microservice that can handle extremely high concurrency. You will design a system that can process purchases for multiple events concurrently while ensuring each ticket is sold only once, even under millions of requests per second. In addition, you'll implement robust error handling, fallback mechanisms, and real-time metrics reporting.

## Task Details

Your task is to extract the high-throughput ticket purchasing component and extend it into a full-featured multi-event system. The service will:

- **Multi-Event Support:** Pre-seed a Redis store with multiple sets of tickets. Each event should have its own ticket pool (e.g., stored with keys like `event:{eventId}:tickets`).
- **API Endpoints:**
  - `POST /buy/:eventId`: Allow a user to purchase a ticket for a specific event.
  - `GET /metrics`: Expose real-time metrics (e.g., tickets sold, available tickets per event) in a format compatible with Prometheus.
- **Atomic Ticket Purchase:** Implement the purchase logic using a Redis Lua script to atomically verify and pop a ticket from the ticket pool. This ensures no duplicate ticket sales even under massive concurrent access.
- **Fallback Mechanism:** If Redis is unavailable or fails during a purchase operation, gracefully fallback to an in-memory store (with appropriate warnings and logs), ensuring the system remains responsive (for demonstration purposes only, as in-memory stores are not persistent).
- **Performance and Load Testing:** The service must be designed to handle tens of thousands of requests and be tested under a simulated load of at least 5000 concurrent connections. You should include logging of key performance metrics and purchase statistics.
- **Design Documentation:** Provide a detailed design document (`design.md`) that explains your architectural decisions, how you ensure scalability, measures to handle potential bottlenecks, and details on your fallback strategy.
- **Dockerization:** Extend the docker-compose setup to include not only Redis but also (optionally) a Prometheus container to scrape and monitor the metrics from your service.
- **PDF Ticket Generation:** For every successful ticket purchase, a PDF receipt must be generated. You are free to choose any open-source PDF generation package of your choice (e.g., pdfkit, jsPDF, etc.) to implement this functionality. Ensure that PDF generation is integrated into the purchase flow without significant performance degradation.

## Requirements

1. **Technology Stack:** Use Node.js and Express for the API, and Redis for persistent ticket management.
2. **Multi-Event Architecture:** Design the system to handle multiple events concurrently. Pre-seed each event with a configurable number of tickets.
3. **Atomic Operations with Lua:** Replace simple atomic operations (like LPOP) with a Redis Lua script that handles the ticket purchase process atomically.
4. **Fallback to In-Memory Store:** Implement a fallback mechanism that activates if Redis operations fail, ensuring continued functionality with clear logging that this is a non-persistent backup.
5. **Metrics Endpoint:** Provide a `/metrics` endpoint that returns JSON data with real-time statistics (tickets sold, tickets remaining per event, errors, etc.).
6. **Robust Testing:** Write comprehensive unit tests and integration tests. The integration tests must simulate high load (>=5000 concurrent requests) and prove that no ticket is sold more than once.
7. **Logging:** Implement detailed logging for purchase operations, errors, and fallback activations.
8. **Design Document:** Include a `design.md` file that outlines your architecture, scalability considerations, and design rationale.
9. **Docker Support:** Update the docker-compose file to run Redis and optionally Prometheus. Provide clear instructions for running the entire stack.

## Deliverables

- Complete codebase with all source files and scripts.
- An updated `README.md` with setup, running instructions, and performance testing guidelines (this file).
- A `design.md` document explaining your design choices, scalability strategies, and fallback mechanisms.
- Docker-compose file supporting all necessary services (Redis, your app, and optionally Prometheus for metrics scraping).

## Getting Started

### Prerequisites

- Node.js (v18+ recommended)
- npm or yarn
- Docker and Docker Compose
- Git

### Environment Variables

The following environment variables can be configured in your `.env` file:

| Variable                    | Default                  | Description                              |
| --------------------------- | ------------------------ | ---------------------------------------- |
| `PORT`                      | `3049`                   | Server port number                       |
| `NODE_ENV`                  | `development`            | Environment mode                         |
| `REDIS_URL`                 | `redis://localhost:6379` | Redis connection string                  |
| `LOG_LEVEL`                 | `info`                   | Logging level (error, warn, info, debug) |
| `LOG_FILE`                  | `logs/app.log`           | Log file path                            |
| `PDF_OUTPUT_DIR`            | `tickets`                | Directory for generated PDF tickets      |
| `PDF_CLEANUP_MAX_AGE_HOURS` | `24`                     | Maximum age for PDF cleanup              |
| `TEST_URL`                  | `http://localhost:3049`  | Base URL for load testing                |

### Quick Start

1. **Clone the repository**

   ```bash
   git clone <repository-url>
   cd module4_backend_project
   ```

2. **Install dependencies**

   ```bash
   npm install
   ```

3. **Set up environment**

   ```bash
   cp env.example .env
   # Edit .env file if needed
   ```

   > **Note**: The `env.example` file contains default configuration values. Copy it to `.env` and modify as needed for your environment.

   > **Important**: Create the `logs` directory if you want to use file logging: `mkdir logs`

4. **Start with Docker (Recommended)**

   ```bash
   # Start core services (Redis + App)
   docker-compose up -d

   # Or start with monitoring (Prometheus + Grafana)
   docker-compose --profile monitoring up -d
   ```

   > **Note**: For Docker deployment, you can also set environment variables directly in `docker-compose.yml` or use the `.env` file for local development.

5. **Seed the database**

   ```bash
   # Seed 5 events with 10,000 tickets each
   npm run seed

   # Custom seeding: 3 events with 5,000 tickets each
   npm run seed 3 5000
   ```

### Manual Setup (Development)

If you prefer to run components separately:

1. **Create necessary directories**

   ```bash
   mkdir -p logs tickets
   ```

2. **Start Redis**

   ```bash
   docker-compose up -d redis
   ```

3. **Start the application**
   ```bash
   npm run dev  # Development with auto-reload
   # or
   npm start    # Production mode
   ```

### API Endpoints

Once running, the following endpoints are available:

| Method | Endpoint                 | Description                             |
| ------ | ------------------------ | --------------------------------------- |
| GET    | `/health`                | System health check                     |
| GET    | `/events`                | List all events with statistics         |
| GET    | `/events/:eventId`       | Get specific event details              |
| POST   | `/buy/:eventId`          | Purchase a ticket for an event          |
| GET    | `/tickets/:purchaseId`   | Download ticket PDF                     |
| GET    | `/metrics`               | Real-time system metrics                |
| GET    | `/admin/pdf-stats`       | PDF management statistics               |
| POST   | `/admin/cleanup-tickets` | Cleanup old ticket files                |
| POST   | `/admin/seed-fallback`   | Manually seed fallback store from Redis |

### Load Testing

The system includes a comprehensive load testing framework:

````bash
# Run full test suite (5000+ concurrent connections)
npm run test:load -- --full

# Test specific event
npm run test:load -- --event 1 --connections 5000 --duration 30

# Multi-event concurrent testing
npm run test:load -- --multi --events 1,2,3 --connections 6000

# Custom load test
node tests/load-test.js --event 2 --connections 1000 --duration 10

# Test fallback store functionality
npm run test:fallback

### Monitoring & Metrics

#### Application Metrics
Access real-time service metrics at: http://localhost:3049/metrics

#### Prometheus (if enabled)
Prometheus dashboard: http://localhost:9090

#### Grafana (if enabled)
Grafana dashboard: http://localhost:3000
- Username: `admin`
- Password: `admin`

### Fallback Store Management

The system includes a robust fallback mechanism that automatically activates when Redis is unavailable:

- **Automatic Seeding**: The fallback store is automatically seeded during server startup and when activated
- **Data Synchronization**: When Redis becomes available again, the fallback store can be manually synced
- **Manual Seeding**: Use `/admin/seed-fallback` to manually populate the fallback store from Redis data

```bash
# Manually seed fallback store from Redis
curl -X POST http://localhost:3049/admin/seed-fallback

# Check fallback store status
curl http://localhost:3049/health
````

### Docker Commands

```bash
# Start core services
docker-compose up -d

# Start with monitoring
docker-compose --profile monitoring up -d

# View logs
docker-compose logs -f app

# Stop services
docker-compose down

# Rebuild and restart
docker-compose up -d --build
```

## Evaluation Criteria

- **Correctness:** The system must ensure that no ticket is sold more than once per event even under extreme load.
- **Scalability:** The design should be able to handle high concurrency and large volumes of requests, with proven integration tests demonstrating the capability.
- **Atomicity & Resilience:** Use of Redis Lua scripting for atomic operations, with a robust fallback mechanism to handle failures gracefully.
- **Code Quality and Testing:** Write clean, well-documented code; include comprehensive unit and integration tests.
- **Logging & Metrics:** Proper logging of operations and a functional metrics endpoint suitable for Prometheus scraping.
- **Design Rationale:** The design document (`design.md`) should clearly articulate your architectural decisions, potential bottlenecks, and design solutions.

## Troubleshooting

### Common Issues

1. **Redis Connection Failed**

   - Ensure Redis is running: `docker-compose up -d redis`
   - Check Redis URL in `.env` file
   - Verify Redis port (default: 6379) is not blocked

2. **Port Already in Use**

   - Change `PORT` in `.env` file
   - Kill process using the port: `lsof -ti:3049 | xargs kill -9`

3. **Missing Dependencies**

   - Run `npm install` to install all dependencies
   - Ensure Node.js version is 18+ (check with `node --version`)

4. **Permission Denied for Logs/Tickets**

   - Create directories with proper permissions: `mkdir -p logs tickets`
   - Check file permissions: `chmod 755 logs tickets`

5. **Fallback Store Not Working**
   - Ensure Redis is seeded: `npm run seed`
   - Check fallback store status: `npm run test:fallback`
   - Manually seed fallback: `curl -X POST http://localhost:3049/admin/seed-fallback`

### Getting Help

- Check application logs in the `logs` directory
- Verify Redis connection: `curl http://localhost:3049/health`
- Test fallback store: `npm run test:fallback`

## Final Challenges

- Enhance your docker-compose setup to include a Prometheus container for live monitoring.
- Optimize your Redis Lua script for even higher performance and lower latency.
- Integrate a distributed tracing solution (e.g., Jaeger) to track purchase request flows.

Good luck and happy coding!