This challenge tests your ability to build a highly scalable, multi-event ticket purchasing microservice that can handle extremely high concurrency. You will design a system that can process purchases for multiple events concurrently while ensuring each ticket is sold only once, even under millions of requests per second. In addition, you'll implement robust error handling, fallback mechanisms, and real-time metrics reporting.

Task Details

Your task is to extract the high-throughput ticket purchasing component and extend it into a full-featured multi-event system. The service will:

Multi-Event Support: Pre-seed a Redis store with multiple sets of tickets. Each event should have its own ticket pool (e.g., stored with keys like event:{eventId}:tickets).
API Endpoints:
- POST /buy/:eventId: Allow a user to purchase a ticket for a specific event.
- GET /metrics: Expose real-time metrics (e.g., tickets sold, available tickets per event) in a format compatible with Prometheus.
Atomic Ticket Purchase: Implement the purchase logic using a Redis Lua script to atomically verify and pop a ticket from the ticket pool. This ensures no duplicate ticket sales even under massive concurrent access.
Fallback Mechanism: If Redis is unavailable or fails during a purchase operation, gracefully fallback to an in-memory store (with appropriate warnings and logs), ensuring the system remains responsive (for demonstration purposes only, as in-memory stores are not persistent).
Performance and Load Testing: The service must be designed to handle tens of thousands of requests and be tested under a simulated load of at least 5000 concurrent connections. You should include logging of key performance metrics and purchase statistics.
Design Documentation: Provide a detailed design document (design.md) that explains your architectural decisions, how you ensure scalability, measures to handle potential bottlenecks, and details on your fallback strategy.
Dockerization: Extend the docker-compose setup to include not only Redis but also (optionally) a Prometheus container to scrape and monitor the metrics from your service.
PDF Ticket Generation: For every successful ticket purchase, a PDF receipt must be generated. You are free to choose any open-source PDF generation package of your choice (e.g., pdfkit, jsPDF, etc.) to implement this functionality. Ensure that PDF generation is integrated into the purchase flow without significant performance degradation.

Requirements

Technology Stack: Use Node.js and Express for the API, and Redis for persistent ticket management.
Multi-Event Architecture: Design the system to handle multiple events concurrently. Pre-seed each event with a configurable number of tickets.
Atomic Operations with Lua: Replace simple atomic operations (like LPOP) with a Redis Lua script that handles the ticket purchase process atomically.
Fallback to In-Memory Store: Implement a fallback mechanism that activates if Redis operations fail, ensuring continued functionality with clear logging that this is a non-persistent backup.
Metrics Endpoint: Provide a /metrics endpoint that returns JSON data with real-time statistics (tickets sold, tickets remaining per event, errors, etc.).
Robust Testing: Write comprehensive unit tests and integration tests. The integration tests must simulate high load (>=5000 concurrent requests) and prove that no ticket is sold more than once.
Logging: Implement detailed logging for purchase operations, errors, and fallback activations.
Design Document: Include a design.md file that outlines your architecture, scalability considerations, and design rationale.
Docker Support: Update the docker-compose file to run Redis and optionally Prometheus. Provide clear instructions for running the entire stack.

Deliverables

Complete codebase with all source files and scripts.
An updated README.md with setup, running instructions, and performance testing guidelines (this file).
A design.md document explaining your design choices, scalability strategies, and fallback mechanisms.
Docker-compose file supporting all necessary services (Redis, your app, and optionally Prometheus for metrics scraping).

Getting Started

Prerequisites

Node.js (v18+ recommended)
npm or yarn
Docker and Docker Compose
Git

Quick Start

Clone the repository

git clone <repository-url>
cd module4_backend_project

Install dependencies
```
npm install
```

Set up environment

cp .env.example .env
# Edit .env file if needed

Start with Docker (Recommended)

# Start core services (Redis + App)
docker-compose up -d

# Or start with monitoring (Prometheus + Grafana)
docker-compose --profile monitoring up -d

Seed the database

# Seed 5 events with 10,000 tickets each
npm run seed

# Custom seeding: 3 events with 5,000 tickets each
npm run seed 3 5000

Manual Setup (Development)

If you prefer to run components separately:

Start Redis
```
docker-compose up -d redis
```

Start the application

npm run dev  # Development with auto-reload
# or
npm start    # Production mode

API Endpoints

Once running, the following endpoints are available:

Method	Endpoint	Description
GET	`/health`	System health check
GET	`/events`	List all events with statistics
GET	`/events/:eventId`	Get specific event details
POST	`/buy/:eventId`	Purchase a ticket for an event
GET	`/tickets/:purchaseId`	Download ticket PDF
GET	`/metrics`	Real-time system metrics
GET	`/admin/pdf-stats`	PDF management statistics
POST	`/admin/cleanup-tickets`	Cleanup old ticket files

Load Testing

The system includes a comprehensive load testing framework:

# Run full test suite (5000+ concurrent connections)
npm run test:load -- --full

# Test specific event
npm run test:load -- --event 1 --connections 5000 --duration 30

# Multi-event concurrent testing
npm run test:load -- --multi --events 1,2,3 --connections 6000

# Custom load test
node tests/load-test.js --event 2 --connections 1000 --duration 10

Monitoring & Metrics

Application Metrics

Access real-time service metrics at: http://localhost:3049/metrics

Prometheus (if enabled)

Prometheus dashboard: http://localhost:9090

Grafana (if enabled)

Grafana dashboard: http://localhost:3000

Username: admin
Password: admin

Docker Commands

# Start core services
docker-compose up -d

# Start with monitoring
docker-compose --profile monitoring up -d

# View logs
docker-compose logs -f app

# Stop services
docker-compose down

# Rebuild and restart
docker-compose up -d --build

Evaluation Criteria

Correctness: The system must ensure that no ticket is sold more than once per event even under extreme load.
Scalability: The design should be able to handle high concurrency and large volumes of requests, with proven integration tests demonstrating the capability.
Atomicity & Resilience: Use of Redis Lua scripting for atomic operations, with a robust fallback mechanism to handle failures gracefully.
Code Quality and Testing: Write clean, well-documented code; include comprehensive unit and integration tests.
Logging & Metrics: Proper logging of operations and a functional metrics endpoint suitable for Prometheus scraping.
Design Rationale: The design document (design.md) should clearly articulate your architectural decisions, potential bottlenecks, and design solutions.

Final Challenges

Enhance your docker-compose setup to include a Prometheus container for live monitoring.
Optimize your Redis Lua script for even higher performance and lower latency.
Integrate a distributed tracing solution (e.g., Jaeger) to track purchase request flows.

Good luck and happy coding!