This challenge tests your ability to build a highly scalable, multi-event ticket purchasing microservice that can handle extremely high concurrency. You will design a system that can process purchases for multiple events concurrently while ensuring each ticket is sold only once, even under millions of requests per second. In addition, you'll implement robust error handling, fallback mechanisms, and real-time metrics reporting.

Task Details

Your task is to extract the high-throughput ticket purchasing component and extend it into a full-featured multi-event system. The service will:

Multi-Event Support: Pre-seed a Redis store with multiple sets of tickets. Each event should have its own ticket pool (e.g., stored with keys like event:{eventId}:tickets).
API Endpoints:
- POST /buy/:eventId: Allow a user to purchase a ticket for a specific event.
- GET /metrics: Expose real-time metrics (e.g., tickets sold, available tickets per event) in a format compatible with Prometheus.
Atomic Ticket Purchase: Implement the purchase logic using a Redis Lua script to atomically verify and pop a ticket from the ticket pool. This ensures no duplicate ticket sales even under massive concurrent access.
Fallback Mechanism: If Redis is unavailable or fails during a purchase operation, gracefully fallback to an in-memory store (with appropriate warnings and logs), ensuring the system remains responsive (for demonstration purposes only, as in-memory stores are not persistent).
Performance and Load Testing: The service must be designed to handle tens of thousands of requests and be tested under a simulated load of at least 5000 concurrent connections. You should include logging of key performance metrics and purchase statistics.
Design Documentation: Provide a detailed design document (design.md) that explains your architectural decisions, how you ensure scalability, measures to handle potential bottlenecks, and details on your fallback strategy.
Dockerization: Extend the docker-compose setup to include not only Redis but also (optionally) a Prometheus container to scrape and monitor the metrics from your service.
PDF Ticket Generation: For every successful ticket purchase, a PDF receipt must be generated. You are free to choose any open-source PDF generation package of your choice (e.g., pdfkit, jsPDF, etc.) to implement this functionality. Ensure that PDF generation is integrated into the purchase flow without significant performance degradation.

Requirements

Technology Stack: Use Node.js and Express for the API, and Redis for persistent ticket management.
Multi-Event Architecture: Design the system to handle multiple events concurrently. Pre-seed each event with a configurable number of tickets.
Atomic Operations with Lua: Replace simple atomic operations (like LPOP) with a Redis Lua script that handles the ticket purchase process atomically.
Fallback to In-Memory Store: Implement a fallback mechanism that activates if Redis operations fail, ensuring continued functionality with clear logging that this is a non-persistent backup.
Metrics Endpoint: Provide a /metrics endpoint that returns JSON data with real-time statistics (tickets sold, tickets remaining per event, errors, etc.).
Robust Testing: Write comprehensive unit tests and integration tests. The integration tests must simulate high load (>=5000 concurrent requests) and prove that no ticket is sold more than once.
Logging: Implement detailed logging for purchase operations, errors, and fallback activations.
Design Document: Include a design.md file that outlines your architecture, scalability considerations, and design rationale.
Docker Support: Update the docker-compose file to run Redis and optionally Prometheus. Provide clear instructions for running the entire stack.

Deliverables

Complete codebase with all source files and scripts.
An updated README.md with setup, running instructions, and performance testing guidelines (this file).
A design.md document explaining your design choices, scalability strategies, and fallback mechanisms.
Docker-compose file supporting all necessary services (Redis, your app, and optionally Prometheus for metrics scraping).

Getting Started

Prerequisites

Node.js (v18+ recommended)
npm or yarn
Docker and Docker Compose
Git

Environment Variables

The following environment variables can be configured in your .env file:

Variable	Default	Description
`PORT`	`3049`	Server port number
`NODE_ENV`	`development`	Environment mode
`REDIS_URL`	`redis://localhost:6379`	Redis connection string
`LOG_LEVEL`	`info`	Logging level (error, warn, info, debug)
`LOG_FILE`	`logs/app.log`	Log file path
`PDF_OUTPUT_DIR`	`tickets`	Directory for generated PDF tickets
`PDF_CLEANUP_MAX_AGE_HOURS`	`24`	Maximum age for PDF cleanup
`TEST_URL`	`http://localhost:3049`	Base URL for load testing
`ALLOWED_ORIGINS`	`localhost:3000,3049`	CORS allowed origins
`RATE_LIMIT_ENABLED`	`true`	Enable rate limiting
`SECURITY_HEADERS_ENABLED`	`true`	Enable security headers
`REDIS_SCAN_BATCH_SIZE`	`100`	Redis SCAN batch size for performance

Quick Start

Clone the repository

git clone <repository-url>
cd module4_backend_project

Install dependencies
```
npm install
```
Set up environment
```
cp env.example .env
# Edit .env file if needed
```
Note

: The env.example file contains default configuration values. Copy it to .env and modify as needed for your environment.

Important

: Create the logs directory if you want to use file logging: mkdir logs
Start with Docker (Recommended)
```
# Start core services (Redis + App)
docker-compose up -d

# Or start with monitoring (Prometheus + Grafana)
docker-compose --profile monitoring up -d
```
Note

: For Docker deployment, you can also set environment variables directly in docker-compose.yml or use the .env file for local development.

Seed the database

# Seed 5 events with 10,000 tickets each
npm run seed

# Custom seeding: 3 events with 5,000 tickets each
npm run seed 3 5000

Manual Setup (Development)

If you prefer to run components separately:

Create necessary directories
```
mkdir -p logs tickets
```
Start Redis
```
docker-compose up -d redis
```

Start the application

npm run dev  # Development with auto-reload
# or
npm start    # Production mode

API Endpoints

Once running, the following endpoints are available:

Method	Endpoint	Description
GET	`/health`	System health check
GET	`/events`	List all events with statistics
GET	`/events/:eventId`	Get specific event details
POST	`/buy/:eventId`	Purchase a ticket for an event
GET	`/tickets/:purchaseId`	Download ticket PDF
GET	`/metrics`	Real-time system metrics
GET	`/admin/pdf-stats`	PDF management statistics
POST	`/admin/cleanup-tickets`	Cleanup old ticket files
POST	`/admin/seed-fallback`	Manually seed fallback store from Redis

Load Testing

The system includes a comprehensive load testing framework:

# Run full test suite (5000+ concurrent connections)
npm run test:load -- --full

# Test specific event
npm run test:load -- --event 1 --connections 5000 --duration 30

# Multi-event concurrent testing
npm run test:load -- --multi --events 1,2,3 --connections 6000

# Custom load test
node tests/load-test.js --event 2 --connections 1000 --duration 10

# Test fallback store functionality
npm run test:fallback

# Test security features
npm run test:security

# Run comprehensive test suite
npm test

# Run specific test categories
npm run test:unit          # Unit tests only
npm run test:integration   # Integration tests only
npm run test:performance   # Performance tests only

# Run critical duplicate prevention tests
npm run test:duplicate-prevention

# Run with coverage report
npm run test:coverage

# Run tests in watch mode (development)
npm run test:watch

### Monitoring & Metrics

#### Application Metrics
Access real-time service metrics at: http://localhost:3049/metrics

#### Prometheus (if enabled)
Prometheus dashboard: http://localhost:9090

#### Grafana (if enabled)
Grafana dashboard: http://localhost:3000
- Username: `admin`
- Password: `admin`

### Fallback Store Management

The system includes a robust fallback mechanism that automatically activates when Redis is unavailable:

- **Automatic Seeding**: The fallback store is automatically seeded during server startup and when activated
- **Data Synchronization**: When Redis becomes available again, the fallback store can be manually synced
- **Manual Seeding**: Use `/admin/seed-fallback` to manually populate the fallback store from Redis data

```bash
# Manually seed fallback store from Redis
curl -X POST http://localhost:3049/admin/seed-fallback

# Check fallback store status
curl http://localhost:3049/health

Docker Commands

# Start core services
docker-compose up -d

# Start with monitoring
docker-compose --profile monitoring up -d

# View logs
docker-compose logs -f app

# Stop services
docker-compose down

# Rebuild and restart
docker-compose up -d --build

Evaluation Criteria

Correctness: The system must ensure that no ticket is sold more than once per event even under extreme load.
Scalability: The design should be able to handle high concurrency and large volumes of requests, with proven integration tests demonstrating the capability.
Atomicity & Resilience: Use of Redis Lua scripting for atomic operations, with a robust fallback mechanism to handle failures gracefully.
Code Quality and Testing: Write clean, well-documented code; include comprehensive unit and integration tests.
Logging & Metrics: Proper logging of operations and a functional metrics endpoint suitable for Prometheus scraping.
Design Rationale: The design document (design.md) should clearly articulate your architectural decisions, potential bottlenecks, and design solutions.

Testing Suite

The project includes a comprehensive testing framework to ensure reliability and prevent critical issues:

Test Categories

Unit Tests (tests/unit/): Test individual components in isolation
Integration Tests (tests/integration/): Test component interactions and API endpoints
Performance Tests (tests/performance/): Verify system behavior under high load

Critical Test Coverage

Duplicate Prevention: Automated verification that no ticket is sold more than once
High Concurrency: Tests with 100+ concurrent requests to ensure data integrity
Fallback Mode: Comprehensive testing of Redis failure scenarios
API Endpoints: Full coverage of all REST endpoints with edge case handling
Security Features: Validation of rate limiting, input validation, and security headers

Running Tests

# Run all tests
npm test

# Run specific test categories
npm run test:unit          # Unit tests only
npm run test:integration   # Integration tests only
npm run test:performance   # Performance tests only

# Run critical duplicate prevention tests
npm run test:duplicate-prevention

# Generate coverage report
npm run test:coverage

# Run tests in watch mode (development)
npm run test:watch

# Use the test runner script for easier test execution
node run-tests.js all           # Run all tests
node run-tests.js validate      # Run core requirement validation
node run-tests.js duplicate     # Run duplicate prevention tests only
node run-tests.js quick         # Run quick test suite

Test Requirements

No Duplicate Tickets: Core requirement verified by automated tests
High Concurrency: System tested with 100+ concurrent requests
Data Consistency: Redis and fallback store synchronization verified
Performance: Response times and memory usage monitored under load
Security: All security features validated with comprehensive tests

Security Features

The system includes comprehensive security measures to protect against common threats:

Rate Limiting

General API: 100 requests per 15 minutes
Purchase Endpoints: 10 requests per minute
Admin Endpoints: 20 requests per 5 minutes

Input Validation

Event IDs: Must be positive integers
Purchase IDs: Must be valid UUIDs
Request Parameters: Validated and sanitized

Security Headers

Content Security Policy: Prevents XSS attacks
HSTS: Enforces HTTPS connections
XSS Protection: Additional XSS prevention
Frame Guard: Prevents clickjacking

Request Security

Size Limits: Maximum 1MB request size
CORS Protection: Configurable allowed origins
Security Logging: Suspicious request monitoring

Troubleshooting

Common Issues

Redis Connection Failed
- Ensure Redis is running: docker-compose up -d redis
- Check Redis URL in .env file
- Verify Redis port (default: 6379) is not blocked
Port Already in Use
- Change PORT in .env file
- Kill process using the port: lsof -ti:3049 | xargs kill -9
Missing Dependencies
- Run npm install to install all dependencies
- Ensure Node.js version is 18+ (check with node --version)
Permission Denied for Logs/Tickets
- Create directories with proper permissions: mkdir -p logs tickets
- Check file permissions: chmod 755 logs tickets
Fallback Store Not Working
- Ensure Redis is seeded: npm run seed
- Check fallback store status: npm run test:fallback
- Manually seed fallback: curl -X POST http://localhost:3049/admin/seed-fallback

Getting Help

Check application logs in the logs directory
Verify Redis connection: curl http://localhost:3049/health
Test fallback store: npm run test:fallback

Final Challenges

Enhance your docker-compose setup to include a Prometheus container for live monitoring.
Optimize your Redis Lua script for even higher performance and lower latency.
Integrate a distributed tracing solution (e.g., Jaeger) to track purchase request flows.

Good luck and happy coding!