A production-ready distributed log processing system built with Spring Boot, Apache Kafka, Redis, and PostgreSQL. This system demonstrates key distributed system patterns including event-driven architecture, circuit breakers, distributed caching, and comprehensive observability.

The system consists of three main services:

• API Gateway (Port 8080): Routes requests, handles rate limiting, and provides unified API access
• Log Producer (Port 8081): REST API that accepts log events and publishes them to Kafka
• Log Consumer (Port 8082): Kafka consumer that processes events and stores them in PostgreSQL

• Apache Kafka: Message streaming platform for event-driven architecture
• Redis: Distributed caching and rate limiting
• PostgreSQL: Persistent storage for processed log events
• Prometheus: Metrics collection and monitoring
• Grafana: Metrics visualization and dashboards

• Java 17+
• Maven 3.6+
• Docker and Docker Compose

If you haven't already, run the generation script:

chmod +x generate_system_files.sh
./generate_system_files.sh
cd distributed-log-processor

./setup.sh

This will start all infrastructure services (Kafka, Redis, PostgreSQL, Prometheus, Grafana) and create necessary topics.

Open three separate terminals and run:

# Terminal 1: Start Log Producer
cd log-producer
mvn spring-boot:run

# Terminal 2: Start Log Consumer  
cd log-consumer
mvn spring-boot:run

# Terminal 3: Start API Gateway
cd api-gateway
mvn spring-boot:run

# Run integration tests
./integration-tests/system-integration-test.sh

# Run load tests
./load-test.sh

• Grafana Dashboard: http://localhost:3000 (admin/admin)
• Prometheus: http://localhost:9090
• API Gateway Health: http://localhost:8080/actuator/health
• Producer Health: http://localhost:8081/actuator/health
• Consumer Health: http://localhost:8082/actuator/health

• log_events_received_total: Total log events received by producer
• log_events_processed_total: Total log events processed by consumer
• log_events_errors_total: Total processing errors
• log_event_processing_time: Time to process individual events

curl -X POST http://localhost:8080/api/logs \
  -H "Content-Type: application/json" \
  -d '{
    "organizationId": "my-org",
    "level": "INFO",
    "message": "Application started successfully",
    "source": "my-service"
  }'

{
  "status": "success",
  "id": "uuid-generated-id",
  "message": "Log event queued for processing"
}

• Pattern: Decoupled services communicate via events
• Implementation: Kafka topics with persistent messaging
• Benefits: High availability, natural backpressure, replay capability

• Pattern: Prevents cascade failures in distributed systems
• Implementation: Resilience4j circuit breakers
• Benefits: System stability, graceful degradation, automatic recovery

• Pattern: Single entry point for client requests
• Implementation: Spring Cloud Gateway with reactive routing
• Benefits: Centralized cross-cutting concerns, simplified client logic

• Pattern: Application manages cache alongside database
• Implementation: Redis with Spring Cache
• Benefits: Improved performance, reduced database load

• Pattern: Multi-level health verification
• Implementation: Spring Boot Actuator with custom health indicators
• Benefits: Early failure detection, automatic recovery

• Throughput: 1,000+ events/second per producer instance
• Latency: 99th percentile under 100ms
• Scalability: Horizontal scaling via Kafka partitions

1. Producer Scaling: Add more producer instances behind load balancer
2. Consumer Scaling: Increase consumer instances (max = partition count)
3. Database Scaling: Implement read replicas and connection pooling
4. Cache Scaling: Redis cluster for high availability

• Retry Logic: Exponential backoff with max 3 attempts
• Dead Letter Queue: Failed messages routed to error topic
• Circuit Breakers: Automatic failure isolation
• Health Checks: Proactive failure detection

• Input validation on all API endpoints
• Rate limiting to prevent abuse
• Health check endpoints exposed for monitoring
• Database credentials externalized via environment variables

./integration-tests/system-integration-test.sh

Tests include:

• Service health verification
• End-to-end log processing
• Metrics endpoint availability

./load-test.sh

Generates 100 concurrent log events to test system under load.

# Submit various log levels
curl -X POST http://localhost:8080/api/logs \
  -H "Content-Type: application/json" \
  -d '{"organizationId": "test", "level": "ERROR", "message": "Test error", "source": "manual-test"}'

mvn clean install

mvn test

Key configuration files:

• application.yml: Service-specific configuration
• docker-compose.yml: Infrastructure setup
• prometheus.yml: Metrics collection configuration

1. Infrastructure services (Kafka, Redis, PostgreSQL)
2. Log Consumer (to create consumer group)
3. Log Producer (to handle incoming requests)
4. API Gateway (to route external traffic)

1. API Gateway (stop accepting new requests)
2. Log Producer (finish processing current requests)
3. Log Consumer (finish processing current messages)
4. Infrastructure services

This system architecture patterns are used by major tech companies:

• Netflix: Event-driven microservices with Kafka
• Uber: Real-time data processing with similar patterns
• Amazon: Distributed systems with circuit breakers and health checks

1. Authentication: Implement JWT or OAuth2
2. Authorization: Role-based access control
3. SSL/TLS: Encrypt all network communication
4. Monitoring: Enhanced alerting and dashboards
5. Deployment: Kubernetes orchestration
6. Backup: Automated backup strategies

This is a learning project demonstrating distributed system patterns. Focus areas for enhancement:

• Additional consumer patterns (batch processing)
• Advanced monitoring (distributed tracing)
• Performance optimizations (connection pooling)
• Security enhancements (authentication)

• Apache Kafka Documentation
• Spring Boot Reference
• Resilience4j Documentation
• Microservices Patterns

Architecture Insight: This system demonstrates that scalable distributed systems are built on predictable failure rather than perfect reliability. Every component assumes others will fail and provides mechanisms for graceful degradation and automatic recovery.