Phase 1 — Foundation
Tasks 1–2Local dev environment, monorepo scaffold, and Kubernetes namespaces.
CompleteReal-Time Fraud Detection System
Build an end-to-end real-time fraud detection pipeline using stream processing, ML models, and production monitoring.
Built over two weeks of intensive self-directed work: a production-style real-time fraud detection pipeline from local WSL2 setup through ML training, Kafka streaming, Kubernetes deployment, observability, and GitHub Actions CI/CD. Source code is live on GitHub; modules are published here sequentially.
Project Modules
Follow these build guides in order. Published modules are ready to start.
1
Local Environment Setup & Master Roadmap
WSL2, Docker, Minikube, kubectl, namespaces, and the full monorepo scaffold.
Published
2
Dockerfiles, Requirements & Service Skeletons
Base Docker images and Python service structure for producer, consumer, and dashboard.
Coming Soon
3
Data Exploration & Feature Engineering
Colab notebook: explore transaction data and engineer fraud-detection features.
Coming Soon
4
Model Training, Evaluation & Export
Train XGBoost, evaluate metrics, export model artifacts for inference.
Coming Soon
5
Kafka & Zookeeper on Kubernetes
Deploy Zookeeper and Kafka StatefulSets in the fraud-kafka namespace.
Coming Soon
6
The Data Producer Service
Python service that reads CSV transactions and publishes events to Kafka.
Coming Soon
7
Consumer & Inference Service
Real-time XGBoost inference with Redis-backed feature state.
Coming Soon
8
MongoDB StatefulSet & Schema Design
Persistent fraud-event storage with production schema patterns.
Coming Soon
9
Streamlit Dashboard with Secure Login
Live fraud feed and analytics dashboard for operators.
Coming Soon
10
Prometheus & Grafana Observability
Custom business metrics, dashboards, and alert rules.
Coming Soon
11
GitHub Actions CI/CD Pipeline
Build, test, and push Docker images on every push to main.
Coming Soon
12
Kubernetes Secrets & Rollout Strategy
Secure secrets management and controlled manual rollouts.
Coming Soon
Before You Start
- Comfortable with Python (functions, classes, pip)
- Basic Linux terminal usage
- Conceptual understanding of REST APIs and databases
- Familiarity with Docker containers (helpful but covered in Module 1)
What You'll Learn
- ✓Design real-time stream processing architectures for financial transactions
- ✓Train and deploy XGBoost fraud classifiers with sub-100ms inference latency
- ✓Build Kafka producer/consumer microservices with Redis-backed state
- ✓Deploy multi-namespace Kubernetes stacks on Minikube
- ✓Persist fraud events in MongoDB with production schema patterns
- ✓Ship a Streamlit ops dashboard with secure authentication
- ✓Instrument custom business metrics in Prometheus and Grafana
- ✓Automate Docker builds and pushes via GitHub Actions
Core Skills
Real-time stream processing Advanced
ML model training & serving (XGBoost) Advanced
Kubernetes deployment & namespaces Intermediate
Event-driven architecture (Kafka) Advanced
Observability (Prometheus/Grafana) Intermediate
CI/CD pipeline design Intermediate
Feature engineering for fraud detection Advanced
Production troubleshooting & optimization Advanced
Implementation Roadmap
Phase 2 — Data & ML
Tasks 3–4Feature engineering in Colab and XGBoost model training/export.
CompletePhase 3 — Streaming Pipeline
Tasks 5–7Kafka deployment, producer service, and consumer inference layer.
CompletePhase 4 — Storage & State
Task 8MongoDB StatefulSet and fraud-event persistence schema.
CompletePhase 5 — Frontend
Task 9Streamlit dashboard with authentication and live fraud analytics.
CompletePhase 6 — Observability
Task 10Prometheus metrics, Grafana dashboards, and alert tuning.
CompletePhase 7 — CI/CD & Security
Tasks 11–12GitHub Actions pipeline, Docker Hub pushes, and K8s secrets.
CompleteDeveloper Notes
Troubleshooting & lessons learned
Minikube fails to start
Challenge: minikube start hangs or errors because Docker Desktop is not running on Windows.
Fix: Open Docker Desktop first, verify wsl --status shows VERSION 2, then rerun minikube start --driver=docker with 4 CPUs and 6GB RAM.
Kafka pods stuck in CrashLoopBackOff
Challenge: Zookeeper not ready before Kafka starts, or services deployed in wrong namespace.
Fix: Deploy Zookeeper first, wait for Ready status, then Kafka. Keep Kafka in fraud-kafka namespace and verify bootstrap server DNS matches the service name.
Feature column mismatch at inference
Challenge: Consumer predictor throws KeyError because live events miss engineered features from training.
Fix: Centralize FEATURE_COLUMNS in predictor.py, add pytest tests that assert every column exists after engineer_features(), run tests in CI before deploy.
Prometheus shows decimal fraud counts
Challenge: increase(counter[30s]) returns 0.3 frauds when scrape interval is 60 seconds.
Fix: Use a range window at least 4× the scrape interval. increase() is rate × seconds — coarse windows extrapolate fractional counts.
Alert flapping on threshold breaches
Challenge: Fraud-rate alert fires and resolves every scrape when metric oscillates around threshold.
Fix: Add a for: duration (e.g. 2m) to alert rules so the condition must hold before firing. Configure resolve_timeout in Alertmanager.
Optimization tricks
Allocate enough Minikube resources upfront
Start with --cpus=4 --memory=6144 --disk-size=20g. Running Kafka, Redis, MongoDB, and three app services on 2GB RAM causes OOM kills that are hard to diagnose.
Cache pip layers in GitHub Actions
Use actions/cache on ~/.cache/pip and consider a slim requirements-dev.txt for CI tests instead of installing full xgboost on every lint job.
Partition Kafka by account ID
Hash nameOrig as the partition key so all transactions for one account land on the same consumer — enables accurate rolling-window features without cross-partition state.
Redis for hot inference features
Store rolling aggregates in Redis with TTL instead of recomputing from MongoDB on every event — cuts p99 latency below 100ms.
Separate K8s namespaces by concern
fraud-app, fraud-kafka, and fraud-monitoring namespaces mirror how platform teams isolate workloads — makes RBAC and blast-radius containment easier later.