About the project

A production-style, real-time fraud detection system built end to end — from streaming ingestion to an ML scorer to a live dashboard, deployed on Kubernetes with GitOps.

The problem

Credit-card fraud is rare and highly imbalanced — under 0.2% of transactions in the public Kaggle Credit Card Fraud dataset are fraudulent. Rather than train on scarce labels, this system uses an unsupervised anomaly detector (Isolation Forest) that learns what "normal" spending looks like and flags outliers in real time.

How it works

End-to-end flow

Producer→ Kafka→ Scoring (ML)→ Query API→ Postgres + SSE Dashboard

A producer replays held-out transactions into a Kafka topic. The scoring service consumes them, runs the model, and publishes risk scores to a second topic. The query API (FastAPI) persists results to Postgres and pushes aggregated metrics to the dashboard over Server-Sent Events every 200 ms — live rate, latency p95, risk distribution and fraud flags.

The services

ProducerReplays the held-out test set onto transactions.raw at a configurable rate; exposes a small HTTP control endpoint to pause/resume the stream

ScoringConsumes transactions.raw + a priority transactions.manual topic, scores each transaction with the Isolation Forest, publishes to transactions.scored; exports Prometheus metrics on its own port

Query APIFastAPI service — consumes scored transactions into Postgres, serves the REST endpoints + SSE stream, exposes its own /metrics for request rate/latency/errors, and hosts the static landing/demo pages

The model

AlgorithmIsolation Forest (scikit-learn), unsupervised anomaly detection

PreprocessingRobustScaler on Time/Amount, fit on normal-only rows to avoid label leakage

ThresholdTuned for F2 (recall-weighted) with a minimum-precision floor

ServingReplays only the held-out test split, so the model scores unseen transactions

Architecture & deployment

StreamingKafka (KRaft) — raw, priority-manual, and scored topics

ServicesPython 3.13 · FastAPI · confluent-kafka, packaged as containers

OrchestrationSingle-node k3s (Kubernetes) on a cloud VM

DeliveryGitHub Actions builds & pushes to GHCR, then commits the new image tag into the Helm values in Git; ArgoCD auto-syncs from there (GitOps)

PackagingOne generic Helm chart, per-service values

InfrastructureTerraform manages the VM, its static IP, and the SSH firewall rule — imported from a hand-created instance, terraform plan stays clean

ObservabilityPrometheus scrapes app-level metrics; Grafana dashboards for throughput, scoring latency, fraud rate, and API request/error rates

TLScert-manager + Let's Encrypt, Traefik ingress

▶ Monitor dashboard Source ↗