Real-Time Fraud Detection System
Scoring payments in under 100 ms with streaming features, rules, ML models and graph signals, plus case management, label feedback loops and model governance.
[!ASSUMPTION] Scope A hypothetical payments company "PayGuard" that processes card and wallet payments for merchants and wants to cut fraud losses while keeping good customers flowing. Numbers are assumptions.
1. Business problem and objectives
Fraud losses are 0.18% of payment volume and rising; the current batch rules engine flags fraud hours later and declines too many good customers (false positives cost more in lost sales than fraud itself). Objectives:
- Score every transaction in real time (p99 < 100 ms added latency).
- Reduce fraud loss rate by 40% while reducing false-positive declines by 30%.
- Let analysts deploy new rules in minutes without code releases.
- Explain every decline for customer service, merchants and regulators.
2. Business model and revenue generation
PayGuard earns a percentage per transaction. Fraud losses, chargeback fees and lost good transactions directly hit margin. A risk product ("fraud protection as a service" for merchants) is an additional revenue line priced per scored transaction or as a share of protected volume.
3. Functional requirements
- Real-time decision API: approve, decline, or challenge (step-up such as 3-D Secure or OTP).
- Streaming features: velocity (transactions per card per hour), amount anomalies, device and IP reputation, geo-velocity.
- Rules engine editable by analysts with simulation/backtesting.
- ML model scoring with explanations (top contributing features).
- Graph signals: linked accounts, shared devices, mule networks.
- Case management for manual review queues.
- Feedback: chargebacks, confirmed fraud, customer disputes as labels.
- Model lifecycle: training, validation, shadow deployment, monitoring.
- Merchant and regulator reporting.
4. Non-functional requirements
| Attribute | Target |
|---|---|
| Latency | Decision p99 < 100 ms, p50 < 25 ms |
| Availability | 99.99%; fail-safe default decision if scoring is unavailable |
| Throughput | 30K decisions/s peak |
| Freshness | Velocity features reflect events within 1 s |
| Explainability | Reason codes for every decision; audit of model and rule versions used |
| Privacy | PII minimized in features; data retention policies |
5. Expected user base and traffic assumptions
- 2 B transactions/month ≈ 770 TPS average, 30K TPS peak (holiday sales).
- 400 M cards/accounts, 3 M merchants.
- Fraud rate ~0.1–0.3% of transactions; labels arrive 30–90 days later (chargebacks).
- Feature store: ~1 B entity keys with ~200 features each for online lookup.
6. High-level architecture
Two paths:
- Online decision path: payment service calls the decision API → fetch online features → evaluate rules → score model → combine into decision with reason codes. Must meet the latency budget.
- Streaming path: payment events → stream processor (Flink) → update velocity aggregates in the online store within a second → also write to lakehouse.
- Offline path: lakehouse + labels → feature engineering → model training → validation → registry → deployment (shadow, then champion/challenger).
7. Detailed component architecture
| Component | Responsibility | Choice and reason |
|---|---|---|
| Decision API | Orchestrates scoring within budget | Stateless Java/Go service, parallel feature fetches, strict timeouts |
| Online feature store | Low-latency feature lookups by card, device, merchant, IP | Redis/Aerospike/DynamoDB or managed feature store online tier; single-digit ms |
| Stream processor | Sliding-window aggregates, sessionization | Apache Flink with keyed state, exactly-once checkpoints |
| Rules engine | Analyst-authored rules (e.g. "amount > 3× 30-day avg AND new device") | Rules compiled to an in-memory evaluator; versioned; hot reload |
| Model server | Gradient-boosted trees (XGBoost/LightGBM) for tabular data; optional deep models | In-process ONNX/treelite inference for < 5 ms, or Triton/SageMaker/Vertex endpoint |
| Graph service | Entity linking (devices, emails, cards, addresses) | Graph DB (Neptune/Neo4j/TigerGraph) for offline community detection; precomputed risk scores pushed to feature store |
| Case management | Review queues, decisions, notes | Workflow app with SLA tracking |
| Label service | Joins chargebacks/disputes to transactions | Batch + streaming; handles delayed and noisy labels |
| ML platform | Training pipelines, registry, monitoring | SageMaker/Vertex AI/Azure ML or Kubeflow/MLflow |
8. Architecture diagram
flowchart LR PAY[Payment service] -->|score request| DEC[Decision API] DEC --> FS[(Online feature store)] DEC --> RULES[Rules engine<br/>in-memory] DEC --> MODEL[Model scoring<br/>in-process ONNX] DEC -->|approve / decline / challenge<br/>+ reason codes| PAY DEC -->|decision event| K[[Kafka]] PAY -->|payment events| K K --> FL[Flink: velocity & session features] FL --> FS K --> LAKE[(Lakehouse)] CB[Chargebacks & disputes] --> LBL[Label service] --> LAKE LAKE --> TRAIN[Training pipeline] TRAIN --> REG[Model registry] REG -->|shadow → champion| MODEL LAKE --> GRAPH[Graph analytics<br/>community detection] GRAPH -->|entity risk scores| FS DEC -->|review cases| CASE[Case management] CASE -->|analyst labels| LBL
9. Frontend architecture
- Analyst console (React): rule authoring with syntax help, backtest on last 90 days before activation, rule performance dashboards (hit rate, precision, false-positive impact).
- Case management UI: queue with priority, transaction timeline, linked-entity graph view, one-click actions, keyboard shortcuts for throughput.
- Merchant portal: risk settings within guardrails, decline reason explanations, dispute evidence upload.
10. Backend architecture
- Decision API with a latency budget: feature fetch 10 ms (parallel, batched by entity), rules 2 ms, model 5 ms, overhead 5 ms.
- Timeouts and graceful degradation: if a feature group is unavailable, the model uses missing-value handling and rules use defaults; if the decision service itself fails, the payment service applies a static fallback policy (approve low-risk under a threshold, challenge others).
- Champion/challenger: challenger models score in shadow; results logged but not used.
- All inputs, feature values, model version and rule versions logged per decision for audit and reproducibility.
11. API design with sample endpoints
POST /v1/decisions
{
"transactionId": "tx_01JD8...",
"amount": { "minor": 129900, "currency": "USD" },
"card": { "fingerprint": "cf_9a2..." , "bin": "411111" },
"merchant": { "id": "m_771", "mcc": "5732" },
"device": { "id": "dv_3k...", "ip": "203.0.113.7", "userAgentHash": "..." },
"customer": { "id": "c_88", "accountAgeDays": 4 },
"channel": "ECOM"
}
-> 200
{
"decision": "CHALLENGE",
"score": 0.83,
"reasons": ["NEW_DEVICE", "AMOUNT_ABOVE_PROFILE", "HIGH_RISK_IP"],
"modelVersion": "gbm-2026-11-03",
"rulesetVersion": "rs-412",
"latencyMs": 21
}
POST /v1/labels { "transactionId": "...", "label": "FRAUD", "source": "CHARGEBACK", "reasonCode": "10.4" }
POST /v1/rules { "name": "...", "expression": "...", "action": "CHALLENGE", "mode": "SHADOW" }
POST /v1/rules/{id}/backtests { "from": "2026-08-01", "to": "2026-10-31" }12. Database schema and data model
Feature keys and examples:
| Entity | Example features |
|---|---|
| card | txn_count_1h, txn_count_24h, amount_sum_24h, distinct_merchants_24h, avg_amount_30d, last_country |
| device | cards_seen_7d, accounts_seen_30d, emulator_flag, first_seen_at |
| merchant | chargeback_rate_90d, refund_rate_30d, mcc_risk |
| ip | geo, asn_risk, txn_count_1h, distinct_cards_1h |
| customer | account_age_days, address_changes_30d, graph_risk_score |
Decision log (append-only, lakehouse): transaction_id, timestamp, all feature values used, rule hits, model score, decision, versions. Labels table: transaction_id, label, source, label_date, confidence.
13. Database technology selection and reasoning
| Need | Choice | Reason |
|---|---|---|
| Online features | Redis Cluster / Aerospike / DynamoDB | Millisecond reads at 30K TPS × several keys |
| Streaming state | Flink keyed state (RocksDB) | Exactly-once sliding windows |
| Decision log and training data | Lakehouse (Iceberg/Delta) on object storage | Cheap, time-travel for point-in-time correct training |
| Graph | Neptune/Neo4j/TigerGraph (offline) | Multi-hop entity traversal; online use via precomputed scores |
| Rules and case data | PostgreSQL | Transactions, workflows |
Point-in-time correctness is critical: training features must be computed as they were at the decision time, not today, or the model learns from future data (label leakage).
14. Authentication and authorization
- Service-to-service: mTLS, workload identity; only the payment service may call the decision API.
- Analysts: SSO + MFA; RBAC (rule author, rule approver, reviewer, model owner); four-eyes approval to activate blocking rules.
- Merchant portal: OAuth/OIDC; merchants see only their own data.
15. Security architecture
- PII tokenized before entering features (card fingerprints, hashed emails).
- Strict access to decision logs; audit of who viewed which case.
- Model artifacts signed; registry with approval gates.
- Adversarial considerations: fraudsters probe rules, so avoid leaking exact reasons to end users; rate limit card-testing patterns.
16. Caching strategy
- Static reference data (BIN tables, MCC risk, country risk) in process memory with versioned refresh.
- Model and rules loaded in memory; hot swap on version change.
- No caching of velocity features beyond the online store (freshness is the point).
17. Messaging and event-driven architecture
- Payment authorization events, decisions, refunds, chargebacks on Kafka.
- Flink jobs keyed by card, device, IP, merchant compute aggregates and write to the online store.
- Decision events feed monitoring (approval rate by segment) and the lakehouse.
18. Cloud deployment architecture
- Decision service co-located in the same regions/AZs as the payment service to minimize network latency.
- Multi-AZ online store with replicas; Flink with checkpoints to object storage.
- Separate ML account/project for training with access to de-identified data only.
19. CI/CD and DevOps
- Code: standard pipelines with contract tests against payment service.
- Rules: rule-as-config pipeline: author → backtest → shadow → approval → active, with automatic rollback if approval rate shifts beyond threshold.
- Models (MLOps): reproducible training pipeline, evaluation gates (AUC/precision at fixed recall, fairness checks), shadow period, champion/challenger with traffic split, registry audit trail.
20. Monitoring and observability
- Technical: latency per stage, timeout counts, feature fetch errors.
- Business: approval rate, decline rate, challenge rate, fraud basis points, false-positive rate (from customer contacts and later labels), per merchant/segment.
- Model: feature drift (PSI), score distribution shift, performance on newly arrived labels.
- Alert on sudden approval-rate drops (a bad rule can decline millions).
21. Scalability and performance optimization
- Batch multiple feature lookups into one round trip (pipelining/MGET).
- In-process model inference for tree models; reserve remote model servers for heavy models used only on risky subsets (two-stage scoring).
- Shard the online store by entity key; avoid hot keys (very large merchants) by pre-aggregating.
- Flink parallelism scaled with Kafka partitions.
22. High availability and disaster recovery
- Decision service active-active across AZs and regions with payment traffic.
- Online feature store replicated; on loss, features can be rebuilt by replaying Kafka (retention ≥ longest window, e.g. 30 days) or from lakehouse snapshots.
- Documented fallback policy when the whole fraud system is unavailable, approved by risk leadership.
23. Cost optimization
- Tree models in-process are cheap; GPU only for training deep or graph models.
- Tiered feature computation: expensive features only for transactions above a risk/amount threshold.
- Lakehouse storage lifecycle; sample approved low-risk transactions for training rather than keeping full feature vectors forever.
24. Architecture trade-offs and alternatives
| Decision | Chosen | Alternative | Consider when |
|---|---|---|---|
| Build vs buy | Build core, buy device fingerprinting/consortium data | Buy fraud platform (Featurespace, Sardine, Stripe Radar for Stripe merchants, cloud fraud services) | Small team, need fast coverage |
| Model type | Gradient-boosted trees | Deep learning / GNNs | Massive data and sequence patterns; higher cost and latency |
| Graph online | Precomputed scores | Real-time graph queries | When ring detection must react within seconds |
| Fail mode | Fallback policy | Fail closed (decline all) | Extremely high-risk segments only |
25. Technology stack recommendations
Kafka, Flink, Redis/Aerospike, Iceberg lakehouse with Spark/Trino, XGBoost/LightGBM with ONNX runtime, MLflow or SageMaker/Vertex AI pipelines, Neo4j/Neptune, PostgreSQL, React analyst console, OpenTelemetry, Evidently/WhyLabs or cloud model monitoring.
26. Common architecture mistakes
- Training on features computed with future information (leakage), producing great offline metrics and poor production results.
- Optimizing only fraud catch rate and ignoring false positives.
- Letting rules accumulate without performance review (hundreds of overlapping rules).
- No shadow mode or backtesting before activating rules.
- Synchronous calls to slow external enrichment services on the critical path.
- No explanation/reason codes, making disputes and regulator questions painful.
27. Senior architect interview questions and answers
How do you keep the decision under 100 ms?
Budget each step, fetch features in parallel and batched, keep rules and tree models in-process, avoid synchronous external calls (precompute enrichment asynchronously into the feature store), set hard timeouts with defaults, and co-locate with the payment service.
How do you handle labels that arrive 60 days later?
Train on data old enough to have mature labels, weight or exclude immature periods, use proxy labels (confirmed fraud reports, account takeovers) for faster feedback, and monitor recent model performance with early signals like analyst decisions.
How do you prevent a bad rule from declining everyone?
Shadow mode and backtesting before activation, staged rollout (percentage of traffic), approval workflow, and an automated guardrail that disables a rule if approval rate or rule hit rate deviates from expectations.
28. Enterprise architect interview questions and answers
How do you govern ML models used for customer decisions?
Model inventory with owners and risk tiers, documented model cards, validation by an independent team (model risk management, e.g. SR 11-7 style practices in banking), fairness and explainability checks, monitoring and periodic revalidation, and audit trails linking each decision to model and data versions.
Should fraud be a shared enterprise capability?
Yes for signals and platform (feature store, device intelligence, case management) to reuse across cards, wallets and lending; product-specific models and rules can remain with domain teams.
29. CTO-level business and technology questions
How do you measure ROI of the fraud platform?
Fraud losses avoided + chargeback fees avoided + revenue recovered from fewer false declines + analyst productivity, minus platform and data costs. Track basis points of fraud and approval rate together.
What is the risk of relying on a vendor?
Less control over models and data, consortium data benefits vs privacy, cost per transaction at scale, and integration lock-in. A hybrid (vendor signals + in-house decisioning) balances speed and control.
30. Real-world implementation roadmap
| Phase | Duration | Outcomes |
|---|---|---|
| 1. Real-time rules | 0–3 months | Decision API, streaming velocity features, analyst rules with backtesting |
| 2. First ML model | 3–6 months | Lakehouse, labels pipeline, GBM model in shadow then champion |
| 3. Case management | 4–8 months | Review queues, analyst feedback labels |
| 4. Graph and device | 6–12 months | Device intelligence, entity graph risk scores |
| 5. Optimization | 12+ months | Champion/challenger, per-segment models, adaptive step-up authentication |