UPI-like Real-Time Payment Platform
Design of a national-scale, real-time account-to-account payment switch with PSP apps, bank connectors, exactly-once debit/credit semantics, risk checks, dispute handling and active-active data centers.
[!FACT] How UPI works today (simplified) India's Unified Payments Interface is operated by NPCI. A payer uses a PSP app (third-party app or bank app) linked to a bank account and a virtual payment address (UPI ID). NPCI's central switch routes requests between the payer's PSP, the remitter bank (debit) and the beneficiary bank (credit). UPI processed on the order of 20 billion transactions per month in 2025 (NPCI publishes monthly statistics; check current figures).
[!ASSUMPTION] Scope We design a hypothetical "RTP-X" switch for a new country plus a reference PSP app, inspired by UPI's model. Volumes, timeouts and rules below are design assumptions.
1. Business problem and objectives
A central bank wants instant, interoperable, low-cost digital payments between any bank accounts, 24×7, usable from any app. Objectives:
- Interoperability: any PSP app can pay any bank account through one switch.
- Settlement finality within seconds for the customer; inter-bank net settlement multiple times a day.
- Success rate ≥ 99.5% excluding business declines (insufficient funds, wrong PIN).
- Peak capacity of 50,000 TPS by year 3 with linear scaling.
- Zero tolerance for money loss or double debit; complete audit trail.
2. Business model and revenue generation
| Party | Revenue |
|---|---|
| Switch operator | Per-transaction switching fee (often small, policy-driven), value-added services (mandates/AutoPay, credit lines, international links) |
| PSP apps | Indirect: user acquisition, cross-sell of credit, insurance, merchant services, ads |
| Banks | Interchange where permitted, float, cross-sell |
| Merchants | Lower acceptance cost than cards (policy dependent; P2M often zero-MDR for small values under some regimes) |
Architecture implication: per-transaction cost must be tiny, so the switch must run efficiently on commodity infrastructure at high utilization.
3. Functional requirements
- Register user: bind device + SIM, link bank account, create payment address (alias), set PIN via bank.
- Pay (push): payer initiates payment to an address, account number or QR.
- Collect (pull): payee requests money; payer approves (with anti-fraud limits).
- Address resolution: alias → bank + account (masked to PSP).
- Transaction status check and history.
- Reversals: automatic reversal if debit succeeded but credit failed.
- Disputes and complaints with defined turnaround times.
- Mandates (recurring payments) and merchant QR payments.
- Net settlement between banks via the central bank's RTGS.
4. Non-functional requirements
| Attribute | Target |
|---|---|
| Availability | 99.99% for the switch (≈ 4 min/month) |
| End-to-end latency | p95 < 3 s user-perceived; switch processing p99 < 100 ms excluding bank time |
| Throughput | 20K TPS sustained, 50K TPS peak (festival and salary days) |
| Consistency | Exactly-once business effect per transaction ID; no lost transactions |
| Durability | RPO 0 within metro; RPO ≤ seconds across geo DR |
| RTO | < 5 min metro failover, < 30 min geo DR |
| Security | HSM-based PIN encryption, mutual TLS with all participants, message signing, device binding |
| Compliance | Data localization, audit retention (e.g. 10 years), regulator reporting |
5. Expected user base and traffic assumptions
- 300 M registered users, 120 M monthly active, 500 participating banks, 50 PSPs.
- 5 B transactions/month in year 3 ≈ 1,900 TPS average; daily peaks ~5× (lunch, evening); festival/salary days ~10× → plan for 20K TPS sustained and 50K TPS burst.
- Each transaction involves ~6 to 8 messages (request, debit request/response, credit request/response, status, notifications) → ~300K messages/s at peak.
- Transaction record ~2 KB with audit metadata → 10 TB/month raw, ~360 TB over 3 years before compression and replication.
6. High-level architecture
- PSP layer (many companies): mobile apps + PSP backend that authenticates users, binds devices, and calls the switch.
- Switch (central): API front end, message validation, routing, transaction state machine, timeout management, risk checks, address resolution directory, reversal and dispute engine, settlement engine.
- Bank connectors: banks expose standard debit/credit APIs to the switch (bank side runs a "UPI adapter" in front of the core banking system).
- Data: partitioned strongly consistent transaction store, immutable audit log, analytics lake.
- Two metro data centers active-active plus a geo DR site.
7. Detailed component architecture
| Component | Responsibility | Design choice |
|---|---|---|
| Participant gateway | mTLS, signature verification, schema validation, rate limits per participant | Stateless, horizontally scaled; per-PSP quotas |
| Address directory | Alias → PSP/bank mapping | Replicated in-memory cache backed by a consistent store; changes via events |
| Transaction orchestrator | State machine per txn: INITIATED → DEBIT_PENDING → DEBITED → CREDIT_PENDING → SUCCESS / REVERSAL_PENDING → REVERSED | Partitioned by txn ID; single owner per partition |
| Timer service | Timeouts per state, status checks | Hierarchical timing wheels per partition, persisted deadlines |
| Risk engine | Velocity limits, device risk, mule detection, collect-request fraud | Streaming features + rules + ML; < 30 ms budget |
| Reversal engine | Auto-reversal when credit fails/unknown | Idempotent reversal requests to remitter bank |
| Dispute system | Complaints, chargeback-like flows, TAT tracking | Workflow engine; integrates with banks |
| Settlement engine | Net positions per bank per cycle, files to RTGS | Batch on immutable transaction log |
| Reporting | Regulator and participant reports | Lakehouse + warehouse |
8. Architecture diagram
flowchart LR
subgraph Payer side
U1[Payer app] --> PSP1[Payer PSP backend]
end
subgraph Switch [Central switch: active-active DC1 + DC2]
GW[Participant gateway<br/>mTLS, signatures, quotas]
DIR[Address directory]
ORC[Transaction orchestrator<br/>partitioned state machines]
TMR[Timer / status service]
RISK[Risk engine]
REV[Reversal engine]
LOG[(Transaction store<br/>+ immutable journal)]
SET[Settlement engine]
end
subgraph Banks
RB[Remitter bank adapter] --> CBS1[(Core banking)]
BB[Beneficiary bank adapter] --> CBS2[(Core banking)]
end
PSP2[Payee PSP] --> U2[Payee app]
PSP1 --> GW
GW --> ORC
ORC --> DIR
ORC --> RISK
ORC --> LOG
ORC -- debit --> RB
ORC -- credit --> BB
ORC --> TMR
TMR --> REV
REV --> RB
ORC -- notify --> PSP2
LOG --> SET
SET --> RTGS[(Central bank RTGS)]9. Frontend architecture
- Native Android/iOS apps (low-end Android performance matters); app size budget; offline-tolerant UI with clear pending states.
- Device binding: on registration the app sends an SMS from the user's SIM to verify the mobile number on the device; device ID and keys stored in secure hardware (Android Keystore/iOS Secure Enclave).
- PIN entry via a common library supplied by the switch operator so PSPs never see the PIN: the PIN is encrypted on device with the switch/bank's public key.
- QR scanning, contact lookup, multi-language UI, accessibility (screen readers, large fonts), voice and feature-phone channels (USSD/IVR) as alternative fronts.
- Never show "failed" for an unknown outcome; show "pending, we will update you".
10. Backend architecture
- Switch core in Java 21/25 (or C++/Rust for the hottest path) with an event-loop or virtual-thread model; ISO 20022-style or JSON/XML message schemas.
- Partitioned single-writer design: transactions hashed by transaction ID to partitions; each partition owned by one orchestrator instance at a time (leases via a consensus store). This gives per-transaction serialization without distributed locks.
- State persisted to a replicated log before acknowledgement (write-ahead), then applied to the state store.
- Strict idempotency: the transaction ID is globally unique (PSP prefix + unique number); duplicates return the current state.
- Bank calls asynchronous with correlation IDs; callbacks resume the state machine.
11. API design with sample endpoints
Participant-facing (simplified, JSON over HTTPS with mTLS and detached JWS signatures):
POST /v1/address/resolve { "address": "ravi@bankx" }
POST /v1/pay { "txnId": "PSPA9F3K2...", "payer": {...}, "payee": {...},
"amount": { "value": "250.00", "currency": "INR" },
"credBlock": "<encrypted PIN block>", "deviceInfo": {...}, "note": "lunch" }
POST /v1/collect { "txnId": "...", "payeeAddress": "...", "payerAddress": "...", "amount": {...}, "expiry": "PT15M" }
GET /v1/txn/{txnId}/status
POST /v1/disputes { "txnId": "...", "reasonCode": "CREDIT_NOT_RECEIVED" }Switch → bank:
POST /bank/v1/debit { "txnId": "...", "account": "...", "amount": {...}, "credBlock": "..." }
POST /bank/v1/credit { "txnId": "...", "account": "...", "amount": {...} }
POST /bank/v1/reverse { "txnId": "...", "originalDebitRef": "..." }
GET /bank/v1/status/{txnId}Every response carries a result code (success, business decline with reason, technical failure, deemed/unknown), which drives the state machine.
12. Database schema and data model
CREATE TABLE txn (
txn_id varchar(35) PRIMARY KEY,
partition_id int NOT NULL,
type varchar(10) NOT NULL, -- PAY, COLLECT, REVERSAL
payer_psp varchar(8) NOT NULL,
payee_psp varchar(8) NOT NULL,
remitter_bank varchar(11) NOT NULL,
beneficiary_bank varchar(11) NOT NULL,
amount_minor bigint NOT NULL CHECK (amount_minor > 0),
currency char(3) NOT NULL,
state varchar(20) NOT NULL,
debit_ref varchar(40),
credit_ref varchar(40),
risk_score smallint,
created_at timestamptz NOT NULL,
updated_at timestamptz NOT NULL,
settlement_cycle varchar(12)
);
CREATE TABLE txn_event ( -- append-only journal, source of truth for audit
txn_id varchar(35),
seq int,
event varchar(30) NOT NULL, -- DEBIT_REQUESTED, DEBIT_SUCCESS, CREDIT_TIMEOUT ...
payload jsonb NOT NULL,
at timestamptz NOT NULL,
PRIMARY KEY (txn_id, seq)
);Money stored in minor units (paise/cents) as integers. Address directory: address → (psp_id, bank_ifsc, account_token, status).
13. Database technology selection and reasoning
| Need | Choice | Why | Alternatives |
|---|---|---|---|
| Transaction state, exactly-once | Distributed SQL with synchronous replication across the two metro DCs (e.g. YugabyteDB/CockroachDB/TiDB on-prem, or Spanner in cloud) | Strong consistency, horizontal write scaling, survives a DC loss with RPO 0 | Sharded PostgreSQL with synchronous standby per shard (simpler, more manual) |
| Append-only journal | Kafka (replication factor 3, min.insync.replicas 2, acks=all) + archived to object storage | High-throughput durable log, replay for recovery and analytics | Dedicated ledger DB |
| Directory cache | In-memory replicated cache (Hazelcast/Redis) with consistent store behind | Sub-ms lookups at 50K TPS | Direct DB reads (too slow) |
| Analytics and settlement | Lakehouse (Iceberg) + Spark/Trino | Large batch computations, regulator reports | Data warehouse |
Data localization often requires on-premises or in-country cloud regions; the design avoids proprietary single-cloud services on the critical path.
14. Authentication and authorization
- Participants (PSPs, banks): mutual TLS with certificates issued by the switch's PKI; every message signed (JWS/XML-DSig); per-participant API keys bound to certificates; IP allow-lists.
- End users: authenticated by the PSP (device binding + app PIN/biometric) and authorized by the bank (UPI-style PIN validated by the remitter bank in its HSM). The switch never sees the clear PIN.
- Operators: privileged access via PAM, MFA, just-in-time access, four-eyes approval for production changes.
15. Security architecture
- HSMs for key management, PIN block translation and signing.
- Network segmentation: participant DMZ, switch core, management plane; no internet egress from core.
- DDoS protection and per-participant rate limiting; circuit breakers per bank so one slow bank does not exhaust switch threads.
- Fraud controls: collect-request limits and warnings (collect is a common social-engineering vector), new-device cooling period limits, velocity limits per payer and payee, mule account detection with graph analytics, merchant risk scores.
- Tamper-evident audit logs (hash chaining); WORM storage for long retention.
- Continuous compliance: vulnerability management, red teaming, participant security certification before onboarding.
16. Caching strategy
- Address directory and bank routing tables: in-memory per node, refreshed via change events, versioned so stale entries are detected.
- Bank health (up/degraded/down) cached and updated every few seconds from success rates; used to fail fast and show users "bank unavailable".
- No caching of transaction state outside the owning partition (single source of truth).
17. Messaging and event-driven architecture
- Internally, every state transition is an event in the journal (event-sourced transaction state). The current-state table is a projection, enabling replay after failures.
- Notifications to payee PSPs, SMS and analytics consume journal topics.
- Settlement, reporting and fraud model training consume from the archived log.
stateDiagram-v2 [*] --> INITIATED INITIATED --> REJECTED: validation / risk decline INITIATED --> DEBIT_PENDING DEBIT_PENDING --> DEBIT_FAILED: bank decline DEBIT_PENDING --> DEBIT_UNKNOWN: timeout DEBIT_UNKNOWN --> DEBITED: status check success DEBIT_UNKNOWN --> DEBIT_FAILED: status check failed DEBIT_PENDING --> DEBITED DEBITED --> CREDIT_PENDING CREDIT_PENDING --> SUCCESS CREDIT_PENDING --> CREDIT_UNKNOWN: timeout CREDIT_UNKNOWN --> SUCCESS: status check success CREDIT_UNKNOWN --> DEEMED_SUCCESS: beneficiary confirms later CREDIT_PENDING --> REVERSAL_PENDING: credit failed REVERSAL_PENDING --> REVERSED DEBIT_FAILED --> [*] REJECTED --> [*] SUCCESS --> [*] REVERSED --> [*]
18. Cloud deployment architecture
- Two metro data centers (or two in-country cloud regions/zones within synchronous distance, < 2 ms RTT) running active-active; transaction partitions distributed across both with consensus-based replication.
- A third geo-distant DR site with asynchronous replication for regional disasters.
- Kubernetes for stateless services; stateful stores on dedicated nodes or managed in-country services.
- Participant connectivity via private links/MPLS plus internet with mTLS as backup.
19. CI/CD and DevOps
- Strict change management: automated tests including protocol conformance suites and replay of anonymized production traffic; staged rollout by partition (canary on 1% of partitions).
- Participant sandbox and certification environment with simulators for banks/PSPs.
- Infrastructure as code, immutable images, signed artifacts; two-person approval for production.
- Change freeze around known peaks (salary days, festivals).
20. Monitoring and observability
- Real-time success rate per bank, per PSP, per transaction type, with technical vs business decline split.
- Latency distributions per hop (switch internal, bank debit, bank credit).
- Count of transactions in UNKNOWN/REVERSAL states and their age (critical customer-impact metric).
- Public bank-wise uptime dashboards to push participants to improve.
- Distributed tracing with transaction ID as the correlation key across participants (each logs it).
21. Scalability and performance optimization
- Linear scaling by adding partitions; rebalancing partitions across orchestrator nodes via leases.
- Batched, pipelined writes to the journal; group commit.
- Non-blocking I/O to banks; per-bank bulkheads and adaptive concurrency limits (bank capacity is the true bottleneck).
- Back-pressure to PSPs with clear "try later" codes when a bank is saturated.
- Precomputed risk features in memory per partition owner to keep risk checks < 30 ms.
22. High availability and disaster recovery
| Failure | Behaviour |
|---|---|
| Node failure | Partition leases move to other nodes in seconds; state recovered from replicated store/journal |
| One metro DC down | Remaining DC owns all partitions (capacity planned at 2× per DC); RPO 0 via synchronous replication |
| Bank down | Circuit breaker opens for that bank; users get immediate decline "bank unavailable"; no impact on others |
| Regional disaster | Declared failover to geo DR; replay journal tail; RPO seconds |
| Unknown outcomes | Status-check loops, then reversal workflows with defined TAT; customer refunds automated |
23. Cost optimization
- High utilization via shared stateless tiers; partition density tuned by load tests.
- Commodity hardware and open-source data stores to avoid per-transaction licence costs.
- Tiered storage for the journal (hot 30 days, warm 1 year, cold archive for the remaining retention).
- Cost per million transactions tracked as the key efficiency metric.
24. Architecture trade-offs and alternatives
| Decision | Chosen | Alternative | Reasoning |
|---|---|---|---|
| Central switch | Hub-and-spoke | Bilateral bank-to-bank links | N² integrations do not scale to 500 banks |
| Settlement | Deferred net settlement in cycles | Real-time gross settlement per txn | Liquidity efficiency; customer still gets instant confirmation |
| Unknown credit outcome | Deemed success after confirmation window, else reversal | Synchronous blocking until certain | Avoids holding resources; defined TAT |
| State storage | Event-sourced journal + projection | CRUD state table only | Auditability and recovery |
| PIN validation | At remitter bank | At switch | Banks own customer authentication; reduces switch liability |
25. Technology stack recommendations
Java (switch services) with Netty/virtual threads; Kafka for journal and events; distributed SQL (YugabyteDB/CockroachDB/TiDB) or sharded PostgreSQL; Redis/Hazelcast for directory cache; Flink for streaming risk features; Kubernetes; HSMs (Thales/Utimaco); Prometheus/Grafana and OpenTelemetry; Spark/Trino on Iceberg for settlement and reporting.
26. Common architecture mistakes
- Treating a timeout as a failure and telling the user "failed" while money was debited.
- Retrying debits without idempotent transaction IDs (double debits).
- Global locks or a single database primary for all transactions.
- Letting one slow bank consume shared thread pools (no bulkheads).
- Weak controls on collect requests, enabling social-engineering fraud.
- No reconciliation between switch, remitter and beneficiary records.
- Logging sensitive data (account numbers, device keys) in clear text.
27. Senior architect interview questions and answers
The debit succeeds but the credit to the beneficiary bank times out. Walk through what happens.
The transaction moves to CREDIT_UNKNOWN. The switch performs status checks with the beneficiary bank at increasing intervals. If the bank confirms credit, mark SUCCESS. If it confirms failure, start reversal to the remitter bank (idempotent, keyed by transaction ID). If still unknown after the defined window, follow the scheme rule (deemed success with later reconciliation, or reversal). The user sees "pending" throughout and gets a notification on resolution. Reconciliation files catch any residual mismatch.
How do you achieve exactly-once without distributed transactions across banks?
Banks process requests idempotently keyed by transaction ID. The switch persists each state transition before acting, and resumes after crashes from the journal. Every external call can be retried safely, and compensations (reversals) are themselves idempotent. The combination yields an exactly-once business effect.
How would you scale from 5K to 50K TPS?
Increase partitions and orchestrator nodes, scale the distributed SQL cluster horizontally, batch journal writes, and most importantly work with banks on capacity, because bank adapters and core banking systems usually saturate first. Add per-bank adaptive concurrency limits and publish bank performance.
28. Enterprise architect interview questions and answers
How do you onboard 500 banks with different core banking systems?
Define a strict, versioned interface specification and certification process; provide reference adapters and simulators; require banks to run a standard adapter layer in front of their core; publish conformance test suites; and govern versions with deprecation timelines.
How do you govern scheme rule changes?
Rules (limits, timeouts, dispute TATs) are configuration with effective dates, versioned and approved through a change board with regulator involvement, tested in the participant sandbox and rolled out by partition.
29. CTO-level business and technology questions
How would you justify on-premises vs public cloud?
Data localization, regulatory comfort and very predictable high utilization favour on-prem or sovereign/in-country cloud for the core switch; elastic analytics and fraud model training can run in cloud. Compare 5-year TCO including staff, and design for portability (Kubernetes, open-source stores).
What KPIs would you report to the board?
Transaction volume and value growth, technical decline rate (switch and per bank), p95 latency, uptime, fraud loss rate per million transactions, dispute resolution within TAT, cost per million transactions.
30. Real-world implementation roadmap
| Phase | Duration | Outcomes |
|---|---|---|
| 0. Scheme design | 0–6 months | Rules, message specs, participant agreements, security standards |
| 1. Core switch MVP | 6–12 months | P2P pay with 10 banks and 3 PSPs in pilot; sandbox and certification |
| 2. Scale and merchants | 12–18 months | QR merchant payments, collect with fraud controls, disputes, 100 banks |
| 3. Resilience | 15–24 months | Active-active metro DCs, geo DR, bank performance dashboards |
| 4. Value-added services | 24+ months | Mandates/AutoPay, credit lines on the rail, international links, offline/feature-phone payments |