UPI-like Real-Time Payment Platform

Design of a national-scale, real-time account-to-account payment switch with PSP apps, bank connectors, exactly-once debit/credit semantics, risk checks, dispute handling and active-active data centers.

[!FACT] How UPI works today (simplified) India's Unified Payments Interface is operated by NPCI. A payer uses a PSP app (third-party app or bank app) linked to a bank account and a virtual payment address (UPI ID). NPCI's central switch routes requests between the payer's PSP, the remitter bank (debit) and the beneficiary bank (credit). UPI processed on the order of 20 billion transactions per month in 2025 (NPCI publishes monthly statistics; check current figures).

[!ASSUMPTION] Scope We design a hypothetical "RTP-X" switch for a new country plus a reference PSP app, inspired by UPI's model. Volumes, timeouts and rules below are design assumptions.

1. Business problem and objectives

A central bank wants instant, interoperable, low-cost digital payments between any bank accounts, 24×7, usable from any app. Objectives:

  • Interoperability: any PSP app can pay any bank account through one switch.
  • Settlement finality within seconds for the customer; inter-bank net settlement multiple times a day.
  • Success rate ≥ 99.5% excluding business declines (insufficient funds, wrong PIN).
  • Peak capacity of 50,000 TPS by year 3 with linear scaling.
  • Zero tolerance for money loss or double debit; complete audit trail.

2. Business model and revenue generation

Party Revenue
Switch operator Per-transaction switching fee (often small, policy-driven), value-added services (mandates/AutoPay, credit lines, international links)
PSP apps Indirect: user acquisition, cross-sell of credit, insurance, merchant services, ads
Banks Interchange where permitted, float, cross-sell
Merchants Lower acceptance cost than cards (policy dependent; P2M often zero-MDR for small values under some regimes)

Architecture implication: per-transaction cost must be tiny, so the switch must run efficiently on commodity infrastructure at high utilization.

3. Functional requirements

  1. Register user: bind device + SIM, link bank account, create payment address (alias), set PIN via bank.
  2. Pay (push): payer initiates payment to an address, account number or QR.
  3. Collect (pull): payee requests money; payer approves (with anti-fraud limits).
  4. Address resolution: alias → bank + account (masked to PSP).
  5. Transaction status check and history.
  6. Reversals: automatic reversal if debit succeeded but credit failed.
  7. Disputes and complaints with defined turnaround times.
  8. Mandates (recurring payments) and merchant QR payments.
  9. Net settlement between banks via the central bank's RTGS.

4. Non-functional requirements

Attribute Target
Availability 99.99% for the switch (≈ 4 min/month)
End-to-end latency p95 < 3 s user-perceived; switch processing p99 < 100 ms excluding bank time
Throughput 20K TPS sustained, 50K TPS peak (festival and salary days)
Consistency Exactly-once business effect per transaction ID; no lost transactions
Durability RPO 0 within metro; RPO ≤ seconds across geo DR
RTO < 5 min metro failover, < 30 min geo DR
Security HSM-based PIN encryption, mutual TLS with all participants, message signing, device binding
Compliance Data localization, audit retention (e.g. 10 years), regulator reporting

5. Expected user base and traffic assumptions

  • 300 M registered users, 120 M monthly active, 500 participating banks, 50 PSPs.
  • 5 B transactions/month in year 3 ≈ 1,900 TPS average; daily peaks ~5× (lunch, evening); festival/salary days ~10× → plan for 20K TPS sustained and 50K TPS burst.
  • Each transaction involves ~6 to 8 messages (request, debit request/response, credit request/response, status, notifications) → ~300K messages/s at peak.
  • Transaction record ~2 KB with audit metadata → 10 TB/month raw, ~360 TB over 3 years before compression and replication.

6. High-level architecture

  • PSP layer (many companies): mobile apps + PSP backend that authenticates users, binds devices, and calls the switch.
  • Switch (central): API front end, message validation, routing, transaction state machine, timeout management, risk checks, address resolution directory, reversal and dispute engine, settlement engine.
  • Bank connectors: banks expose standard debit/credit APIs to the switch (bank side runs a "UPI adapter" in front of the core banking system).
  • Data: partitioned strongly consistent transaction store, immutable audit log, analytics lake.
  • Two metro data centers active-active plus a geo DR site.

7. Detailed component architecture

Component Responsibility Design choice
Participant gateway mTLS, signature verification, schema validation, rate limits per participant Stateless, horizontally scaled; per-PSP quotas
Address directory Alias → PSP/bank mapping Replicated in-memory cache backed by a consistent store; changes via events
Transaction orchestrator State machine per txn: INITIATED → DEBIT_PENDING → DEBITED → CREDIT_PENDING → SUCCESS / REVERSAL_PENDING → REVERSED Partitioned by txn ID; single owner per partition
Timer service Timeouts per state, status checks Hierarchical timing wheels per partition, persisted deadlines
Risk engine Velocity limits, device risk, mule detection, collect-request fraud Streaming features + rules + ML; < 30 ms budget
Reversal engine Auto-reversal when credit fails/unknown Idempotent reversal requests to remitter bank
Dispute system Complaints, chargeback-like flows, TAT tracking Workflow engine; integrates with banks
Settlement engine Net positions per bank per cycle, files to RTGS Batch on immutable transaction log
Reporting Regulator and participant reports Lakehouse + warehouse

8. Architecture diagram

flowchart LR
  subgraph Payer side
    U1[Payer app] --> PSP1[Payer PSP backend]
  end
  subgraph Switch [Central switch: active-active DC1 + DC2]
    GW[Participant gateway<br/>mTLS, signatures, quotas]
    DIR[Address directory]
    ORC[Transaction orchestrator<br/>partitioned state machines]
    TMR[Timer / status service]
    RISK[Risk engine]
    REV[Reversal engine]
    LOG[(Transaction store<br/>+ immutable journal)]
    SET[Settlement engine]
  end
  subgraph Banks
    RB[Remitter bank adapter] --> CBS1[(Core banking)]
    BB[Beneficiary bank adapter] --> CBS2[(Core banking)]
  end
  PSP2[Payee PSP] --> U2[Payee app]
  PSP1 --> GW
  GW --> ORC
  ORC --> DIR
  ORC --> RISK
  ORC --> LOG
  ORC -- debit --> RB
  ORC -- credit --> BB
  ORC --> TMR
  TMR --> REV
  REV --> RB
  ORC -- notify --> PSP2
  LOG --> SET
  SET --> RTGS[(Central bank RTGS)]

9. Frontend architecture

  • Native Android/iOS apps (low-end Android performance matters); app size budget; offline-tolerant UI with clear pending states.
  • Device binding: on registration the app sends an SMS from the user's SIM to verify the mobile number on the device; device ID and keys stored in secure hardware (Android Keystore/iOS Secure Enclave).
  • PIN entry via a common library supplied by the switch operator so PSPs never see the PIN: the PIN is encrypted on device with the switch/bank's public key.
  • QR scanning, contact lookup, multi-language UI, accessibility (screen readers, large fonts), voice and feature-phone channels (USSD/IVR) as alternative fronts.
  • Never show "failed" for an unknown outcome; show "pending, we will update you".

10. Backend architecture

  • Switch core in Java 21/25 (or C++/Rust for the hottest path) with an event-loop or virtual-thread model; ISO 20022-style or JSON/XML message schemas.
  • Partitioned single-writer design: transactions hashed by transaction ID to partitions; each partition owned by one orchestrator instance at a time (leases via a consensus store). This gives per-transaction serialization without distributed locks.
  • State persisted to a replicated log before acknowledgement (write-ahead), then applied to the state store.
  • Strict idempotency: the transaction ID is globally unique (PSP prefix + unique number); duplicates return the current state.
  • Bank calls asynchronous with correlation IDs; callbacks resume the state machine.

11. API design with sample endpoints

Participant-facing (simplified, JSON over HTTPS with mTLS and detached JWS signatures):

POST /v1/address/resolve        { "address": "ravi@bankx" }
POST /v1/pay                    { "txnId": "PSPA9F3K2...", "payer": {...}, "payee": {...},
                                  "amount": { "value": "250.00", "currency": "INR" },
                                  "credBlock": "<encrypted PIN block>", "deviceInfo": {...}, "note": "lunch" }
POST /v1/collect                { "txnId": "...", "payeeAddress": "...", "payerAddress": "...", "amount": {...}, "expiry": "PT15M" }
GET  /v1/txn/{txnId}/status
POST /v1/disputes               { "txnId": "...", "reasonCode": "CREDIT_NOT_RECEIVED" }

Switch → bank:

POST /bank/v1/debit     { "txnId": "...", "account": "...", "amount": {...}, "credBlock": "..." }
POST /bank/v1/credit    { "txnId": "...", "account": "...", "amount": {...} }
POST /bank/v1/reverse   { "txnId": "...", "originalDebitRef": "..." }
GET  /bank/v1/status/{txnId}

Every response carries a result code (success, business decline with reason, technical failure, deemed/unknown), which drives the state machine.

12. Database schema and data model

CREATE TABLE txn (
  txn_id          varchar(35) PRIMARY KEY,
  partition_id    int NOT NULL,
  type            varchar(10) NOT NULL,             -- PAY, COLLECT, REVERSAL
  payer_psp       varchar(8)  NOT NULL,
  payee_psp       varchar(8)  NOT NULL,
  remitter_bank   varchar(11) NOT NULL,
  beneficiary_bank varchar(11) NOT NULL,
  amount_minor    bigint NOT NULL CHECK (amount_minor > 0),
  currency        char(3) NOT NULL,
  state           varchar(20) NOT NULL,
  debit_ref       varchar(40),
  credit_ref      varchar(40),
  risk_score      smallint,
  created_at      timestamptz NOT NULL,
  updated_at      timestamptz NOT NULL,
  settlement_cycle varchar(12)
);

CREATE TABLE txn_event (       -- append-only journal, source of truth for audit
  txn_id   varchar(35),
  seq      int,
  event    varchar(30) NOT NULL,   -- DEBIT_REQUESTED, DEBIT_SUCCESS, CREDIT_TIMEOUT ...
  payload  jsonb NOT NULL,
  at       timestamptz NOT NULL,
  PRIMARY KEY (txn_id, seq)
);

Money stored in minor units (paise/cents) as integers. Address directory: address → (psp_id, bank_ifsc, account_token, status).

13. Database technology selection and reasoning

Need Choice Why Alternatives
Transaction state, exactly-once Distributed SQL with synchronous replication across the two metro DCs (e.g. YugabyteDB/CockroachDB/TiDB on-prem, or Spanner in cloud) Strong consistency, horizontal write scaling, survives a DC loss with RPO 0 Sharded PostgreSQL with synchronous standby per shard (simpler, more manual)
Append-only journal Kafka (replication factor 3, min.insync.replicas 2, acks=all) + archived to object storage High-throughput durable log, replay for recovery and analytics Dedicated ledger DB
Directory cache In-memory replicated cache (Hazelcast/Redis) with consistent store behind Sub-ms lookups at 50K TPS Direct DB reads (too slow)
Analytics and settlement Lakehouse (Iceberg) + Spark/Trino Large batch computations, regulator reports Data warehouse

Data localization often requires on-premises or in-country cloud regions; the design avoids proprietary single-cloud services on the critical path.

14. Authentication and authorization

  • Participants (PSPs, banks): mutual TLS with certificates issued by the switch's PKI; every message signed (JWS/XML-DSig); per-participant API keys bound to certificates; IP allow-lists.
  • End users: authenticated by the PSP (device binding + app PIN/biometric) and authorized by the bank (UPI-style PIN validated by the remitter bank in its HSM). The switch never sees the clear PIN.
  • Operators: privileged access via PAM, MFA, just-in-time access, four-eyes approval for production changes.

15. Security architecture

  • HSMs for key management, PIN block translation and signing.
  • Network segmentation: participant DMZ, switch core, management plane; no internet egress from core.
  • DDoS protection and per-participant rate limiting; circuit breakers per bank so one slow bank does not exhaust switch threads.
  • Fraud controls: collect-request limits and warnings (collect is a common social-engineering vector), new-device cooling period limits, velocity limits per payer and payee, mule account detection with graph analytics, merchant risk scores.
  • Tamper-evident audit logs (hash chaining); WORM storage for long retention.
  • Continuous compliance: vulnerability management, red teaming, participant security certification before onboarding.

16. Caching strategy

  • Address directory and bank routing tables: in-memory per node, refreshed via change events, versioned so stale entries are detected.
  • Bank health (up/degraded/down) cached and updated every few seconds from success rates; used to fail fast and show users "bank unavailable".
  • No caching of transaction state outside the owning partition (single source of truth).

17. Messaging and event-driven architecture

  • Internally, every state transition is an event in the journal (event-sourced transaction state). The current-state table is a projection, enabling replay after failures.
  • Notifications to payee PSPs, SMS and analytics consume journal topics.
  • Settlement, reporting and fraud model training consume from the archived log.
stateDiagram-v2
  [*] --> INITIATED
  INITIATED --> REJECTED: validation / risk decline
  INITIATED --> DEBIT_PENDING
  DEBIT_PENDING --> DEBIT_FAILED: bank decline
  DEBIT_PENDING --> DEBIT_UNKNOWN: timeout
  DEBIT_UNKNOWN --> DEBITED: status check success
  DEBIT_UNKNOWN --> DEBIT_FAILED: status check failed
  DEBIT_PENDING --> DEBITED
  DEBITED --> CREDIT_PENDING
  CREDIT_PENDING --> SUCCESS
  CREDIT_PENDING --> CREDIT_UNKNOWN: timeout
  CREDIT_UNKNOWN --> SUCCESS: status check success
  CREDIT_UNKNOWN --> DEEMED_SUCCESS: beneficiary confirms later
  CREDIT_PENDING --> REVERSAL_PENDING: credit failed
  REVERSAL_PENDING --> REVERSED
  DEBIT_FAILED --> [*]
  REJECTED --> [*]
  SUCCESS --> [*]
  REVERSED --> [*]

18. Cloud deployment architecture

  • Two metro data centers (or two in-country cloud regions/zones within synchronous distance, < 2 ms RTT) running active-active; transaction partitions distributed across both with consensus-based replication.
  • A third geo-distant DR site with asynchronous replication for regional disasters.
  • Kubernetes for stateless services; stateful stores on dedicated nodes or managed in-country services.
  • Participant connectivity via private links/MPLS plus internet with mTLS as backup.

19. CI/CD and DevOps

  • Strict change management: automated tests including protocol conformance suites and replay of anonymized production traffic; staged rollout by partition (canary on 1% of partitions).
  • Participant sandbox and certification environment with simulators for banks/PSPs.
  • Infrastructure as code, immutable images, signed artifacts; two-person approval for production.
  • Change freeze around known peaks (salary days, festivals).

20. Monitoring and observability

  • Real-time success rate per bank, per PSP, per transaction type, with technical vs business decline split.
  • Latency distributions per hop (switch internal, bank debit, bank credit).
  • Count of transactions in UNKNOWN/REVERSAL states and their age (critical customer-impact metric).
  • Public bank-wise uptime dashboards to push participants to improve.
  • Distributed tracing with transaction ID as the correlation key across participants (each logs it).

21. Scalability and performance optimization

  • Linear scaling by adding partitions; rebalancing partitions across orchestrator nodes via leases.
  • Batched, pipelined writes to the journal; group commit.
  • Non-blocking I/O to banks; per-bank bulkheads and adaptive concurrency limits (bank capacity is the true bottleneck).
  • Back-pressure to PSPs with clear "try later" codes when a bank is saturated.
  • Precomputed risk features in memory per partition owner to keep risk checks < 30 ms.

22. High availability and disaster recovery

Failure Behaviour
Node failure Partition leases move to other nodes in seconds; state recovered from replicated store/journal
One metro DC down Remaining DC owns all partitions (capacity planned at 2× per DC); RPO 0 via synchronous replication
Bank down Circuit breaker opens for that bank; users get immediate decline "bank unavailable"; no impact on others
Regional disaster Declared failover to geo DR; replay journal tail; RPO seconds
Unknown outcomes Status-check loops, then reversal workflows with defined TAT; customer refunds automated

23. Cost optimization

  • High utilization via shared stateless tiers; partition density tuned by load tests.
  • Commodity hardware and open-source data stores to avoid per-transaction licence costs.
  • Tiered storage for the journal (hot 30 days, warm 1 year, cold archive for the remaining retention).
  • Cost per million transactions tracked as the key efficiency metric.

24. Architecture trade-offs and alternatives

Decision Chosen Alternative Reasoning
Central switch Hub-and-spoke Bilateral bank-to-bank links N² integrations do not scale to 500 banks
Settlement Deferred net settlement in cycles Real-time gross settlement per txn Liquidity efficiency; customer still gets instant confirmation
Unknown credit outcome Deemed success after confirmation window, else reversal Synchronous blocking until certain Avoids holding resources; defined TAT
State storage Event-sourced journal + projection CRUD state table only Auditability and recovery
PIN validation At remitter bank At switch Banks own customer authentication; reduces switch liability

25. Technology stack recommendations

Java (switch services) with Netty/virtual threads; Kafka for journal and events; distributed SQL (YugabyteDB/CockroachDB/TiDB) or sharded PostgreSQL; Redis/Hazelcast for directory cache; Flink for streaming risk features; Kubernetes; HSMs (Thales/Utimaco); Prometheus/Grafana and OpenTelemetry; Spark/Trino on Iceberg for settlement and reporting.

26. Common architecture mistakes

  1. Treating a timeout as a failure and telling the user "failed" while money was debited.
  2. Retrying debits without idempotent transaction IDs (double debits).
  3. Global locks or a single database primary for all transactions.
  4. Letting one slow bank consume shared thread pools (no bulkheads).
  5. Weak controls on collect requests, enabling social-engineering fraud.
  6. No reconciliation between switch, remitter and beneficiary records.
  7. Logging sensitive data (account numbers, device keys) in clear text.

27. Senior architect interview questions and answers

The debit succeeds but the credit to the beneficiary bank times out. Walk through what happens.

The transaction moves to CREDIT_UNKNOWN. The switch performs status checks with the beneficiary bank at increasing intervals. If the bank confirms credit, mark SUCCESS. If it confirms failure, start reversal to the remitter bank (idempotent, keyed by transaction ID). If still unknown after the defined window, follow the scheme rule (deemed success with later reconciliation, or reversal). The user sees "pending" throughout and gets a notification on resolution. Reconciliation files catch any residual mismatch.

How do you achieve exactly-once without distributed transactions across banks?

Banks process requests idempotently keyed by transaction ID. The switch persists each state transition before acting, and resumes after crashes from the journal. Every external call can be retried safely, and compensations (reversals) are themselves idempotent. The combination yields an exactly-once business effect.

How would you scale from 5K to 50K TPS?

Increase partitions and orchestrator nodes, scale the distributed SQL cluster horizontally, batch journal writes, and most importantly work with banks on capacity, because bank adapters and core banking systems usually saturate first. Add per-bank adaptive concurrency limits and publish bank performance.

28. Enterprise architect interview questions and answers

How do you onboard 500 banks with different core banking systems?

Define a strict, versioned interface specification and certification process; provide reference adapters and simulators; require banks to run a standard adapter layer in front of their core; publish conformance test suites; and govern versions with deprecation timelines.

How do you govern scheme rule changes?

Rules (limits, timeouts, dispute TATs) are configuration with effective dates, versioned and approved through a change board with regulator involvement, tested in the participant sandbox and rolled out by partition.

29. CTO-level business and technology questions

How would you justify on-premises vs public cloud?

Data localization, regulatory comfort and very predictable high utilization favour on-prem or sovereign/in-country cloud for the core switch; elastic analytics and fraud model training can run in cloud. Compare 5-year TCO including staff, and design for portability (Kubernetes, open-source stores).

What KPIs would you report to the board?

Transaction volume and value growth, technical decline rate (switch and per bank), p95 latency, uptime, fraud loss rate per million transactions, dispute resolution within TAT, cost per million transactions.

30. Real-world implementation roadmap

Phase Duration Outcomes
0. Scheme design 0–6 months Rules, message specs, participant agreements, security standards
1. Core switch MVP 6–12 months P2P pay with 10 banks and 3 PSPs in pilot; sandbox and certification
2. Scale and merchants 12–18 months QR merchant payments, collect with fraud controls, disputes, 100 banks
3. Resilience 15–24 months Active-active metro DCs, geo DR, bank performance dashboards
4. Value-added services 24+ months Mandates/AutoPay, credit lines on the rail, international links, offline/feature-phone payments