UPI-style payments switch

A generic UPI-style switch that routes payments between banks using a virtual payment address (VPA, such as ravi@okbank). It handles about 100 million payments a day, with festival peaks. The hard parts are idempotency, the unknown outcome after a timeout, and reconciliation (matching our records against each bank's records).

Design a payments switch that moves money between bank accounts using a virtual payment address, about 100 million payments a day, with festival-hour peaks.

Requirements

Functional

  • A payer's app starts a payment through its payment service provider (PSP: the company that connects the app to the switch). The payment goes to a VPA such as ravi@okbank, with an amount and a transaction reference that the payer generates.
  • Resolve the VPA to the beneficiary bank and an account handle. Ask the payer's bank (the remitter bank) to debit, then ask the beneficiary bank to credit.
  • If the credit fails after a successful debit, reverse the debit automatically.
  • Report a final status (success, failed, reversed) to both sides, and let either side query the status of a payment.
  • Produce settlement files at the end of each cycle, so banks can settle their net positions with each other.

Non-functional

  • Scale: about 100 million payments a day (about 1,200 a second on average), and about 12,000 a second in a festival-hour peak.
  • Correctness: a payment is never debited twice, and never left debited without a credit or a reversal. Every state change is durable and auditable.
  • Availability: 99.99% for the switch itself, which is about 52 minutes of downtime a year. A slow bank must only affect its own customers.
  • Latency: the switch adds under 100 ms. End to end, a payment takes under a few seconds at p95 (95 out of 100 payments), mostly spent in bank calls. After a hard overall timeout, for example 30 seconds, the payment is pending, not failed.
  • Retention: transaction records are kept for years, for audit and disputes.

Out of scope

  • The payer's app, PIN entry and device binding. The PSP and the remitter bank handle authentication.
  • Inter-bank settlement itself. The switch produces settlement files; money moves between banks separately.
  • Fraud scoring models (mention a hook for them).
  • Disputes and the chargeback workflow, beyond recording the data they need.

Estimate

  • Average: 100M payments a day / 86,400 s ≈ 1,160, call it about 1,200 payments a second.
  • Peak: festival hours run about 10 times the average → about 12,000 payments a second; design for that, not the average.
  • Bank calls: each payment means about 3 outbound calls (resolve VPA, debit, credit) → about 36,000 bank messages a second at peak, spread over many banks.
  • Ledger writes: about 3 state transitions persisted per payment (created, debited, credited/final) → 12,000 x 3 = 36,000 writes a second at peak. At about 2,000 comfortable writes a second per primary that is 18 shards; plan 24 for headroom, each with a synchronous replica.
  • Status reads: about 3 per payment (both apps poll or query) → about 36,000 reads a second at peak, served from replicas and a short-lived cache of final states.
  • Storage: about 1 KB per payment with its history → 100M x 1 KB = 100 GB a day → about 36 TB a year; 10 years ≈ 365 TB, x3 replication ≈ 1.1 PB. Keep about 90 days hot (9 TB) and archive the rest.

API

  • POST /v1/payments {payer_ref (idempotency key), payer_vpa, payee_vpa, amount, currency: "INR", note} -> 202 {txn_id, status: "PENDING"}
  • GET /v1/payments/{txn_id} -> 200 {status, debit_status, credit_status, updated_at}
  • GET /v1/vpa/{vpa} -> 200 {name, bank_code} (name confirmation before paying; rate-limited)
  • POST /v1/payments/{txn_id}/bank-callbacks {leg: "debit"|"credit", bank_ref, result} -> 200 (asynchronous responses from banks)
  • GET /v1/settlements/{cycle_id}?bank= -> 200 {file_url, totals}

Data model

  • transactions(txn_id PK, payer_psp, payer_ref, payer_vpa, payee_vpa, amount_paise BIGINT, state, created_at, updated_at, UNIQUE(payer_psp, payer_ref)): sharded by txn_id.
  • txn_events(txn_id, seq, from_state, to_state, bank_ref, payload_hash, ts): append-only audit log, never updated.
  • vpa_directory(vpa PK, bank_code, account_token, name, status): read-heavy, cached; the switch stores a token, not the account number.
  • bank_health(bank_code PK, success_rate_1m, p95_ms, breaker_state): drives routing and circuit breakers.
  • settlement_entries(cycle_id, bank_code, txn_id, direction, amount_paise): the per-cycle position each bank settles.

A request, step by step

  1. Priya pays ₹450 to ravi@okbank. Her PSP sends POST /payments with its own reference P-77812. The gateway checks the PSP's certificate and signature.
  2. The orchestrator inserts a transaction row in state CREATED, with UNIQUE(psp, payer_ref). A retry of the same request hits that unique constraint and gets the existing txn_id back.
  3. The orchestrator resolves ravi@okbank in the VPA directory, usually from its in-process cache. The result is the beneficiary bank and an account token.
  4. It moves the row to DEBIT_PENDING, then asks Priya's bank to debit ₹450 through that bank's adapter. It passes txn_id as the bank's idempotency reference. The bank answers DEBITED, the row moves to DEBITED, and an event is appended to txn_events.
  5. It moves to CREDIT_PENDING and asks Ravi's bank to credit. On success, the row becomes SUCCESS and both PSPs are told. Each transition is a conditional update that works only from the expected previous state. So two workers can never both advance the same payment.
  6. If the credit is declined, the orchestrator sends a reversal to Priya's bank with the same txn_id, so the reversal is idempotent. The payment ends in REVERSED. If any bank call times out, the payment stays in a *_PENDING state and the status workers take over.
  7. Every transition is published to Kafka. Status workers ask banks about stuck payments, with backoff. At the end of the cycle, reconciliation compares the ledger with each bank's report and produces settlement files.

Deep dives

Idempotency and the payment state machine

Weak: a boolean "paid" column, and retries on error. The PSP retries a request that timed out. The switch treats it as a new payment, and Priya is debited twice. The status is a guess, and nothing records how the payment reached it.

Good: an idempotency key with a unique constraint. An idempotency key is an id that makes a repeated request harmless. Here it is the PSP's reference, which is unique per PSP. A retry finds the existing row and returns its txn_id and current status, instead of starting again. The same txn_id is passed to each bank as its reference, so the bank also recognises a repeated debit request.

Great: an explicit state machine with guarded transitions and an append-only log. A state machine lists every allowed state and every allowed move between them. The main states are CREATED, DEBIT_PENDING, DEBITED, CREDIT_PENDING and SUCCESS. The side branches are FAILED, REVERSAL_PENDING, REVERSED and DEEMED (outcome unknown, being resolved).

Each move is a guarded update: UPDATE ... SET state = new WHERE txn_id = ? AND state = expected. If zero rows are updated, someone else already moved it. Every transition is also appended to txn_events in the same database transaction. That gives auditors a full history and lets you rebuild the state.

Trade-off: you get more writes per payment and more states to test. About three state changes plus events, at 12,000 payments a second, means the ledger must be sharded. In payments, that is cheap insurance.

The debit succeeded but the response timed out

This is the question the interview turns on. The remitter bank may have debited Priya, but the switch never heard back.

Weak: mark it failed and tell Priya to try again. If the debit did happen, she is now short ₹450. If she retries, she pays twice.

Good: mark it pending and ask the bank. After the timeout, the payment stays DEBIT_PENDING and the user sees "pending, do not retry". A status worker asks the bank about the same txn_id, with backoff: after 30 seconds, 2 minutes, then 10 minutes. If the bank says debited, the switch continues to the credit. If it says not debited, the payment fails cleanly.

Great: pending, plus reconciliation, plus a safe reversal. Status checks resolve most cases within minutes. For the rest, reconciliation at the end of the cycle compares the ledger with each bank's own report, line by line.

Any payment debited without a matching credit gets an idempotent reversal. It is keyed by txn_id, so issuing it twice reverses only once. There is a fixed deadline: no payment can stay unresolved past it, and the customer is told when the money will be back.

Trade-off: being honest about uncertainty costs some user experience, because "pending" is frustrating. But guessing either way loses money. Never turn "unknown" into "failed".

One slow bank must not take down the switch

During a festival peak, one bank's systems slow down to 20-second responses.

Weak: one shared thread pool for all bank calls. Calls to the slow bank hold threads for 20 seconds each, and the pool runs out. Then payments between two healthy banks start timing out too. One bank's problem becomes everyone's problem.

Good: timeouts and a bulkhead per bank. A bulkhead is a separate, limited pool of resources, so one failure cannot use up everything. Each bank gets its own connection pool and concurrency limit, sized from its normal traffic. Its timeout matches its usual p99, not 30 seconds. When the slow bank's pool is full, only that bank's payments wait or fail fast.

Great: bulkheads, circuit breakers and honest feedback. A circuit breaker per bank watches the success rate. When the rate drops below a threshold, the switch stops sending new payments to that bank for a short time. It answers at once with a clear "bank unavailable, try later". Without this, pending payments pile up, and each one needs reconciliation.

Half-open probes (a few test requests) check whether the bank has recovered. Bank health is published, so PSP apps can warn users before they pay.

Trade-off: failing fast declines some payments that might have succeeded slowly. But it keeps the pile of pending payments small, and it protects the customers of every other bank.

What breaks

  • A bank times out or slows down: per-bank alerts on success rate and p95 fire. The bulkhead keeps other banks healthy, and the circuit breaker fails new payments to that bank fast. Payments already in flight move to pending and are resolved by status checks.
  • A ledger shard primary fails: its synchronous replica is promoted, so no acknowledged state change is lost. Payments on that shard pause for the failover, which takes seconds, and PSPs see pending.
  • The orchestrator crashes mid-payment: the row stays in a *_PENDING state. A sweeper (a background job that scans for stuck rows) finds rows past their deadline. It hands them to status workers, which continue from the recorded state.
  • Kafka is down: state changes are still committed in the ledger. Events are written to an outbox table and published when Kafka returns. (An outbox is a table of events waiting to be published, written in the same transaction as the state change.) Reconciliation is delayed, not wrong.
  • Reconciliation finds a mismatch, such as a payment debited but never credited or reversed. It raises a high-priority alert and an idempotent reversal. The count of unresolved items past their deadline is a key metric.

What each level is expected to cover

Mid level: A Mid candidate should give the rates: about 1,200 a second, and 12,000 at peak. They should use an idempotency key so retries do not debit twice, and store payments in a replicated relational database. They should separate the debit and credit steps, and say that a timeout leads to a pending state that is checked later.

Senior: A Senior candidate also draws the full state machine, with guarded transitions and an append-only event log. They handle the unknown outcome with status checks, reconciliation and idempotent reversals, and shard the ledger from the write estimate. They isolate banks with bulkheads and circuit breakers, and use an outbox for events. They also discuss audit, retention and settlement.

Common mistakes

  • Treating a timeout as a failure, which either loses customer money or invites a double payment.
  • No idempotency key, so a PSP retry starts a second payment.
  • Updating a status column without checking the expected previous state, so two workers both advance or reverse the same payment.
  • Putting the whole payment in one distributed transaction across banks, which external banks do not support.
  • Using floating-point numbers for money instead of integer paise.
  • Sizing for the daily average (about 1,200 a second) and missing the tenfold festival peak.

Follow-up questions

  • The credit to the beneficiary bank keeps failing. When do you reverse, and how do you make sure you reverse only once?
  • How would you support "collect" requests, where the payee asks the payer to approve?
  • How do you run this switch across two data centres without ever processing a payment twice?
  • How would you add a fraud check without adding much latency?
  • What does reconciliation compare, and what happens to the items that do not match?

Lessons behind this answer