Design a payment processing system that can charge customers, handle refunds, and integrate with external payment processors — while guaranteeing no duplicate charges and full auditability, even across network failures. (For the implementation-level mechanics referenced throughout — idempotency, the authorize/capture state machine, reconciliation, PCI scope reduction — see the Payment Systems chapter under Spring Boot REST API Development.)
For a mid-size e-commerce platform: 1M orders/day, each with one charge attempt (occasionally two, for retries).
Client (browser)
│ card details go DIRECTLY to processor's hosted SDK, never through our servers
▼
[Payment Processor's SDK] ──returns a TOKEN──▶ Client
│
▼
Client sends { token, amount, idempotencyKey } to OUR backend
│
▼
[API Gateway] → [Payment Service]
│
├─▶ [Payment DB] (strongly consistent — the source of truth for state)
├─▶ [Payment Processor API] (authorize → capture)
└─▶ [Reconciliation Job] (scheduled, compares DB vs processor's log)
The Payment Service is the only component that talks to the payment processor directly — this centralizes idempotency enforcement, state-machine validation, and audit logging in one place rather than scattering payment logic across every service that might need to charge a customer (Order Service, Subscription Service, etc. all call Payment Service, never the processor directly).
CREATED → AUTHORIZED → CAPTURED → SETTLED
│ │
▼ ▼
VOIDED REFUNDED
(any state) → FAILED
Every transition is validated against the current state before being allowed — this is what prevents a double-capture or a refund on a never-captured payment from being possible even under a bug or a race condition upstream.
POST /api/v1/payments
Headers: Idempotency-Key: <client-generated-uuid>
Body: { "token": "tok_abc", "amount": 4999, "currency": "USD", "orderId": "ord_123" }
→ 201 { "paymentId": "pay_456", "status": "AUTHORIZED" }
POST /api/v1/payments/{id}/capture
→ 200 { "status": "CAPTURED" }
POST /api/v1/payments/{id}/refund
Body: { "amount": 4999, "reason": "customer_request" }
→ 200 { "status": "REFUNDED", "refundId": "ref_789" }
GET /api/v1/payments/{id}
→ 200 { "status": "CAPTURED", "history": [...timestamped transitions...] }
Every mutating call requires the client to supply an idempotency key; GET calls (naturally idempotent) don't.
payments
id, orderId, amount, currency, status, idempotencyKey (UNIQUE INDEX),
processorReference, createdAt, updatedAt
payment_transitions (append-only audit log)
id, paymentId, fromStatus, toStatus, timestamp, metadata
refunds
id, paymentId, amount, reason, status, processorReference, createdAt
The unique index on idempotencyKey is the core correctness mechanism (see Payment — Idempotency Implementation) — it's a database-level guarantee, not application logic, which is what makes it safe under concurrent requests. payment_transitions being append-only (never updated, only inserted) is what makes the audit trail trustworthy — a mutable history could be silently altered after the fact.
Payment volume is typically much lower than overall site traffic (checkout is one step in a much larger user journey), so the Payment Service rarely needs the same horizontal scale as, say, a product catalog service. The actual bottleneck is usually the external payment processor's own rate limits and latency, not internal capacity — scaling strategy here is more about resilience (circuit breakers, retries with idempotency) than raw throughput. The database benefits more from strong consistency and good indexing (on idempotencyKey and orderId) than from sharding at this volume.
The external payment processor is the hard bottleneck — its rate limits, latency, and occasional downtime are entirely outside this system's control. The mitigation isn't scaling harder; it's resilience (Circuit Breaker Pattern around the processor call, a bounded retry policy, and the reconciliation job as a safety net for whatever slips through). A secondary bottleneck is the idempotency-key unique index under very bursty retry storms — mitigated by the index being a lightweight, well-optimized database operation relative to the processor call itself.
Processor timeout (ambiguous outcome): query the processor directly using the idempotency key to resolve ground truth rather than guessing (see Payment — Failure Handling & Reconciliation) — this is the single most important failure path to get right in this entire design.
Processor fully down: fail closed (reject new charges with a clear "try again later" error) — unlike a rate limiter, payment should never fail open, since failing open here would mean accepting orders with no way to actually charge for them.
Database write succeeds, processor call never made (crash between steps): the reconciliation job catches this — a local record stuck in CREATED with no corresponding processor transaction is flagged and either retried fresh or investigated.
Duplicate webhook delivery: webhook handlers must themselves be idempotent (most processors deliver webhooks at-least-once, not exactly-once) — process each webhook event ID exactly once regardless of delivery count.