Cross-region failover for payment platforms

Cross-region failover for a general web application is a well-understood pattern: replicate state, route around outages, reconcile on recovery. For a payment platform, the same pattern needs to preserve properties that a general web application doesn't have to worry about.

Written October 2024 from a DR design engagement.

What must survive a failover

• In-flight transaction identity. A transaction that was authorizing when the primary region failed must be recoverable, either by continuation in the secondary or by explicit cancellation.
• Batch integrity. A batch in mid-construction must not be lost, duplicated, or partially submitted.
• Idempotency-key state. The receiver on the secondary must recognize keys from before the failover to prevent double-processing on retry.
• Reconciliation state. The last reconciliation checkpoint must transfer, so the recovery reconciliation doesn't reprocess history.

What actually goes wrong

• Asynchronous replication lag. A transaction committed to the primary but not yet replicated to the secondary is invisible on failover.
• Split-brain risk. If both regions think they're primary for a moment, the same transaction gets processed twice.
• Time skew. Clocks between regions drift; a failover event that treats timestamps naively produces confusing ordering.
• Retention windows. Idempotency keys retained for 24 hours in-region may not survive a cross-region failover cleanly.

Design decisions worth making early

• Synchronous replication for authorization-critical state, asynchronous for reporting. The trade-off is latency for durability; for auth, the durability wins.
• Explicit region tagging on every event. So reconciliation after failover can distinguish "processed in primary" from "processed in secondary".
• A rehearsed failover drill. Not a document; an actual drill, run at least twice a year with real traffic diverted.

Cross-region failover for payments is expensive to build and expensive not to have. The middle case — thinking you have it and finding out you don't — is the worst.