Routing payments across two or more gateways looks like a redundancy gain. It is, and it introduces a new family of failure modes that single-gateway systems never see. Here are the ones we have actually paid for.
Mode 1 · Double-authorisation across failover.
Symptom: the same card charged twice within seconds, once at the primary gateway (which timed out) and once at the secondary (which succeeded). Root cause: the timeout window on the primary was 30s; the gateway actually responded with success at 32s, after we had already routed to the secondary. The customer saw two charges. The bank saw two authorisations.
Fix: idempotency key derived from the internal payment ID, not from anything either gateway returns. Both gateways are sent the same key; both reject the duplicate at the gateway end. Cost of the fix: two weeks of careful coding plus one full settlement cycle of dual-running before we trusted it. Cost of the bug, before the fix: refunds, and a conversation with the scheme.
Mode 2 · Reconciliation drift on hour-of-day boundaries.
Symptom: month-end reconciliation off by 0.3%: small enough to be noise, large enough to fail a regulator. Root cause: gateway A reported settlements in UTC; gateway B in the local timezone of its acquiring bank. A transaction at 23:55 local time appeared in different settlement days on the two gateways.
Fix: every settlement file is normalised to UTC at the ingest boundary. The internal ledger has one timezone. The day-end cutover is explicit, not implicit. The drift went to zero on the next month-end.
Mode 3 · Routing rule that disagrees with the issuer database.
Symptom: a small percentage of transactions routed to gateway B that should have gone to gateway A, based on the issuing-bank BIN range. Root cause: the BIN range table was last updated 14 months earlier. New BIN ranges allocated to a major issuer were not in the table; the routing code defaulted to gateway B for unknown BINs.
Fix: BIN range table now refreshed weekly from the scheme-published source, with a diff that is reviewed by an engineer. Unknown BINs are routed to the gateway with the higher historical authorisation rate, not to a default. Lesson: a default that was correct at the time of writing becomes wrong over time, silently.
Mode 4 · The retry storm.
Symptom: gateway A degraded for 90 seconds; the platform issued five times its normal request volume in those 90 seconds, then continued for ten minutes after the gateway recovered. Root cause: every failed transaction triggered an automatic retry in the application layer, plus a second retry at the queue layer, plus a third at the integration layer. Three retries multiplied across three layers is 27 attempts per transaction.
Fix: retry budget is set at one layer only: the integration layer. Every layer above it treats a failure as final and surfaces it. Token bucket on the integration layer caps the global retry rate at 1.5× the steady-state request rate. The retry storm has not recurred.
Mode 5 · The reconciliation that succeeds because it ignores the discrepancy.
Symptom: reconciliation reports "match" every day. Root cause: the reconciliation job had a clause that filtered out transactions older than 7 days, on the grounds that they were "out of scope." Some genuine discrepancies were older than 7 days because the gateway delayed certain settlement reports. The reports were green every morning; the discrepancy was real.
Fix: the filter was removed. The reconciliation now reports the full set; the operations team triages discrepancies by age and amount, not by an arbitrary cut-off. A green reconciliation report is meaningful again.
The principle behind all five.
Routing across gateways introduces a distributed-system problem in a place that used to be a single integration. Distributed-system problems require distributed-system rigour: idempotency keys you control, time normalised at the boundary, retries budgeted at one layer, defaults that are loud, and reconciliation that does not silently exclude the awkward cases.
Redundancy is not a feature you add. It is a class of correctness problems you take on, with discipline.