A few months back I got a message that made my stomach drop: “Your site said my payment failed, but my bank already took the money.” I pulled up the payment gateway’s dashboard — the transaction was sitting there marked Success. My own database had the same transaction logged as Failed. Same charge, two different truths, and a customer stuck in the middle.

If you’ve built anything that takes online payments, you’ve probably hit some version of this. It’s an unsettling bug, because nothing actually crashes — every system involved is working exactly as designed. They’re just not agreeing with each other.

That diagram is almost exactly what was happening on my end. My server sent the charge request, waited three seconds, heard nothing, and wrote status: failed to the order table. The bank approved the charge one second later. The gateway’s webhook, confirming success, showed up two seconds after that — but by then nothing was listening for it. The order was already closed.

Three systems, one shared truth (that nobody guarantees)

Every online payment involves at least three parties talking to each other: your server, the payment gateway, and the bank or card network behind it. The gateway’s entire job is to relay what happened between them back to you — but that relay can travel over more than one channel: a direct API response, a browser redirect, or an asynchronous webhook. When one of those channels is slow, dropped, or misread, your server can end up holding a different version of events than the one that actually occurred.

The usual suspects

What the logs actually looked like

Once I started logging every status transition with a timestamp and a source, the bug stopped being a mystery and became something I could just read:

The fix wasn’t one line of code. It was accepting that payment status is eventually consistent, not instantaneous, and rebuilding the flow so the system reflects that instead of fighting it.

The fix, step by step

1. Stop treating the immediate response as final

   A “failed” result from the redirect or the immediate API call now only sets the order to pending_verification — never a final state on its own.

2. Make the webhook the actual source of truth

   A dedicated endpoint, called only by the gateway, verifies the signature and updates the order from the webhook payload — independent of anything the browser reported.

   POST /webhooks/payment-gateway
   1. verify signature using the gateway's secret key
   2. read transaction_id and status from the payload
   3. look up the order by transaction_id
   4. if status == success and order != success → mark success
   5. if status == failed and order != success → mark failed
   6. return 200 immediately, so the gateway stops retrying

3. Add an idempotency rule

   Gateways retry webhooks if they don’t get a fast 200. Once an order is marked success, nothing can downgrade it — only a refund or chargeback event can change it, and that’s handled separately.

4. Reconcile on a schedule, not by hand

   A job runs every 15 minutes, finds anything still pending_verification for more than 5 minutes, and pulls the transaction status directly from the gateway’s API. This catches the rare case where a webhook never arrives at all.

5. Separate the two timeouts

   The “wait for the browser redirect” timeout can stay short — it’s just UX. The “wait for backend confirmation” logic no longer has a hard timeout; it waits for the webhook or the reconciliation job instead.

6. Log every transition, not just the final one

   Every status change is now stored with a timestamp and a source — redirect, webhook, or reconciliation. A mismatch went from a guessing game to a two-minute read of the log.

7. Tell the customer the truth

   Instead of a hard “Payment failed” the instant the redirect looks bad, the confirmation page now reads “Confirming your payment — this can take a minute” whenever the status is still pending. Support tickets dropped noticeably once customers stopped seeing “failed” while their money was already moving.

The bigger lesson

Payment status isn’t a single fact your server learns once. It’s a conversation between three systems, spread across more than one channel, arriving at more than one speed. Build the system to expect that — webhooks as the source of truth, idempotent updates, and a reconciliation job as a safety net — and the “successful on their end, failed on mine” mismatch stops being a mystery and becomes just another status your code already knows how to handle.

If you’re chasing this bug right now, start with one question: is your “failed” status coming from the redirect or the immediate API response? Nine times out of ten, that’s exactly where the disconnect begins.