Webhook Replay and Ordering Controls for Payment Operators

Teams that master HMAC verification still lose money to replay and ordering failures. A perfectly signed duplicate posts twice. A Confirmed event arrives before Paid during a partial outage. A handler crashes after side effects but before idempotency persistence—then retries amplify the damage.

Replay and ordering controls sit beside verification in the control plane. They define how long events remain valid, how duplicates suppress, and how state machines behave when the network does not respect your preferred narrative sequence.

Replay protection layers

• Signature verification on raw bytes.
• Idempotency store with uniqueness on provider event identity.
• Optional webhook replay window rejecting stale timestamps.
• Secret rotation with overlapping verification keys.
• Audit logs without storing secrets or full payload archives insecurely.

Duplicate delivery handling

Duplicates are often legitimate retries—not attacks. Duplicate webhook suppression should return success when the logical event already applied, so providers stop retrying without re-running side effects.

Ordering guarantees your state machine must enforce

Do not assume causal ordering from provider queues. Implement transition tables: allowed source states, required fields, and whether late events upgrade or no-op. Buffering is acceptable when bounded; unbounded buffers become hidden debt.

Provider retry orchestration

Document when providers retry (timeouts, non-2xx, specific error classes), backoff behavior, and maximum attempt windows. Your 2xx should mean “safe for you to stop”—either because work is durable or because idempotent skip occurred.

Event sequencing assumptions to write down

1. Which events may arrive multiple times?
2. Which pairs may arrive out of order?
3. Which events are terminal for a payment_id?
4. Which side effects require Confirmed versus Paid?
5. What happens when an event references unknown payment_id?

Testing replay and order scenarios

Fixture tests for duplicate delivery bursts, reversed event order, and crash between side effect and idempotency write. Chaos tests should validate that non-2xx responses produce intentional retry behavior—not accidental infinite loops.