Playbook
Duplicate payment investigation workflow
Separate duplicate webhooks, duplicate chain observations, and duplicate commerce intents with evidence-first triage.
01
Objective
Determine whether duplicate signals are benign retries, duplicate commerce intents, or reconciliation errors—before double fulfillment or double posting.
02
Prerequisites
- Idempotency keys logged with payment_id and event type.
- Commerce reference uniqueness rules documented.
- Duplicate taxonomy distinguishes webhook vs chain vs order duplication.
03
Operational signals
- Same payment_id with multiple Confirmed transitions.
- Two payment_ids for one order reference.
- Idempotency store shows skipped duplicates rising.
04
Decision points
- Refund versus credit versus apply to balance.
- Whether to halt fulfillment pending investigation.
- Provider dispute versus internal matcher bug.
05
Escalation paths
- Support → finance for customer balance impact.
- Engineering → payment operations for handler bugs.
06
Failure modes
- Issuing refund and fulfillment for both signals without investigation.
- Deleting idempotency records to unblock handler.
07
Recovery patterns
- Collect provider event timeline for all involved payment_ids.
- Map commerce references across three planes.
- Apply resolution code from exception taxonomy.
- Retries are normal. Webhook delivery is at-least-once. Design consumers to tolerate duplicates and out-of-order arrivals where possible.
- Asynchronous by design. Payers, chains, and your servers operate on different clocks. UI and finance should not assume synchronous finality.
- Eventual consistency. API reads, webhooks, and portal views may briefly diverge during transitions. Reconciliation jobs exist to converge truth.
Walkthroughs: /operations