Postmortem · Bug 02 of 03

Stripe webhook ordering: how an old event downgraded a paying customer

Bug 02 from our own hardening suite — Stripe delivers at-least-once, never in-order.

The one-sentence version: an out-of-order Stripe event could overwrite newer subscription state with older state — meaning a customer who had just upgraded could be silently downgraded by a delivery that was minutes late.

Stripe does not promise order

Stripe's webhook delivery guarantee is at-least-once. Not in-order, not exactly-once. Retries, network delays, and concurrent delivery to your endpoint mean events can arrive in any sequence, and the same event can arrive many times. In development — one event at a time, on demand — everything always arrives in order. The bug only appears in production traffic, under exactly the conditions you can't easily reproduce by hand.

The bug

The naive handler pattern — the one in most tutorials, and the one in our code — is: on a subscription event, take the payload and write the plan and status into the database. Last write wins. Now look at this timeline:

09:00  customer upgrades to Pro            (Stripe state: pro)
09:01  subscription.updated (pro) arrives  → we write: pro  ✓
09:02  a delayed subscription.updated (free)
       — an older event, retried and stalled
       in transit — arrives                → we write: free ✗

The customer paid for Pro.
Their account now says Free.

Every individual line of the handler was correct. The state machine as a whole was wrong, because it assumed delivery order matches event order. Stripe never made that promise.

How we caught it

Our hardening suite's stripe-webhook-reliability test deliberately replays subscription events out of order against the real database and asserts the final state equals the state of the newest event. Against the naive handler, the assertion failed: the stale event won. No amount of reading the handler would have found this — the code handled every event correctly in isolation. The bug lived in the interaction between events.

The fix: compare before you write

Each subscription row now carries the timestamp of the newest event applied to it (last_stripe_event_at, added by migration 000026_stripe_event_ordering.sql), and the sync layer refuses to apply anything older:

// lib/billing/sync.ts — simplified
const eventCreatedAt = new Date(subscription.created * 1000).toISOString();

const { data: existingSub } = await admin
  .from("subscriptions")
  .select("id, organization_id, last_stripe_event_at")
  .eq("stripe_subscription_id", subscription.id)
  .maybeSingle();

// Out-of-order protection: ignore stale events
if (
  existingSub?.last_stripe_event_at &&
  new Date(existingSub.last_stripe_event_at) > new Date(eventCreatedAt)
) {
  return; // stale event — acknowledge, never apply
}

// ...write subscription state, including:
//     last_stripe_event_at: eventCreatedAt

Every write records the event time it applied, so the next event — however late it arrives — is compared against real state, not arrival order. The hardening suite's webhook-reliability test replays stale and fresh updates against the real database and asserts the final state equals the newest event. It ships with the product and runs in CI on every push: if billing logic ever becomes order-dependent again, the build fails — not a paying customer's plan.

The checklist rule

This is the ordering check in our Hardening Checklist. Ask yourself one question about your own webhook handler: before writing subscription state, does it compare the event's timestamp against the state it already holds? If the honest answer is “we process events in the order they arrive” — that's not a neutral choice. That's the bug.