One expired secret looked like 861 revoked customers

On 2026-10-21 a background job in a marketplace seller tool decided that 861 sellers had revoked its access, and parked all of them for a week. None of them had. The app's own OAuth client secret had expired, so every call made on behalf of every seller failed the same way. The data was fine; the classifier was wrong. The rule we took from it: an error that hits everyone at once is an outage of something shared, not hundreds of separate decisions, and it must not be booked as hundreds of separate facts.

What happened?

The tool calls a marketplace API on behalf of each seller. Each seller grants access once through OAuth. To use any seller's grant, the tool also authenticates as itself with a client ID and a client secret. That secret belongs to the app and has an expiry date.

On 2026-10-21 it expired. From then on every per-seller call came back as 403 "Access to requested resource is denied", with a detail line saying the app's secret had expired. A nightly sweep goes through every connected seller, checks that the connection still works, and parks sellers whose access is gone so the tool stops trying them. It parked the first seller on a 403, then the second. By morning 861 sellers were parked for a week, each with a record saying they had revoked access.

Why did the sweep believe it?

Because it looked at one call at a time. To a single request, a revoked seller and a broken app credential look the same: a 401 or 403. The difference is only visible across the fleet. A real revocation is one seller on one night. A broken app credential is every seller, minutes apart, with the same error text.

The sweep had no memory between sellers, and it wrote long-lived state on every guess. One shared cause became 861 individual records, each correct-looking on its own.

How do you tell per-user from fleet-wide?

Start with what the error says. OAuth 2.0 (RFC 6749, section 5.2) separates the two cases in its error codes. invalid_grant means the grant or refresh token is "invalid, expired, revoked" or was issued to another client: that is about one user's grant. invalid_client means "Client authentication failed": that is about the app itself, and it is the same for every user. Without a useful code, the pattern across users decides.

The RFC defines unauthorized_client as the client not being allowed to use a grant type, so it is not obviously per-user. We count it as per-user because of how this API uses it. Check how your provider uses it before copying that row.

The classifier

The fix keeps one counter across the whole run and checks the error class before touching any user's record:

streak = 0                          # generic 401/403 in a row, across users
for user in users:
    result = call_api(user)
    if result.ok:
        streak = 0
        continue
    kind = classify(result)
    if kind == "shared":            # invalid_client, "secret expired"
        stop_run("shared credential")
        break
    if kind == "per_user":          # invalid_grant, unauthorized_client
        streak = 0
        mark_revoked(user, run_id)
        continue
    streak += 1                     # generic 401/403, cause unknown
    if streak >= 5:
        stop_run("suspected outage")
        break
    rest(user, hours=6, run_id)     # short, and tagged with the run

When the run stops, it stamps nobody. The next run starts clean. Five is small enough to stop before much damage; real revocations carry a per-user code and reset the streak. The few users rested before the breaker trips carry the run id, so they are easy to lift.

How did the cleanup go?

Each park had been written with the run that wrote it and the time. Once the secret was renewed, recovery was one pass over the parks from that window: 841 of the 861 were cleared automatically.

That only worked because the stamp said where it came from. A bare "revoked" flag with no cause and no run would look the same as a real revocation, and the cleanup would have been a guess.

The same mistake, elsewhere

Any batch job that writes per-user state from API errors can do this:

• Billing retries. The payment processor has an outage and every renewal fails. A job that marks each card as failed turns one incident into thousands of dunning emails.
• Token refresh. The identity provider is down, every refresh fails, and a job that signs out users whose refresh failed signs out everyone.
• Webhook delivery. Your own outbound network breaks, every delivery times out, and a job that disables endpoints after N failures disables every customer's endpoint.

It is the same shape as a draft PR that never gets reviewed, where an absent result was read as a negative one. Here it is errors: a shared cause booked as many individual facts.

Checklist

• List your error classes. For each code or message, decide whether it is about one user or about something shared (your credential, your network, the provider).
• Add a circuit breaker across users. N of the same unexplained error in a row, from different users, stops the run. We use 5.
• Write no long-lived per-user state during a suspected outage. Stop first, decide later.
• Stamp every write with its cause, run and time. Then undoing a bad run is one query, not an investigation.
• Prefer short rests to long parks for unknown errors. A 6-hour rest that was wrong costs little; a week-long park that was wrong costs a week.

Common questions

What is the difference between invalid_grant and invalid_client?

In RFC 6749, invalid_grant is about the grant or refresh token, which belongs to one user. invalid_client is about the client failing to authenticate, which affects every user of that client at once.

What is a circuit breaker?

Martin Fowler describes it as wrapping a call in an object "which monitors for failures"; once failures pass a threshold, it trips and stops making the call. Here it trips on a streak across users.

Should a 403 ever disable a user automatically?

Only when the error says it is about that user, such as invalid_grant. A streak of generic 403s is a reason to stop and look, never a reason to disable anyone.

Sources

• RFC 6749: The OAuth 2.0 Authorization Framework, section 5.2 (Error Response)
• Martin Fowler: CircuitBreaker