The cheapest guard for a mirrored pipeline is a count against the source

A marketplace seller tool copies each seller's orders from the marketplace's API into its own database, and every profit number it shows comes from that copy. In October an audit found that for 26 accounts the orders had quietly stopped arriving. Some had been missing for weeks, some since February, about 13,200 orders in all. No screen said so. Every page showed what the copy had, and none showed what the source said it should have. The rule we took from it: for any pipeline that mirrors another system, the first guard to build is a count against the source, per account.

What happened?

The tool learns about new orders in two ways. The marketplace sends a notification when an order changes, and the tool fetches the order and stores it. The seller then sees revenue, fees, costs and profit, all computed from the stored orders.

That design has a blind spot. If notifications stop, nothing fails. No request returns an error, no job crashes, and the numbers on screen still add up. They are just smaller than they should be. A seller who sells less than usual for a while has no way to tell a slow month from missing data, and neither did we.

It came to light only because someone asked a different question. Instead of "is this order right?", the audit asked "how many orders does the marketplace say this account had, and how many do we have?" For 26 accounts the two numbers did not match.

Two silent causes

Both causes were about one seller having more than one account record, for example when someone cancels and signs up again, or connects twice.

A lookup that picked the wrong twin. When a notification arrives, it carries the marketplace's seller id. The tool turned that into an account with a rule that amounted to "one account per seller id, lowest record id wins". When a seller had two records, the lowest id was the older one, and often that was the cancelled one. Notifications were routed to an account nobody looked at. The live account received nothing, and nothing logged that a choice had been made.

A disconnect that deleted shared state. The notification subscription lives at the marketplace and belongs to the seller, not to one of our records. When one of the twin records was disconnected, the cleanup deleted that subscription. The other record still depended on it. From then on the marketplace sent nothing for that seller at all.

Neither cause produced an error. Each one was a reasonable line of code that assumed a seller has exactly one account.

The count check

The fix that matters most is the one that would have caught both causes, and any future third cause, within two days. Each night a scheduler runs once per account:

for each account, nightly:
  source = marketplace.count(orders, yesterday)
  mirror = db.count(orders, account, yesterday)
  short  = mirror < source
  if short two days running:          alert
  if short and no subscription:       alert

Two short days, not one, because orders can arrive late and a single short day is often just a slow notification. One short day is enough when the subscription is also missing, because then the cause is already known. On its first run across about 800 accounts the check raised 0 false alerts and found 2 real stragglers.

The check does not know why the counts differ, and it does not need to. It only knows that the copy is behind the source. That is enough to send a person to look, and it is cheap: one count call per account per night.

Pick one of several, on purpose

The lookup now ranks matches explicitly instead of taking whichever comes first. A connected account beats a disconnected one. Then a paying account beats a cancelled one. Then the newest record wins. And whenever more than one record matched, it writes a log line saying so, with the ids it saw and the one it picked.

The log line is the part that is easy to skip. A rule that picks well today can still pick badly when the data changes, and a tie that nobody sees is a tie nobody fixes. Any "pick one of several" in your code is a decision; make it visible.

Shared subscriptions

A disconnect on one record must not delete a subscription that other records rely on. Before deleting anything that lives outside the record, check whether another live record still uses it, and keep it if so. This is the same habit as in one expired secret that looked like 861 revoked customers: know which state belongs to one user and which is shared, and treat them differently.

Checklist

• Count against the source, per account, first. Before any per-line accuracy work, compare how many records the source has with how many you have.
• Rank every multi-match lookup explicitly. Write down the order (connected, paying, newest for us) and log whenever more than one record matched.
• Never let one record's disconnect delete shared state. Check for other users of a subscription before removing it.
• Alert on two signals, not one. Two short days, or one short day plus a missing subscription, keeps late arrivals from becoming noise.

Common questions

Why not check every order line?

Line checks tell you whether the orders you have are right. They say nothing about orders you never received. A count against the source is cheaper and catches the bigger failure, so it comes first.

Why wait two days before alerting?

Orders often arrive late, so a single short day is usually noise. Two short days in a row, or one short day with no subscription, is a real gap worth a person's time.

Sources

• One expired secret looked like 861 revoked customers
• A draft PR never gets reviewed: four false rounds in one day