The integration that never errored.

Somebody reconciles two reports by hand at quarter end and the numbers are close but not equal. Not wildly wrong — close enough that the first assumption is a timing difference.

It is not a timing difference. Records have been going missing for eleven months, and nobody can say which ones or since when.

Why it does not error

An integration is written against the case where both systems are up, the payload is well formed, and nothing is rate-limited. That case is real and it covers most days, which is why the integration passes testing and runs cleanly for months.

The rest of the time something is briefly unavailable, a field arrives empty that never has before, or a downstream API quietly changes what it accepts. If none of that is handled, the record does not error. It goes nowhere.

The sending side sent. The receiving side was never asked. Both report success, and both are telling the truth.

That is what makes this worse than an outage. An outage produces an alert, a page, a post-mortem and a fix. This produces nothing at all, and the absence of an alert is read as the absence of a problem.

The three failures, in order of how often we find them

Retries without backoff

Something is briefly down, so the integration retries — immediately, repeatedly, from every worker at once. A thirty-second blip becomes a rate-limit ban, and the ban lasts an hour. Everything sent during that hour is lost unless it was queued, and it usually was not.

The fix is exponential backoff, a retry ceiling, and a dead-letter queue a person can actually inspect. A retry policy belongs in the design, not in the response to the first incident.

No reconciliation

Both sides believe they are consistent because neither has ever checked. Divergence is invisible by construction and compounds daily.

A scheduled job that counts records on both sides and reports the difference is the cheapest thing on this list. It is also the one most often missing, and it is the only mechanism that would have caught every case described here.

Validation in the middle rather than at the edges

Bad data is accepted, transformed, stored, and discovered three systems later — where the original context is gone and the fix is a migration rather than a rejection.

Validate at the boundary, reject loudly, and keep the rejected payload somewhere a person can look at it. Cheap at the edge; expensive everywhere else.

What it actually costs

Not the missing records, usually. Those can often be replayed once you know which ones they are.

The cost is what happens to the numbers afterwards. Once an organisation has discovered that two systems quietly diverged for eleven months, every figure drawn from either one carries a hedge. Decisions get made with a caveat attached, and reports get sanity-checked by hand — which is exactly the manual work the integration was built to remove.

In one case we reviewed, the reconciliation that would have caught it took a day to write. The investigation that eventually found it took three weeks, and the confidence has not fully returned.

The same shape appears when an auditor asks what a system decided and the answer has to be assembled after the fact: the question that arrives in month nine.

Engagement observation across integration work rather than a counted study. The three failure modes are consistent enough to write down; the sample is projects we were called into, which skews toward integrations somebody already suspected.

The check worth running this week

Pick your most important integration. Count the records on both sides for the same period.

If the numbers match, you have learned something in an afternoon. If they do not, you have found it before it turns into a quarter-end conversation, and the difference between those two positions is most of what this piece is about.

Then ask three questions about the same integration:

• What happens when the far end is down? If the answer is that it retries, ask how many times and how fast.
• Where does a rejected record go? If the answer is a log, it is gone. A queue somebody inspects is a different thing.
• Who would notice if nothing arrived for a week? If nobody, add the alert on silence — not on error, on silence. The worst failures produce no errors at all.

Counting both sides is the diagnostic and it takes a morning. What follows — reconciliation, retries, a record somebody can inspect — is the work.