Designing integrations that fail well

The first version of an integration is written for the happy path, because the happy path is what the vendor documentation describes and what the sandbox returns. The second version is written after an incident, in a hurry, by whoever was on call.

The difference between those two versions is not sophistication. It is a handful of assumptions: that the call arrives once, that the response comes back, that the other system agrees with you afterwards. None of those are true, and designing for that from the start costs surprisingly little.

Start from the assumption that it is down right now

For any external system you depend on, four questions decide the design, and they are worth answering in writing before the first request is sent:

1. What happens to a user action if this system is unavailable? Reject, queue, or degrade — pick deliberately per operation. "Show an error" is a valid answer for an address lookup and an unacceptable one for a payment that already succeeded.
2. What is the worst case if a call executes twice? A duplicate charge, a duplicate shipment, and a duplicate email are three very different severities and deserve three different amounts of protection.
3. How do we find out we disagree? If nobody compares the two systems on a schedule, the answer is: from a customer.
4. Who owns the retry? Us, them, a queue, or a human. Two parties both retrying the same operation is a reliable way to duplicate it.

That is a fifteen-minute exercise per integration and it determines almost everything below.

Idempotency is a key, not a hope

A timeout tells you nothing about whether the operation happened. The request may have been dropped before arriving, or executed and lost on the way back. From your side those are identical, which is why "did it work?" is unanswerable and the wrong question to build on.

The answer is to make repeating the call harmless. Generate a key that is derived from the intent — not from the attempt — and send it with every retry:

// Derived from what we are trying to do, so every retry
// of the same intent produces the same key.
const idempotencyKey = createHash('sha256')
  .update([orderId, 'capture', attemptGroupId].join(':'))
  .digest('hex');

await payments.capture(
  { orderId, amount },
  { headers: { 'Idempotency-Key': idempotencyKey } },
);

Two mistakes to avoid. Do not generate the key per attempt — a fresh UUID on each retry gives you no protection at all, which is a bug that passes every test that does not simulate a timeout. And do not include a timestamp in the derivation, for the same reason.

On the receiving side, apply the same rule to your own webhook endpoints. Store the provider event id with a unique constraint and let the database reject the duplicate:

create table webhook_events (
  provider      text        not null,
  event_id      text        not null,
  received_at   timestamptz not null default now(),
  payload       jsonb       not null,
  processed_at  timestamptz,
  primary key (provider, event_id)
);

Insert first, then process. A unique-violation on insert is not an error — it is the system working. Providers redeliver, sometimes out of order, sometimes weeks later, and always at least once more than you expect.

Never write to two systems in one breath

This is the single most common source of quiet data divergence:

// Broken in a way that only shows up under load.
await db.orders.update({ id, status: 'paid' });
await erp.createInvoice(order); // <- crash here and the two disagree forever

There is no arrangement of these two lines that is safe, because there is no transaction spanning them. Swapping the order just changes which side is wrong.

The outbox pattern fixes it with one insert. Write the intent to your own database, in the same transaction as the state change, and let a separate worker deliver it:

await db.transaction(async (tx) => {
  await tx.orders.update({ id, status: 'paid' });
  await tx.outbox.insert({
    topic: 'erp.invoice.create',
    payload: { orderId: id },
    availableAt: new Date(),
  });
});

Now the write is atomic with respect to your own database, and delivery becomes an independent, retryable concern. The guarantee you get is at-least-once delivery — which is exactly why the receiving side needs the idempotency work above. The two patterns are not alternatives; they are two halves of one design.

The worker itself stays small: claim a batch with select ... for update skip locked, deliver, mark done or reschedule. A few dozen lines of code that remove an entire category of incident.

Retries that do not make things worse

Retrying everything is nearly as bad as retrying nothing, because a retry storm against a struggling dependency is how a slow integration becomes an outage. Classify the failure first:

• Retryable: timeouts, connection resets, 429, 502, 503. Exponential backoff with jitter — without jitter, every stuck job retries at the same instant and the dependency gets hit by a wave every time it comes back.
• Terminal: 400, 422, a rejected card, a validation error. Retrying will never help. Fail loudly, once, and stop.
• Ambiguous: anything where the request may have been applied — a 409, or a timeout on a write. Reconcile by asking the other system for the current state rather than guessing. Never retry blindly.

Give every retryable job a bounded attempt count and a dead-letter destination. An unbounded retry with no visibility is not resilience, it is a job that will still be failing in six months with nobody aware of it. The dead-letter queue is not a failure state — it is the queue a human is supposed to look at, and it should be small enough that they can.

Reconciliation is a feature

Even with all of the above, two systems drift. Someone edits a record in the ERP directly, a webhook was dropped during a provider incident, a deploy went out mid-batch.

So schedule the comparison. A nightly job that walks yesterday's orders and compares your view against the other system, then reports differences to a channel a human reads. It takes an afternoon to build. Every project I have worked on that had one found real problems in the first week — which is the point: those problems already existed, and the only question was whether you or your customer would find them first.

When it finds a difference, the log needs to be able to explain the sequence of events. This is where an append-only event history pays for itself; the same argument shows up in building a marketplace MVP that survives its own success.

Four numbers per integration

Dashboards accumulate until nobody reads them. For each integration, four numbers cover almost every real question:

• Success rate over a moving window, not since the start of time.
• p95 latency, because the average hides the timeouts that generate your ambiguous cases.
• Outbox depth and oldest pending age. Depth alone looks fine while one poisoned message sits there for a day; the age is what tells you.
• Dead-letter count since the last time someone looked. If this is never zero, it is decoration.

Alert on the oldest-pending age rather than on individual failures. Individual failures are normal and will train the team to ignore the channel. A message that has been stuck for an hour is not normal, and it is the earliest signal that something needs a person.

None of this is advanced distributed systems work. It is a handful of patterns that assume the network is unreliable and the other side is imperfect — which, over a long enough window, is just an accurate description of production.