Queues, Retries, and Idempotency: Stop Processing Messages Twice

1. Assume a message can come back#

A worker finishes a write and loses its connection before acknowledging the message. The queue has no way of knowing whether the effect happened, so it delivers the message again. That single scenario is enough to break a consumer that increments a counter without identifying the event. SQS standard queues are documented as at-least-once delivery; the consumer has to tolerate duplicates.

Think of idempotency as a property of the effect: repeating the same intent leaves the business state exactly as it would be after a single execution. It does not mean ignoring any request that looks similar. You need a stable identity for the same operation and a policy for what happens when the same key arrives with different content.

Documentation: AWS: at-least-once delivery in SQS ↗

2. Assign an identity before publishing#

The producer should generate event_id and persist it together with the business operation. Retrying the publish keeps that same identifier. The consumer uses a key that includes its own name or effect version plus the verified tenant of the event, so two different consumers can legitimately process the same event.

Authenticate the channel and validate the schema before trusting those fields. Store a fingerprint of the relevant content so you can detect a key being reused incorrectly. Decide how long the deduplication record lives based on your real window of retries, replays, and retention; deleting it too early lets a replay apply the effect again.

3. Couple deduplication with the local effect#

When the effect lives in the same database, a unique constraint and a transaction let you tie the processed marker to the write. Inserting the marker, committing, and only then applying the effect opens a window where a crash loses work. Applying the effect first, outside the transaction, opens the opposite window: duplicating it.

CREATE TABLE processed_events (
  tenant_id uuid NOT NULL,
  consumer text NOT NULL,
  event_id uuid NOT NULL,
  processed_at timestamptz NOT NULL DEFAULT now(),
  PRIMARY KEY (tenant_id, consumer, event_id)
);

begin transaction
insert dedup key on conflict do nothing returning event_id
if inserted:
  apply business change in this same database transaction
else:
  verify stored payload fingerprint matches this event
commit
ack message

ON CONFLICT gives you the mechanism for resolving a uniqueness collision. Your code still has to tell a legitimate collision apart from an inconsistent message, and check that the business precondition was met. A write that did not affect the expected row may call for an error and a rollback, not a silent success.

Documentation: PostgreSQL: INSERT and ON CONFLICT ↗

4. Separate publishing from external side effects#

The transaction above does not cover an external API. If you send an email or call another service before the commit, a local rollback does not undo it. If you do it after the commit, a crash can prevent it from ever being sent. Design that boundary with a transactional outbox and, where the destination supports one, an idempotency key it accepts.

The outbox stores the intent to publish together with the business change, in the same transaction. A relay sends it and records progress. That relay can also resend if it crashes between publishing and recording success, so the consumer still needs deduplication. The outbox solves message loss between the database and the broker; it does not turn every system into a single transaction.

Documentation: AWS: transactional outbox pattern ↗

5. Retry with a budget and jitter#

Classify failures: a timeout or a temporary outage may recover, while an invalid schema will not improve no matter how many times you retry. Define a maximum number of attempts, a total time limit, and a maximum delay. Add jitter so that many workers do not hit a service again at the same instant. Honor the destination's retry hints when they apply.

function retryDelayMs(attempt: number) {
  const base = 500;
  const cap = 30_000;
  const ceiling = Math.min(cap, base * 2 ** Math.min(attempt, 16));
  return Math.floor(Math.random() * ceiling);
}

Match the visibility timeout or lease to your processing time, and extend it if the provider allows. If it expires while you are still working, another consumer can receive the same message. Do not use a long delay as a substitute for idempotency.

6. Design a DLQ you can actually operate#

A dead-letter queue needs an owner, an alert, and a review procedure. Keep the failure cause, the attempt count, and the consumer version without dumping secrets into logs. Reproduce the failure in your authorized environment, fix the root cause, and redrive a small batch with the original identity before attempting a bulk recovery.

Track the age of the oldest message, errors, backlog, and time to recovery. The price per million requests helps you estimate spend, but it says nothing about how long your team will take to fix a stuck event. That operational capability is part of choosing a queue.