A backup job that hit a flaky S3 endpoint used to fail the whole run. Now it retries three times with growing gaps, and if it still fails it lands in a dead-letter set that pages me instead of vanishing. BullMQ makes that easy once you configure it deliberately rather than accepting the defaults.

Backoff, not hammering

The default of retrying immediately is the worst option for a transient failure, because the thing you are retrying is usually still busy. Exponential backoff spaces the attempts out, giving the downstream service room to recover.

await queue.add("backup", data, {
  attempts: 3,
  backoff: { type: "exponential", delay: 5000 },
});

Three attempts at 5s, 10s, 20s covers nearly every blip I see in practice, from a momentary network drop to a brief upstream restart.

Idempotent jobs or nothing

Retries only help if running a job twice is safe. A job that charges a card or sends an email needs an idempotency key, so the second attempt recognizes the work is already done and skips it. I learned this the unglamorous way, with two of the same notification in a tester's inbox.

const already = await redis.set(`job:${job.id}:done`, "1", "NX", "EX", 86400);
if (!already) return; // a prior attempt finished this

The dead-letter set

When all attempts fail, the job sits in BullMQ's failed set. That set is not a graveyard, it is a worklist. A failed listener writes an alert, and the failed jobs stay inspectable so I can read the exact payload and stack that broke.

worker.on("failed", (job, err) => {
  logger.error({ jobId: job?.id, err: err.message }, "job dead-lettered");
  alerts.page("backup job failed after retries", job?.data);
});

Watch the queue, not just the app

The failure mode that bit me hardest was not a crash, it was a queue quietly backing up while everything looked green. Now a small check reports queue depth and the age of the oldest waiting job. A growing backlog is the earliest signal that a worker is stuck or starved.

const waiting = await queue.getWaitingCount();
const oldest = await queue.getWaiting(0, 0);

Concurrency with intent

Workers default to one job at a time. Bumping concurrency speeds throughput, but only up to what the downstream can take. For the backup worker I cap it low on purpose, because twenty parallel uploads to the same bucket just trade one bottleneck for another. The retries, the dead-letter alerts, and the depth metric together mean a failure is loud and recoverable instead of silent and lost.