Background jobs and queues done right

Assume every job runs more than once

Queues deliver at least once, not exactly once. A worker can crash after doing the work but before acknowledging it, and the job comes back around. If your job charges a card or sends an email, running twice is a real problem. Design each job to be idempotent — key the side effect on a stable id and check whether it already happened — so a redelivery is harmless rather than a duplicate charge.

• Make side effects idempotent with a natural key or an idempotency token.
• Pass a job an id to load fresh, not a serialized snapshot that goes stale in the queue.
• Keep jobs small and single-purpose so a retry re-runs the minimum work.

Retries need backoff and a dead end

A job that fails and retries instantly, forever, turns a transient blip into a self-inflicted denial of service against whatever it is calling. Use exponential backoff with jitter so retries spread out, cap the attempts, and send exhausted jobs to a dead-letter queue instead of dropping them. The dead-letter queue is where you inspect, fix and replay — a failed job should never simply vanish.

You cannot operate what you cannot see

An asynchronous system fails silently by nature — the user already got their response, so a broken job produces no error page. Instrument queue depth, job latency and failure rate, and alert when the backlog grows faster than workers drain it. A rising queue depth is the earliest warning that something downstream is struggling.

• Monitor queue depth, processing time and dead-letter counts.
• Separate slow bulk jobs onto their own queue so they cannot starve latency-sensitive work.
• Alert on backlog growth, not just on individual job failures.