Knowledgebase

Managing Background Workers Across Stacks Print

  • backenddevelopment, backend, performance, database, guide, howto, solution, zillionkinghost
  • 0

Running work outside requests.

WHAT EVERY STACK NEEDS

A queue Worker processes consuming it A scheduler for periodic work

WHAT THE QUEUE HOLDS

Jobs: a type, a payload, and metadata.

WHAT TO KEEP SMALL

The payload.

WHY

Large payloads slow the queue; pass identifiers and fetch the data.

WHAT TO CONFIGURE

Retry count Increasing delay between attempts A destination for permanently failed jobs

WHAT TO ENSURE

That processing twice is safe.

HOW

Record completion, and check before acting.

WHAT TO MONITOR

Queue depth Failure rate Whether workers are alive Time jobs wait before starting

WHAT A GROWING QUEUE INDICATES

Insufficient workers, or jobs failing repeatedly.

WHAT TO SEPARATE

Queues by priority, so slow bulk work does not delay urgent work.

WHAT TO BE CAREFUL WITH

Jobs holding database connections for a long time Jobs with no timeout Workers running old code after deployment

WHAT TO ALERT ON

Failed jobs, and a job type that stops appearing.


Was this answer helpful?
Back

Are you happy with your experience? Leave us a review on Trustpilot.


Trustpilot