Knowledgebase

Handling Long-Running and Batch Workloads Print

  • 0

Work that is not a web service.

WHAT THIS COVERS

Data processing Report generation Media handling Machine learning tasks

WHAT DIFFERS FROM SERVICES

They finish They may run for hours They may need far more resources, briefly

WHAT TO USE

Jobs, rather than deployments.

WHAT TO SET

A deadline Retry limits Resource requests reflecting actual need

WHAT TO BE CAREFUL WITH

Jobs being interrupted by node maintenance or scaling.

WHAT TO DO ABOUT IT

Make the work resumable, with progress recorded Or protect the node from disruption while it runs

WHAT RESUMABLE MEANS

Processing in chunks, recording what is complete.

WHY THAT MATTERS MOST

An eight-hour job that restarts from zero is unusable.

WHAT TO CONSIDER FOR VERY LARGE WORK

Splitting it into many small jobs.

WHAT THAT PROVIDES

Parallelism, and failure affecting only one piece.

WHAT TO WATCH

Jobs consuming resources needed by services.

WHAT TO DO ABOUT IT

Separate node pools, or scheduling priorities.

WHAT TO SET

Lower priority for batch work, so services are protected.

WHAT TO MONITOR

Duration, and whether it is growing.


Was this answer helpful?
Back

Are you happy with your experience? Leave us a review on Trustpilot.


Trustpilot