Knowledgebase

Batch and Streaming Approaches Print

  • dataengineering, data, guide, howto, solution, zillionkinghost, hosting, support
  • 0

When data is processed.

WHAT BATCH PROCESSING IS

Processing accumulated data at intervals.

WHAT STREAMING IS

Processing each record as it arrives.

WHAT BATCH PROVIDES

Simplicity Easy reprocessing Efficient use of resources Straightforward correctness

WHAT IT COSTS

Latency: results are as fresh as the last run.

WHAT STREAMING PROVIDES

Low latency Continuous results

WHAT IT COSTS

Substantially more complexity Harder correctness, particularly around ordering and late data Harder debugging and reprocessing

WHAT TO CHOOSE

Batch, unless latency genuinely requires otherwise.

WHY

Most requirements described as real-time are satisfied by processing every few minutes.

WHAT TO ASK

What decision changes if this is an hour old rather than a second old?

WHAT MICRO-BATCHING IS

Frequent small batches, approximating streaming with batch simplicity.

WHAT IT SUITS

Most cases between the two.

WHAT TO AVOID

Building a streaming system for a requirement nobody stated.


Was this answer helpful?
Back

Are you happy with your experience? Leave us a review on Trustpilot.


Trustpilot