Knowledgebase

Streaming: Everything That Matters, Briefly Print

  • dataengineering, data, guide, howto, solution, zillionkinghost, hosting, support
  • 0

The summary.

ORDERING IS GUARANTEED WITHIN A PARTITION, NOT A TOPIC

Events for one entity must share a partition, keyed by entity identifier, or they may be processed out of order.

Partition count sets maximum parallelism, and increasing it later is disruptive.

EXACTLY-ONCE REFERS TO THE EFFECT, NOT THE DELIVERY

The practical approach is at-least-once delivery with idempotent processing. Commit offsets late — duplication is recoverable, loss is not.

AGGREGATING BY PROCESSING TIME PRODUCES WRONG RESULTS WHEN DATA IS DELAYED

Use event time, set watermarks from observed lateness, and decide deliberately what happens to late records.

A DEAD LETTER DESTINATION IS ESSENTIAL

Without it, one unprocessable record blocks its partition indefinitely.

CONSUMER LAG IS THE DIRECT MEASURE OF WHETHER PROCESSING KEEPS UP

NEVER PUT SENSITIVE DATA IN AN EVENT THAT CONSUMERS DO NOT NEED

Events are retained and widely readable.

BATCH FOR MOST THINGS; STREAM ONLY WHERE LATENCY GENUINELY DEMANDS IT

And if both exist, check automatically that they agree — divergence destroys confidence in both.


Was this answer helpful?
Back

Are you happy with your experience? Leave us a review on Trustpilot.


Trustpilot