Running them reliably.
WHAT TO MONITOR
Consumer lag, per partition Throughput, in and out Error and retry rates State size, for stateful processing Retention against disk
WHY LAG MATTERS MOST
It is the direct measure of whether processing keeps up.
WHAT GROWING LAG INDICATES
Consumers too slow, or a partition skewed.
WHAT TO DO ABOUT IT
Add consumers, up to the partition count.
WHY THAT LIMIT
One partition is consumed by one consumer in a group.
WHAT THAT MEANS FOR DESIGN
Partition count sets the maximum parallelism, and increasing it later is disruptive.
WHAT A DEAD LETTER DESTINATION IS
Where records that repeatedly fail are sent.
WHY IT IS ESSENTIAL
Without it, one unprocessable record blocks the partition indefinitely.
WHAT TO DO WITH IT
Monitor it, and investigate.
WHAT TO PLAN
Reprocessing: replaying from a point after fixing a bug.
WHAT THAT REQUIRES
Retention long enough, and consumers that can be reset.
WHAT TO TEST
Consumer restart Platform node failure A poison record
