Architectures using both.
WHAT THE TENSION IS
Streaming provides freshness; batch provides correctness and simplicity.
WHAT THE LAMBDA APPROACH WAS
Running both, with a batch layer producing authoritative results and a streaming layer providing recent approximations.
WHAT IT COSTS
Logic implemented twice, in two systems, which diverge.
WHAT THE KAPPA APPROACH PROPOSES
One streaming path, with reprocessing by replaying history.
WHAT IT REQUIRES
Retention of all events, and the ability to replay at scale.
WHAT MOST ORGANISATIONS ACTUALLY DO
Batch for the majority, with streaming only where latency demands it.
WHY THAT IS SENSIBLE
It confines complexity to where it is justified.
WHAT TO STREAM
Operational alerting Fraud and abuse detection Live dashboards where decisions depend on them Feeding operational systems
WHAT TO BATCH
Reporting Financial reconciliation Model training data Anything reviewed rather than reacted to
WHAT TO ENSURE IF BOTH EXIST
That they agree, checked automatically.
WHY
Divergence between a live dashboard and a daily report destroys confidence in both.
