Why ELT Can't Keep Up in the Era of High-Scale Data Engineering

I've watched batch ELT hit its ceiling repeatedly across enterprises. The real pain emerges not when data volume grows—it's when business teams demand answers within hours, not days. Confluent's argument rings true: batch windows create stale copies that multiply storage costs and fragment truth across systems. But I'd push back on positioning this as purely a streaming versus batch problem. The issue is architectural laziness. Most organizations implement streaming as a bolt-on, leaving batch infrastructure untouched. True scale requires rethinking data freshness requirements per use case. Some analytics genuinely don't need real-time; others absolutely do. I've found success building tiered pipelines: streaming for operational decisions (fraud, personalization), batch for historical analytics. The governance gap Confluent mentions is the real killer—without proper metadata and lineage tracking, faster data just means faster propagation of bad decisions. My recommendation: audit your pipelines by decision latency requirements, not by technology preference. Then architect accordingly. Kafka becomes valuable only when it solves a specific freshness problem, not as a status symbol.