Topic: Batch vs. streaming data pipelines: With the demand for more timely information, over 80% of data engineers will incorporate streaming data pipelines into their data stack by 2023. Do you agree or disagree and why? Reply to this thread to comment.
That sounds like a pretty solid trend in next 5 years, with more and more realtime use cases emerging and customers are becoming less patient to wait for business insights longer than 30 minutes. However, 80% is something rather a ballpark estimate, which makes more sense if breaking down by industry to discuss.
1 Like
@boyang - what industries do you see the change happening more rapidly, where data engineers are incorporating streaming data pipelines?
I could name a few, for example:
- Financial institutions need realtime streaming for fast anomaly detection
- Internet advertisement streams in realtime auction results and billable user events to produce accurate budgeting and pacing data
- Body training platform needs to have accurate trainee numbers to adjust the program workload to maximize effect
1 Like