Weekly discussion thread: Oct 13-Oct 19

Topic: Batch vs. streaming data pipelines: With the demand for more timely information, over 80% of data engineers will incorporate streaming data pipelines into their data stack by 2023. Do you agree or disagree and why? Reply to this thread to comment.

Cartoon Thinking GIF by SpongeBob SquarePants

That sounds like a pretty solid trend in next 5 years, with more and more realtime use cases emerging and customers are becoming less patient to wait for business insights longer than 30 minutes. However, 80% is something rather a ballpark estimate, which makes more sense if breaking down by industry to discuss.

1 Like

@boyang - what industries do you see the change happening more rapidly, where data engineers are incorporating streaming data pipelines?

I could name a few, for example:

  • Financial institutions need realtime streaming for fast anomaly detection
  • Internet advertisement streams in realtime auction results and billable user events to produce accurate budgeting and pacing data
  • Body training platform needs to have accurate trainee numbers to adjust the program workload to maximize effect
1 Like

@rogoben what do you think?