The world beyond batch: Streaming 101

A high-level tour of modern data-processing concepts.

By Tyler Akidau

August 5,2015

Three women wading in a stream gathering leeches (source: Wellcome Library,London).

Editor's note: This is the first post in a two-part series about the evolution of data processing,with a focus on streaming systems,unbounded data sets,and the future of big data. See part two.

Streaming data processing is a big deal in big data these days,and for good reasons. Amongst them:

• Businesses crave ever more timely data,and switching to streaming is a good way to achieve lower latency.
• The massive,unbounded data sets that are increasingly common in modern business are more easily tamed using a system designed for such never-ending volumes of data.
• Processing data as they arrive spreads workloads out more evenly over time,yielding more consistent and predictable consumption of resources.

Despite this business-driven surge of interest in streaming,the majority of streaming systems in existence remain relatively immature compared to their batch brethren,which has resulted in a lot of exciting,active development in the space recently.

Get O'Reilly's weekly data newsletter

As someone who’s worked on massive-scale streaming systems at Google for the last five+ years (MillWheel,Cloud Dataflow),I’m delighted by this streaming zeitgeist,to say the least. I’m also interested in making sure that folks understand everything that streaming systems are capable of and how they are best put to use,particularly given the semantic gap that remains between most existing batch and streaming systems. To that end,the fine folks at O’Reilly have invited me to contribute a written rendition of my Say Goodbye to Batch talk from Strata + Hadoop World London 2015. Since I have quite a bit to cover,I’ll be splitting this across two separate posts:

1. Streaming 101: This first post will cover some basic background information and clarify some terminology before diving into details about time domains and a high-level overview of common approaches to data processing,both batch and streaming.
2. The Dataflow Model: The second post will consist primarily of a whirlwind tour of the unified batch + streaming model used by Cloud Dataflow,facilitated by a concrete example applied across a diverse set of use cases. After that,I’ll conclude with a brief semantic comparison of existing batch and streaming systems.

So,long-winded introductions out of the way,let’s get nerdy.

Background

To begin with,I’ll cover some important background information that will help frame the rest of the topics I want to discuss. We’ll do this in three specific sections:

• Terminology: To talk precisely about complex topics requires precise definitions of terms. For some terms that have overloaded interpretations in current use,I’ll try to nail down exactly what I mean when I say them.
• Capabilities: I’ll remark on the oft-perceived shortcomings of streaming systems. I’ll also propose the frame of mind that I believe data processing system builders need to adopt in order to address the needs of modern data consumers going forward.
• Time domains: I’ll introduce the two primary domains of time that are relevant in data processing,show how they relate,and point out some of the difficulties these two domains impose.

Terminology: What is streaming?

Before going any further,I’d like to get one thing out of the way: what is streaming? The term “streaming” is used today to mean a variety of different things (and for simplicity,I’ve been using it somewhat loosely up until now),which can lead to misunderstandings about what streaming really is,or what streaming systems are actually capable of. As such,I would prefer to define the term somewhat precisely.

The crux of the problem is that many things that ought to be described by what they are (e.g.,unbounded data processing,approximate results,etc.),have come to be described colloquially by how they historically have been accomplished (i.e.,via streaming execution engines). This lack of precision in terminology clouds what streaming really means,and in some cases,burdens streaming systems themselves with the implication that their capabilities are limited to characteristics frequently described as “streaming,” such as approximate or speculative results. Given that well-designed streaming systems are just as capable (technically more so) of producing correct,consistent,repeatable results as any existing batch engine,I prefer to isolate the term streaming to a very specific meaning: a type of data processing engine that is designed with infinite data sets in mind. Nothing more. (For completeness,it’s perhaps worth calling out that this definition includes both true streaming and micro-batch implementations.)

As to other common uses of “streaming,” here are a few that I hear regularly,each presented with the more precise,descriptive terms that I suggest we as a community should try to adopt:

Get O'Reilly's weekly data newsletter

1. Unbounded data: A type of ever-growing,essentially infinite data set. These are often referred to as “streaming data.” However,the terms streaming or batch are problematic when applied to data sets,because as noted above,they imply the use of a certain type of execution engine for processing those data sets. The key distinction between the two types of data sets in question is,in reality,their finiteness,and it’s thus preferable to characterize them by terms that capture this distinction. As such,I will refer to infinite “streaming” data sets as unbounded data,and finite “batch” data sets as bounded data.
2. Unbounded data processing: An ongoing mode of data processing,applied to the aforementioned type of unbounded data. As much as I personally like the use of the term streaming to describe this type of data processing,its use in this context again implies the employment of a streaming execution engine,which is at best misleading; repeated runs of batch engines have been used to process unbounded data since batch systems were first conceived (and conversely,well-designed streaming systems are more than capable of handling “batch” workloads over bounded data). As such,for the sake of clarity,I will simply refer to this as unbounded data processing.
3. Low-latency,approximate,and/or speculative results: These types of results are most often associated with streaming engines. The fact that batch systems have traditionally not been designed with low-latency or speculative results in mind is a historical artifact,and nothing more. And of course,batch engines are perfectly capable of producing approximate results if instructed to. Thus,as with the terms above,it’s far better describing these results as what they are (low-latency,and/or speculative) than by how they have historically been manifested (via streaming engines).

From here on out,any time I use the term “streaming,” you can safely assume I mean an execution engine designed for unbounded data sets,and nothing more. When I mean any of the other terms above,I will explicitly say unbounded data,or low-latency / approximate / speculative results. These are the terms we’ve adopted within Cloud Dataflow,and I encourage others to take a similar stance.

On the greatly exaggerated limitations of streaming

（编辑：安卓应用网_ASP源码网）

【声明】本站内容均来自网络，其相关言论仅代表作者个人观点，不代表本站立场。若无意侵犯到您的权利，请及时与联系站长删除相关内容!