Jeff Meyerson talks with Frances Perry about Apache Beam, a unified batch and stream processing model. Topics include a history of batch and stream processing, from MapReduce to the Lambda Architecture to the more recent Dataflow model, originally defined in a Google paper. Dataflow overcomes the problem of event time skew by using watermarks and other methods discussed between Jeff and Frances. Apache Beam defines a way for users to define their pipelines in a way that is agnostic of the underlying execution engine, similar to how SQL provides a unified language for databases. This seeks to solve the churn and repeated work that has occurred in the rapidly evolving stream processing ecosystem.
📆 2016-10-11 18:44 / ⌛ 00:52:51
📆 2016-10-11 18:31 / ⌛ 00:52:51
📆 2016-10-04 23:42 / ⌛ 00:51:51
📆 2016-10-04 23:32 / ⌛ 00:51:51
📆 2016-09-27 20:13 / ⌛ 00:52:08