Digging Into Dagster: An Opinionated Open Source Framework For Data Orchestration

The Python Podcast.__init__

• Episode: Digging Into Dagster: An Opinionated Open Source Framework For Data Orchestration
• Website: The Python Podcast.__init__
• Feed URL: https://www.podcastinit.com/feed/mp3/
• Duration: 00:59:28
• Published: 2020-09-08 00:35

Data applications are complex and continually evolving, often requiring collaboration across multiple teams. In order to keep everyone on the same page a high level abstraction is needed to facilitate a cross-cutting view of the data orchestration across integration, transformation, analytics, and machine learning. Dagster is an innovative new framework that leans on the power and flexibility of Python to provide an extensible interface to the complete lifecycle of data projects. In this episode Nick Schrock explains how he designed the Dagster project to allow for integration with the entire data ecosystem while providing an opinionated structure for connecting the different stages of computation. He also discusses how he is working to grow an open ecosystem around the Dagster project, and his thoughts on building a sustainable business on top of it without compromising the integrity of the community. This was a great conversation about playing the long game when building a business while providing a valuable utility to a complex problem domain.