Dagster Labs

I immediately became uv-pilled once I have it a try and it's been central to any python project I do these days. With that being said the dg cli is a vastly more ergonomic experience with building out Dagster projects and doing local development.

I write a lot about what makes Dagster great but I am not afraid to tell you why you shouldn't use it. Here's five reasons you should not use Dagster: 1. You are happy with cron. If it works, don't touch it. If you aren't being hit by dependency issues, or having to debug failing jobs, or wishing for more observability, then cron is great! I love cron! Some of my best friends use cron! 2. You don't have a data platform owner. If you don't have someone invested in creating and building a data platform, then Dagster isn't the right fit. Dagster won't magically fix your people or process problems. You still need to implement it and roll it out to the team like any other tool. 3. Your team isn't comfortable with version control. Dagster was built with software engineering best practices in mind. That means it relies on version control for pipelines as code. If your team can only operate point-and-click ETL tools then they're not ready for Dagster. 4. You need an interactive workflow, rather than one based on schedules and jobs. If you are building your pipelines in a Jupyter Notebook and want to examine the results as each cell runs, then that might be just fine! Fire up your notebook and run it. 5. You don't care about data. Dagster's operating model is centered around data. We're focused on making data a first-class citizen in orchestration through lineage, metadata, insights, and data quality checks. If you don't care about data, you won't really appreciate Dagster.

Simplifying Dagster development with our new dg CLI + uv We've just released dg, our new CLI tool that makes getting started with Dagster faster and more intuitive. By integrating uv, the high-performance Python package manager, we're streamlining project scaffolding and dependency management for data engineers. The combination tackles one of Python's long-standing pain points while giving you a solid foundation for building data assets and pipelines. Setting up complex, collaborative data projects just got a whole lot easier. See how it works! Link in the comments.

Terrible docs kill good tools. Pedram Navid, Head of Data Eng + DevRel at Dagster Labs, walked us through how they rebuilt Dagster’s docs from 0 to intuitive, searchable + actually helpful. Full talk being posted to datacouncil.ai and youtube following the conference!

Terrible docs kill good tools. Pedram Navid, Head of Data Eng + DevRel at Dagster Labs, walked us through how they rebuilt Dagster’s docs from 0 to intuitive, searchable + actually helpful. Full talk being posted to datacouncil.ai and youtube following the conference!

Stuck with outdated Airflow DAGs? Our new dagster-airflow library makes migration a breeze! One team migrated 73 Apache Airflow DAGs to Dagster in just two weeks! With Airlift, you can now: - View your Airflow instance within Dagster. - Map DAGs to full lineage assets. - Migrate execution step-by-step. - Decommission your legacy systems. Why switch? Dagster focuses on data assets, not just tasks, giving you better visibility into data lineage and dependencies and our branch deployments mean you'll never have to "test in production" again. Ready to accelerate your data team's velocity? Check out our complete migration guide today.

🚨 New Video 🚨 I created a tutorial on how to build Custom Dagster Components using a text extraction PDF pipeline as an example. The power of components is that data platform owners can embed organizational context and abstract away complexity for less technical stakeholders. So they can unblock themselves and contribute to their data platform with a simplified interface while still adhering to software engineering and Dagster best practices. Check it out today! Link in the comments.

Our Head of Engineering Eric Thanenthiran ran a great session at PyCon DE & PyData yesterday -- showing you how to build modern data platforms with open source Python tools. Here's a summary, and we will share the recording in the comments once it's available so bookmark this post! Key components of the modern stack are going to be discussed in the session: 👉 Sources: Connecting to APIs and files with declarative pipelines 👉 Pipelines: Moving data efficiently with tools like dltHub 👉 Data Stores: Centralising data in fast, flexible stores like DuckDB 👉 Transformation: Turning raw data into insight with dbt Labs 👉 Orchestration: Coordinating workflows with Dagster Labs 👉 Visualisation: Building interactive dashboards with Streamlit Practical Considerations When Implementing Open Source Stacks: 💰 Build vs. buy decisions remain crucial (sometimes you need that SLA guarantee) 🕳️ Understanding batch/micro-batch processing complexities is essential 📊 Most appropriate for gigabytes-scale data, not necessarily petabytes 👀 Always consider the right tool for each component rather than one-size-fits-all. We're really quite passionate about helping businesses navigate these practical, modern approaches to turn data chaos into business clarity -- so don't hesitate to reach out if that is of interest to you! #DataEngineering #PyCon #OpenSource #Python #DataStack #BusinessIntelligence #TasmanOnTour

"How we think about data orchestration needs to fundamentally change, and Dagster represents that shift in thinking." In this new post, from our Chief Dashboard Officer, Pedram Navid, he explores why modern data teams are moving beyond Airflow toward asset-centric orchestration. While others are busy implementing yesterday's innovations, we've been building the future: Low-code pipeline authoring that works with Components, rich data quality assertions, column-level lineage, and a unified data catalog that makes teams more productive. Don't just orchestrate tasks - empower your data platform.