Dagster Labs

It’s a great feeling when I find out an organization I love is using the open source version of the software I sell (paid too, but I usually know about those already!). Today’s discovery: Wikimedia!

I'm seeing a lot of folks talking about developers getting LLM psychosis from their heavy Claude code usage. Personally, it has changed how I develop and dramatically increased my velocity. But as many of the greybeards have stated, writing code was never really the hard part of building software. Standards, architecture, security, scalability, enabling collaboration, and documentation are only going to be more important as more organizations have tech debt superspreaders unleashed on their code bases. At Dagster, we have been heavily experimenting and investing in ways to leverage the power of LLMs while still adhering to sound software engineering best practices. We'll be hosting a webinar on January 27th to discuss and demo the best practices for using LLMs to develop with Dagster! Register today. Link in the comments.

You need to answer a question about pipeline, ad spend, or what's happening in your sales calls. The in app reporting for many SAAS tools are woefully inadequate unless you want to answer simple questions about the tools current state. You want to be self-sufficient. You're comfortable with data, you just don't have direct access to it in a way that's actually useful. Compass now connects directly to Salesforce, Google Ads, and Gong. Ask questions about your GTM data in Slack. Get answers in seconds. No more waiting on someone else to pull numbers you could analyze yourself. No more exporting CSVs just to ask a follow-up question. No more rebuilding dashboards every time you want to slice things differently. Your data team still owns the definitions and governance. You just get to finally use the data without the bottleneck.

LLMs don't know your codebase. They don't know Dagster's idioms. They don't know your pipeline conventions. And that disconnect is why so many engineers try AI-assisted development, get frustrated, and walk away, leaving massive productivity gains on the table. The abstractions have come a long way since 2023. Planning, sub-agents, skills, MCP. Tools like Claude Code and Cursor have changed what's possible. But without the right approach, you're just generating code that doesn't fit. Our DevRel team has been building infrastructure to close that gap. Join Alexander Noonan, Colton P., and Dennis Hume for a practical session on what actually works: → Best practices for LLM development in data engineering → Common pitfalls that derail productivity → Live demos of effective workflows 📅 January 27 | 12:00 PM ET Register today! Link in the comments

Conventional wisdom says AI is flooding us with slop. But median code was already slop: After all LLMs are producing code that looks like what it was trained on. Yes, used carelessly, you get vibe-coded, buggy software churned out by people who don't know what they're doing. What the slop pessimists miss is what happens when someone who actually cares about craft uses it. Here's a subtle example that's changed how I work: When I'm building my own tools, I constantly notice small things. A confusing error message. A formatting inconsistency. A sharp corner in the UX that could be smoother. Before, I'd try to write these down. Maybe I'd batch them up and fix them later. But the activation energy was high. You lose context. The list grows. Things slip through. Now the calculus is completely different. The moment I notice anything, I open a new Claude Code session. I describe the problem. It creates a plan. I look at it, nod, and launch a Github Action (our internal tools can do this easily) to code to address it. I move on and I’ll review and merge it the next time I have a minute. The cost of fixing has dropped so dramatically that I've shipped more PRs on my internal tooling in the past few months than I would have thought possible. Many of them are tiny improvements I never would have bothered with before. Individually these fixes seem trivial, but added together it makes a huge difference to the user experience. There's an old productivity framework: Do it, Defer it, Delegate it, or Drop it. What AI does is systematically move an entire class of work from "defer" and "delegate" into "just do it" because doing it is now nearly free, as long as you have the right workflow. The result isn't sloppier software. It's software where someone who cares can actually act on that caring, in the moment, without friction.

We use Dagster to run Dagster. And now we're using Compass to understand what's happening across our Dagster+ platform, directly in Slack. Coming soon to Dagster+ users. Dagster+ makes it easy to orient yourself around your data platform's state and stay ahead of anomalies. Pairing it with Compass dramatically speeds up debugging those complex pipeline failures: diagnose issues, check platform health, and spot patterns worth fixing, all without leaving Slack. Want to go deeper? Our Analytics Lead Anil Maharjan is doing a Deep Dive on Feb 17 where he walks through real workflows and shows how Compass identified the root cause of a pipeline that was failing 40-50% of the time. Link in the comments

We're hiring 👀 One of my favorite parts of working at Dagster is talking to customers every day. They’re building incredible things, they push our thinking, and they often make my day. We’re looking for a PM to help Dagster+ grow and scale alongside our enterprise customers. You’ll spend a lot of time with customers and customer-facing teams: learning how they work, what they need, and turning that into what we build next as adoption expands across complex orgs. Come work with us! 👉 https://lnkd.in/gdtFAJvU DM me if you’re curious (or if you have someone great to refer) 🙏

The running joke in data engineering is that data engineers don't look at the data. Data analysts and other stakeholders who use the data to deliver value are rightly frustrated when the infrastructure they need to do their jobs is generating incorrect data that doesn't show up on time. Some teams are stuck in a reactive cycle of implementing data quality checks reactively after an issue has been surfaced. This process is suboptimal as erodes trust in your data platform and keeps the data engineering team in a reactive workflow. Alexander Noonan wrote a practical guide and example Dagster project for enforcing data quality at every stage of your pipeline, so you catch issues early instead of cleaning up messes later. Check it out today!