Executive Summary
Your data team's sprints often miss their goals. This can cause friction with the rest of the business and lead to your best people feeling burnt out.
It's easy to mistake activity for progress. If the underlying data architecture isn't sound, running two-week sprints just means you produce broken dashboards more quickly. This can frustrate your best engineers enough that they look for work elsewhere.
In my experience, a pure software-agile model doesn't quite fit data architecture work. A hybrid approach often works better: a more traditional, time-boxed project for foundational work, like defining metrics, and a flexible, Kanban-style flow for day-to-day reporting and analysis.
The tension between long-term roadmaps and urgent requests
If you're a Head of Data, you're likely familiar with a certain tension. The CEO and board want a 12-month roadmap with clear delivery dates. They want to see the big picture. At the same time, the Head of Product is messaging you for a quick analysis that could change priorities for next week. The business wants the stability of a long-term plan but the responsiveness of a more agile approach, and your team is often caught in the middle.
This is a situation I've seen in many companies, usually around the Series B to D stage. They've hired good people, invested in the modern data stack, and started using the language of software development: sprints, story points, stand-ups. The trouble is, they've often just moved the same chaotic processes into the cloud. Automating a messy process just means you get the wrong answers, faster. Trying to manage it with Scrum can give you a more organised view of the same problems every two weeks.
Why data engineering isn't the same as software engineering
The difficulty, I think, is that data work isn't quite the same as software engineering. A software engineer often builds a self-contained component, tests it in isolation, and then ships it. A data request is usually a different sort of thing. It's often more like archaeology.
Asking for what seems like a 'simple' change to a dashboard metric is rarely simple. It usually means an engineer has to trace the data's lineage back through a tangle of SQL scripts, undocumented dbt models, and source data of varying quality. What looks like a two-hour task can uncover an underlying problem in how you define a customer, which can then turn into a week-long investigation. This unpredictability is what makes sprint commitments so difficult to meet. It's not really a failure of estimation. It suggests the model itself might not be the right fit.
When you try to force this investigative work into a rigid sprint, you can create a Data Engineering Bottleneck. Engineers might cut corners to meet a deadline, which adds to the technical debt. Or, they miss the deadline, which can erode trust with stakeholders. The constant switching between deep, foundational work and the stream of ad-hoc reporting requests is a common cause of burnout.
A hybrid approach: separating foundational work from daily requests
What I've found works well is to stop trying to apply a single methodology to all data work. It's often better to separate the work into two different streams, each with its own way of working. This is an important part of a good Data Team Strategy.
#### 1. Foundational work as a distinct project
This is the less glamorous, but essential work that no one really wants to budget for, but whose absence causes problems for everyone. It includes things like defining your core business entities, establishing a Single Source of Truth, and building a properly governed Semantic Layer. This kind of work doesn't fit neatly into sprints. It's better treated as a capital project. You need to set aside dedicated time for it, perhaps a full quarter, where the team's main goal is to build that stable foundation. It needs a clear, top-down plan with as few interruptions as possible. It's a project, not a series of sprints.
#### 2. Day-to-day requests managed with Kanban
Once that foundation is in place, you can handle the final steps of analysis and reporting with a more flexible system. A Kanban board showing the flow of requests, for example from 'Backlog' to 'In Progress', 'In Review', and 'Done', is a good fit here. It lets the business see what's being worked on and what the priorities are, without the pressure of a fixed sprint commitment. It accepts that a request might be quick or it might require a deeper investigation, and it helps the team manage their capacity. This helps build trust, as you aren't making promises that are difficult to keep.
Getting agreement for this change
The main challenge in my experience isn't technical, it's about managing expectations. You'll need to be able to explain to stakeholders, "We will be delivering fewer dashboards for the next three months." You have to explain that you are pausing the delivery of new reports to rebuild the foundations. You can expect some pushback.
The trade-off is this: you have to move a bit slower for one quarter so that you can move much faster for the next three years. It takes a bit of courage and very clear communication, but it's often the only way I've seen to escape the cycle of missed deadlines and build a data function that provides real value.
The goal here is a data team that is calm, focused, and trusted. The business gets a clearer understanding of what is a major architectural project versus a quick analytical query. And you, as the Head of Data, can spend less time apologising for missed deadlines and more time delivering insights that make a real difference.