Data Engineering Bottleneck: The Ad-Hoc Reporting Trap

Executive Summary

---

How the ad-hoc model starts

In the early days of a company, it feels good to be the central point for all data questions. Your team are the go-to people for numbers, the ones who help power decisions. Every Slack message asking for a quick number is a sign your team is valued. You hire smart, helpful people, and they build a reputation for getting things done.

This model works very well, right up until the moment it doesn't. The success that made your team essential is now the thing holding it back. As the business scales, the volume of questions grows quickly. Your team's capacity, however, does not.

Why the ad-hoc model stops scaling

You've likely invested in a modern data stack, with tools like Snowflake, dbt, and a BI tool like Looker or Lightdash. The risk is that you've just found a faster way to manage the same messy process. You end up with a faster way to answer random questions, while the more important work gets pushed back.

This is a classic Data Engineering Bottleneck. In my experience, senior engineers who should be building your core data infrastructure end up spending their days writing one-off SQL scripts to find out why student engagement dipped last Tuesday. The business sees a queue of tickets. You see a real waste of your team's skills and a risk to your roadmap.

I’ve seen this pattern at quite a few EdTech companies and other businesses with high transaction volumes. At one scale-up, we found the data team was spending 60% of its time on questions that stakeholders could, and should, have been able to answer themselves. We set up a 'Self-Serve Hub' that cut down the 'Can you just pull this number?' messages on Slack by half in one quarter. This freed up the core team to focus on building new data products.

Moving from a service desk to a product team

To get out of this situation, you have to change how the team operates. The goal isn't to answer questions faster. It's to build a system where most questions don't need to be asked of your team in the first place. The solution is to Reduce Data Tickets by treating your internal data platform as a product, not a service desk.

Step 1: Build the core dashboards. First, you need to pause and take stock. If you do an honest audit of all incoming requests, you'll probably find that 80% of the questions come from 20% of the business areas. Your first job is to build a solid set of core management dashboards that answers these common queries, and that everyone agrees on. This becomes the single source of truth.

Step 2: Centralise your business logic. Often, Self-serve Analytics fails because people don't trust the data. If users can't find what they need or don't believe the numbers, they'll go back to asking your team directly. There are two parts to fixing this. First, you use a Semantic Layer where all your business logic is written down, version-controlled, and easy to see. No more SQL scripts hidden on someone's laptop. Second, you create a central 'Self-Serve Hub' in a tool like Notion or Confluence. This is a permanent, documented place with 'how-to' guides that becomes the first place people go for data.

Step 3: Find and train champions in other teams. It's usually better to stop running generic training sessions. Instead, try to identify the 'data-curious' person in each department, the one who is always asking the most thoughtful questions. Make them the data expert for their department. Train them on the data they use every day, working backwards from their most common reports so they properly understand the logic. They can then become the first point of contact for their own teams.

The difficulty of making the change

I should be direct: this transition isn't always easy. It can be a bit of a culture shock. Your team is used to being helpful and saying 'yes'. You now need to teach them to say, "Have you checked the Self-Serve Hub first?".

Stakeholders who are used to a personal, immediate service might not like it at first. They may complain that you are slowing them down. You have to be ready to explain that you are moving a bit slower for one quarter so the entire business can move faster for the next three years. This is as much a project about managing people as it is about data architecture. You are not just managing data; you are managing expectations and changing habits across the company.

What the new model looks like in practice

Once the changes bed in, the result is significant. Your team is no longer a reactive service desk but the respected team that builds and looks after the company's core data assets. The ad-hoc queue gets smaller, replaced by a manageable list of high-impact projects. Your engineers are more engaged, your colleagues in other teams feel more in control, and the business can move faster, confident in the data they use every day. You're no longer managing a bottleneck; you're helping the business to scale.