Executive Summary
During your busiest periods, like Black Friday or the summer holidays, your main reports slow down, sometimes to the point of being unusable. This leaves your commercial teams without the data they need when it matters most.
When dashboards are down, it can lead to poorly priced products, campaigns that aren't optimised, and slow reactions to stock levels. This can cost you money and, over time, can mean the business has less confidence in the data team.
In my experience, this isn't something you can fix by simply increasing your Snowflake budget. It's usually a sign of an architectural problem. The solution is to adjust how your data models are built, moving away from a 'query everything at once' approach to one that's designed for efficiency at a larger scale.
How systems that once worked start to struggle with scale
In the early days, things probably worked quite well. Your dashboards were quick, your data models were straightforward, and the commercial team trusted the numbers. But as the business grew, so did the data. A hundred times more transactions, perhaps. The setup that served you well to begin with is now what's slowing you down.
The first sign of a problem is often a worried Slack message: "The main booking dashboard is down!" This usually isn't a one-off glitch, but a symptom of a deeper issue. The methods that worked for ten thousand bookings a day start to struggle with the load of ten million. It's a common problem for a business that's scaling up.
This is one of the most common Scale-up Data Challenges I see. Companies in high-volume sectors like travel and e-commerce invest in a modern data stack: Snowflake, dbt, Looker. The problem is that the logic running on top of those tools often doesn't evolve along with the business. You've made your existing processes run on a faster platform, but the underlying approach hasn't changed. Eventually, that catches up with you.
Why your modern data stack can feel slow
The problem isn't usually the tools themselves. It's more likely that your reporting is trying to do very complex calculations on raw, event-level data every time someone refreshes a page. Your dashboards are being asked to do an enormous amount of work each time they're opened.
I worked with a travel marketplace recently where their main performance dashboard took twenty minutes to load during the January sales peak, which meant it wasn't much use. The commercial team had to rely on guesswork. When we looked into it, the dashboard was trying to recalculate user-session data from billions of rows of raw clickstream events for every single query. It was an approach that just couldn't scale.
This is a classic sign that your team is dealing with a Data Engineering Bottleneck. Instead of building robust, long-term solutions, your best engineers spend their time firefighting inefficient queries and apologising for downtime. It's understandable that the business starts to lose confidence, not because the team isn't working hard, but because the system wasn't designed for the scale it now has to support.
Moving from raw queries to a more structured approach
To get out of this cycle, the team's focus often needs to shift. Instead of just answering one-off questions, the goal becomes building a reliable system that delivers timely information, no matter the load.
In my experience, the way to fix this involves three main ideas:
The trade-offs of fixing the problem
I should be honest: this isn't a quick fix. It's a proper project. It often means explaining to stakeholders that you need to pause work on new dashboards for a little while to improve the underlying engine. It can involve a bit of negotiation to get the time to pay down the technical debt that is causing problems for the business.
What good looks like afterwards
The result of this work is predictability. Your next peak trading season should be much calmer for the data platform. The dashboards are stable, fast, and trusted. Your team is no longer spending all its time firefighting and can focus on providing the kind of insights that help the business grow. You end up with a data setup that helps the business, rather than holding it back.