Product Analytics: Why Your User Data is a Liability

Executive Summary

---

When the numbers you present are met with silence

I'm sure you've been in the meeting. You present a chart showing a 15% adoption for a new clinical workflow feature. You're about to suggest the next phase of work when the CTO says, "That number doesn't feel right. The backend logs seem to be showing something quite different."

The confidence in the room just drains away. The conversation grinds to a halt. Your roadmap is now based on feelings, and your own credibility has taken a knock. This isn't a personal failing: it's what happens when a data system is broken. For Product Managers in sensitive areas like HealthTech, this situation is all too common. You've likely spent a good deal on a modern data stack, perhaps Snowflake, dbt, Looker, or Amplitude, but find you've only managed to automate the confusion. Automating a broken process just means you get the wrong answers faster.

I've seen this happen quite a bit in scale-ups, typically around Series B to D. The push to ship features and show growth often gets ahead of the discipline needed to track them properly. The result is a tangle of data that seems to cause more arguments than it settles.

Why the product metrics don't seem to add up

The problem usually isn't the dashboard itself. It's that your definition of 'user engagement' hasn't been properly agreed upon, and it's being tracked inconsistently. In my experience, the root of the problem is usually in how the Product Analytics was first set up.

Here’s what has probably happened:

Inconsistent event tracking: Your client-side analytics, what the user clicks in the app, and your server-side database, which should be the real record, are telling two different stories. This is common when the tracking logic isn't kept up to date across different app versions or platforms.

Ambiguous definitions: What makes an "active user"? Is it a login? Viewing a page? Completing a key action? Without a single, written-down definition, every team, from Product to Marketing to Finance, will have a slightly different, self-serving answer. This is one of the main reasons for Data Trust Issues.

Technical debt in the analytics: No one really budgeted time to maintain the analytics setup. I once looked at a product analytics system where 60% of the tracked events were for features that no longer existed, yet they were still being used in core KPI calculations. The system was creating more confusion than clarity.

A way to fix the foundations: from untrusted events to reliable insights

This isn't a problem you can patch with another dashboard or by hiring more analysts. It's about rebuilding the foundations so that data is trusted by default.

Get everyone to agree on the key metrics: The first step is about people, not technology. You need to get the leads from Product, Engineering, and Commercial in a room and have them agree on a single, written Metric Definition for the 10-15 KPIs that really run the business. It can be a painful meeting, but it's the foundation for getting data you can trust.

Define your business logic in one place: The rules that define your metrics should not live in the settings of a BI tool. They need to be written in code in your data transformation layer, for example, using dbt. This centralisation means that whether you're looking at a report in Looker or a CSV file, the definition of User Retention is the same everywhere. This is how you build a single source of truth that everyone can rely on.

Introduce some light governance: Governance doesn't have to mean bureaucracy: think of it as quality control. In a HealthTech setting, some sensible HealthTech Data Governance is essential. This could be as simple as adding automated checks to your deployment pipeline that validate tracking schemas before they go into production. This stops inconsistent tracking from ever getting out into the wild.

This is more of a political problem than a technical one

Putting this fix in place means having some difficult conversations. You're effectively telling teams that the numbers they've relied on might not be right. You're asking engineers to add what they might see as extra process to their work. It's sensible to expect a bit of resistance.

It often means slowing down for a quarter to be able to move much faster for the next few years. It's an investment in the speed of your decision-making. Replacing a department's trusted, if slightly creaky, Google Sheet is a political act. The goal is to give them an automated, reliable source of truth in its place. They'll likely thank you for it later, but you should be prepared for some debate now.

What you're working towards

The result of this work is clarity and confidence. It's being able to walk into any meeting and show a Feature Retention chart that everyone in the company trusts. It means you can stop having the same arguments about whose numbers are right, and start making good, quick decisions that move the product forward. You stop guessing at user behaviour and start truly understanding it.