Blog / Metrics & Learning Products
The Metric We Almost Shipped the Wrong Feature For
The dashboard said engagement was up and to the right. A smaller, uglier number, buried two tabs deeper, told us we were about to build the wrong thing.
The following is a composite of a pattern I've seen play out more than once in learning products, told as a single story for clarity — the specific numbers are illustrative, but the shape of the mistake is very real.
A team I was close to was reviewing a quarter of strong-looking numbers. Time in app was up 22%. Daily active users was up. Session count per user was up. Someone on the team, reasonably, proposed doubling down: build a bigger version of the feature driving those numbers — a daily streak system with more elaborate rewards — since it was clearly working.
The number that didn't match the story
Before greenlighting it, someone pulled a different number: lesson completion rate, broken out by how long a user had been on the app. It told a much less flattering story. Users in their first two weeks were finishing lessons at roughly the same rate as before. The extra time in the app wasn't going toward learning — it was going toward checking the streak, tapping through the reward animation, and closing the app again.
Time in app had gone up. Engagement, in the sense that actually mattered for a learning product, hadn't moved at all.
Why the flattering number won the room at first
Time in app and DAU are easy numbers to love. They're big, they're simple to explain in a slide, and in most product reviews they trend the "right" direction often enough that nobody questions them by default. They also measure attention, which is not the same thing as value — and for a product whose entire promise is "help you learn something," attention without learning is close to the least useful kind of growth available.
A number going up and to the right feels like evidence. Sometimes it's just a receipt for where people's attention went, with no information at all about whether that was good for them.
What changed once the real number was visible
The streak feature didn't get scrapped — it stayed, because a light habit-forming layer isn't inherently bad. But it stopped being the quarter's big bet. Instead, the team looked at why lesson completion wasn't moving despite all that extra attention, and found a much less exciting-sounding culprit: the practice sets after each lesson were too long for a mobile session, so people were opening lessons, glancing at them, and leaving without finishing — which the "time in app" number couldn't distinguish from real engagement at all.
Shortening those practice sets moved lesson completion in a way the streak redesign never would have, because it fixed the actual mechanism, not a number that merely correlated with it.
The question worth asking before you trust a good-looking metric
Not "is this number going up," but "what would have to be true about user behavior for this number to go up for a bad reason?" If you can construct even one plausible bad-reason story — and for time-in-app and DAU, you almost always can — that's your cue to go find the less flattering number sitting one layer underneath, before you commit a quarter to the metric on the slide.
If you've got a dashboard in front of you right now and a nagging feeling it's not telling you the whole story, that's a genuinely good use of an hour with someone who isn't attached to the number already.
Stuck on something like this yourself?
I take on a small number of 1:1 calls, career conversations, and resume reviews each month.
See how we could work together