Chapter 6 · Product Analyst
6. Product sense & diagnosing metric changes
~8 min read
'Engagement dropped 15% week over week, what happened?' is the most common case question and a real Tuesday-morning task. Interviewers are watching for structure, not a lucky guess.
6.1 The diagnostic framework#
- Clarify and confirm. What exactly is the metric, over what window? Is the drop real, or a tracking bug? Rule out instrumentation first.
- Segment aggressively. Break by platform, channel, geo, app version, new vs returning. A change in one segment points at the cause.
- Internal vs. external. Did we ship a release or change pricing? Or is it seasonality, a competitor, an outage?
- Hypothesis + test. Land on the likeliest cause and say exactly what query you'd run to confirm.
6.2 Measuring a feature: the structure#
- Goal: what user problem and business objective does this serve?
- Primary metric: the one number that captures success.
- Secondary & guardrails: supporting signals and what must not break.
- Segments & timeframe: for whom, and over what horizon.
6.3 Second-order effects#
| A change that improves… | May quietly hurt… | Watch this counter-metric |
|---|---|---|
| Short-term engagement | Long-term retention | 30/90-day retention, churn |
| Conversion rate | Revenue per user | ARPU, AOV |
| Watch time / clicks | User satisfaction | CSAT, complaints, uninstalls |
| Acquisition volume | User quality | Activation & retention by channel |
Get the next chapter and weekly interview tips by email
One short email per week. Skim in a minute. Unsubscribe anytime.