11. Worked case studies

11.1 'DAU dropped 12% on Tuesday'#

1. Clarify & validate. Confirm DAU definition and window. Is this real or a tracking issue? Check the latest release and pipeline.
2. Segment. Break by platform, app version, geo, new vs returning. Suppose the drop is on iOS 5.3.
3. Internal vs. external. Version-correlated and platform-specific → strongly internal. Likely a 5.3 iOS bug or logging regression.
4. Hypothesis & test. Compare event-firing rates for key actions between 5.2 and 5.3 on iOS; check crash rates. Recommend rollback or hotfix.

11.2 Design a metric for a new Stories feature#

1. Goal. Deepen daily engagement.
2. Primary. Share of DAU who view at least one Story per day.
3. Secondary. Stories viewed per viewer, creation rate, completion rate, session length.
4. Guardrails. Core feed engagement and overall retention must not fall; watch complaint rates.
5. Segments & horizon. Creators vs viewers, new vs existing; measure over weeks to rule out novelty.

11.3 'Should we ship this experiment?'#

Prompt: A test shows the variant lifted signup-to-activation by 1.5%, p = 0.04, after 4 days. Ship it?

• Duration & peeking: 4 days may not cover a full weekly cycle; was sample size fixed in advance?
• Practical significance: is 1.5% meaningful, and what's the CI, could the true effect be near zero?
• Guardrails: did retention, revenue, or latency move the wrong way?
• Novelty: is the lift fading day over day?

Recommendation: don't ship on 4 days alone. Run to the pre-planned sample size and a full cycle, then decide.

11.4 'Why is week-1 retention falling?'#

1. Cohort the decline. Plot by signup-week to see slide vs cliff.
2. Segment by channel. A retention drop is often a mix-shift (Simpson's paradox). Check retention within each channel.
3. Inspect activation. If retention fell within channels, look at the onboarding funnel and activation rate.
4. Synthesize. 'Roughly 70% mix-shift from the new campaign; 30% a real activation dip after the onboarding redesign.' Recommend a targeted fix for each.