From Flag to Decision: Engineering and Running A/B Tests on Android You Can Actually Trust

Adding an A/B testing SDK to your app takes an afternoon. Shipping a decision you can defend takes two skills most teams only half-learn: engineering the flag infrastructure so it produces honest data, and designing and reading the experiment so the data means something. Miss either half and the failure is silent — no crash, no exception, just a dashboard that confidently tells you to ship the wrong thing. This session walks the full lifecycle of an experiment on Android, drawn from production patterns in a travel app used by millions of travelers monthly, where every feature ships behind a flag. In one continuous arc we'll cover both halves: Running it: - Designing a test that can actually answer your question: IF/THEN/BECAUSE hypotheses, one primary metric, guardrails, and a pre-committed duration - Reading results with confidence: significance in plain language, mechanism metrics that explain what the headline number can't, and the peeking trap Engineering it: - One experimentation abstraction over Statsig: type-safe keys, a single injected interface, gates vs. experiments behind one call site - Cold start done right: async init and deterministic defaults, so the UI never blocks or flickers between variants - Resolving variations once in your MVI reducer / ViewModel — and exposure logging at the moment of view, not assignment - The debt side: testing every arm, QA overrides, and deleting dead flags before they rot your codebase Woven through the whole session: I'll walk you through several pain points every experimenting team eventually hits (underpowered tests, peeking, variant flicker, diluted exposure, sample ratio mismatch, colliding experiments, and more), candid takeaways from years of running experiments in production, and a take-home checklist of the practices that make an A/B test succeed - before you build, while you build, while it runs, and after it ends. No prior experimentation experience needed. If you've never run an A/B test, this session is designed as your on-ramp; if you run them every week, you'll recognize the scars and leave with an architecture you can implement next sprint. Key learning points 1. Design a trustworthy experiment before writing any code: an IF/THEN/BECAUSE hypothesis, one primary metric, guardrail metrics that protect what you're not trying to move, and a pre-committed duration. 2. Architect a single experimentation abstraction over any SDK (Statsig used for concreteness): type-safe feature keys, one injected interface, gates vs. experiments behind one call site — testable, vendor-portable, and with one place for policy like defaults and kill switches. 3. Handle the mobile-specific engineering edges: async cold-start init with deterministic default-to-Control (no blocking, no flicker, no mid-session variant swaps), variations resolved exactly once in the MVI reducer and driven through immutable state into Compose. 4. Log exposure honestly — assignment is not exposure: manual exposure events for below-the-fold and conditional components, and how getting this wrong dilutes real effects until a harmful variant reads as "neutral, ship it." 5. Read results and close the loop: what statistical significance means in plain language, why mechanism metrics explain what the primary metric can't, the peeking trap that flips decisions, and the cleanup ritual — test every arm, own every flag, delete the loser the week you decide.