Experimentation Playbook

Build Products With Confidence. Make Decisions With Evidence!

The Experimentation Playbook Collection is a 6-part guide designed for Product Managers, Data Scientists, Analysts, and Engineers who want to make confident, evidence-based product decisions.

Created by Manisha Arora (Google) and Banani Mohapatra (Walmart), this collection walks you through the full experimentation lifecycle—how to design clean tests, run them reliably in production, analyze outcomes correctly, navigate common pitfalls, and learn from real industry case studies.

Whether you’re testing UI changes, ranking models, onboarding flows, or pricing strategies, this playbook gives you a complete, practical system to build products with clarity and scientific rigor.

What’s inside the 6 parts:

Part 1: A modern intro to experimentation and the experimentation lifecycle

Part 2: How to design clean experiments—hypotheses, metrics, test units, MDE, sample sizing

Part 3: How to run experiments in production—logging checks, assignment consistency, avoiding contamination & SRM

Part 4: How to analyze results—p-values, CIs, effect sizes, practical vs statistical significance

Part 5: Challenges—noise, errors, bias, risk, and real-world pitfalls

Part 6: Case studies from Google, Bing & Meta, connecting theory to real product decisions

Ideal for anyone shaping product outcomes or making data-backed decisions.

Level up your experimentation game — get it today!