Experimentation Velocity: How to Run 100+ Experiments Monthly
Experimentation Velocity: How to Run 100+ Experiments Monthly
The fastest-growing companies like Booking.com, Netflix, or Amazon run hundreds to thousands of experiments monthly. How do they do it? And what do you need to get closer to them?
Why Velocity Matters
The math is simple:
- More experiments = more learnings
- More learnings = faster optimization
- Faster optimization = faster growth
Booking.com runs 25,000+ experiments yearly. Even if only 10% succeed, that's 2,500 product improvements per year. Your competition running 10 tests monthly doesn't stand a chance.
Prerequisites for High Velocity
1. Technical Infrastructure
Without the right tools, you won't scale.
| Component | Purpose | Examples |
|---|---|---|
| Feature flags | Quick on/off switching | LaunchDarkly, Unleash, custom |
| Event tracking | Behavior measurement | Segment, Amplitude, Mixpanel |
| Statistical engine | Automated analysis | Eppo, Statsig, custom |
| Dashboards | Real-time visibility | Looker, Metabase, custom |
2. Organizational Setup
Decentralization is key:
- Every team can run experiments
- Central team provides tooling and best practices
- Clear ownership — every experiment has a DRI
- Streamlined approval — no more than 24h for approval
3. Cultural Aspects
- 🧪 Experimental mindset — "we don't know until we test"
- 📊 Data-driven decisions — intuition is a hypothesis, not fact
- ❌ Tolerance for failure — 70% of tests "fail" — and that's OK
- 📚 Learning culture — every test brings insight
Build vs. Buy: Decision Matrix
| Factor | Build | Buy |
|---|---|---|
| Upfront cost | High (eng time) | Low-medium |
| Ongoing cost | Maintenance | Subscription |
| Customization | Unlimited | Limited |
| Time to value | Months | Days |
| Scalability | Depends on implementation | Usually good |
Recommendation:
- <50 experiments/month: Buy ready-made solution
- 50-200 experiments/month: Hybrid (purchased + custom integrations)
- 200+ experiments/month: Consider custom platform
Platform Comparison
| Platform | Best for | Price | Pros | Cons |
|---|---|---|---|---|
| Optimizely | Enterprise | $$$$ | Robust, full feature set | Expensive, complex |
| LaunchDarkly | Feature flags focus | $$$ | Excellent flags, fast | Weaker analytics |
| Statsig | Data-driven teams | $$ | Strong statistics, AI | Newer player |
| Eppo | Product teams | $$ | Warehouse-native | Smaller ecosystem |
| GrowthBook | Startups | $ (open source) | Cheap, flexible | Requires setup |
Process for Scaling
Experiment Proposal Template
Every experiment should have:
📋 EXPERIMENT PROPOSAL
1. Hypothesis: [What we're testing and why]
2. Primary metric: [One metric for decision]
3. Secondary metrics: [Guardrail metrics]
4. Segment: [Who's in the test]
5. Sample size: [How many users needed]
6. Duration: [How long it runs]
7. Success criteria: [When is test successful]
8. Owner: [Who's responsible]
Prioritization: ICE Framework
Score each experiment 1-10:
- Impact: How big an impact do we expect?
- Confidence: How confident are we in the hypothesis?
- Ease: How easy is implementation?
ICE Score = (I + C + E) / 3
Highest score = highest priority.
Weekly Review Ritual
| Time | Activity | Participants |
|---|---|---|
| 0:00-0:10 | New results review | Everyone |
| 0:10-0:30 | Deep dive top 3 learnings | Everyone |
| 0:30-0:45 | New experiments proposal | Owners |
| 0:45-0:55 | Prioritization voting | Everyone |
| 0:55-1:00 | Action items | Lead |
Measuring Program Health
| Metric | Definition | Target |
|---|---|---|
| Experiments launched/month | Number of tests started | 100+ |
| Win rate | % of tests with positive result | 15-30% |
| Avg. experiment duration | Average test length | 2-4 weeks |
| Time to decision | Time from launch to decision | <3 weeks |
| Learnings documented | % of tests with documentation | 100% |
| Impact delivered | Cumulative lift from wins | Tracking |
Case Study: Booking.com
How they run 25,000+ experiments yearly:
- Democratization — any employee can run a test
- Low barrier — simple tooling, quick setup
- Automation — automatic analysis, automatic stopping
- Culture — testing is the company's DNA
- Learning loops — sharing learnings across teams
Results:
- Conversion improvements every week
- Thousands of small wins = massive competitive advantage
- Culture of continuous improvement
Conclusion
Experimentation velocity isn't about chaotically launching tests. It's about a systematic approach to learning and optimization.
Action steps:
- Assess current velocity (experiments/month)
- Identify bottlenecks (tooling? process? culture?)
- Select or upgrade experimentation platform
- Implement proposal template and review ritual
- Measure program health metrics
- Iterate and scale
