# CRO Testing Methodology

Deep-dive into experiment design, statistical rigor, and test prioritization from the CRE Methodology.


## Table of Contents
1. [The Philosophy of Bold Testing](#the-philosophy-of-bold-testing)
2. [A/B Testing vs. Multivariate Testing](#ab-testing-vs-multivariate-testing)
3. [Statistical Significance](#statistical-significance)
4. [ICE Prioritization Framework](#ice-prioritization-framework)
5. [Test Documentation](#test-documentation)
6. [When Tests Fail](#when-tests-fail)
7. [CRO Team Dynamics](#cro-team-dynamics)
8. [Testing Platform Comparison](#testing-platform-comparison)

---

## The Philosophy of Bold Testing

### Why "Meek Tweaks" Fail

Most A/B tests fail because they're too small to detect. Button color changes, minor copy tweaks, and micro-optimizations suffer from:

1. **Insufficient sample size** - Small changes require massive traffic to reach significance
2. **Interaction effects** - Minor changes get lost in noise
3. **Opportunity cost** - Time spent on 2% wins could find 200% wins

**Rule: Test big changes that could double conversion, not small changes that might move it 5%.**

### The 10x Mindset

Before any test, ask: "Could this 10x our results?" If not, is it worth testing?

- **Worth testing:** Complete page redesign, new value proposition, fundamentally different offer
- **Not worth testing:** Button color, font size, image swap

---

## A/B Testing vs. Multivariate Testing

### A/B Testing (Split Testing)

Compare two (or more) complete versions against each other.

| Aspect | Details |
|--------|---------|
| **Best for** | Testing big concepts, page redesigns, offers |
| **Traffic needed** | Lower (split between 2-4 variants) |
| **Insights** | Which version wins overall |
| **Limitation** | Doesn't show which elements contributed |

**When to use:**
- You have a hypothesis about a major change
- Traffic is limited
- You're comparing conceptual approaches

### Multivariate Testing (MVT)

Test multiple elements simultaneously to find optimal combination.

| Aspect | Details |
|--------|---------|
| **Best for** | Optimizing elements after winning concept proven |
| **Traffic needed** | Much higher (combinations multiply) |
| **Insights** | Which specific elements drive results |
| **Limitation** | Requires significant traffic |

**When to use:**
- You have a winning page to optimize further
- High traffic (100k+ monthly visitors)
- Clear, isolated elements to test

### Traffic Requirements

**A/B Test:**
```
Minimum sample per variant = 250-500 conversions
For 2 variants with 5% conversion: 10,000-20,000 visitors needed
```

**Multivariate Test:**
```
Combinations = (Options for Element 1) × (Options for Element 2) × ...
Example: 3 headlines × 3 images × 2 CTAs = 18 combinations
Each combination needs 250+ conversions = 90,000+ conversions total
```

**Recommendation:** Start with A/B tests. Only move to MVT when you have:
- Proven winning page concept
- 100k+ monthly visitors
- Mature testing program

---

## Statistical Significance

### What It Means

Statistical significance tells you: "How likely is this result due to chance vs. a real effect?"

**Industry standard:** 95% confidence (p-value < 0.05)
- 95% confident the difference is real
- 5% chance it's random noise

### Common Mistakes

**1. Peeking and stopping early**

Checking results daily and stopping when you see a winner leads to false positives.

- **Wrong:** "We're at 95% confidence after 3 days—ship it!"
- **Right:** Pre-determine sample size and test duration; don't stop early

**2. Calling tests with insufficient data**

| Visitors | Conversions | Can you call it? |
|----------|-------------|------------------|
| 500 | 15 vs 20 | No |
| 5,000 | 150 vs 200 | Possibly |
| 50,000 | 1,500 vs 2,000 | Yes |

**3. Ignoring practical significance**

A statistically significant 0.1% lift isn't worth implementation complexity.

**4. Multiple comparison problem**

Testing 20 variants? One will show "significance" by chance alone.

### Sample Size Calculation

Before testing, calculate required sample size:

**Inputs needed:**
- Baseline conversion rate
- Minimum detectable effect (MDE) you care about
- Statistical power (typically 80%)
- Significance level (typically 95%)

**Rule of thumb:**
```
For 5% baseline, 20% relative lift detection:
~25,000 visitors per variant needed

For 5% baseline, 50% relative lift detection:
~4,000 visitors per variant needed
```

**Key insight:** The smaller the effect you want to detect, the more traffic you need. This is why bold changes are better—they're detectable with less traffic.

### Test Duration

**Minimum test duration:**
- At least 1 full business cycle (typically 1-2 weeks)
- Include weekdays AND weekends
- Account for seasonality

**Why?**
- Visitor behavior differs by day of week
- Friday buyers differ from Monday researchers
- Monthly cycles affect B2B especially

---

## ICE Prioritization Framework

Prioritize test ideas using ICE scores:

### Impact (1-10)
"If this wins, how big would the impact be?"

| Score | Impact Level |
|-------|--------------|
| 10 | Could double conversion rate |
| 7-9 | Major improvement (30-50%+) |
| 4-6 | Moderate improvement (10-30%) |
| 1-3 | Minor improvement (<10%) |

### Confidence (1-10)
"How confident are we this will work?"

| Score | Confidence Level |
|-------|------------------|
| 10 | Proven in research, worked before |
| 7-9 | Strong research supports it |
| 4-6 | Reasonable hypothesis |
| 1-3 | Gut feeling, unvalidated |

### Ease (1-10)
"How easy is this to implement and test?"

| Score | Ease Level |
|-------|------------|
| 10 | Text change only |
| 7-9 | Design change, no dev needed |
| 4-6 | Requires development |
| 1-3 | Major technical lift |

### ICE Score Calculation

```
ICE Score = (Impact + Confidence + Ease) / 3
```

Or weighted:
```
ICE Score = (Impact × 2 + Confidence × 1.5 + Ease × 1) / 4.5
```

### Sample Prioritization

| Test Idea | Impact | Confidence | Ease | Score |
|-----------|--------|------------|------|-------|
| New headline from customer research | 8 | 9 | 10 | 9.0 |
| Add video testimonial | 7 | 7 | 6 | 6.7 |
| Redesign checkout flow | 9 | 6 | 3 | 6.0 |
| Change button color | 2 | 2 | 10 | 4.7 |

---

## Test Documentation

### Before the Test

Document:
1. **Hypothesis:** "If we [change X], then [metric Y] will improve because [reason based on research]"
2. **Primary metric:** One metric that determines winner
3. **Secondary metrics:** Additional metrics to monitor
4. **Guardrail metrics:** Metrics that shouldn't decrease
5. **Sample size requirement**
6. **Test duration**
7. **Traffic allocation**

### After the Test

Document:
1. **Results:** Raw numbers, conversion rates, confidence interval
2. **Statistical significance:** p-value, confidence level
3. **Practical significance:** Is the lift worth implementing?
4. **Learnings:** What does this teach us about our customers?
5. **Next steps:** Ship winner, iterate, or abandon?

### Learnings Database

Every test should add to organizational knowledge:

| Test | Hypothesis | Result | Learning | Applicable to |
|------|------------|--------|----------|---------------|
| Homepage headline A/B | Customer language converts better | Winner: +27% | Customers care about outcomes, not features | All landing pages |
| Form length test | Shorter forms convert better | Loser: no diff | Our audience expects detailed forms | Lead gen pages |

---

## When Tests Fail

### Types of "Failure"

**1. No winner (inconclusive)**
- Sample size too small
- Effect size too small to detect
- Test needed to run longer

**2. Control wins**
- New version is worse
- Hypothesis was wrong
- Still a learning!

**3. Technical problems**
- Tracking broke
- Experience differed from plan
- Sample contamination

### What to Do

1. **Document the learning** - "We learned customers prefer X"
2. **Investigate why** - Go back to research
3. **Don't give up on the page** - The opportunity exists, you just haven't found the solution
4. **Try a bolder change** - Maybe the change wasn't big enough

**Critical insight:** A failed test that teaches you something is more valuable than a winning test you don't understand.

---

## CRO Team Dynamics

### Roles in a CRO Program

| Role | Responsibility |
|------|----------------|
| **CRO Lead** | Strategy, prioritization, stakeholder management |
| **Researcher** | User research, surveys, analytics analysis |
| **Designer** | Wireframes, mockups, user flows |
| **Developer** | Test implementation, technical QA |
| **Analyst** | Results analysis, statistical rigor |

### Getting Stakeholder Buy-In

**Common objections:**

| Objection | Counter |
|-----------|---------|
| "We already know what works" | "Then testing will confirm it quickly" |
| "Testing takes too long" | "Shipping wrong things costs more" |
| "Our traffic is too low" | "Then we test bigger changes" |
| "The CEO wants X" | "Let's test to validate the idea" |

**Building credibility:**
1. Start with quick wins (high-traffic pages, obvious problems)
2. Document and share learnings widely
3. Quantify impact in revenue terms
4. Build testing into the culture, not just a project

### Test Velocity

**Goal:** Increase valid tests per month over time.

| Maturity | Tests/Month | Characteristics |
|----------|-------------|-----------------|
| Beginner | 1-2 | Manual processes, ad-hoc |
| Developing | 4-6 | Established backlog, regular cadence |
| Advanced | 10-20 | Parallel testing, mature process |
| Expert | 20+ | Multiple simultaneous tests, automated |

---

## Testing Platform Comparison

| Platform | Best For | Limitations |
|----------|----------|-------------|
| Google Optimize | Beginners, free tier | Sunsetting, limited features |
| VWO | Mid-market, visual editor | Can be slow, limited targeting |
| Optimizely | Enterprise, complex tests | Expensive, learning curve |
| LaunchDarkly | Dev-centric, feature flags | Not optimized for marketing |
| Custom | Full control | Development cost |

### Key Features to Look For

- Visual editor for non-developers
- Robust statistical engine
- Segment targeting
- Integrations (analytics, CDP, etc.)
- Flicker prevention
- Mutually exclusive experiments
