name: ab-test-analyzer description: Statistical significance calculator for A/B test results with sample size requirements, segment breakdowns, and hypothesis generation. Use when feeding test results, checking statistical significance, calculating sample sizes, analyzing experiment outcomes, or generating next test ideas based on results. Platform: Google and Meta. metadata: platform: Google and Meta

A/B Test Analyzer

Evaluate A/B test results with statistical rigor and generate actionable next steps.

Process

Collect test data - Variations, sample sizes, conversions, time period
Check validity - Runtime, sample size, peeking issues
Calculate significance - Z-score, p-value, confidence interval
Segment analysis - Device, source, new vs returning
Interpret results - Statistical vs practical significance
Generate hypotheses - "Why it worked" and next test ideas

Sample Size Calculator

n = 2 × (Zα/2 + Zβ)² × p(1-p) / δ²

Where:
- n = sample size per variation
- Zα/2 = 1.96 (95% confidence) or 2.58 (99%)
- Zβ = 0.84 (80% power) or 1.28 (90%)
- p = baseline conversion rate
- δ = minimum detectable effect (absolute)

Quick Reference (95% confidence, 80% power):

Baseline CR	10% Relative MDE	20% Relative MDE
2%	78,000/var	19,500/var
5%	30,000/var	7,500/var
10%	14,300/var	3,600/var

Use Case	Approach
Large samples (>10K), yes/no decision	Frequentist
Continuous monitoring, early stopping	Bayesian
Regulatory/legal requirements	Frequentist
Want probability statements	Bayesian

ab-test-analyzer

Install

Related skills

A/B Test Analyzer

Process

Sample Size Calculator

Significance Calculation

Validity Checklist

Bayesian vs Frequentist

Output Format

Example

Guidelines

Data access (Ryze MCP)