A/B Test Analysis Report Template: SQL Summary and Launch Recommendation

What the report must answer

An A/B test report should answer one business question: should the team launch, stop, extend, or rerun the experiment? A table of metrics is not enough. Stakeholders need a decision, the evidence behind it, and the caveats that could change the decision.

Copy this report outline

1. Recommendation
2. Experiment setup
3. Primary metric result
4. Guardrail metric result
5. Segment findings
6. Data quality and assignment checks
7. Caveats
8. Rollout or follow-up plan

Start with the decision

Put the recommendation at the top. Busy readers should understand the answer before reading the SQL details.

Recommendation: Launch to 100% of eligible users.
Primary metric: checkout conversion increased from 8.2% to 8.9%.
Impact: +0.7 percentage points, +8.5% relative lift.
Guardrails: refund rate and support complaint rate were stable.
Caveat: mobile segment lift was stronger than desktop, so monitor desktop after rollout.
Next step: staged rollout over 3 days with daily guardrail monitoring.

Include experiment setup

The setup section makes the report auditable. Include control, treatment, target users, randomization unit, date range, traffic allocation, and the pre-defined primary metric.

SQL example: report-level summary

This query produces one row per variant with users, conversions, conversion rate, revenue per user, and refund rate. Extend it with your own guardrails.

WITH assigned AS (
  SELECT
    user_id,
    variant,
    assigned_at
  FROM experiment_assignments
  WHERE experiment_id = 'checkout_form_test'
),
user_metrics AS (
  SELECT
    a.user_id,
    a.variant,
    MAX(CASE WHEN e.event_name = 'purchase_completed' THEN 1 ELSE 0 END) AS converted,
    SUM(CASE WHEN e.event_name = 'purchase_completed' THEN e.revenue ELSE 0 END) AS revenue,
    MAX(CASE WHEN e.event_name = 'refund_requested' THEN 1 ELSE 0 END) AS refunded
  FROM assigned a
  LEFT JOIN events e
    ON e.user_id = a.user_id
   AND e.event_time >= a.assigned_at
   AND e.event_time < a.assigned_at + INTERVAL '7 days'
  GROUP BY 1, 2
)
SELECT
  variant,
  COUNT(*) AS users,
  SUM(converted) AS conversions,
  1.0 * SUM(converted) / NULLIF(COUNT(*), 0) AS conversion_rate,
  1.0 * SUM(revenue) / NULLIF(COUNT(*), 0) AS revenue_per_user,
  1.0 * SUM(refunded) / NULLIF(COUNT(*), 0) AS refund_rate
FROM user_metrics
GROUP BY 1
ORDER BY 1;

Segment findings without fishing

Segment analysis should explain the result, not create unlimited chances to find a winner. Start with planned segments from the design document: device, user type, traffic channel, geography, or customer tier.

SELECT
  variant,
  device_type,
  COUNT(DISTINCT user_id) AS users,
  COUNT(DISTINCT CASE WHEN converted = 1 THEN user_id END) AS conversions,
  1.0 * COUNT(DISTINCT CASE WHEN converted = 1 THEN user_id END)
    / NULLIF(COUNT(DISTINCT user_id), 0) AS conversion_rate
FROM experiment_user_metrics
GROUP BY 1, 2
ORDER BY device_type, variant;

How to explain an inconclusive test

An inconclusive result is not always a failed idea. It can mean the effect is smaller than expected, the metric is noisy, traffic was too low, or the experiment design was not strong enough.

If sample size was too low, recommend extending the test only if guardrails are stable.
If tracking was broken, mark the result invalid and rerun.
If guardrails worsened, recommend against launch even if the primary metric improved.
If lift was positive but small, recommend a cost-benefit review before rollout.

Download the report workbook

The workbook includes an A/B Test Analysis tab with control and treatment rates, lift formulas, chart-ready summary fields, and a decision column.

Download Excel template Download analysis checklist