AFAnalyst Field Notes
A/B Testing

A/B Test Design Template for Data Analysts

A good experiment starts before launch. Use this structure to make sure the team agrees on the hypothesis, audience, traffic split, duration, and decision rule.

By Analyst Field Notes8 min read

1. Experiment background

Describe the product change, campaign, or page variation being tested. Include the launch context, owner, and the business problem that triggered the test.

2. Hypothesis

Write the hypothesis in a way that can be measured. For example: making deposit rewards clearer will increase user trust and improve deposit conversion rate.

3. Test setup

Define the experiment type, target users, test scenario, splitting moment, and experiment duration. If the sample size is small, plan to observe at least two full business cycles instead of stopping too early.

Control group: existing experience
Treatment group: new experience
Traffic split: 50% / 50%
Primary metric: conversion rate
Guardrail metric: refund rate or complaint rate

4. Sample size

Estimate baseline conversion rate, expected lift, significance level, and power. If the required sample is much larger than available traffic, adjust the design before launching.

For example, a product team might expect conversion to move from 8.0% to 8.8%. That looks small, but at scale it can be commercially meaningful. The analyst should calculate whether the available traffic can detect that lift within a reasonable test window.

5. Decision rule

Before the experiment starts, define what will count as launch, rollback, or continue observing. This prevents the team from changing the goal after seeing early results.

6. Metric dictionary

Add a metric dictionary to the design document. A clear dictionary prevents debates after launch, especially when different teams calculate conversion, activation, or retention differently.

deposit_conversion_rate =
  users_with_completed_deposit / users_assigned_to_experiment

complaint_rate =
  users_with_support_complaint / users_assigned_to_experiment

7. Segment plan

Decide which segments matter before the test starts. Common segments include new versus returning users, channel, device, geography, product category, and user value tier.

Do not over-interpret every tiny segment. Segment analysis should answer a business question, such as whether the treatment works only for new users or whether mobile users experience more friction.

8. Pre-launch QA

  • Confirm users are assigned to only one group.
  • Check the event names and timestamps.
  • Verify control and treatment traffic split.
  • Exclude internal, test, and bot traffic.
  • Confirm dashboards use the same metric definitions as the analysis query.

SQL tracking check

Before launch, make sure users are assigned once and events can be joined back to the experiment group.

SELECT
  experiment_group,
  COUNT(DISTINCT user_id) AS assigned_users,
  COUNT(DISTINCT CASE WHEN event_name = 'deposit_complete' THEN user_id END) AS converters,
  1.0 * COUNT(DISTINCT CASE WHEN event_name = 'deposit_complete' THEN user_id END)
    / COUNT(DISTINCT user_id) AS conversion_rate
FROM experiment_events
WHERE experiment_name = 'reward_page_test'
GROUP BY 1;

Downloadable template

The workbook includes an A/B Test Design tab for hypotheses, traffic split, metrics, and decision rules.

Download Excel template Download launch checklist

FAQ

How long should an A/B test run?

It should run long enough to reach the planned sample size and cover normal business cycles. A test that runs only on weekdays may miss weekend behavior.

Can I change the hypothesis after launch?

You can document new observations, but the launch decision should use the pre-defined hypothesis and primary metric. New hypotheses should become follow-up tests.