← All articles
email-marketingab-testingdeliverability

A/B testing email campaigns without fooling yourself

Open and click rates are noisy. Here is how to run an email A/B test that produces a decision you can defend, and when an automatic winner send is worth using.

Transmitly Team·November 15, 2026·4 min read

Most email A/B tests are decided by a difference that would disappear with another hour of data. Variant B gets two more opens than variant A, someone declares a winner in the dashboard, and the next campaign inherits a lesson that was never there.

The fix is not a more complicated statistics package. It is deciding what counts as a result before you send, and refusing to over-read a thin sample.

Pick one metric before the send

Open rate is the fastest signal and the noisiest. An open is a request for the tracking pixel. Privacy proxies, image blocking and security scanners all request or suppress pixels without a human reading anything. That does not make opens useless — it makes them directional.

Click rate is slower but usually more intentional. A click still is not proof of attention: link prefetching and mail scanners can click, too. But it is a stronger signal than a pixel fetch.

Choose one metric per test. If you choose opens, do not switch to clicks when the open result is inconvenient. If you choose clicks, expect to wait longer and expect fewer events.

Change one thing

A subject-line test and a body-copy test answer different questions. If variant B changes the subject, the hero image and the call to action, a win tells you that the combination worked — not which part mattered.

Start with the highest-leverage variable: the subject line. Then test the sender name, the opening line, one call to action, or plain text versus HTML. One meaningful change per test keeps the result explainable.

Use delivered mail as the denominator

A bounce is not a non-open. If variant A has more invalid addresses, dividing opens by the full audience makes A look worse for a data-quality reason. Compare events against messages that actually reached a remote MX.

This also means cleaning the list before the test. A campaign sent to stale addresses measures deliverability, not copy.

Give the sample room

There is no universal minimum, but small samples produce unstable rates. If each variant has only a few dozen delivered messages, a one-open difference can flip the result. More recipients per variant is better; a test with a few hundred per variant is more useful than one with twenty.

Time matters as much as volume. Opens can trickle in for hours, and clicks can arrive the next morning. A test window that ends after ten minutes mostly measures your most online recipients. A few hours is a reasonable start; longer is better for click tests and for audiences spread across time zones.

Treat a close result as no result

If variant B wins by a fraction of a percentage point, the honest conclusion is “no clear difference.” Running the same test again is more valuable than declaring a permanent brand rule.

Ties should not be broken by enthusiasm. In Transmitly, ties go to variant A. That is not because A is better; it is because a deterministic default is better than silently picking whichever variant happened to lead when the timer fired.

When an automatic winner send helps

A split test sends both variants to the whole audience and compares them afterward. An automatic winner send does something different: it withholds most of the audience, tests A and B on a sample, waits, then sends the winning variant to the remainder.

That pattern is useful when the campaign is large enough to afford a sample and when you would rather optimise the majority of the send than learn for next time. It is not useful for small audiences. If there is no meaningful remainder, you are just delaying part of the campaign for no benefit.

Two guardrails matter:

  • The metric must be tracked. An open-rate winner needs open tracking; a click-rate winner needs click tracking. Otherwise both variants score zero and the choice is meaningless.
  • The decision must be visible. Record the metric, the window, the winning variant and both sample rates. “The system picked B” is not an audit trail.

Transmitly stores those fields on the broadcast and emits broadcast.winner_selected. The analytics keep the test sample separate from final totals, so you can see what the decision was based on even after the remainder has been sent.

A short checklist

  1. Decide the metric before sending.
  2. Change one meaningful thing.
  3. Compare against delivered mail, not the whole list.
  4. Use enough recipients and a window longer than ten minutes.
  5. Treat close results as inconclusive.
  6. Make the winner decision and its evidence visible.

That will not make email testing perfectly clean. It will stop a two-open difference from becoming policy.

Campaigns, tracking and automatic winner sends are included on every Transmitly plan. See email campaigns and A/B testing or read the broadcast API guide.

Transmitly

Durable acceptance, honest delivery events.

Verify a domain, send through the API, and see exactly what the remote MX said. No shared-IP guesswork.

Related reading