Skip to content

The subject A/B test: which subject gets more opens

A newsletter can test two subjects before being sent to everyone. One part of the recipients receives subject A and another equal part subject B; after a few hours, the program checks which was opened most —or which had more clicks— and sends the rest with the winner. You set it up in More options in the newsletter.

How do I set up a subject A/B test?

  1. Prepare the newsletter as usual in Mailing → Newsletters. The subject at the top is subject A.
  2. Open More options and, in Subject A/B test, type another subject in Subject B.
  3. Choose how many take part in the test in Tested on (from 10 % to 50 % of the recipients, split equally between the two subjects; by default, 20 %).
  4. Choose how long to wait in Wait (from 1 to 72 hours; by default, 4).
  5. Choose what decides it in Winner: the one with most: opens or Clicks.
  6. Carry on with the recipients and the launch as in your first newsletter.

In Who to?, the count says how the split will look: ‘A/B test: 100 with subject A, 100 with B and 800 wait for the winner’. It is the same split that is made on launch. To remove the test, leave Subject B empty.

How is the winning subject decided?

When all the test emails have gone out, the wait begins. When it ends, the program compares the two groups with the same metrics you see in Mailing → Metrics:

The highest rate wins. Rates are compared, not raw figures, so a group with an extra bounce or two is not at a disadvantage.

Automatic opens do not count. Apple Mail's privacy protection or the security filters of some companies ‘open’ emails by themselves; those opens, and clicks from filters, are left out. What they are is explained in what the people who receive your emails see.

If they tie, subject A wins. Also when neither of the two groups has any delivered emails: there is nothing to compare and the newsletter carries on with its subject.

Where do I see how the test is going and which subject won?

In the newsletter's control panel, by opening it in Mailing → Newsletters. The Subject A/B test section says what stage it is at —going out, waiting until a date and time, or decided— and shows one row per group: subject A, subject B and the rest, with how many receive it, how many have gone out and their open and click percentages. When it is decided, the winner is marked and the sentence says why, with the figures: ‘B wins: 18,2 % opens (20 of 110 delivered) against 12,7 % for A (14 of 110)’.

While it waits, the control panel says so: ‘A/B test sent: on 12/10/2026 at 14:30 the winning subject is decided and the rest goes out with it’.

What happens to the rest while it waits?

They are not sent. Those who were not in the test wait in the newsletter as pending and only go out when there is a winner, with its subject. If you pause the newsletter during the wait, the test is decided when you resume it; if you cancel it, the rest are not sent.

The same rules as for any email apply to the rest: anyone who unsubscribes during the wait, or whose address stops being valid, no longer receives it. And if the test has too many rejections, the brake stops the newsletter before the rest go out, as in any newsletter: it is explained in the control panel of a newsletter.

How are the people in the test chosen?

At random, among those who receive the newsletter (those left out do not count). The split does not depend on the order of the list or on how old the records are, and two newsletters to the same list do not split the same way. A minimum number of recipients is needed so that each subject reaches at least one person: with 20 %, ten. If that is not reached, launching says so and you either remove the test or raise the percentage.

What should I know before running a test?

More about Mailing