The subject A/B test: which subject gets more opens
A newsletter can test two subjects before being sent to everyone. One part of the recipients receives subject A and another equal part subject B; after a few hours, the program checks which was opened most —or which had more clicks— and sends the rest with the winner. You set it up in More options in the newsletter.
How do I set up a subject A/B test?
- Prepare the newsletter as usual in Mailing → Newsletters. The subject at the top is subject A.
- Open More options and, in Subject A/B test, type another subject in Subject B.
- Choose how many take part in the test in Tested on (from 10 % to 50 % of the recipients, split equally between the two subjects; by default, 20 %).
- Choose how long to wait in Wait (from 1 to 72 hours; by default, 4).
- Choose what decides it in Winner: the one with most: opens or Clicks.
- Carry on with the recipients and the launch as in your first newsletter.
In Who to?, the count says how the split will look: ‘A/B test: 100 with subject A, 100 with B and 800 wait for the winner’. It is the same split that is made on launch. To remove the test, leave Subject B empty.
How is the winning subject decided?
When all the test emails have gone out, the wait begins. When it ends, the program compares the two groups with the same metrics you see in Mailing → Metrics:
- By opens: the estimated opens rate of each group, that is, people who opened it or clicked a link, out of the emails delivered.
- By Clicks: the click rate, people who clicked any link, out of those delivered.
The highest rate wins. Rates are compared, not raw figures, so a group with an extra bounce or two is not at a disadvantage.
Automatic opens do not count. Apple Mail's privacy protection or the security filters of some companies ‘open’ emails by themselves; those opens, and clicks from filters, are left out. What they are is explained in what the people who receive your emails see.
If they tie, subject A wins. Also when neither of the two groups has any delivered emails: there is nothing to compare and the newsletter carries on with its subject.
Where do I see how the test is going and which subject won?
In the newsletter's control panel, by opening it in Mailing → Newsletters. The Subject A/B test section says what stage it is at —going out, waiting until a date and time, or decided— and shows one row per group: subject A, subject B and the rest, with how many receive it, how many have gone out and their open and click percentages. When it is decided, the winner is marked and the sentence says why, with the figures: ‘B wins: 18,2 % opens (20 of 110 delivered) against 12,7 % for A (14 of 110)’.
While it waits, the control panel says so: ‘A/B test sent: on 12/10/2026 at 14:30 the winning subject is decided and the rest goes out with it’.
What happens to the rest while it waits?
They are not sent. Those who were not in the test wait in the newsletter as pending and only go out when there is a winner, with its subject. If you pause the newsletter during the wait, the test is decided when you resume it; if you cancel it, the rest are not sent.
The same rules as for any email apply to the rest: anyone who unsubscribes during the wait, or whose address stops being valid, no longer receives it. And if the test has too many rejections, the brake stops the newsletter before the rest go out, as in any newsletter: it is explained in the control panel of a newsletter.
How are the people in the test chosen?
At random, among those who receive the newsletter (those left out do not count). The split does not depend on the order of the list or on how old the records are, and two newsletters to the same list do not split the same way. A minimum number of recipients is needed so that each subject reaches at least one person: with 20 %, ten. If that is not reached, launching says so and you either remove the test or raise the percentage.
What should I know before running a test?
- Only the subject is tested. The preview text and the design are the same for both groups.
- With opens, the newsletter must track opens; with Clicks, clicks. If not, To be able to launch it says so and does not let you launch it.
- A small test is decided by very little: with 20 people per subject, a difference of two opens is ten points. With short lists, use a high percentage or do not run a test.
- Wait long enough: many people open the email hours later. With 4 hours you see the trend; with 24, almost everything.
- Resend to non-openers on a newsletter with a test suggests the subject that won: it is explained in resending a newsletter to people who did not open it.
More about Mailing
- Newsletters and lists from your customers, suppliers and contacts
- Mailing campaigns
- How to design an email template
- Consent and legal basis of each address
- Newsletter control: progress, pausing, resuming and cancelling
- Managing mailing addresses
- The ERP sending service: sending your newsletters without your own email account
- Sending invoices and other documents by email
- Importing mailing addresses from a file
- Mailing: newsletters, lists and who you can write to
- The quality of an email address
- Mailing settings: quality, consent, tracking and retention