This is the part most subject line advice skips, and it is the part that determines whether your testing program produces knowledge or noise.
An A/B test on a subject line is a comparison of two proportions. The difference you observe is a mix of the real effect and sampling error. With small samples, sampling error dominates. Run a test on 400 recipients per arm and a 3-point gap in open rate is well inside what chance alone produces, so the "winner" you pick is closer to a coin flip than to a finding.
You do not need a statistics package to apply the right rule. You need a decision rule:
| List size per arm | What a test can realistically detect | What to do instead |
|---|
| Under about 2,000 | Only very large differences, well above typical subject line effects | Skip A/B tests; use sequential cohort comparisons across several sends |
| 2,000 to 10,000 | Moderate differences, if you test one variable at a time | Test deliberately, accept that only big winners are real |
| Over 10,000 | Realistic differences on opens and clicks | Standard A/B testing works; still confirm winners on a holdout |
Two additional constraints push you toward larger, simpler tests.
Metric noise. Opens are inflated by privacy features that pre-fetch messages, so an open-rate difference of a few points may partly reflect which recipients use Apple Mail rather than which subject line they preferred (Letterhead). Clicks are cleaner but rarer, so they need even bigger samples to separate signal from noise.
List heterogeneity. A list of 5,000 built over three years contains several populations with different reading habits and different inbox apps. Splitting it randomly helps, but if you split by send date instead, seasonality and content quality contaminate the comparison.
Designing a test that yields an answer
- Write the hypothesis as a mechanism, not a preference. Not "shorter is better" but "moving the specific term into the first 30 characters will raise clicks because mobile readers will see it."
- Change one thing. Subject length, preheader angle, or sender name. Two changes at once and you cannot attribute the result.
- Split randomly within a single send. Half the list gets A, half gets B, same body, same time. This is the only design that isolates the variable.
- Pre-commit to the metric and the threshold. Decide before sending whether you will judge on clicks or on opens, and how big a gap you will act on.
- Repeat before you believe it. A subject line pattern that wins three sends in a row is a finding. One win is an anecdote.
- Keep a running log. Subject line, preheader, character count, open rate, click rate, list size, and what else changed that week. After twenty sends you will have your own benchmark, which beats every published average.
Letterhead's guidance that your own historical data matters more than generic benchmarks applies directly here: external averages describe other people's lists (Letterhead).
A worked example
Suppose you have 6,000 subscribers, 5,700 delivered, and you want to test a short specific subject against your current long descriptive one. You split 2,850 and 2,850.
- Variant A (your current line, 78 characters): 1,026 opens, 36.0 percent. Clicks: 114, or 4.0 percent.
- Variant B (specific term front-loaded, 44 characters): 1,083 opens, 38.0 percent. Clicks: 148, or 5.2 percent.
The open rate difference is 2 points, which on this sample size is suggestive but not decisive. The click difference is 1.2 points, roughly a 30 percent relative lift, and it shows up on a metric that privacy features cannot fake. You ship B, log it, and run the same comparison twice more before you call it a permanent rule. That is a result you can defend, and it took three sends rather than one.
If your list were 800 subscribers, the same percentages would produce gaps of a few dozen people. Your job then is not to test faster but to test structurally: rotate one element per month across the whole list and compare month over month, accepting that content quality moves the number as much as the line does.
Antonio Evans
