The validation test is rank correlation, not exact agreement. You do not need the model to predict that a line will get 26.4 percent; you need it to put your best lines above your worst. Spearman's rank correlation between total score and open rate is the right statistic. With around 100 sends, a correlation above roughly 0.5 is a usable signal; near zero means the rubric is measuring something other than what your audience rewards; a negative value means one of your criteria is inverted.
Worked example with numbers. Suppose you score 100 sends and get a Spearman correlation of 0.58. That is above the threshold, so you proceed. Now split the sends into the top quartile by total score and the bottom quartile. Say the top quartile averaged a 31 percent open rate and the bottom quartile averaged 19 percent. That 12-point spread is your expected effect size, and it is what you should design your split test to detect.
A quick check in code, using scipy:
import pandas as pd
from scipy.stats import spearmanr
df = pd.read_csv("scored_sends.csv")
rho, p = spearmanr(df["total"], df["open_rate"])
print(f"spearman={rho:.2f} p={p:.4f}")
q = df["total"].quantile([0.25, 0.75])
top = df[df["total"] >= q[0.75]]["open_rate"].mean()
bottom = df[df["total"] <= q[0.25]]["open_rate"].mean()
print(f"top quartile={top:.3f} bottom quartile={bottom:.3f}")
The sample size matters for interpreting that p-value. With 100 observations, a correlation of 0.5 is comfortably distinguishable from zero; with 20 observations it is not, and you risk chasing noise. If your list is small, pool sends across campaigns before you judge the rubric, and accept that validation will take a quarter rather than a week.
There is one more comparison worth making: score the same lines twice, in two separate runs, and measure how many changed. If more than around a tenth of the lines move by two or more points on a criterion, the anchor for that criterion is too vague and needs a sharper example. This is a reliability check, and it is separate from the validity check against opens. A rubric can be perfectly reliable (same score every time) and still be invalid (predicts nothing). You want both.
Treat the correlation as a gate, not a verdict. A rubric that clears 0.5 has earned the right to guide drafts. It has not earned the right to send without a test. The next step is a live split: take your next send, use the highest-scoring and lowest-scoring candidates from the modeled batch as the two test arms, and split the list evenly. Omeda's analysis warns that correlation is not causation and recommends testing length and content directly with your own audience in its length study; the same caution applies to any rubric score.
For a list under about 1,000 recipients, do not test five variants. Test two, and build understanding across campaigns instead, because each arm needs enough recipients to separate a real effect from chance. Practical guidance for small senders puts the usable threshold at roughly 1,000 recipients per variation per Ruzuku's testing walkthrough.
One side note on what a subject line test can and cannot fix. If your emails are landing in the spam folder, a better subject line will not rescue them. Deliverability depends on authentication and complaint rates: SPF, DKIM and DMARC for bulk senders, plus keeping spam complaint rates below 0.3 percent per Gmail's sender guidelines and the parallel Yahoo requirements in Yahoo's sender best practices. AWS summarizes both providers' rules as three buckets: authentication, easy unsubscribe, and complaint monitoring in its bulk sender overview. Score subject lines only after those are in place.
Antonio Evans
