A Beginner’s Guide to Split Testing

Think of split testing (or AB testing) like a deathmatch for ideas. It’s the best tool we have for figuring out what works and what doesn’t. When you run a split test, two or more versions of your content are served to randomized audiences. The best performing content wins.

You can run split tests on both emails and web pages to learn things like:

• Effectiveness of long vs short-form copy
• Your audience’s tolerance for confronting images
• Best placement and appearance of buttons
• Which calls-to-action are most persuasive
• Does it help or hinder to use a supporter’s first name in a subject line? etc.

How to conduct a useful split test

1. Define your research question. What are you trying to figure out or optimize? Name it.
2. Know which metric to measure. For example, if you’re optimizing a subject line, your metric will be your email open rate. If you’re optimizing an email layout, your metric will be click rate. If you’re optimizing anything on a web page, your metric will probably be your web page KPI.
3. Isolate one variable to test. If the results of your test are definitive, you’ll want to know precisely which factor is influencing behavior. So, test one thing at a time, such as: a headline, an image, content length, button color, message framing, subject line, etc. Don’t take that too literally; “one thing” might still require you to tweak a few elements together. For example, a change in hero image might require a respective tweak to a headline, if they reference one another. If you want to test more things, pace yourself—there will be time for more tests.
4. Choose a big enough sample size. How many people do you want to expose to your test before the result is called? Your sample size needs to be large enough to generate a reliable result. Hundreds is too small. Thousands is better. Tens of thousands is better still. Most split testing systems will ‘lock in’ the winning test version once this threshold is met.
5. Run your test. Go get a coffee. This could take a while.
6. Learn from your result. Hopefully you’ve been delivered a result that clearly answers your research question. If so, nice work. If not, find the lesson. Does it really make zero difference to open rates if you use your supporter’s first name in a subject line? That’s important, too! Share your insights to fast-track your team’s learning curve.
7. Repeat. Surely there are more things you want to learn?

Split testing tools

Most email broadcast systems have in-built split (or “AB”) testing modules. For web pages, Optimizely (paid) and Google Optimize (free) come well-recommended.