I've written quite a bit about the value of testing every part of your online campaigns, including ad text and ad copy, landing pages, banners, forms, subject lines, and more. Michael Egan of Yahoo! Search Marketing further explains the value of testing, even for the best marketers:

It’s hard to argue that Tiger Woods is pretty darn good at what he does. But even he is not perfect. Imagine if he were allowed to hit four balls each time and then choose the shot that worked the best. Scary good.

However, as I wrote in Test Everything - But Don't Over Test, it's easy to over-design your test in a way that never results in valid, confident results. To help with this, Marketo has created a new Online Test Calculator that will tell you how many test versions your landing page (or ad, email, etc.) can support.

Test Everything... But Don't Over Test

As explained by DestinationCRM in Screen Testing: Testing Can Drive Better Marketing Results, But Which Test Is Best?, there are two main types of online tests: "A/B testing" and "multivariate" testing. In A/B testing, you create distinct versions of each of page or offer, rotate through them, and see which performs best. To tease out what causes the difference, you should only vary one thing on each page at a time. In contrast, multivariate testing tests more than one thing at a time.

Marketo's software helps marketers both create and test landing pages; we use A/B testing. The best known tool for multivariate testing is Google Website Optimizer.

Multivariate testing is great if you get lots of traffic and responses each day, but it can quickly result in a test that's too complicated. For example, let's say you want to test the headline, image, caption, call to action, and form (five elements) on a landing page, and each element has four test variants. Using Google Website Optimizer, this would create a total of 5 x 5 x 5 x 5 = 625 test variants! As we'll see, that's way too many.

Some more advanced testing tools use a technique called "Design of Experiments" or Taguchi to reduce the number of recipes down to 10 or 20 versions. This is a great improvement, but it is still too many for any site that would not be considered "high volume".

How Many Tests Can You Run?

To understand this, we need something that tells us how many tests we can run. This is where Marketo's new Online Test Calculator comes in. It lets you enter how many responses you get per day, and returns the number of test versions you can support.

(If you're curious about the statistics behind this calculator, check out Landing Page Testing - The Ultimate Guide To Test Statistics.)

Here are a few examples. First, assume you want to test a landing page, and that you currently get 20 conversions each day. How many landing page test versions can you run if you want significant results within 2 weeks (14 days)? Plugging this into the calculator returns "You can run 2 test versions, including 1 champion and 1 challenger."

So far, so good. But how long will it take to get valid results if you have eight test versions and get 20 conversions per day? Entering this the calculator returns "This test will generate statistically valid results in 52 days".  This is much longer than most marketers are willing to wait for results.

Even worse, most small- and mid-size companies get much fewer than 20 conversions per landing page per day. At Marketo, I spend about $100 a day on one of my ad groups. The landing page for that ad group has a great conversion rate (17%), but even this page only gets about five conversions a day (e.g. $20 per conversion). If I wanted to test 10 versions, I'd have to wait 239 days (or at least 87 days if I'm willing to accept Type II errors). Instead, I test only two versions — one champion and one challenger — and act on the results every 18 days.

Summary

Don't get me wrong: I'm a fan of multivariate approaches to testing for sites that can support those volumes. The trick is to pick the right approach for the volumes your landing pages receive. However, no matter what approach you use, just remember that doing any testing is more important than arguing about what kind of testing to use.

Note: You can read more on this topic in my recent article at Search Engine Land.