Landing Page Testing: How Much Is Too Much?


Posted: October 24, 2007 | Landing Page, Testing

I’ve written quite a bit about the value of testing every part of your online campaigns, including ad text and ad copy, landing pages, banners, forms, subject lines, and more. Michael Egan of Yahoo! Search Marketing further explains the value of testing, even for the best marketers:

It’s hard to argue that Tiger Woods is pretty darn good at what he does. But even he is not perfect. Imagine if he were allowed to hit four balls each time and then choose the shot that worked the best. Scary good.

However, as I wrote in Test Everything – But Don’t Over Test, it’s easy to over-design your test in a way that never results in valid, confident results. To help with this, Marketo has created a new Online Test Calculator that will tell you how many test versions your landing page (or ad, email, etc.) can support.

Test Everything… But Don’t Over Test

As explained by DestinationCRM in Screen Testing: Testing Can Drive Better Marketing Results, But Which Test Is Best?, there are two main types of online tests: “A/B testing” and “multivariate” testing. In A/B testing, you create distinct versions of each of page or offer, rotate through them, and see which performs best. To tease out what causes the difference, you should only vary one thing on each page at a time. In contrast, multivariate testing tests more than one thing at a time.

Marketo’s software helps marketers both create and test landing pages; we use A/B testing. The best known tool for multivariate testing is Google Website Optimizer.

Multivariate testing is great if you get lots of traffic and responses each day, but it can quickly result in a test that’s too complicated. For example, let’s say you want to test the headline, image, caption, call to action, and form (five elements) on a landing page, and each element has four test variants. Using Google Website Optimizer, this would create a total of 5 x 5 x 5 x 5 = 625 test variants! As we’ll see, that’s way too many.

Some more advanced testing tools use a technique called “Design of Experiments” or Taguchi to reduce the number of recipes down to 10 or 20 versions. This is a great improvement, but it is still too many for any site that would not be considered “high volume”.

How Many Tests Can You Run?

To understand this, we need something that tells us how many tests we can run. This is where Marketo’s new Online Test Calculator comes in. It lets you enter how many responses you get per day, and returns the number of test versions you can support.

(If you’re curious about the statistics behind this calculator, check out Landing Page Testing – The Ultimate Guide To Test Statistics.)

Here are a few examples. First, assume you want to test a landing page, and that you currently get 20 conversions each day. How many landing page test versions can you run if you want significant results within 2 weeks (14 days)? Plugging this into the calculator returns “You can run 2 test versions, including 1 champion and 1 challenger.”

So far, so good. But how long will it take to get valid results if you have eight test versions and get 20 conversions per day? Entering this the calculator returns “This test will generate statistically valid results in 52 days”.  This is much longer than most marketers are willing to wait for results.

Even worse, most small- and mid-size companies get much fewer than 20 conversions per landing page per day. At Marketo, I spend about $100 a day on one of my ad groups. The landing page for that ad group has a great conversion rate (17%), but even this page only gets about five conversions a day (e.g. $20 per conversion). If I wanted to test 10 versions, I’d have to wait 239 days (or at least 87 days if I’m willing to accept Type II errors). Instead, I test only two versions — one champion and one challenger — and act on the results every 18 days.


Don’t get me wrong: I’m a fan of multivariate approaches to testing for sites that can support those volumes. The trick is to pick the right approach for the volumes your landing pages receive. However, no matter what approach you use, just remember that doing any testing is more important than arguing about what kind of testing to use.

Note: You can read more on this topic in my recent article at Search Engine Land.

Related Resources

  • Patrick Schaber

    Great points in this article. A low amount of conversions doesn’t always mean it’s a bad campaign. It could just be related to the business.

  • Karl

    Excellent post, you’ve crystalized the point so well I’m going to use this to bring some folks up the curve on my end. BTW- congrats on getting the nurturing product to beta.

  • Chris

    Good read, I’ve had some of those same problems and your page really helped us out!

Jon (@jonmiller) leads Marketo's product marketing, strategy, and content marketing initiatives. He is the author of multiple Definitive Guides including Marketing Automation, Engaging Email Marketing, and Marketing Metrics & Analytics. In 2010, The CMO Institute named Jon a Top 10 CMO for companies under $250 million revenue. Jon holds a bachelor’s degree in physics from Harvard College and has an MBA from the Stanford Graduate School of Business.

Read Jon's Blogs

Landing Page Testing: How Much Is Too Much?

Follow Us

Most Shared

true colors feat

True Colors: What Your Brand Colors Say About Your Business [Infographic]

Visual Gold 2

Visual Gold! The New Revolution of Content Marketing [Infographic]

Instagram Infographic

What Your Instagram Filter Says About You [Infographic]


Killing Time: How to Destroy Your Productivity [Infographic]


The Evolution of Modern Marketing Automation [Infographic]

Top Articles

Tighten your belt!

7 Strategies for B2B Marketing during a Recession: The Definitive Guide


The Evolution of Modern Marketing Automation [Infographic]


Big List of B2B Marketing Blogs

The ROI of Paid Social Media Advertising

The ROI of Paid Social Media Ads

100th blog post

Marketo’s 1,000th Blog Post: Our Modern Marketing Definition Revisited