Testing and Optimization

When The Boss Has A Crazy Idea, It’s Time To Start Testing – An Interview with a Chief Data Officer


Many marketers have been in a situation where their boss comes up with a crazy idea they would like to have implemented.  Staying focused on initiatives that work often requires pushing back on the idea, but the problem is that staying employed often requires entertaining the idea. This results in a conundrum for marketers.

Nicole Delma is the Chief Data Officer for the up and coming music startup RCRD LBL, and during a panel discussion at the Online Trust Alliance Forum she shared advice on how to manage “crazy ideas” that come from top management. At the time, Ms. Delma advised the following:

“Whenever an idea emerges from the top down that seems questionable – I almost always respond the same way. First, I’ll say, ‘Great – let’s set up a test and see what the numbers tell us’. The data is a pretty quick and foolproof indicator of whether a concept will resonate with our audience.“

Recently, I was helping a marketer implement some best practices when some “crazy ideas” from the marketer’s management started coming in. Remembering Ms. Delma’s solid advice on this issue, I sent her a few questions which she graciously answered:

Josh: During your session at the Online Trust Alliance Forum you provided some great advice on how to deal with marketing suggestions that might be coming out of left field.  How did you develop this approach?

Nicole: From many, many years of working in highly-competitive environments where extremely passionate stakeholders often held strongly opposing views – think of a Creative Director vs a CMO. In the end, most companies want to be successful and it’s hard to argue for an idea, no matter how good it seems, if the numbers don’t support its effectiveness.

Josh: When you do move forward with a test, what’s your general testing methodology? What kinds of sample size do you test with?

Nicole: I’m partial to A/B testing when I need to reach a conclusion quickly. When I have more time and the luxury of automating analytics, I’ll move forward with a multivariate test. Sample size depends largely on the level of confidence I need to reach in the results. In general, the more money that’s at stake – the higher the level of confidence I need to reach. You don’t need to be a mathematician to figure sample size out either. I’ve found that there are many great tools for non-statisticians available for free online that can quickly calculate the sample size you need in order to achieve statistical significance at various levels of confidence.

Josh: What was one of the more interesting and strange ideas that tested out well? Did you expand the program when you saw the positive tests results?

Nicole: I once had a request to deploy an email marketing campaign for a chemical dependency treatment center on behalf of a very well-reputed publication. I remember thinking it was a very bad idea to send such a controversial solicitation to a fairly conservative audience and was certain we would get a ton of unsubscribes and abuse complaints. Because this was a high-paying advertiser, I didn’t have the luxury of turning down the campaign without first running a test to back my suspicions. After some very crafty targeting, we managed to isolate a group of likely candidates to get this promotion. To my surprise, the campaign was one of the best performing we’d sent that year and the email delivery of the ad was a tremendous hit for both the advertiser and the publication. According to the data, my suspicions were unwarranted and I would have missed a big opportunity if I wasn’t open to running that initial test. The advertiser was so happy with the results, they bought more advertising the following year – largely due to the fact that the digital campaigns converted so well.

Thanks for all great advice Nicole!

For more great advice and tips, stay connected to Marketo’s Modern B2B Marketing Blog.