Yesterday, we talked about the key metrics you want to look at in Excel – 13-14 indicators that speak to you about progress and testing results.

However, a direct marketing Muggle will look and these data and say “Huh.  Interesting.” This is direct marketing Muggle code for “this is not interesting and it makes me think of my Algebra II class, which was taught by a nun.”

While you will want all of the data, you will want a skinnier, clearer chart for others, preferably with colors that call out what is actually important.  Let’s look at a fairly standard test – your thesis was that extra personalization in the letter would increase average gift versus your control.  Here’s what this could look like:

The first thing to notice is that your hypothesis was wrong – average gift didn’t go up.  But now you have another decision – should you pay for the additional personalization in the future?

You, as a direct marketing professional, can read this chart.  The increased personalization caused response rate to increase.  As a result, gross income per piece went up and net income per piece went up.  However, return on investment went down; the additional investment didn’t bring in as much as the investments before it.  What would you recommend to your boss?

This is a judgment call based on your goals for your program.  One good approach would be to call for a retest – possibly with even more personalization or to see if you can get the personalization costs down or different ask strings to try to boost average gift.  This is clearly not in the 95% percentile one way or the other (which are other good fields to add to your spreadsheet when you get more advanced), so more testing would be good.

But I know which one I would mail more quantity of when the next test is done – I would use the personalization version as the control.  For me, net per piece matters more than ROI.  Our donors’ time is a scarce and valuable commodity.  There are only so many times you have the opportunity to get in front of them, so if you have the opportunity to maximize their investment of their time, versus trying to go for cost control in borderline cases like this one.

Charity Navigator would disagree with me, as they focus on cost of fundraising, so that’s another point in my argument’s favor.  Remember the Charity Navigator Constanza test – hear what they have to say and do the opposite and it will be to your benefit.

So now you have your course of action.  Now you have to have other people see it your way.  Time to explain it:

The first thing to note is that it’s legible.  The second is quantity and absolute gross, net, and cost numbers are gone.  These don’t have any relevance to the decision over what to roll out with.  If you leave them in, there’s a natural human temptation to think biggest = better, especially when it’s called revenue and has a dollar sign in front of it.  For a layperson, it’s good to eliminate these distractions.

Then we’ve color-coded the winning parts.  Control wins on cost; test wins on response rate and ROI, gross income and net.  This helps draw attention to the salient bits.  It is amazing how much these little steps can help focus minds.

You will note that I left ROI in there, even though it is evidence that does not support the case you are trying to make.  I’ve talked about testing as a central commandment on the direct marketer’s tablets.  But testing is nothing if there isn’t intellectual honesty.  You have to make the case, but also give your team all of the information to challenge you and make your arguments better.

This is usually where the aesthetic marketers get us data-driven marketers.  They tell quality stories based not on what is true, but on what we wish were true.

We must become equally good storytellers, because a good story plus data beats just a good story.  On Thursday, I’ll talk about how to present data in a compelling way, but first, we have to figure out how to measure our metrics.