Month: November 2015

Acquire your own online donors

nickellinger, , , , Leave a comment

I know, it’s an odd title.  But every year, 30% of your online constituents go away not because they aren’t interested in you anymore, but because they changed their online contact information.  Seventeen percent change after six months.  That means that the half-life of your list is less than two years.

halffull.jpgIt doesn’t matter if it’s half full or half empty, just that it’s half of what it was.

This is going to be online acquisition week and I’ll go through a lot of tactics for bringing folks into your organization online.  But because of this, the best and the easiest way to acquire new online constituents is not to lose the old ones.  Conversely, it doesn’t make sense to start acquiring new online people if you aren’t ready to keep them.

So we’re going to go back to that old saw: data hygiene.  If you don’t believe in the necessity of data hygiene, then there’s a PSA you need to read.  We’ll look at the online specific ways to keep your data clean.

Scrubbing user-entered data.  Users will misspell their own names, then blame you when you address them by that wrong name.  Email addresses are no better, but those at least have a standard format that you can check.  Hopefully, you have an email validator on your forms, but you probably have data that predates your validation.  Some things to look for:

  • Does the email address end with a top-level domain (e.g., .com, .gov, .net, .org, .edu, .mil, or a country code)? Many an email has bounced back because it went to aol.con.
  • Does the email address have an @? If the email has a ! or # in it, chances are that, because these are right next to the @ symbol, @ is what was intended.
  • Does it bounce? That is, when you send the email, does it come back to you?  This is one reason that it’s important to send your email from a real email address – so you can change bounced emails or mark them for further work.  (Also, a real email address will help with charges that you are spamming people).

Put an email validator on your forms if you haven’t already.  Just to make that perfectly clear.

Run an ECOA service.  ECOA, or electronic change of address, is a service that functions a bit similarly to the national change of address (NCOA) registered with the USPS.  These services look at innumerable services across the Web to determine where a person might have gone.  You should ideally run both bad email addresses and email addresses that have not opened an email in a certain amount of time (say, six months).  If they haven’t opened an email in six months and they have a good email address…

Suppress chronic nonresponders from most emails.  If people aren’t opening your emails continually, they won’t miss you not sending emails (and, when you do send a very occasional email, it will be a bit more of a surprise).  And they won’t drag down your open rate or mark your emails as spam, making you more likely to survive email providers spam filters.

E-append your file.  Take your offline donor file and give it to an e-append service; they will return email addresses of people who would probably like to interact with you online, but haven’t yet given you their email address.  This also works like ECOA for bounced and non-opening addresses.

This isn’t going to stop your attrition from bad email addresses.  But it will help you hold on a little longer to the people who want to hear from you.

Acquire your own online donors

Breaking down your thank you silos

nickellinger, , , , , Leave a comment

As mentioned on Wednesday, there are a few different ways to thank your donors.   Thanking donors well, as we know, increases retention, average giving, and good karma in the world.

That said, we too often treat the way someone came into our organization, or the way they made their gift this most recent time, as the entirety of who a person is and how they are going to interact with our organizations.

Most donors have some combination of a mail box, an email box, a phone, a mobile phone, social media accounts, and more.  Yet we insist on assume that mail donors gonna mail checks, walkers gonna walk, phone-ees gonna give over the phone, and haters gonna hate.

If I were to create the stone tablets of nonprofit direct marketing, “Origin ≠ destiny” would be only slightly below “test everything” on the list of commandments.  Of course, I’m not going to create stone tablets, because we tested out of that in ancient Egypt.

It is certainly true that someone who started by donating through the mail is more likely to donate through the mail than someone who has only donated online.  This is because these two people have proven responsive to these two media.  However, it is not true that the mail donor will donate only through the mail.  It’s not even true that the mail donor prefers to donate through the mail – their origin may just have been how you reached out to her first.

Much is invested in creating multichannel donors – e-appends, email captures, telemarketing campaigns, mail conversion series, etc.  Yet we continue to acknowledge offline gifts offline and online gifts online and rarely the twain shall meet.

This is a pity because, while you will receive phone calls if someone is oversolicited by channels they didn’t want to be solicited through, you will rarely receive angry calls resulting from thanking someone too much.  Try to count the number of times someone has yelled at you “YOU ARE BEING TOO DAMN APPRECIATIVE OF MY SUPPORT!”.  If that number is existent, it’s at maximum finger-countable.

So I’ve just started trying some of these and while the juries are still out, early results are showing that they are bearing fruit.  This is in part because some of these are so darn low cost, if you can get the system right up front:

Thank your mail donors by email when you have the email.  I mentioned on Tuesday that speed of thank you is a key predictor of future support.  Let’s say that you are working with a caging vendor that will get acknowledgments out the day after the gift is received.  Chances are you aren’t going to improve significantly on that.  While neither snow nor rain nor heat nor gloom of night stays these couriers from their appointed rounds, neither will these or any other disaster or incentive get them to speed up.

But what if at the same time as your caging vendor sent the letter, they also triggered an email to the donor that said “[Name], thank you so much for your [amount] gift.  We just got it and you are already [the great thing your cause does].  You are going to get your official receipt and thank you in the mail in a couple days, but I couldn’t wait to tell you how much your support means to all of us here at [organization].”

Send an outbound voice mail as a thank you for online and mail gifts.  This is another way to get thank you’s out quickly when you have a phone number for the donor.  This also works for event donors.

Have a mail-based welcome series for online donors and/or constituents. There’s no reason a thank you and welcome needs to stop at the edge of the Internet.

Send a post-event package or series for your event participants.  This will help those participants go beyond donating just to the event and forging a deeper tie with your organization.

Thank you for reading.  Please leave other ideas for multichannel thank you’s in the comments, so we can all learn from each other.

Breaking down your thank you silos

Thank you to the bloggers who showed me how it’s done

nickellinger, , Leave a comment

I’m absolutely forgetting people, but here are some of the ones to thank:

Have a happy Thanksgiving, everyone!

Thank you to the bloggers who showed me how it’s done

There must be 50 ways to thank your donors

nickellinger, , , , , Leave a comment
  1. Write them a letter, Eddie Vetter.
  2. Send them a birthday card, Renard.
  3. Remember them on important holidays, Rutherford B Hayes.
  4. Acknowledge their support on important dates like their first gift’s anniversary, Mercy.
  5. Thank them with a prerecorded outbound voice message, Fezzig. (If you can’t tell already, not all of these rhymes are going to be winners…)
  6. Try that prerecorded outbound voice message to see if it will increase fulfillment rates among your telemarketing pledgers, Medgar Evers.
  7. Handwrite them a note, billy goat.
  8. Send them a copy of your annual report with a kind note and their name circled, Erkel.
  9. Make a personal call, Saul.
  10. Ask them to volunteer, dear. (Yes, really, some of your donors may want to become more involved in your organization)
  11. Thank them in person, Orson.
  12. Have a special area/table/zone for them at your next event, Clark Kent.
  13. Send them a member card, Jean-Luc Picard.
  14. Invite them to special briefings that are only for a member, December.
  15. Create a specialized donor thank you newsletter, Irish setter.
  16. Send them a copy of a book written by one of your in-house experts, Howard Kurtz.
  17. Create a year-end statement of their giving and the impact it has made, Sade.
  18. Use a survey to get their thoughts, Don Knotts.
  19. Ask your ED or another luminary to write a card in blue ink, then to make it look handwritten in bulk on a budget, run copies of it on the color printer, Harold Pinter.
  20. Conduct donor telephone calls in a town hall style, Kyle.
  21. Write quality stories, Jason Vorhees.
  22. Make high-quality and personalized online after-action pages and automated emails, Outlaw Josey Wales.
  23. Send them a staff white paper, Don Draper.
  24. Create a personalized support statement in infographic form, Norm. (NORM! How’s it going out there, Norm? It’s a dog eat dog world and I’m wearing Milkbone underwear (laughter from studio audience))
  25. Have excellent donor service, Neal Purvis (screenwriter on six James Bond movies. If you already knew that, you might be interested in one of my books here).
  26. Write them memos about the impact they’ve made and what is left to be done, hon.
  27. Send them pictures about the impact they are making and not of someone handing someone else a giant check, Beck.
  28. Welcome them strategically with a cross-channel series, Aries.
  29. Invite them to share their personal story, Rory (aka Mr. Amy Pond).
  30. Ask for why they give and personalize your thanks to what meaning to them, Clem.
  31. DM them on Twitter, Senator Vitter.
  32. Send them a letter that is written by someone whose life they’ve changed, Danny Ainge.
  33. Wish that the song was about 30 ways to leave your lover, Crispin Glover.
  34. Shoot a thank you video, Hideo.
  35. Throw a donor and volunteer appreciation party, Aarti (Sequeira of Food Network fame, of course).
  36. Ask them to vote on issues where you can live with any of the selections like member card design, Robert Irvine.
  37. Have a donor appreciation wall at your headquarter, Michael Porter.
  38. Message them on Facebook, Captain Hook.
  39. Make it easy for them to tell others about their support, Queen Consort.
  40. Honor and make sure they know you honor their particular and individual connection to your cause, Santa Claus.
  41. Have a phone bank thankathon from your employees and volunteers, Mouseketeers.
  42. Talk to them about the meaning they are giving to people’s lives, Douglas Adams.
  43. Call them for their opinion, Virginian.
  44. Send them an impact-focused news clipping, Rudyard Kipling. (Do you like Kipling? I don’t know; I’ve never kippled.)
  45. Invite them to hear, online or in-person, a guest speaker, Bunson and Beaker.
  46. Thank them with a celebrity if you have one connected to your nonprofit, Stephen Moffit (again, if you know who this is, one of my books might be up your alley).
  47. Reach out on Mother’s Day, Father’s Day, and/or Grandparent’s Day, depending on their gender and age, Larry Page.
  48. Send a February 14th valentine to the donors you love, turtle dove.
  49. Allow virtual access to whatever form of annual meeting you have, be it a conference, jamboree, or lobby day, Auntie May.
  50. Above all, write from, and to, the heart, Bart.
There must be 50 ways to thank your donors

5 simple rules of thanking donors

nickellinger, , , , , 2 Comments

Your acknowledgment/thank you’s should be:

For everyone.  E-very-one.  I once worked with a nonprofit that thanked everyone who gave $250+ on letterhead, $10-249 on copy paper, and under $10 not at all.  My first step was to thank everyone.  I know that the love discussion from yesterday can come under pressure when finances are tight.  But as an exercise, go back and look at the first donors of your last ten large bequests.  My guess is that the majority were under $20 and some under $10.  Thanking everyone is not only right and polite; it is a great investment in your long term.

That doesn’t mean that you have to ask for a $2 gift again, or in the same way.  You still have a responsibility to maximize your contribution toward your cause. But you do have to be grateful that they gave a gift.

mayathanks2

That doesn’t all mean that you shouldn’t differentiate your thank you’s.

Differentiated by reason for giving. Part of making people feel special is to treat them specially.*

Your different types of donors are supporting different types of things for different reasons.  Your monthly sustaining donors are giving, presumably, because of appeals you have make about the need for steady, predictable income.  Your advocacy donors – those who donated in conjunction with an urgent appeal for change – are going to be the exact opposite.  They will be looking to support the urgent rather than the constant need.  Thus, the messaging should be dissimilar for these.

Differentiated by lifecycle.  If someone is a lapsed donor who is reactivating, remember the prodigal son.  Now is the time to kill the metaphorical fatted calf and welcome them back and letting them know you appreciate that they are coming back, especially if you had been using lapsed-type “why has thou forsaken us?” language to get them back.

Similarly, new donors should have a whole new set of acknowledgment and onboarding messages.  I won’t repeat my blog post on onboarding for new donors and supporters, except to commend that piece to you.

Differentiated by amount given/quality of supporter.  This in part pragmatic – you want to invest more in keeping your better donors.  But it is oft said that smaller gifts are given from the heart and major gifts are given from the brain.  This is partly misleading, in that you have to engage the heart of your major donors first, but the pitch that you make to a major donor is more about the long-term impact that they are going to make with their investment.  Similar language just isn’t appropriate for a $10 donor, who is helping your mission, but not because of a transformative legacy they are looking to leave.  There too is a difference in messaging necessitated by a difference in reasoning.

And then there’s the obvious part – your largest donors should have higher touch acknowledgments.  That includes handwritten notes, personal phone calls, cards for special occasions like birthdays or holidays.  The key that many, many organizations forget is not to let high touch get in the way of a timely thank you.  If you normally send out thank you letters every day, but your high dollar donors get a letter from your ED that s/he sends out every 1-2 weeks, you are falling into this trap.  You are essentially differentiating backwards – your best donors are receiving the worst donor service.

The way to avoid this is to get the standard receipt and thank you immediately as you normally would do, then to follow up with your high-touch thank yous.  Few will mind if you say “I know you got our standard thank you last week, but I wanted to personally reach out to tell you how much your gift meant to me.”  Rather the opposite in most case.

This is imperative because one of the best predictors of whether someone will give again is how quickly and well they are thanked.  So, the final rule is:

Timely.  Get your receipts out as soon as you can, because of the impact on the next gift.  If it’s for a high-dollar donor, consider differentiating even on timeliness, with first-class postage on those thank yous.  Take a look at Blackbaud’s mystery shopper experience here.  Your donors are used to get receipts in week one (for the above average) or week two (for the just average).  You want to be above average to get those additional donations.

Thank you for reading.  Tomorrow, we’ll talk about different ways to thank your donors: some that are a bit nonstandard, all of which help express your gratitude.

* It’s statements like that that are the reason I make the big bucks.

5 simple rules of thanking donors

Why say thank you?

nickellinger, Leave a comment

Since this is the week of Thanksgiving here in the United States, I thought it would be fitting to do a week on thanking donors for their support.

Also, since I’m nothing if not data-driven, I didn’t call this blog post “Wherefore thank yous” because my previous attempt to go Shakespeare – Wherefore segmentation –  was one of my least trafficked posts so far.  So I’m going to have to get my fix some other way.

shakespearethanks

So, why do we say thank you to our donors and supporters?

If you just said because there is a legal requirement to do so know that I am virtually very disappointed in you.

As nonprofit marketers, love is our business.  Our goal is to have people who fall in love with our causes, who are interested to read our next mail piece or email, watch our next video, or take our next advocacy action.  For these people, the people who love us, donating is a pleasurable experience, connecting them to something they care about.

You may remember upwards of two things* from your freshman year Econ class:

  1. Something about supply and demand
  2. Individuals act to maximize their individual utility rationally.

Donating to a nonprofit does not fit the second one.  If we were logical, coldly rational beings, we wouldn’t donate to charity any more than we would appreciate a sunset, cry with our friends for their losses, and know that our children are the cutest of all possible children**.

Thank goodness people aren’t like this.  Thank goodness we give to those we don’t know, care for people we’ll never meet, and plant trees who will give shade to someone else’s grandkids.

As I write this, well in advance of its publication, Americans are caring about the people of a nation that many were so mad at a while back that they renamed potato side dishes to avoid using that country’s name.  Here you see both sides of the coin – the maddening demons in human nature exploding violently on to the innocent versus the millions better angels of our natures working together to heal, repair, and care.

My point, and I do have one, is that giving is an irrational act in the absolutely best possible meaning of the word irrational.  People love our causes.  In return, it is vital that we love them back.  Saying thank you is part of the social contract of giving and even if it were not, we would still do it because we are as good or aspire to be as good as those who are giving of themselves to our causes.

So was a little bit more flowery than I had intended.  Let me assure you, tomorrow, we’re going to get back to how acknowledgments can help us raise net revenues again.  Because make no mistake, a quality acknowledgment program can and should net additional money in the long term.  Just because it is the right thing to do doesn’t mean it will require sacrifice.

* I hate to brag, but I was an Econ minor, so I remember a third thing: “Something about inflation.”

** Clearly, this is wrong, because mine are and everyone else’s are competing for second.

Why say thank you?

Testing beyond individual communications

nickellinger, , Leave a comment

So far, the testing that I’ve discussed is how to optimize a communication or overall messaging.  The next step is trying to answer fundamental questions about the nature of your program – things like how many times to communicate and through what means.

There is a pretty good chance that you are not communicating enough to many of your constituents.

But wait, you say.  We send out a mail piece a month, have multiple telemarketing cycles per year, and have both a monthly e-newsletter and semi-frequent emails on other topics.  Our board members and staff who are on our seed lists are consistently on me, you say, that we are communicating too much.  And we get donors who complain that they are getting a mail piece before their last one was acknowledged.

However, remember in the discussion of segmentation that more donors are saying their nonprofits are undercommunicating, not over. That means that the average number profit needs to be communicating more than it is.

And the concern that you are annoying people with asking for money comes from an oft-quoted and concerning inferiority complex from the nonprofit.  We have to believe that we are good enough to merit a gift and making an appropriate ask to be effective.  We want to give our donors an opportunity to be a part of something powerful and transformative.  Remember that if we do our jobs well, donating to our organization is a positive experience.

So how would you test whether you are communicating often enough/too often?  The first step is to figure out where you are as a control with a cross-medium communications calendar.  This is easy said than done, but it’s a necessary first step.  This need not be perfect; as you are going to want to have some communications that are timely and focused on current events, you may have to have some placeholders in place that simply indicates “we’re going to email something here.”

Then split test your file and test, so that part of your file gets X communications and another gets X plus or minus 1.  I’d suggest plus.  Then measure the total success of the communications.

I once helped lead a test where we took mail pieces out of our schedule during membership recruitment.  We would send a piece or two, then wait to see if those donors would donate before sending to them against to make sure that we were addressing them properly as either a renewed donor or as someone who has not yet renewed.  Each individual piece in the resting membership series had a significantly better ROI and better net than the more consistent appeal series.

Yet the appeal series brought in more money for the organization and the mission overall.  I would argue, as I did at the time, this is the actual important metric.  If you want to look at metrics like ROI or response rate, your best opportunity is to send one letter to your single best donor – you’ll get a 100% response rate and ROI percentages in the tens of thousands or more.

But for real life, the goal is more money for more mission.  So overall net is the metric of choice.

The easiest campaigns to add to are the ones that already have a multistage component.  Let’s say you have a matching gift campaign that goes mail piece 1, email 1, mail piece 2, email 2 (with two weeks between each).  A way of testing up would be to look at doing mail piece 1, email 1 + mail piece 1.5, mail piece 2 + email 1.5, email 2 (so there’s still two weeks between each set of communications, but they double up in the middle).  That would be adding a mail piece and an email and if you test both of these with net as your goal, you will have a better framework for the campaign in the following year as well as for additional testing throughout the year.

With email only campaigns, there’s another way of checking whether you are over-emailing your file – looking to see if your total opens and clicks fall.  There is a point at which open rates and click rates will begin to fall; however, you shouldn’t worry too much until adding another email not only lowers your open and click rates but lowers your total number of opens and clicks (similar to a focus on total net, rather than net per piece).

This tipping point in email is probably well past where you think it is.  Hubspot did a study of emails per month on both open and click-through rates.  The sweet spot with the highest open and click rates was between 15 and 30 email per month.

That’s right – opens and clicks went up until you got in the range of daily emails.  Things went downhill after 30 days.  So if you are sending more than daily emails (on any day but December 31 or the last day of a matching opportunity), you might be emailing too much – so take that as a cautionary tale for the .0001% of you who are doing this.  For the other 99.9999%, hopefully this will give support for the business case for testing up on your emails.

There are three tricks to cross-platform testing:

  1. There is a whole science of attribution testing. If you have the ability to look at this literature and your data systems will support this, go for it.  However, most organizations of my experience don’t have all of their data in the same place initially, making this exceedingly hard.  Thus, this sort of testing up/down for cadence should look at sources of revenue by audience test panel rather than through what medium the donation is made.  You may be surprised how much adding a mail piece increases your online revenue or adding a telemarketing cycle boosts the mail piece.
  2. Unlike with strictly piece-based attributes, I’d argue you have to test every cell here because there are interactions among the means of communication. It may be that mail + mail is better than mail and mail + phone is better than mail, but that when you have mail + phone + mail, you have diminishing returns that don’t compensate for doing both mail pieces.
  3. You will have to be vigilant about the creation of your testing cells. ft_15-07-23_notonline_200pxAs much as you would like to call everyone who has a phone number or email everyone who has an email address, and use those who don’t have a phone number or email on file as a control audience, those are different types of donors.  Pew has a great summary of the non-Internet users of the US at right.  Even if you looked just at the age and income variables, you can see how this would make your control audience look very different from your non-control.In reverse, 66% of 25-29 year olds live in houses where there is no landline, compared with 14% of 65+ year olds, according to the National Center for Health Statistics.

    So, if you think of the average person for whom you have a phone number, but not an email address, that person looks very different from the one where you have an email address, but not a phone number.  Thus, you have to either control for all demographic variables in your assessment (hard) or split test people by means of communication that you have available. (marginally easier)

Thanks for reading and be sure to let me know at nick@directtodonor.com what future topics you’d like to see.

Testing beyond individual communications

Testing for smaller lists

nickellinger, , , , , , Leave a comment

One of my favorite non-Far Side single panel cartoons is

miracle

 

This is often what it feels like to be a small nonprofit or small division of a nonprofit.  You know exactly what you would do if you were big.  But you aren’t (yet).  And absent that miracle in the middle, you aren’t going to be there soon.  It feels like a Catch-22 – you aren’t big enough to test, but you aren’t going to enough to test unless you test.

A lot of people have this problem.  One of my favorite conversion sites, unbounce.com, recommends that you have 1000 conversions per month to do A/B testing.  That takes a large nonprofit to accomplish.  Like the Oakland As in Moneyball (both book and movie are recommended), you have fewer resources, so you are going to have to be smarter than your competition other worthy causes.  Here are some tips on how:

Learn what’s important first: Before you do your first test with online traffic, look at your analytics reports (do you have Google Analytics on your site?).  Where are people bouncing from your site?  Where are they dropping out of the donation process?  What forms aren’t converting?  You may be able to do more with one-tenth the traffic or donor list if you are testing the things that will matter to you.

Steal from other people first: There are some things that are almost immutably true.  Requiring more information on a form means lower conversion rates.  Having a unique color for your donate button that stands out from the other colors on your Web site will increase clicks.  Using a person’s name, unless it’s in a subject line, will likely increase response rate.  I commend the site whichtestwon.com to you.  I’ve had the privilege of presenting at their live events and the type of information that comes of them in terms of what others have tested first will save you time and money on things you can do, rather than test.

Go big: I’ve talked about things like envelopes and teasers and things to test.  If you don’t have a large donor or traffic base, ignore that.  You want to be testing audience and offer – the things that can be global and game changing.

Test across time: If you are testing an audience, an offer, or a theme, that doesn’t have to be accomplished in one piece or email.  Rather, you can test it over a year if you want.  Let’s say you want 25,000 people in each testing group, but only have 3,000, you can get a similar feel for the response to large-scale changes over nine pieces, rather than testing it all in one.

Require less proof: Chances are you are used to doing more with less already.  If you are Microsoft, you can run your test until you get 99.9% certain you are correct.  You should be willing to be less certain.  Some nonprofits choose 80% certainty as their threshold.  Even 60% can give you directional results.  Bottom line, this is a restriction you may be willing to relax.

Test cheaply:  Testing direct mail and telemarketing is expensive.  You want to do your learnings on your site with Google Analytics and either Google’s optimization tool or Optimizely, in email, or on social media.  I would go so far as to say that even larger nonprofits don’t want to test an envelope teaser that they haven’t already tested as a subject line to see if it grabs attention.  Survey tools like SurveyMonkey or Zoomerang can also help you pre-test your messaging either with your core audience (free) or with a panel of people who fit your demographic target (cheap, if you can keep your number of questions down).

Get testing subjects cheaply: I know it sounds like I’m in Google’s pocket, but they have many nonprofit solutions at the right price for smaller nonprofits – free.  One of these is Google Grants, which allows you to use their AdWords solution with in-kind donated advertising.  Get this now, if you don’t have it.  We’ll do a whole week on AdWords at some point, but in the meantime, if you have a form you are testing and you don’t have enough traffic, pause all of your campaigns except the ones directed to that form.  You will get your results a lot more quickly.

Test by year: It’s not an ideal solution, but if you test one thing one year and then another tactic the next year at the same time, you can get a gut feeling as to what is more effective.

Avoid word salad: Consider the time on West Wing (which I remember better than many real-life presidencies) when the Majority Leader who was running for president was asked why he wanted to be president:

 

“The reason I would run, were I to run, is I have a great belief in this country as a country and in this people as a people that go into making this country a nation with the greatest natural resources and population of people, educated people … with the greatest technology of any people of any country in the world, along with the greatest, not the greatest, but very serious problems confronting our people, and I want to be President in order to focus on these problems in a way that uses the energy of our people to move us forward, basically.”

Good writing converts.  Good writing mandates active verbs and few adverbs (my personal crutch).

 

“It’s an adverb, Sam. It’s a lazy tool of a weak mind.”
— Kevin Spacey in Outbreak

Good writing ignores the mission statement, discards stats, eschews your jargon, and touches you in a very personal place.  OK, perhaps not that active a verb.  I’m talking about your heart, you sicko.

Don’t test good copy versus bad copy.  Come up with your best before you test, lest you learn what you already should know.

Conspire.  You have coalition partners and people who are in similar positions around you.  Get out into the big blue room and see what they are doing.  And be generous with your own tests – deposits in the karma bank rarely fail to pay interest.

Finally, embrace the advantage of being small.  As a smaller nonprofit, you are going to have to be smarter about testing than bigger ones.  But you will be able to swing for the fences while they are still trying to get their different versions of teaser copy through the Official Teaser Copy Review Subcommittee.  You can be bold and find your voice honed to what works, rather than what your boss’s boss’s boss’s brother-in-law said you should try out over Thanksgiving dinner.

Tomorrow, we’ll go into some testing modalities that allow you to test things beyond a single communication or theme.

Testing for smaller lists

How to test more than one variable at a time

nickellinger, , , , Leave a comment

Yesterday, I articulated why you would want to test more than one variable at a time.  The trick is testing multiple things and still getting results you can act on.  Here’s how to do this without taxing your math skills and without exceeding the capacity of your list.  We’ll talk about mail testing as online testing is far easiest, given instant results, lower testing costs, the ability to see what specific calls to action (i.e., links) are getting the most action, and the ability to sequence tests (e.g., testing a subject line earlier in the day, then rolling out with a winning subject line and two test copy versions in the afternoon).

Let’s look at a simple testing matrix, where you would want to test two variables (let’s say envelope and copy) with two options each.  It’s easy enough to do the multiplication and say that, for a “test one variable at a time” approach, you would want four test cells; let’s call them A1, A2, B1, and B2.

Copy 1 Copy 2
Envelope A
Envelope B

By testing every option, you get the best testing results.  Likewise, you can see intuitively that no two testing cells will give you full results: if you tested A1 and B2, for example, if B2 did better than A1, you wouldn’t know whether it was envelope B or copy 2 (or both) that caused the lift.  And any other combination would not test both variables.

But three testing cells will get you the data you need.  Let’s say you did not test cell B2 and get the following results:

Copy 1 Copy 2
Envelope A  $.50  $.75
Envelope B  $1.00  ?

Here, you can see envelope B did 50% better on a gross per piece basis than envelope A when it had copy 1.  You can also see that copy 2 did 100% better than copy 1 when in envelope A.  From this, you can deduce that B2, which wasn’t tested, would have the best results at about $1.50 gross per piece.  This is because that is both 50% better than B1 (the result of the envelope lift) and 100% better than A2 (the result of the copy lift).

And if you ever do have results this clean, please let me know about them.

This is just a 2 x 2 matrix, but it illustrates a point that will apply to greater multivariate testing – any time you have significant results in three out of four of a 2×2 matrix, you can deduce the fourth.  I call this the testing L because of the shape it makes on your testing matrix.  All you need to do is to iterate a traditional A/B test and you can get a robust testing strategy.Let’s say you want to test three envelopes and three sets of copy.  Here you would only need five intersections instead of the nine you might expect:

Copy 1 Copy 2 Copy 3
Envelope A  X  X
Envelope B  X  X
Envelope C  X

You can see how the L technique would help you determine the remaining blank cells one at a time.

What if you wanted to add another dimension?  In addition to your three envelopes and three sets of copy, you’d like to also test three ask strings, for example.  Here, you’d only need to test both ask strings as a replacement for the existing ask string.  So to the above testing matrix, where you are testing:

A1$ (where $ is the existing ask string)
A2$
B2$
B3$
C3$

Adding, say, A2+ and B3% would give you an L over this new dimension (remembering that you were able to solve for the entire testing matrix in two dimensions).  If you were to try to test all 27 possible combinations with a quantity of 25,000 each, that would be a prohibitive 675,000 mailings.  This makes it a much more manageable 175,000 people.

This assumes, of course, that you want 25,000 people in each test panel.  There are some more advanced statistical techniques that would allow you to mail a larger number of intersecting tests.  These make it so that each cell is no longer independently projectable, but can still give you aggregate results.  I personally like having some experience at the cellular level, especially if there is a strong control already in place (in which case I will weigh the testing cells toward the control package’s attributes), but this is a possibility.

But even 25,000 is too large for some.  In fact, one of the more frequent questions I hear is how to do testing when you have a small list.  That’s the topic for tomorrow; then, Friday, I’ll talk about cross-platform and cadence testingcross-platform and cadence testing – testing that goes beyond the “here’s a communication; here’s another communication” type testing.

How to test more than one variable at a time

“Test one variable at a time” is a lie

nickellinger, , , Leave a comment

It’s not an intentional lie and its heart is in the right place, but it’s wrong nonetheless.

The reason people will tell you to test only one variable at a time is that you want to be able to isolate why what happened happened.  So, for example, if you changed the teaser on an envelope and sent it to an equivalent audience at the same time with the same contents in the envelope, if there was an increased response rate, that is a winning test because of the envelope.

This is a fine way to test if there’s only one thing you want to learn at a time.  You can refine your program this way, getting better and better.  This is the direct marketing equivalent of kaizen – the practice of continual improvement popularized in manufacturing, but now applies to much strategic thinking.

But there are some significant problems with this:

  • You can’t test synergy between variables. Let’s say you have a subject line you’d like to test.  However, it may work better with a different version of your email; after all, you wrote the original subject line for this email – the new one may not fit as well.  Testing one thing at a time may not allow us to test the most coherent versions of each of your offers.
  • It can lead to small ball, where you only test things at the most granular level. In his book Fundraising When Money is Tight, Mal Warwick talks about testing teaser copy 25 different times with almost as many clients.  Of the tests, 21 – 84% — showed no difference (and these were at quantities that would have shown a difference had there been one).  This is an OK learning if you can learn other things from the package as well, but if that’s all you learn, you’ve investing in testing without any return more than four out of five times.
  • It can’t make significant leaps forward. Let’s say you have a control piece in decline.  You know it needs to be replaced because of its response rate.  Or maybe, in a more positive outlook, you’ve accomplished the goal you were striving toward.  Either way, the way to get rid of this piece isn’t to test the envelope one year and the response device the next year – you have to test more than one variable at once.

All in all, this violates a rule you should have for yourself – to learn as much as possible whenever possible.  Think of it as if you were trying to reach the highest elevation on Earth.  If you had the rule of “go up from where you until you can’t go up any more,” you will reach a peak higher than you are currently, but by no means the highest point possible.  Similarly, if you had the rule “climb to the highest point you can see, even if it means going down a bit,” you will be doing better and getting higher than you were, but this iterative process will not lead to you having to don an oxygen tank anytime soon.

So it is with testing.  Testing one variable at a time will get you closer and closer to your local maximum, but not the global maximum.

But the basis of the argument for variable isolation is not untrue.  You still need to be able to figure out what works and what doesn’t.  The trick is sussing out what did what in your test.  That’s what we’ll cover tomorrow: how to layer multiple single tests to get results you can act on.

“Test one variable at a time” is a lie