It sounds like a non-sensical question. And it highlights another major difference between offline and online direct marketing — trackability.
Those who live in the digital marketing space are used to being able to track what happens with their emails and campaigns down to the user level. They complain when tracking pixels don’t work quite the way they are supposed to on every device and aim for ever better attribution models to understand where their investments are going.
Those in the offline space are used to sending something out and waiting for results. And waiting. And waiting.
Further, they are used to looking at packages as a whole. They get one result: did someone donate (OK, two: and how much)? Because of this, it’s tempting to think of mail testing as the thumbs up or thumbs down as in the Roman coliseum.
But you can find out things like your offline open rates and tweak them to your heart’s content. Take a simple 2X2 testing matrix.
While you won’t be able to tell what your actual open rate was, you can to content yourself with relative open rates. With online, you have an intuitive feel for whether a 20% open rate is good or bad compared with the emails around it (and whether they generally are opened at 10% or 30%). This same relative weighing works well in mail. If 20% more people donating with envelope A than with envelope B all other things being equal, then you have a 20% better open rate with envelope A.
Similarly, if letter C does better than letter D by 30% with the other parts of the mail piece staying constant, you have a 30% better “click-through” rate.
And you probably already know the trick that you only have to test three of the four quadrants here. If envelope A beats B when they both use letter D and letter C beats D when they both use envelope B, chances are pretty good that the winning test is envelope A with letter C, even though that wasn’t a tested combination.
But what you may not know is the right algorithm can do this writ large with a wide variety of variables. Ask your vendor(s) if they can run permutations that will allow you to figure out what happens when you five envelopes, four offers, three letter permutations, six different ask strings, and so on. They should be able to create a variablized stew that helps you run a number of tests at once.
The other thing that I’d recommend is not just taking a page from the online playbook, but using online tools to test your efforts first. Don’t know if your teaser copy will work well? Try it as an email subject line or a CPC ad headline first. While the audiences are a bit different online and offline, catchy is generally catchy and boring is boring. Working out details like this online can save your testing for things that can actually help you get to know your donor better, leading to more valuable communications and donors.
(Or, better yet, scrap your teaser copy and test a plain white envelope — it may have the best open rate of all.)