Every field does its best to be impenetrable to outsiders and the world of online testing is no different.  We measure KPIs instead of “important things.” The differences among CPA, CPC, CPM, CPR, CTA, CTR, and CTOR are all important (for example, one of these can save your life, but isn’t an online metric) and there are TLAs* that I haven’t even talked about.

So this week I want to look at measuring online successes (and not-yet-successes), but first, we need to get our terms straight so we know what we are trying to impact, starting with email metrics.

For me, this is easiest picturing the steps that would go from email to action.  An email is:

• Sent
• Delivered
• Opened
• Clicked upon (or unsubscribed from)
• Responsible for a completed action

Almost all of the other important metrics are ratios of the number of people who did this (or the number of unique people who did this — unique meaning the number of people who did something, not the number of total times something was done.  For example, 1000 people clicking on a link once and one person clicking on a link 1000 times will have the same click-through rate, but very different unique click-through rates).

The most important of these ratios are:

Delivery rate: emails delivered divided by emails sent.  This is inexact, as different email providers provide different levels of data back to you as to whether an email was a hard bounce (email permanently not delivered) or soft bounce (temporary deliver issues like full email box or email message too large).  But as long as you are using the same email program to send your emails, you will have consistent baselines and be able to assess whether it’s getting better or worse.

Open rate: emails opened divided by emails sent.  There are a couple minor problems with this.  First, opens can’t be calculated on text emails.  That is, only HTML emails have the tracking to determine whether they were opened or not.  Second, some email clients allow people to skim the contents of an email in a preview pane and count it as an open.  Third, some email clients don’t count an open as an open (even if the person interacts with the message) if it is only in a preview pane.  So it’s an inexact science.

However, open rates are still a good, but not perfect, measure for testing the big three things that a person sees before they open a message:

• Sender
• Subject line
• Pre-header.  (If you don’t know what a pre-header is, check out my post on that here)

Why isn’t it a perfect measure?  Because it’s hackable.  Let’s say your control subject line is “Here’s how your gift saved a life.”  If you test the subject line “Your gift just won you a Porsche,” it might win on open rate, but you’ve lied to your donor (unless you have an astounding back-end premium program).  That will spike your unsubscribe rate and lower your click-throughs**.

So you probably want to look at this in combination with click-through rates (CTR).  This is another one of those metric pairs that prevent you from cheating the system that I love so much.  Click-through rate is number of people who clicked (assuming you are using unique click-through rate) by divided by emails sent.  It’s a good way of measuring how well your content gets people to (start to) take action.

Another good way to look at how your email content performs is click-to-open rate (CTOR).  This is number of people who clicked (assuming you are using unique CTOR) by divided by opens.  As you might guess, it fits very nicely with the previous two metrics.  Let’s say two emails both had 1% click-through rates.  One of them might have had a 20% open rate and a 5% click-to-open rate; the other might have had a 5% open rate and a 20% click-to-open rate.  In this case, you’d want to see if you could take the subject, sender, and pre-header of email #1 and combined it with the body copy of email #2.

You also need to look at unsubscribe rate (number of unsubscribes divided by number of emails sent), but not as much as many would think. If it starts going too high, you may want to worry about how you are acquiring your subscribers; likewise, it’s good to compare unsubscribe rates across types of constituents (that is, do your advocacy supporters unsubscribe faster than your white paper downloaders?  Perhaps it’s time for more segmentation and better advocacy content).  But don’t let it drive the boat.

Finally, you want to look at conversion rate: those who took the action divided by those who clicked through.  While not strictly an email metric, I include it here because a person could try the same underhanded tactic with the Porsche to boost click-through rates (to bait and switch the clicker) and because it’s so vital to be measuring and optimizing against.

But that’s another post.

If you want to benchmark your metrics, I strongly recommend M+R’s benchmarks study here.  And please sign up for my free newsletter here.  We have strong metrics (40% open rates and 8% click-throughs) so others are (hopefully) finding it to be useful nonprofit direct marketing content.
* Three-letter acronyms

** Also, it’s wrong.  Don’t do it.