Fundamentals of Direct Mail Testing

Download as pdf

Once during my younger years, I was presenting test results to a client along with recommendations on what we should mail next.

The client acknowledged that the test results supported my recommendation but said, “Our president doesn’t like that particular copy.”

My immediate response was “So?”

Admittedly, it wasn’t my most tactful hour. But when it comes to successful direct mail, it really doesn’t matter what we think is the best offer, copy or package design.

What matters is what our customers and/or donors respond to.

Fortunately, direct mail gives us a reliable method for measuring what motivates our customers to respond. And when we know how new elements of a mailing—lists, offer, copy, format and timing—affect response, we can drive our direct mail campaign toward greater profitability.

What we can’t do is ignore valid test results because they weren’t what we expected. Test results are the voice of our customers and they will lead us to success if we listen.


When to test

 A bad economy isn’t a good reason not to test. In many ways, it makes testing more imperative than ever. Not only will testing make your future mailings more cost effective, it will position you to capitalize on an improving economy.

The best time to test is when you have a successful control.

Too often, clients don’t want to invest in new tests because “our control is still working.” But waiting for the control to fatigue before you begin testing can mean disaster for your cash flow. Test from strength. However successful your control is, and however long it’s been the control, it can be improved.


What to test

Typically, the more dramatic a change you make in a package, the more dramatic the difference in results.

For example, when you need a breakthrough, test the components that have the greatest influence on the mailing’s success—lists, offer, format and copy. Forget about testing minor changes on page 3 or the color of the return envelope. Test big things for big results.

However, for clients mailing larger volumes, “tweaking” the control for incremental gains often makes sense.

For example, Client A and Client B both have an average response rate of 1%, with a $25 average transaction. Both test a new package that lifts results by 10%. The only difference is that Client A has an annual mailing volume of 1 million pieces while Client B mails 10 million pieces every year.

For Client A, with its smaller mailing volume, this 10% lift means an annual difference of $25,000 in income—hardly enough to cover the cost of the test. But for client B, the same 10% lift means $250,000 in additional income and that’s enough to have an impact on the organization.

But whatever you decide to test, do the math first. Testing a package that can’t recover its test cost in a reasonable time, or that requires an unrealistic rate of response to beat the control, doesn’t make sense. (See How to Use a Pre-Event Routine for More Profitable Direct Mail Testing.)


How much to test

When deciding how many pieces to test, the important thing isn’t how many test packages you mail but how many responses you get back.

Your test volume should be no fewer than the number you need to produce statistically reliable results. Personally, I like to see 100 responses. But mathematicians, using formulas I won’t even pretend to understand, say as few as 39 responses can be statistically valid with a 90% to 95% probability of accuracy.

You can use an online calculator to tell you how much to mail, but the concept is easy. Just take the control’s response rate and see how many pieces you need to mail in order to produce 100 responses. For example, with a 1% response rate, you would need to mail 10,000 pieces to produce 100 responses.

As a rule of thumb, you’ll want to limit your total test volume to no more than 20% to 25% of the total mailing. This way, if the tests fail, the returns from your control will help absorb the loss and preserve the operating budget.


How to Test smart

You will always have someone saying, “We’ve never done this before.” Or “I wouldn’t respond to this.” Or “This isn’t creative enough.”

But you can remind the naysayers that the purpose of testing isn’t to be creative. It’s to make money. And of course we haven’t done this before. That’s why we’re testing it now. And whether the office skeptics would respond to the mailing really doesn’t matter. You’re not mailing to them.

Testing is essential to profitable direct mail, but test smart.

  • Don’t take shortcuts. Do the math first.
  • Test things that matter most—lists, offer, copy and format.
  • Don’t be afraid to test things that increase your package cost. Typically, you’ll increase profits more by adding features and benefits than by eliminating components and benefits.
  • Don’t get “test happy” and schedule tests just to satisfy someone’s curiosity.
  • Before you mail, have procedures in place to credit the mailing with responses to the 800 number or website.
  • Evaluate test results using rollout costs. Test costs do not accurately reflect how future mailings will perform.

Taking time to understand and practice the fundamentals of direct mail testing goes a long way toward ensuring your campaign’s success. And whether you’re a startup or a “mature” mailer, the three most important rules of direct mail are the same: Test. Test. And Test.

Test smart and have the discipline to act on the results.


Tags: , , , , ,

Leave a Reply