Portals eNewsletters Web Seminars dataWarehouse.com DM Review Magazine
DM Review | Covering Business Intelligence, Integration & Analytics
   Covering Business Intelligence, Integration & Analytics Advanced Search
advertisement

Resource Portals
Business Intelligence
Business Performance Management
CRM
Data Integration
Data Quality
Data Warehousing Basics
EAI
EDM
EII
ETL
More Portals...

Advertisement

Information Center
DM Review Home
Web Seminars & Archives
Newsletters
Current Magazine Issue
Magazine Archives
Online Columnists
Ask the Experts
Industry News
Search DM Review

General Resources
Bookstore
Industry Events Calendar
Vendor Listings
White Paper Library
Glossary
Software Demo Lab
Monthly Product Guides
Buyer's Guide

General Resources
About Us
Press Releases
Awards
Media Kit
Reprints
Magazine Subscriptions
Editorial Calendar
Contact Us
Customer Service

How to Design a Valid Sampling Test

  Article published in DM Direct Newsletter
August 15, 2003 Issue
 
  By Amoy X. Yang

Direct marketers need a sample test to tell if a list selection will work for a promotion package. How big should this test be? As a rule of thumb, a right-size sampling for a test should be small enough to keep the cost minimum but large enough to ensure its statistical significance for rollout projection. The challenge is how to balance this dilemma and come up with an optimal solution.

Sampling size can be theoretically solved with a well-developed equation as function of three variables: m (mean = response rate), z (confidence related) and e (allowable errors). This approach, however, seems impractical due to its tedious math development with statistical complexity.

Instead, many direct marketers use what they feel comfortable to determine a test size - an intuitive-based method. Using this method, problems occur - either the sampling size is too small to make a valid projection or there is uncertainty whether the rollout responses will mirror the test result. It ends up that the test did not remove uncertainties since sampling size was not statistically designed for its scientific accuracy. 

Let's use an example: a sample test with 2,200 names that generate 2.00 percent response rate. Without math calculations, you take a chance, hoping the rollout would repeat or be close to test result. Realistically, luck doesn't work because you don't follow a statistical rule to gain your winning odds. Testing needs money and time. Why test if it does not provide necessary insights for rollout decision making? The bottom line is the sampling size must be statistically valid to infer the desired population with some certainty.

In most direct marketing analysis, the responding result is a categorical "dummy" variable, since all responses can only be scored as two options: 1 = response or 0 = no response. Under this 0/1 statistical distribution, only a tiny responding portion from total mailings will contribute to the sampling significance study while the majority of no-response does not signal interest or intention to be projective for the population. With most response rates at no more than 5 percent in today's direct marketing arena, sample size could be much larger for its statistical purpose than many people anticipated. This is why sampling methodology used for normal distribution cannot be simply imitated to the case of 0/1 responding distribution.

For a quick solution, let's skip time- consuming math procedures and examine a sampling model used for 0/1 distribution, as show:

n = {-4z2(2e2- m+m2)+[(4z2 (2e2-m+m2)) 2-4(4e2)z4 (4e2-1)]1/2}/ (2*4e2)

Where n is the sampling test size that relates to three variables including m, n and z - m (mean) is a response rate, estimated from past performance(s) or experience; e stands for an allowable error that defines a projective precision range; and a z value is a confidence-related coefficient. Say, z = 1.96 at 95 percent confidence level. Assuming all three variables are given, m = 2.00 percent, error = +/-10 percent and confidence level = 95 percent, n can be solved by the equation above as 18,865.  This is how large sampling size you need to target your pre- established precision and confidence. If 2.00 percent response rate is obtained from 18,865 mailed names, you can statistically predict that a rollout result has 95 percent probabilities to fall between 1.80 percent (10 percent less than m) and 2.20 percent (10 percent more than m). The other 5 percent are chances that actual rollout response could be out of range (1.80 - 2.20 percent). You actual risk is 2.5 percent, which is split a half of 5 percent below lower boundary and another half above upper boundary. Provided your break-even point is 1.80 percent and other things being equal from test to rollout, the test indicates that you have 97.5 chances to win. It is your decision whether you can bear 2.50 percent risks. This sampling model gives you a definite answer about how large you test should be to ensure a statistical valid projection for rollout campaigns. Simply plugging your desired numbers into equation, you can figure out a valid sampling size to meet your specific test goal. 

For your convenience, Figure 1 provides a sampling table that contains precalculated sampling sizes in terms of e and confidence, with a given e at +/-10 percent, which is well accepted in direct marketing. Now, go back to the case at the beginning, you actually have a 50 percent chance of being within projection range when n =  2,231, as exhibited in Figure 1.

Mean(m)=Response Rate

5.00%

4.00%

3.00%

2.00%

1.00%

Allowable Error (e)

(+/-)10%

(+/-) 10%

(+/-) 10%

(+/-) 10%

(+/-) 10%

Lower Boundary

4.50%

3.60%

2.70%

1.80%

0.90%

Upper Boundary

5.50%

4.40%

3.30%

2.20%

1.10%

Z-Value

Confidence

Sample (n)

Sample (n)

Sample (n)

Sample (n)

Sample (n)

0.674

50%

865

1,092

1,472

2,231

4,508

0.842

60%

1,349

1,705

2,297

3,482

7,035

1.036

70%

2,043

2,581

3,477

5,271

10,651

1.282

80%

3,128

3,952

5,325

8,071

16,309

1.645

90%

5,150

6,507

8,767

13,289

26,852

1.960

95%

7,312

9,237

12,446

18,865

38,121

2.576

99%

12,630

15,956

21,499

32,586

65,848

Figure 1: Sampling Table

As you can see,a sampling table provides quick multiple findings in a broad picture to target your test goal.

What is the impact on projection if the actual test mean is different from your estimated one? Since the test was completed, you have to use new actual m to adjust other variables by sampling model. If actual m > estimated m, you will have a  more accurate projection since you did a test with a larger size than it should be. The challenge is the opposite outcome, when you tested a smaller size based on overestimating m. All you can do is lower your projective accuracy to complement a smaller test size. Recall the example where 18,865 names were applied to test with an estimated m at 2.00 percent as well as e = +/-10 percent at 95 percent confidence. If the test, unfortunately, turns out a lower response rate at 1.00 percent, you have to adjust confidence to 80 percent plus/minus under your current situation. Therefore, m should be carefully arranged, especially in cases such as test for big control list, 10-15 percent deduction for rollout mean projection or direct responses at narrow margins.

How big test should be depends on how your direct mail performs by mean and how you obtain predictive accuracy by error and confidence. A statistical valid test size can be mathematically calculated with a sampling model under 0/1 distribution. Alternatively by using a sampling table with precalculated sampling sizes, direct marketers can also do quick searches in different direct marketing scenarios. It is definitely helpful to understand sampling variables and their interactions in order to achieve an optimal sampling size while minimizing potential problems along the way.

...............................................................................

For more information on related topics visit the following related portals...
Database Marketing and Market Segmentation.

Amoy X. Yang is a database marketing manager with AAA Arizona at Phoenix. He can be reached at 623-326-3922 or amoy_y@yahoo.com. With more than ten-year experience in  quantitative analysis, he created this new sampling methodology and applied it to multiple DM campaign tests. The model has been proven consistent and successful in its statistical significance.

Solutions Marketplace
Provided by IndustryBrains

TechExcel CRM
TechExcel CRM sets the standard for high-end CRM: powerful, configurable, affordable and easy to use.

Looking for IT Service Management Software?
A complete service solution, Magic Service Desk combines best-in-class software with ITIL best practices, the industry standard framework for IT service delivery and support. Visit out web site and download our complimentary white papers.

Intuit Track-It! Help Desk & CRM Software
Intuit IT Solutions provides Track-It! - the leading help desk software solution for employee & customer self-help, call tracking, problem resolution, remote control, asset management, LAN/PC auditing, and electronic software distribution. Free demo

Customer Relationship Management for IT
Web-based CRM and more with Autotask: Great business management software optimizes resources and track billable project and service work. Get a demo, then try it free with sample data. Click here for your free trial!

TopLine Leadership Sales & Sales Mgmt. Training
Offering two customized workshops. One for sales managers to improve coaching and leadership skills; and one for sales people to improve consulting/solution selling skills.Visit our website to download free report.

Click here to advertise in this space


E-mail This Article E-Mail This Article
Printer Friendly Version Printer-Friendly Version
Related Content Related Content
Request Reprints Request Reprints
advertisement
Site Map Terms of Use Privacy Policy
SourceMedia (c) 2005 DM Review and SourceMedia, Inc. All rights reserved.
Use, duplication, or sale of this service, or data contained herein, is strictly prohibited.