If you like this blog post and want more expert advice on performance marketing, direct marketing, direct response advertising, please share it. Thank you!

Share on linkedin
Share on twitter
Share on reddit
Share on facebook
Share on email
Share on whatsapp

This Blog Post's Table of Contents

The Principles and Techniques for Testing Media, Markets, Products, Offers, Creative, and Seasonality: The Short Guide.

If you like this blog post and want more expert advice on performance marketing, direct marketing, direct response advertising, please share it. Thank you!

Share on linkedin
Share on twitter
Share on reddit
Share on facebook
Share on email
Share on whatsapp

This Blog Post's Table of Contents

Performance marketing is a science and I know of a few instances where a hit-or-miss approach made anyone a fortune. Countless companies have gone under, thinking that direct response advertising, large-scale Facebook, Google Ads, direct mail or telephone selling could get them out of trouble when they had a large surplus stock of a product which no one seemed to want.

When I am approached by people who want to get into performance marketing, my advice is invariably to start the investigative process from the beginning. Let research be your guide to discover who, if anyone, needs your product or reacts to it favorably. Look at the competition and list the advantages which you can offer. If the answers are hopeful, then research into ways of approaching the relevant markets.

Estimate how much is available for promotion per unit of sale. You can then decide which medium of performance marketing, or which mix of media, is most appropriate and begin your testing.

  1. Media tests. Plan comparative tests of the various forms of online and offline advertising, direct mail, telemarketing, door-to-door promotion, direct response radio and TV ads, online display ads including Google, Facebook, and Taboola, as well as print magazine ads.
  2. Market tests. Prepare comparative tests of a variety of mailing lists, different traffic sources and its segments, different behavioral groups and geographical areas, space and television media appealing to different markets.
  3. Product tests. What response do you get when your product is offered in different forms, sizes and quantity? Which of your product’s virtues should you lead with for the greatest success?
  4. Offer tests. What prices, incentives and supply methods will give you the most profitable response?
  5. Creative tests. How should your promotion be presented and what promotional devices and models (one-step, multi-step or others?), what kind of ads and copy will give you the best results?

Market and Traffic / List Testing

People have often come to me and expressed a keen interest in performance marketing but have been puzzled about the best way to find suitable products for this type of business. One cannot emphasize too often that in marketing the market comes first. Unless you possess intimate knowledge of a market and some kind of unique or exceptionally easy access to it, it takes a great deal of investment to start from scratch.

For instance, if you own a retail store selling men’s clothes and large numbers of customers pass through your door every day and pay by credit card, you have a natural market for repeat sales through performance marketing media.

All you have to do is collect the names and addresses, email, Facebook likes (not for fun, but for targeting purposes of course), and at the very least select those who clearly live too far away to come to your store frequently and then offer them a service through the mail.

After all, you will already have measured them and have some indication not only of their sizes, but of the type of clothes they buy. You can select the best of them by their value of purchase or interest group and create online and print promos which will fit their tastes and buying patterns. They have identified themselves and have already shown a preference for what you have to offer. No one else could put together such an ideal market for you.

It is, of course, possible to identify a market by using your own observation and experience. For instance, the explosion in the sale of dietary supplements and natural medicine has demonstrated that there are hundreds of thousands, if not millions of people with an interest in this subject.

The manufacturers of these products sell them by way of renewable subscriptions in the US but there is, as yet, relatively little sold by way of subscription in Europe. And the appetite for an everyday healthy living there is virtually unlimited. Video courses, ebooks, coaching, dietary done-for-you catering, become a healthy living mentor opportunities, certification courses, niche cookbooks, diet apps – there is hardly a limit to the possibilities.

The field of education and interactive learning programs offer almost unlimited opportunities including home study courses in natural medicine with all its branches and health issues to be solve or alleviated.
There is clearly a case for starting clubs and groups of various kinds for natural health aficionados. And where there is a known market of people with common interest, devising products becomes relatively simple.

It is then a question of finding lists and traffic sources which match your market. How do you discover whether markets are amenable to buying or not? In the first instance, it is possible to conduct ‘orthodox’ research by personal interview, by phone, through online questionnaires and by making specific offers of existing products, merely to get a financial measurement of reaction which will either encourage you or cause you to think again.

Or you can:

Whatever happens, markets should be tested for their basic reactions before you spend too much on creating products for markets which do not exist.

Product Testing

Once you are sure that there is a market and have begun by quantifying it with some initial existing products, you will wish to progress to the testing of new and additional products. These may be things you can purchase ready-made or others which you may need to design specifically for your markets.

It is one of the great virtues of performance marketing that you can test products before you have invested in either purchasing or developing them – a process called ‘dry testing’. This technique was pioneered by old-school print publishers such as Reader’s Digest, Time Life and others who habitually produced concepts of a number of books or series of books and then ‘dry tested’ them.

A synopsis for a book is written; binding, title and two or three sample spreads are designed in great detail and promotional packages are created for several variants. These are then promoted in a series of product tests, described as ‘pre-publication offers’. It is explained to the reader that a book is shortly to be published on a given subject and the reader is given an opportunity to order it prior to publication at a specially reduced price. He is specifically asked not to pay until the book is actually produced and delivered to him, but his order is firm.

This obviously works with other items including services.

By the way, Reader’s Digest was also famous for pre-testing, as in testing before even dry testing, see:

In this way, it is possible to test half-a-dozen proposed products against one another and to discover with a high degree of accuracy which is likely to be the most successful. You can then go ahead with only those items which are sure to succeed.

What do you do about the disappointed buyers of products you decide to abandon? The answer is to write to each one and to explain that you are not now going ahead and to send them some gift as compensation.

Your gift may even be a discount voucher to be used against the purchase of one of the products you have decided to proceed with. In my experience, this technique has never caused offence. Quite the opposite, customers who have ordered a product without paying in advance and have subsequently received a gift have been amazed at this generous gesture and have become firm friends.

Offer and Price Testing

Product tests have to be carefully designed to ensure that one product does not succeed, and another fail because of price advantages alone.

Where products are tested against one another, you have to ensure that the profit to you will be the same whichever of the products succeeds. In this way you are not introducing an unfair bias.

Once you have decided on products, you can then experiment with price. It is a strange thing, but the highest price does not necessary give you the lowest response, nor does the lowest price necessarily give you highest number of orders.

I remember an occasion when we tested an online home-study course which was most attractive and could be produced at a remarkably low cost (it was pre-recorded, streaming-based). We tested three different prices and the highest price not only gave us the greatest profit, but it actually produced the highest number of orders as well.

At first this astonished us and we tested again to ensure that the results were not due to some extraordinary freak circumstance. When the result was repeated, we concluded that the exciting claims which we made for the product were simply not credible at the lower price. People were not only prepared to pay the higher price but were actually more inclined to believe our claims.

When you first mention price testing to people who are unfamiliar with this technique, they tend to be horrified, feeling that there is something intrinsically dishonest in this procedure and that those who have ordered the product at the higher price might meet someone who has bought it at the lower price and would then complain bitterly. In fact, I know of very few instances where this has ever happened.

If, after careful price testing, you find that the lower price is the one to adopt, it is good practice to refund the difference to those who have paid the higher price with a message explaining that it has now become possible to offer the product for less. This obviously creates enormous goodwill. If the higher price is eventually adopted, those who have paid less have obviously no complaint.

If ever your customer who paid the higher price discovered that the product had been offered for less, the answer is to tell the truth. If it is actually more profitable to sell the product at a lower price, this means that the sales achieved by this method make it more economical and that the power of advertising and sales promotion has enabled you to increase sales to such an extent that you are able to reduce your prices. This explanation, together with the refund I have already mentioned, generally pacifies even the most vociferous customer.

But price is not the only incentive which can improve your economics.

There is virtually no limit to the variety of offers which can be made to influence conversion rate and, in the end, these are still a matter of using imagination and getting your sums right. Once a product shows promise, you need to test prices, offers and incentives against truly comparable lists and traffic sources to arrive at the winning formula.

Once this has been devised, you will probably wish to apply it in a major direct response advertising campaign or telemarketing exercise. With every major campaign you should again test a number of new ideas and devices, but it is important at each new test should have a control. It is not good practice to run a small test against the main body of your rollout.

Creative Testing

The material you use in performance marketing, whether online or print, can affect your fortunes as much as the offer and the price. Copy needs to be tested, both headlines and body copy, while illustrations also have an important effect as do colors. I remember one occasion when we had tested all the elements we could think of and finally came to an expensive rollout when our envelope manufacturer rang me up.

He told me that he had unfortunately under­ordered the blue stock from which the envelopes were to be made and wished to substitute green stock of a similar depth of color for the remainder.

I objected strongly and ultimately consented to use this paper only if he agreed to bear the cost of any reduction in response. To our amazement, we found that the green envelopes pulled better by a measurable percentage than the blue ones.

This was something we had not thought of testing, and we were so surprised that we decided to test again on a larger scale. The green envelopes again pulled better than the blue and we had added to our store of experience.

It is highly dangerous to alter several elements of a creative package at the same time. Each element needs to be tested by itself, otherwise it is impossible to know precisely what made the difference.

Whenever I have ignored this principle, I have lived to regret it. Whenever we have changed a number of elements, believing that some were bound to improve the result and therefore did not need to be tested separately, we later wished that we had not broken the rule. Often the overall result declined, and we did not know why.

Testing in Print Magazines, Loose Inserts and by Household Distribution

While fundamentals of online testing needs no explanation due to highly-automated tools available, this is not the case with offline performance marketing media.

How can you test all these variables, not only within one offline medium, but between different media of offline direct response promotion?

In print newspaper and magazine advertising some companies are able to provide a split run. That is to say, half the circulation can carry one variant of your ad, while the other carries a different version. With some newspapers you get for example a north/south split, but the response for these particular two geographic areas may be different in any case.

You therefore run each ad first in one part of the country and then in another and you can obtain a coefficient between the two different geographical regions by comparing the response from the same ad in each region. The only variable is then timing. You need huge samples and consistent samples which makes the task challenging.

Loose inserts, or free-standing inserts, are one of the easiest things to test in print advertising because you supply them and can deliver to the publication a loose insert with any number of different tests interleaved at random. This ensures that there is no geographical difference in your samples and that they are also evenly distributed over newsstand copies and copies sent to subscribers.

Try household, door-to-door distribution which is another way to test ads that is less well known. You can produce six or more different coupon ads and print them in the form of leaflets. These can then be distributed door-to-door in geographic areas which correspond to the readership profile of the print publications you intend to use. Although the response from these leaflets may not, in itself, be viable, this method of testing will provide you with comparative data to show which copy, creative approach or offer is likely to pull best.

I have found this method of pre-testing print ads very useful if magazine ads are several times more expensive than a leaflet distribution.

Testing Online, Direct Mail and Telemarketing

Here, the opportunities for testing are infinite. Because you are in control of every single name you approach, you can take matched samples and promote to them different packages or scripts in order to get accurate results in product, market, price, offer and creative tests.

The vital condition is to ensure that the samples are truly comparable, that the right material is enclosed with each test and that all your response vehicles are carefully and properly keyed. In all testing, response also needs to be evaluated not only by the number of conversions, but right through to the promotion cost per dollar turnover and per dollar profit.

How large should a test sample be? The minimum reliable size depends on your percentage of response. Where you expect a conversion rate of 10% or 100 conversions per 1,000, a sample of 1,000 or 2,000 may be perfectly reliable, if it is truly representative of the universe you are considering. If you reckon that no more than 0.5%, or five per thousand will convert, your sample needs to be at least 20,000 to provide you with reliable data.

(Side note: while most marketers savvy in the online and print arena seem to adhere to these rules, I am yet to find a telemarketing house or in-house team that fulfills sample requirements in day-to-day so-called ‘testing’. The rules are exactly the same.)

However, there are ways to enlarge the sample upon which you base your deductions without using minimum quantities of 20,000 per cell. For instance, if you are testing both lists (or list segments) and creative, you can evaluate a list across five different creative lines and avoid the need to use 20,000 list with each of these creative variants.

The same applies to price and offer tests. What you need to do is produce a matrix of test which will enable you to read the results not only vertically but horizontally. Here is an example of a typical matrix for a test campaign:

ListsCreative Test #1
Highly-personalized, 4-page, brochure
Creative Test #2
Non-personalized, 4-page, brochure
Creative Test #3
As #1 but different copy and graphics
Creative Test #4
As #1 but no brochure
Creative Test #5
As #1 but with extra testimonial leaflet
Creative Test #6
As #1 but with extra free gift of a pencil
List A (total available 200,000)5,0005,0005,000
List B (total available 85,000)5,000
List C (total available 250,000)5,0005,0005,0005,0005,0005,000
List D (total available 35,000)5,000
List E (total available 60,000)5,000
A matrix for a direct mail test campaign.

B2B and Small Samples Issue

In B2B industrial sales where your total market is measured in only hundreds or a few thousand prospects, this type of testing may not be practicable, but testing of some sort and the evaluation of comparative media is, nevertheless, essential. Whatever you do, it is vital to keep statistics so that you can build on your experience. You may have to test and prove for yourself one point or principle at a time. But even so, your experience will build up and your success will grow cumulatively.

Let me give you one example. We once created a test campaign for a consulting firm providing corporate innovation workshops. The unit of sale was around $10,000 and there was a case of testing LinkedIn advertising, followed either by a visit of a sales representative or a phone approach, against old-fashioned direct mail designed to provide leads for sales representatives or for follow-up by phone.

The budget was not large, and we ran several LinkedIn ads to produce leads which were then followed-up by sales representatives. We also conducted a direct mail campaign and studied ways of using the phone initially as a follow-up tool.

Online-only leads proved expensive and the cost, combined with the cost of sales calls, meant they were not viable.

The same turned out to be the case with direct mail, but we discovered that 75% of the leads we achieved could be converted into appointments through carefully planned telephone sales calls.

We then decided to test what would happen if we called individuals who had received a direct mail shot but had not replied to it. To our astonishment we found that a third of these could be turned into appointments and this proved to be the profitable path.

Regression Analysis

So far, I have covered the testing of known factors which you can plan before the event. Regression analysis is a high-sounding phrase which means getting wise after the event. In email, telemarketing, direct mail terms, this means that you promote to a set of prospects and subsequently examine each respondent minutely in relation to a comprehensive list of characteristics.

For instance, what kind of house does the person live in, is he or she married or single, how many children do they have, what education, what hobbies, what car do they drive? What purchases they made and how much they spent? Did they return the product? Or failed to pay? Or abandoned the cart?

Each factor can then be tabulated for every respondent on your lists and analyzed. The same should be done for those who did not convert, and an accurate profile of your ideal prospect can then be obtained. This information will enable you to look for others which match this profile and who will, in all likelihood, convert equally well. Similarly, with those who do not respond. By studying their characteristics, you can avoid touching them.

For instance, we phoned a list of people and carefully analyzed those who bought. From this we discovered that virtually all of them lived in houses and used land lines. We tested our theory and built a list of people with this same characteristic. We discovered that this list converted extremely well, and we were able to pay a premium for leads having these characteristics.

Another quite common example of regression analysis is the case where you are not at all clear about the ideal target groups for your product and decide to advertise it in a variety traffic sources, describing its virtues and characteristics as clearly as possible, but without specifying its precise applications.

For instance, you may have a waterproof suit or a comprehensive, lightweight, and easily portable toolkit. Either one might sell better if you knew precisely what group of people would find them most useful and advertised them to these specific groups.

But, in the first instance, you deliberately approach a large general market. Then, when you have received your first thousand orders, you go back and examine your customers with great care. If you find that the best customers for your waterproof suit are all fishermen, yachtsmen, or motorcyclists, you can make up your mind after the event to market your product to them and even your product’s name may then be subsequently changed to fit one or more of these ideal markets.

Seasonal Indicator and Trend Measurement

We have now seen how the media and traffic sources you use, the offers you make, and the price and material all affect response. Another factor to consider is timing. There is an ideal moment for everything and promotions at various times of the year – and in different years – do not necessarily perform in the same way.

Therefore, it is dangerous to test a particular element one day and to carry out another test two or three months later without having a fresh control.

In some businesses, the September to November period is the best time to promote because the product or service is related to the gift giving season: people buy your product as a Christmas present. (But then paid media are often most expensive this time of year.)

In other instances, the week after Christmas or immediately after the New Year, are the best because people have the festive season out of their system and are ready to consider new ideas and to make a fresh start.

In some businesses the summer months are unproductive. In others these are the best months, perhaps because of the relative dearth of competitive offers. It never pays to jump to conclusions.

Only testing can be your guide. Therefore, seasonal indicator campaigns are so valuable, and this is how they work.

Whatever other tests or promotions are planned for the year, you choose a totally uniform list of reasonably good prospects and divide it into 12 totally comparable samples on an Nth name selection basis. That is to say, you take every 12th name and call this ‘Sample 1’, every 11th name and call it ‘Sample 2’, etc. Or just make sure they are fully random.

You then email, mail, text, phone etc. precisely with the same creative to one portion on the first Monday of every month. Or, if this is possible, divide your test quantity into 52 and promote an equal sample every Monday.

The conversion from of each sample has to be separately recorded so that you can measure the response and chart it. Because each sample is so small, it is not practicable to chart each individual one. Instead, you take moving five-week centered averages. That is to say, you take five weeks together and plot the total against the third. You then move on and take the second to the sixth week and center it on the fourth week, etc. This will give you a graph showing your seasonal variation without regard to trend.

By trend, I mean that your sales, or your product’s popularity, may be generally going up or down. The only way to establish and eliminate your trend is to take longer averages – perhaps of three months’ or five months centered results. If a definite trend can be established in this way, then you can eliminate it from your seasonal and arrive at a true seasonal variation.

This process of establishing both seasonality and trend can be adapted to those online and offline media which use a database approach (i.e., telemarketing, email, direct mail, SMS text).

If you can discipline yourself to follow this procedure, you will produce some amazing results. In one D2C business, we discovered that the weeks between Christmas and the January 14th did, at one time, produce the highest response in the year. In the clothing industry, there are usually two optimum times of the year – when people first think of summer clothes and when they are first prepared to buy winter clothes.

Every industry and business have its own pattern and those who ignore this fact may wrongly condemn a business as unproductive, or test at the best time of the year and plan their rollout at a time when response is relatively poor.

And please, do not succumb to the utter nonsense of “believing” in “best practices” of when to send your email campaigns. Are Wednesdays “best” now? Or are these Fridays because Thursdays were hip last year? Test for yourself, stop “believing”.


  1. The six major factors that affect the viability of a performance marketing operation are the market, the product, the offer, the media, the timing, and the response.
  2. The market refers to the reachable online or offline audience for a product or service, and effective audience targeting is essential in performance marketing.
  3. The product should be based on a study of the market and its needs, and should be tested for its quality, unit size and quantity, frequency of supply, price, and packaging.
  4. The offer is the specific deal being made to the customer, including the price, terms, and bonuses. It should be tested to determine its effectiveness and profitability.
  5. The media refers to the channels through which the offer is communicated to the customer, such as print, radio, TV, or online advertising. It is important to choose the right media for the target audience and to test different options to determine their effectiveness.
  6. The timing of the offer is also important, as it can affect the response rate. Testing different times and seasons can help determine the best time to launch an offer.
  7. The response refers to the customer’s reaction to the offer, including the conversion rate, average order value, and customer lifetime value. It is important to track and analyze the response to determine the success of the campaign.
  8. To start a performance marketing campaign, it is important to first research the market and the competition, and to identify the unique advantages that the product or service offers.
  9. Media, market, product, offer, and creative elements should all be tested to determine their effectiveness.
  10. It is important to have a clear understanding of the target market and to have access to a suitable audience for the product or service.
  11. The success of a performance marketing campaign depends on a variety of factors, including the target market, the product or service being offered, the media being used, the offer being made, and the creative elements of the campaign.
  12. It is important to track and analyze the response to a performance marketing campaign to determine its effectiveness and make any necessary adjustments.
  13. It is important to have a clear understanding of the costs and potential profits associated with a performance marketing campaign to determine its feasibility.

If you like this blog post and want more expert advice on performance marketing, direct marketing, direct response advertising, please share it. Thank you!

Share on linkedin
Share on twitter
Share on reddit
Share on facebook
Share on email
Share on whatsapp

Leave a Reply to the Article

The expert's thoughts on direct response - growth hacking - performance-based marketing activities - DIRECT MARKETING

About Me, Rafal Lipnicki.

the direct / performance marketing consultant with a strange sounding name


Not your usual "guru" but a real-world performance marketing & innovation consultant based in Europe and an experienced senior executive at leading multinational companies.

What and Where.

I am a consultant for hire, working remotely and on-site all over the world (but Europe is always preferred). See my consulting services page for details.


Contrarian advice most of the time. Document-based audits, workshops, one-off projects, mentoring programs, and more.