This article is intended to help you identify and understand the specific elements that can and should be tested when developing your ecommerce site and its promotional campaigns. Since ecommerce sites are far more complicated and expensive to produce than long-form landing pages or microsites, the rules for testing are somewhat different.

Four elements of an ecommerce offering must be isolated for testing:

1. Products
2. Offers
3. Creative
4. Traffic sources

I will examine the diverse ways by which these elements can be tested, both within your already existing traffic sources and through other media.

Product Testing

Ecommerce marketers can test merchandise in two basic ways. The first, and possibly the most obvious and commonly used, is to simply run with a product. If it works, continue to run it. If it does not work, do not run it anymore. The second method is to test merchandise in another media or format.

Of course, the above is a quite simple summation of a rather complex operation. Let’s examine the practical details that goes into making – and constantly reevaluating – these decisions.

Analyzing Product Payout

In a profitable ecommerce venture, every product not only pays for itself (this includes traffic costs and alternative profit calculations) but also pays for part of the total amount it costs to have the business functioning and generates a profit as well.

For instance, if you do CPM-based email shots to outside lists with the intent to acquire new customers, and these email shots consist of multiproduct offer, then to determine if an item will perform profitably, you need to do three things:

1. Establish your cost per one email sent out.
2. Establish the cost of advertising space for each product.
3. Analyze the profit or loss for each item.

There is a simple formula for accomplishing all three tasks in this easy, one medium, multiproduct scenario:

1. The total cost of the email and its development is divided by the number of products presented in it. This establishes cost per product.
2. Additionally, you may want to ‘charge’ each product based upon percentage of space allotted in the email. This would establish cost of advertising space for each item. (This generally concerns only super large businesses sending out tens of millions of emails each month.)
3. When sales are all in, or projectable, a balance sheet is drawn for each item. The profit or loss for each item in the promotional campaign is arrived at by taking total sales, less advertising cost, product and fulfillment cost, and overhead.

Similar calculations need to be done for all your traffic sources, including both paid and owned media (i.e., your own email database). Such calculations work for print package inserts as well.

Once you have all these figures, it is easy to figure your winners and losers. But you do not just run with your winners and throw out your losers. Some of them you will want to “retest” with different media, space allocation or placements within the email promo, paid ads, or distinct parts of your site.

For instance, if you had a big winner that ran on 1/4 of a multiproduct email, you may want to try it on larger space or have a solo mailing just for this one product. Probably the additional “space investment” will more than pay for itself. If it does not, you just go back to your original space allocation.

In the case of marginal winners (or losers), you may want to try reducing placements, clicks or space allotted. For example, by putting additional products on the homepage or reallocating some of the hot spots of your ecommerce site to a proven “big winner,” the marginal product will not have to work so hard and probably can be turned into a profit maker.

Another possibility – and one many ecommerce companies often do not try – is to see in how small a space you can sell a proven money-maker, e.g., your email newsletter’s or actual site’s footer and header or ‘other people browsed’ space. This has a couple of significant advantages.

You can add new products without losing money on the old ones. And you can expand your product line without incurring the additional cost of expanding the number of campaigns you conduct.

One of my B2B clients did this rather successfully in its email newsletters promoting the ecommerce part of the business. The company felt certain products were naturals for it. So, newsletter by newsletter the company gradually reduced the space allocated to these products.

Sales and profits stayed approximately even on these products, but since the company had created more room in the newsletter, it was able to add new products and actually increase the profits per one email sent out.

Test “Hot Spots”

Another method for introducing new products, or “retesting” winners, marginal winners and sometimes even losers, is to run them in a “hotter” spot. These spots include the (1) home page, (2) header, (3) footer, (4) order form (5) ‘others browsed also’ part of the individual product pages.

Good sellers usually will be even better sellers when placed in these areas, and marginal items can be turned into profit-makers.

Test Different Media

You can also test products by using other media or formats. “Custom audience” Facebook ads, Google “Customer Match” ads, package inserts including print mini-catalogs, web push campaigns, and usual retargeting can give you invaluable information about what kinds of products will work for your customer database.

These are good methods if you are looking to expand your ecommerce business, but are not sure which directions to pursue.

Microsites. These can be much smaller, far less elaborate, and much less expensive to develop than “standard” ecommerce sites. They can be used in the following ways:

• Linked from your regular mailings and ecommerce pages to help you evaluate new product directions that your customers are interested in.
• Used in Facebook ads, Google Ads as well as in cold mailings to new lists to help you expand your market while you evaluate products.
• Linked from your bouncebacks in fulfillment packages (“The secret offer available at …”) to give you feedback on what other kinds of products your customers will buy from you.

Facebook ads and Google Ads. Ecommerce merchants can use online display ads basically in two different ways. First, smaller businesses seeking to expand their products base often test new merchandise in display ads. This does double duty – it allows them to expand their customer house list and gives them readings on the types of merchandise that may have a general appeal in their ecoms.

Other ecommerce merchants use display ads to sell proven winners. This allows them to build and expand their customer base.

Bouncebacks. Since bouncebacks ride along postage-free in fulfillment packages, you can often use this vehicle to test new products as well.

Offers Stimulating Sales

Your offer can make or break your performance marketing program. The sweeter your offer is, the more attractive your ecommerce store is to prospects. You might want to consider testing the following sales stimulators or incentives to purchase: on sale items, mix and match discounts, premiums, deadline notices and/or coupons.

“On Sale”

These are two of the most popular words in the English language, and, next to “free” and “new,” possibly two of the best words in performance marketing. But you do not have to have a specific sale campaign to have items “on sale” in your online store.

And you need not group them all together. In fact, many ecommerce marketers have found that scattering these “on sale” items throughout the site actually encourages readership.

The use of handwritten “on sale” signs, crossed-out prices with new prices written in, and other such graphic treatments heighten the excitement of your sale. And they do not have to look “shoddy.”

You need to communicate clearly that you are reducing prices and that these items are better values than ever. If you do not like “on sale,” try “special selling” or “best value.”

Mix and Match Discounts

This type of purchase incentive allows both low-volume and high-volume customers to enjoy savings. It also encourages low-volume or single-item customers to increase their orders. This is an idea borrowed from business-to-business ventures that consumer consumer-based ecoms could well consider.

Let’s say you are selling shoes. If your average sale is usually only one or two pairs, you can encourage people to buy more by giving them a gradual savings based on the total number of items they order. The reason these are called mix and match discounts is because you encourage customers to buy more than one style but still enjoy a “volume” discount.

So, the mother who wants to buy a pair of shoes for herself could end up buying two or three more pairs for her daughters to enjoy the extra savings.

These kinds of “deals” can be tested in the true sense of the word. Often you will sell a much greater volume, so if your margins are good, you end up more than making up for the price reductions and discounts.

Premiums

Another way to sweeten your presentation is to offer premiums with orders. These can be tied to a specific total dollar order, or they can be given for any order at all. Either way, they tend to increase both the quantity of the orders as well as the average order size.

Many B2C marketers have had great success by allowing customers to choose from two or three different premiums. Others have found it is more beneficial to offer just a single premium. To determine what premium offer works best for you, you would probably want to test them. Usually the best place to test your premium offer is on the order for, site header, or a dedicated prelander or email promotion.

Again, you can run an A/B tests on your order form or prelander. Just be sure to have a proper traffic samples so you can read results that are significant in statistical terms.

Deadlines

Deadlines also can be used effectively in an ecommerce promotion, especially when combined with a free gift offer or sale pricing. For instance, some of the cheese niche online stores offer a “free wheel of cheese for orders placed before Thanksgiving.” This technique allows them to pull many of their Christmas orders in-house earlier.

Coupons

Specialized online and print coupons related to products in your online store often help move these products more quickly. For instance, if you are reducing the price on a specific group of items, by including discount coupons for these products you heighten the sense of immediacy and value of the offer.

Again, these coupons can be placed on a prelander or email or given away as a package insert. Some ecommerce marketers actually place them right in the body of the product description.

You can also send coupons as an SMS text, email, retargeting, voice message follow-up to further stimulate sales.

Creative Format Testing

As is the case with merchandise, few ecommerce merchants can afford exhaustive testing of “creative.” It is just too expensive to produce two independent sites or to test headlines, product descriptions or graphic approaches for specific products to see which direction works best.

However, if you are set on testing creative, you could try long-form emails, mini-sites, package inserts and prelanders. It is relatively easy and economical to test in these formats, and you usually can read results fairly clearly. However, results may not necessarily correlate with your overall ecommerce operation.

If you are just starting up, testing can give you some direction on how you should be positioning your site creatively. But if you already have an established look and want to change, you will probably have to go out into the marketplace and do your research.

Basically that is the gist of much ecommerce creative testing. You try something new. If it works, you keep doing it. If it does not, you go back to what you were doing before.

Customer List / Own Database and Traffic Testing

To ensure that your ‘house list’ performs for you at peak efficiency you may want to consider the following testing options.

Touching Your House List as Often as Is Cost Efficient

This does not mean you have to do a full campaign mailing every day. Printouts of your best offers as package inserts, including multiple flyers, direct mail, SMS texts, voice messages, retargeting, custom audience targeting via Facebook and Google ads, etc. can help you maintain the “give and take” relationship that keeps these people buying – and buying more – from you. Just do not stick to email only.

Package Inserts in Orders and Immediate Follow-ups

Many people are ripe for re-order promotion while they are opening the packages they have just received. Take advantage of this proclivity by supplying them with the stimulus and means to place that order. This could be achieved by way of an insert as well as retargeting or SMS texts with a “secret offer available only to those who ordered a product in the last 3 days”.

Special Offers for Good Customers

Take a tip from retail stores that give priority treatment to frequent customers in the form of “Courtesy Days” and special sales. They make your customers feel that you value their business, and they give customers one more reason to order from you.

Outside Lists and Outside Traffic Sources

In order to generate more income and to expand your customer base, you must be involved in testing outside traffic sources, especially in the email channel as prospects coming from this channel tend to respond best in subsequent monetization attempts. The most logical way to pursue this is through a Pyramid Test Structure (or “pyramiding”).

1. Initial Phase. During this phase, for example, you might buy a quantity of 1,000,000 (for statistical reliability; this would me CPM-based buy most of the time) from the strongest email lists in a number of different list categories to allow for the greatest rollout potential.
2. Confirmation Phase. Next you would buy a quantity of 5,000,000 from lists that worked “well” originally to confirm the confidence level of the lists. You would also want to email a quantity of 1,000,000 from other lists from categories that produced “good” results in order to logically broaden the base of your pyramid and, thereby, your prospect universe.
3. Rollout Phase. Now you can rollout to the entire universe of email lists that “confirmed well.” Confirm test lists tested during Phase 2 and continue to test other similar email lists.

To get the most value out of an email list, select lists for “vertical” categories depending upon the product you are working with. For example, if you want to select house wares buyers for cooking accessories, you would also want to select lists from less obvious “horizontal” categories that reach a similar demographic audience.

Remember, if you are looking for other ways to build your email list, you may want to consider advertising your ecommerce site and products in display ads. (This includes Facebooks ads, as mentioned earlier.)

The first thing to realize about email list testing is that it is a management issue, not a statistical exercise. It should be addressed by focusing on ecommerce sales and profitability. Tests are intended to “buy” information, since they usually cost a certain amount of money to conduct and yield results which are used to make better management decisions.

At the simplest level, tests can indicate whether or not an email list should be used for subsequent acquisition campaigns.

While testing of email lists has been performed extensively in the ecommerce industry, current practices can be improved in several areas.

Let’s review some of the key aspects of email list testing, including:

• Test design
• Selection and screening of email lists for testing
• Analysis of test results
• Advanced issues in testing

The following guidelines use email channel as an example. The advice concerns most other outside traffic sources which are based on prospect/customer records stored in a database of some sort.

Test Design

Your tests should be designed to support managerial decision-making. The design should attempt to balance the cost of the test and the value of the test information. However, most ecommerce marketing tests today are designed without reference to financial information, making such balancing impossible.

A test design typically consists of primarily two components:

• a decision rule for rolling out with a particular email list, and
• a quantity of records which will be tested.

The test quantity and the decision rule are interrelated issues.

A decision rule expresses how you will decide whether or not to roll out with a particular email list. An example decision rule is: Roll out if the test indicates that there is a 95% chance that the results will exceed breakeven. Most often, decision rules specify the confidence level which a manager insists on for exceeding breakeven on the rollout.

A confidence of 95% is often used because it is relatively conservative (that is, it will not allow for very many rollouts of losers), but it does not significantly increase the number of lost opportunities, as long as tests are appropriately designed. Sometimes managers will want a specified level of certainty that the rollout will exceed a particular level of profitability.

Once you establish a decision rule for rolling out to a list, you should examine other financial characteristics of the business. In particular, it is usually crucial to examine:

• Web development costs in rollout quantities and test quantities. In some situations the test quantity cost will be higher since there are few economies of scale.
• The average contribution to overhead and profit per order received from the campaign.
• The total number of records available on an email list.
• The expected response rate from the email list prior to testing.

Using these pieces of information, and introducing the manager’s decision rule, a moderately sophisticated Excel formula can examine all possible test quantities and find the one that offers the best odds of balancing test value and cost. This can be referred to as the optimal test quantity.

Selecting Email Lists for Testing

Finding good email lists to test is part art and part science. The art of email list selection involves finding email lists which have the right psychographic profile. Since psychographic characteristics are difficult to measure and quantify, this process is quite intuitive and relies upon expert opinion and judgment.

The science of email list selection involves examination of several measurable email list characteristics. These include:

• The duplication rate between the house list and the candidate list for testing. Higher duplication rates usually correlate with higher response rates. (Contrary to direct mail and telemarketing lists not all email list sources will allow for deduplication.)
• Demographic characteristics of a candidate email list. Good email lists usually have demographic profiles that are similar to the demographics of the house list.
• Rental of your house list by another firm. Often, but not always, if your house list works well for some other performance marketer, then the other performance marketer’s email list will work well in return.

The intuitive and scientific approaches should be used together since they complement each other. Moreover, email lists do change and evolve over time, so an email list that was tested unsuccessfully several months or even years ago now may be usable.

Analysis Of Test Results

The most important issues to be addressed in analyzing email list performance are:

• What is the nature of the result? Have you proven that an email list is profitable, unprofitable or marginal? Have you proven that the email list is better, worse, or equal to other email list alternatives?
• Have all major sources of uncertainty been included in the analysis? In particular, are you only interested in analyzing the response rate, or is the variance in order size also an issue?
• How should the result be expressed? Are you interested in knowing a confidence interval around the result? For example, do you want to know that the response rate is between 2.7% and 3.3% with 95% confidence? Or, do you want to know the confidence that the response rate is above a certain target? For example, do you want to be 97.5% certain that the response rate is above 2.7%?

These are all issues which you should address before any statistics are calculated and presented. Once these issues are resolved clearly, the necessary calculations are mechanical.

In many email list testing situations it is important to look not only at the absolute performance of a list, but also at its relative performance. The overall strength of your test offer will affect the success of an email list. Therefore, when there is a very weak offer, a disproportionate number of lists will be unsuccessful.

When there is an extraordinarily strong offer, a disproportionate number of lists will be successful. It is often a good idea that you evaluate email lists by their relative performance on an offer. This allows you to see which lists are relatively strong or weak. An unsuccessful email list on a weak offer might then be tested again with a strong offer.

Advanced Issues Related To Email List Tests

A number of issues affect the email list testing process. Ecommerce marketers must consider the following:

• Seasonality affects offer and email list performance. Email list size and characteristics (for example, number of 0–6-month buyers) may fluctuate significantly throughout the year. You should be alert to seasonal characteristics in both the email list and the offer which will make interpretation more complex.
• Email list test results often serve only as a basis for making future email list decisions. Most lists offer a variety of selections that can be made and will affect list performance. If the 0-6 month buyers on an email list perform marginally on a test, you may want to add monetary or other selects which will reduce the universe of records but improve its performance. Geographic selects usually can be added and will make some difference in performance.
• The number of names from a test list that duplicate on either your house list or other rental lists is a key characteristic of the email list. Email lists with very high duplication rates will tend to perform well, but they deliver a relatively smaller number of net additional usable records. Moreover, if names hit on multiple email lists will be promoted to two or more times, the financial effects of this should be figured into the email list performance analysis.
• The frequency with which an email list is rented is related to its performance. In particular, an email list that is rented and promoted to repeatedly usually will experience some decline in its performance. It is certainly worth testing the best frequency of use of different email lists in order to maintain their viability. Successful email lists, of course, should be used repeatedly, but not so often that their performance seriously deteriorates.

Summary

• Testing in ecommerce does not concern only the creative part. Make sure you test products, offers, and traffic sources as well.
• Monetizing your customer database or house list via email promotions is only one of a multitude of tactics available. Do use all tactics, formats, and media available to you.
• One of the keys to success in ecommerce marketing is the identification and exploitation of email lists and similar traffic sources. The traffic testing process is the performance marketer’s way of buying information about traffic sources in order to use them most effectively.