Chapter 18 – Measuring the Effectiveness of the Promotional Program
18-1
CHAPTER 18
MEASURING THE EFFECTIVENESS OF THE PROMOTIONAL
PROGRAM
Chapter Overview
This chapter discusses one of the most critical—yet controversial—issues in all of advertising. While it
seems perfectly logical that managers on both the advertising and client side would like to know whether
the programs they have implemented are effective, in reality the majority of advertising and promotional
programs go untested. Compounding this is the fact that many of those programs that are evaluated use
inadequate or inappropriate measures in making this determination.
Learning Objectives
2. To know the various dependent measures that is used in assessing advertising effectiveness.
4. To evaluate alternative methods for conducting measures of advertising effectiveness.
Chapter and Lecture Outline
I. ARGUMENTS FOR AND AGAINST MEASURING EFFECTIVENESS
As noted in the beginning of this chapter, it seems logical that research on the effectiveness of the
promotional program would be conducted. Yet we also find that this is not always the case. Reasons why
research should be conducted and reasons why it is not are presented.
A. Reasons for Measuring Effectiveness—three major reasons are offered for why measures of
effectiveness should be taken. These include:
1. To avoid costly mistakes
Chapter 18 – Measuring the Effectiveness of the Promotional Program
18-2
B. Reasons Not to Measure Effectiveness—a variety of reasons (and excuses) are offered to explain
why the effectiveness of the advertising/promotional campaign is not taken. Perhaps the most
common of these are:
2. Problems with research measures used
4. Objections from the creative department
While some of these arguments have merit, others result from excuse making, politics, or a lack
of understanding of the value associated with conducting such research.
Professor’s Notes
II. CONDUCTING RESEARCH TO MEASURE EFFECTIVENESS
This section discusses how research to measure effectiveness should be conducted. More specifically, it
addresses the issues of what to test, as well as where and how such evaluations should take place.
A. What to Test—each of the elements of the promotional program should be examined. Research
should be designed to determine the effectiveness of:
1. Source factors—an important question is whether the spokesperson being used is effective
and how the target market will respond to him or her.
3. Media strategies—Research may be designed to determine which media class (broadcast vs.
print) generate the most effective results, the location within a particular medium (front page
4. Budgeting decisions—the relationship between the ad budget and sales is often hard to
5. When to Test—Effectiveness testing can generally be classified as:
Chapter 18 – Measuring the Effectiveness of the Promotional Program
18-3
B. Where to Test—there are certain advantages and disadvantages associated with where
effectiveness tests take place. Again, two different testing environments are discussed.
1. Laboratory tests—those tests conducted in a specific location to which consumers are
Professor’s Notes
III. THE TESTING PROCESS
The actual testing of ads may take place throughout the campaign. Four types of tests are discussed.
A. Concept Generation and Testing—Figure 18-5 describes the process involved in concept testing.
As noted, concept testing takes place very early on in the development of the campaign, and is
designed to explore consumers’ responses to ads and/or campaigns being considered. Focus
groups and mall intercepts are commonly employed methods at this stage.
C. Pre-testing Finished Ads—many advertisers prefer to test the ad in its finished form. A variety of
both print and broadcast measures may be used.
1. Pre-testing finished print messages—a number of methods for pretesting print messages are
provided. Some of the more common of these are:
Chapter 18 – Measuring the Effectiveness of the Promotional Program
18-4
2. Pre-testing finished broadcast ads—some of the more popular methods for pre-testing
broadcast ads are:
a. theater tests—one of the most popular lab methods for pretesting finished commercials.
b. on-air tests—often includes single-source ad research.
D. Market Testing of Ads—Many of the methods already mentioned are also used in the posttesting
of advertisements. The text once again discusses methods of market testing ads of both the print
and broadcast varieties.
1. Posttests of print ads—Three of the most common forms of posttesting print ads include:
a. inquiry tests—these are designed to measure advertising effectiveness on the basis of
2. Posttests of broadcast commercials
a. comprehensive tests—The chapter discusses one of the more commonly employed
posttest measures of broadcast commercials. The Ipsos-ASI Next *TV test replaced the
Burke Test which had often been the topic of debate in respect to its value and validity.
Professor’s Notes
Chapter 18 – Measuring the Effectiveness of the Promotional Program
18-5
IV. ESTABLISHING A PROGRAM FOR MEASURING ADVERTISING EFFECTS
To this point in the chapter, a number of methods of testing the effectiveness of print and broadcast ads
have been presented. In each case, these measures were discussed in respect to their contributions as well
as their limitations. In this final section, the chapter discusses how the marketer might establish a sound
program for conducting effectiveness research. This final section focuses on two distinct areas; (1)
problems with current research methods and (2) essentials of effective testing.
B. Essentials of Effective Testing—Again referring to the PACT program, four suggestions are
offered that we believe will improve the effectiveness of the evaluation program. These include:
• establishing communications objectives
• employment of a consumer response model
Professor’s Notes
V. MEASURING THE EFFECTIVENESS OF OTHER PROGRAM ELEMENTS
The majority of this chapter has focused on the measurement of advertising effectiveness. Obviously,
measuring the effectiveness of other program elements is important as well. The final section of this
chapter discusses the measurement of effectiveness of sales promotions programs, sponsorships and
nontraditional media.
A. Measuring the Effectiveness of Sales Promotion—A number of organizations measure sales
promotions. Pages 630-631 discuss some of the methods employed.
B. Measuring the Effectiveness of Support Media—Some progress has been made in the
measurement of the media discussed in chapter 13.
• The effects of shopping cart signage—This study used personal interviews in grocery
stores to measure awareness of, attention to, and influence of this medium.
Chapter 18 – Measuring the Effectiveness of the Promotional Program
C. Measuring the Effectiveness of Sponsorships—A number of companies now measure sports
sponsorship.
Teaching Suggestions
Due to the importance and amount of information presented in this chapter, we suggest that the lecture
cover at least two time periods. The first time module might be used to discuss the importance of
conducting effectiveness studies, some of the reasons why such studies are not conducted, and some of
the pretest measures. The second session could cover posttests, and the PACT guidelines. In addition, the
second session should discuss ways of measuring communications objectives.
Answers to Discussion Questions
1. Explain why it is so difficult to measure the effectiveness of an IMC program that uses multiple
program elements. (LO3)
Think about the argument used to describe why it is so hard to measure the effectiveness of an IMC
program using sales as the dependent measure. Other marketing program elements also contribute to
the sales result, and it is often difficult to parcel out this effect.
Chapter 18 – Measuring the Effectiveness of the Promotional Program
18-7
2. Discuss the various types of pretests that advertisers employ. Why do marketers pretest ads? What are
some reasons why marketers may not pretest? (LO2)
Pretests are those tests that take place before the advertisement or commercial is made available for
viewing by the public. These tests may take place at a number of points from as early on as idea
3. What are some of the problems with existing effectiveness measures? Discuss the essentials of
effective testing methods. (LO3)
Many measures of effectiveness that are used are not truly measuring outputs, but rather they are
testing inputs. For example, marketers talk about the number of people reached, eyeballs,
impressions, etc., which do not really test the impact of the ad, but measure exposure or potential
exposures. To be a measure of effectiveness the test must reveal the impact of the ad, not just that
people see it. This is particularly common in internet advertising, where visitors, clicks, etc. provide
some insights, but not enough indication of the impact of the ad.
Chapter 18 – Measuring the Effectiveness of the Promotional Program
4. The Ogilvy Awards are given to companies and their agencies that have successfully employed
research to develop and measure the effectiveness of their campaigns. Google the Ogilvy Awards ,
pick any of the award winners and discuss why that program was successful. (LO1)
5. Cite examples where advertisers may be able to develop successful ads without the benefit of pre–
testing. Why would an advertiser decide to make the decision not to pretest?(LO2)
Many (if not most) ads are not pre-tested. Your everyday retail ads, ads that are straight forward and
non-controversial will seldom be pre-tested but still can be effective. For most advertisers, pre-testing
occurs when there is some doubt or uncertainty about the ad or commercial. For example, issue ads,
6. What is the difference between a lab testing methodology and a field testing methodology? What are
the advantages and disadvantages associated with each? When should each be used?(LO2)
Lab tests are those that take place in a controlled environment. The major advantage of these tests is
Chapter 18 – Measuring the Effectiveness of the Promotional Program
18-9
7. The chapter discusses the use of neurosciences to assess viewers’ evaluation of commercials. What
are the pros and cons associated with these methods? When should advertisers use them? (LO3)
The chapter discusses the use of this research methodology and the pros and cons associated with it.
As noted in the text,it often requires that multiple measures of effectiveness be taken to truly evaluate
the effectiveness of an ad.
with other methodologies.
8. Eye tracking methodologies have been used to measure advertising effectiveness in the past. This
methodology has no been used in a variety of other ways. Discuss the use of eye tracking and its
applications to advertising and promotion. (LO3)
Eye tracking has been used to observe viewer’s eyes to see where they go in watching a commercial
or print ad, and where they focus and how much time is spent there. This form of tracking has been
used for quite some time has been proven to offer valuable insights.
Chapter 18 – Measuring the Effectiveness of the Promotional Program
18–10
9. Some companies test their commercials in rough formats described in the chapter. Others only test the
commercials in finished form. Discuss some of the advantages and disadvantages of conducting tests
at both stages. (LO2)
The clear advantage of rough testing is the cost savings. Given that roughs correlate so highly with
finished ads , many advertisers will use them to explore the potential effectiveness of them before too
much money has been spent.
10. Discuss some of the reasons why some companies decide not to measure the effectiveness their
promotional programs. Explain why this may or not be a good strategy. (LO1)
The chapter cites a number of reasons why advertising effectiveness measures are not taken. Each of
these is discussed in the text, but a short review is also provided here.
Cost—One of the most commonly cited reasons is the belief that conducting these measures is too
costly, or that the moneys are better spent elsewhere. Many advertisers feel that they would rather
spend the money to purchase more media, improve production, etc. In addition, they cite the time
costs that are involved.
Chapter 18 – Measuring the Effectiveness of the Promotional Program
A variety of reasons can be offered as to why effectiveness measures should be taken:
To Avoid Costly Mistakes—If the ad or commercial is developed is unsuccessful, a great deal of cost
will be incurred by the company. These costs include the actual costs associated with the development
and implementation of the campaign, as well as the opportunity cost.
Additional Discussion Questions (not in text)
11. Most of this chapter focuses attention on the measurement of traditional forms of advertising. How
are companies measuring nontraditional advertising forms? (LO2)
As noted in earlier chapters, there have not been a lot of new methodologies provided to measure the
impact of the newer media options. Of course, the Internet has its measurement criteria, some of
which are unique to the medium (and some of which have questionable validity and reliability), while
others are traditional (tracking, recall, etc.)
Chapter 18 – Measuring the Effectiveness of the Promotional Program
18–12
12. The chapter notes that while companies may spend a substantial portion of their revenues on
advertising, over 70% spend less than 2% of their budgets on measuring advertising effectiveness.
Explain whey this is the case. Do you agree or disagree with this strategy. (LO1)
Chapter 18 provides some interesting information on marketers’ views toward measuring
effectiveness of advertising. As can be seen by examination of this Figure, marketers are not
convinced of the capability to accurately measure the impact of their marketing communications. As
noted in the text, one of the studies cited therein notes that only 2% of ad budgets are spent on
measuring advertising impact.
13. Explain why it is so difficult to measure the effectiveness of an IMC program that uses multiple
program elements. (LO3)
One of the arguments for not using sales as an indicator of advertising success is the fact that other
marketing factors—price, product and distribution as well as external factors will contribute to the
sales of a product. A similar argument exists for measuring the contribution of IMC elements.
Chapter 18 – Measuring the Effectiveness of the Promotional Program
18–13
14. A USA Today survey is conducted each year that has viewers vote on the best Super Bowl
commercial. A discussion as to the usefulness of this survey focuses on the merits (or lack thereof)
of this poll or any other such survey. Discuss some of the reasons why this poll might be used to
determine effectiveness, and why some contend that it is not a good measure. (LO1)
USA’s yearly poll of Super Bowl commercials has been referred to by some as a popularity contest.
Indeed, it is referred to as the most popular commercials. To the extent that liking and popularity are
the criteria that are to be used to determine effectiveness, the commercials and the testing method are
good.
15. Give examples of the various types of rough testing methodologies. Describe why a company might
wish to test at this phase of the process. When might they wish to test only completed ads? (LO2)
Given the high cost (and increasing cost) of finished commercials, rough tests are becoming used
more frequently. The fact that these tests’ results correlate highly with the finished products at a
fraction of the costs allows researchers to rely on the predictability of the test results. Rough tests are
Chapter 18 – Measuring the Effectiveness of the Promotional Program
18–14
16. Major changes have taken place in the way that theater tests are conducted. Describe some of these
changes and the changes in measures that have also occurred in this testing method. (LO2)
As noted in the text, theater tests have been used for quite some time. In the past, participants would
be invited to a theater-type setting, asked to watch commercials and programming, and give their
17. Studies indicate that the combined effects of advertising in various media may lead to more
effectiveness than just using one medium. Explain why you think this might occur, using some of the
examples cited in the chapter. (LO1)
Various elements of the IMC program each make their own contributions to the overall impact of the
campaign. A combination of the program elements is likely to have a greater impact than the effect
of the element used alone. Some examples cited in the text:
Chapter 18 – Measuring the Effectiveness of the Promotional Program
18–15
18. Discuss some of the aspects of the communication program that advertisers may wish to test. Provide
some examples. (LO1)
In testing various aspects of the communication program, advertisers may wish to test various
components including:
Source factors—advertisers would like to know whether the source being used is effective and how
the target market will respond to it. A product spokesperson may be an excellent source initially but
for a variety of reasons may lose impact over time. Changes in the source’s attractiveness or
likeability or other external factors may also lead to changes in source effectiveness.
have also found that some programs are more effective. For example, direct marketers feel that reruns
will generate more response that the first run of a prime time TV program. Another factor is the
vehicle option source effect, i.e., the differential impact that the advertising exposure will have on the
same audience member if the exposure occurs in one media option rather than another. The final
factor in media decisions involves scheduling. The evaluation of flighting versus pulsing or
continuous schedules is important. There is evidence to support the fact that continuity may lead to a
Chapter 18 – Measuring the Effectiveness of the Promotional Program
IMC Exercise
Have students find an example of a current or recent advertising campaign being used by a company.
Have them explain the various measures of effectiveness that might be used by the company from both a
AdForum Exercise: “Examining the Effectiveness of Beer Commercials”
(See Advertising and Promotion Playlist, Chapter 18)
The playlist for Chapter 18 contains a number of beer ads used by some of the top selling U.S. brands of
beer. Review each of these commercials and answer the following:
1. What are the specific objectives of each of these ads?