Chapter 14
Measuring and Evaluating Web Marketing Programs
Learning Objectives:
By the time students complete this chapter they should be able to:
• Discuss the various types of Internet marketing metrics that are available.
• Explain the importance of usability testing.
Chapter Perspective
The importance of this chapter cannot be overstated. Web metrics are at the heart of the
interactivity, trackability and measurability of the Internet. Usability testing is relatively easy for
students to identify with, particularly after you point out that it has many of the elements of focus
group research in the physical world. Internet metrics are a new animal to most students. Many
undergraduate students still have a marketing research perspective, and it can be very difficult to
I would encourage you to spend at least two hour or hour-plus class sessions on this material.
Usability can take up most of a session. I find that even after two or three hours on metrics a lot
of students are still quite fuzzy on even the basics. One thing might be to play one demo that
My personal belief, after many years of experience, is that you simply need to take students
through these examples, through the numbers. Hopefully they will find it fascinating to realize
the depth of understanding that can be gained about how customers behave. Even if they find it
gruesome, they need to understand the role and general outlines of metrics.
subject intelligently in an interview and can demonstrate a willingness to learn.
Marketing Effectiveness and the Importance of Marketing Metrics
The two case studies—Papa Gino’s Pizza and Sybase—are meant to focus students on the
business reasons for using metrics—basically to understand what is happening and how to
improve effectiveness. The uses in B2B and B2C are similar. The issue of the cases is what
metrics to focus on to get the greatest increase in effectiveness.
Ways to Evaluate Website Effectiveness
Figure 14.2 provides an overview of metrics. Usability metrics provide the customer’s
perspective. The business view has two components. Site performance, as discussed in Chapter
12 is the purview of the web master whose job it is to ensure that the site is working properly at
Usability Testing
Visitors to a website need to be able to navigate it smoothly, to find the products or information
they need, and to be pleased—not frustrated—by their experience on the website. All those
things are true. The fundamental basis of web usability, however, is user task performance.
Visitors come to the website motivated to accomplish some goal, to perform some task. Usability
testing is designed to ensure that task performance is not only possible, but hopefully efficient
and entirely satisfactory, a good customer experience.
The stages are:
• Testing site design concepts to see if they make sense. This is primarily site structure,
although design concepts can and often should be tested also. The shot of Nielsen’s card
sorting exercise (see first link in Interactive Exercise 14.1) is an example of how
• Testing the full usability of the site when it is functionally complete and most if not all
of the content is there. This stage is meant to put the site through its paces in a rigorous
manner. Students should be clear that the site is not on the WWW at this point. It is on a
server and can be accessed from a usability laboratory or at a distance. Much of the value
You probably need to point out that this example comes from the early days of the web. We
should be grateful that Dr. Neilsen preserved it because few enterprises will now release this
kind of information because of its competitive significance. The good news is that the structure
of usability testing hasn’t changed much. There is now software to track each mouse click that is
probably more sophisticated than what Dr. Neilsen used in his early tests. Some usability labs
incorporate eye-tracking capability into their set-up. However, the focus group origins of the
process remain intact. Dr. Neilsen’s early and simple set-up can be used to emphasize to
students that this is something they can do for their companies themselves. A test can be set up in
a conference room, quickly and without great expense. It can be outsourced to a firm that has
specialized competence in the area if resources are available. There is simply no excuse for
businesses not to test the usability of their sites before going online and from time to time
thereafter.
be worthwhile to take a few minutes and talk about the general importance of the 80/20 rule in
many areas of marketing.
Enterprise Metrics for Evaluating Websites
This section is introduced with the example of Fresh Direct, a food delivery service. In the U.S.
Specifics included looking carefully at metrics for the recommendation engine, which showed
that recommended (complimentary) items sold well. They were offering a new prime meat
product and were able to target emails only to customers who had previously purchased meat.
They also did a Valentine’s Day email promotion that worked well; this data could have come
from their email platform as well as from an integrated metrics platform. Metrics on keyword
searched that direct visitors to the Fresh Direct site can provide a number of valuable insights. So
does monitoring the metrics specifically related to their shopping cart.
The subject of website performance is treated briefly. The aim is to let students be exposed to
the “big picture.” There are two basic types of site metrics: business metrics, our focus—and
technical metrics, used by the IT professionals who run the site to monitor and fine-tune the
performance of the site. Figure 14.5 is an example of a performance report. It deals with links,
both internal and external, and gives a number of data items that indicate which links are
working properly and which are not. Students should focus on it only long enough to recognize
that it provides no actionable marketing insights. Marketers should visit their own site frequently
and make their judgments about performance as a customer does.
There are three basic types of behavioral data that students need to understand before getting into
the details. As described in the text they are:
• Traffic data describes activity on the site. It includes metrics such as number of visitors,
sessions, and page views.
It is admittedly hard to draw a clear line between “traffic” and “audience.”
Figure 14.6 shows a report from a site that used to be a simple hit counter. It has grown into a
more robust metrics platform. There is a link on the PPT slide so you can substitute a simple hit
counter home page if you wish.
Students need to understand what a server request log really is. They are really nasty things that
no marketer wants to look at, but marketers need to have a general sense of where their data
comes from. The main entries, by IP address, are:
• The IP address of the requesting computer
The data is collected for each hit (file), and the spreadsheet analogy helps to make the enormity
of the resulting database seem real to the students.
A web master at your school could give you a copy of a page from a server request log—that
makes the point quite effectively! Students can move on from that with gratitude for web
metrics programs that make data accessible!
privacy questions in the minds of your students. If so, it is a good time to encourage them to start
reading privacy policies to find out what, how and if marketers are telling visitors about what
they do—and why they do it. It is also a good question to ask of the metrics person at your
school. Table 14.1 summarizes some of the data that can be provided for a single visitor to the
site using tagged pages.
This may be too much for your students, but you should be aware that the page view has been a
key metric, probably because it is much like “exposure” in traditional advertising. However, the
page view is losing currency because of Ajax technology (Old Navy and many retail sites are
good examples). Basically, many pages are loaded as thumbnails. Looking at only that “page
view” does not show which of the thumbnails were clicked on. That can be handled by metrics
platforms, but it has pretty much done away with the original idea of page view, since the
meaning has become inconsistent between Ajax pages and regular product pages.
Server logs collect data without the active participation or permission of the user and often
without their knowledge. Panels of Internet users are recruited who permit tracking software to
These data have the usual advantages of panel data including identified respondents and the
ability to collect demographic and lifestyle data about them. Longitudinal data is also an output
of panel research. Firms that maintain panels are eager to do custom work for individual clients
as well as to sell their standard research products.
Measuring Website Traffic, Audiences, and Campaigns
This section attempts to give a reasoned explanation of traffic and audience measures.
The text lists some of the key metrics in each category. On the PPT they have been summarized
in a table to try to clarify when is a confusing picture, if for no other reason that the almost
limitless number of specific variables that are available. By combining the possible time frame
with only a few of the basic metrics, the magnitude of the metrics challenge becomes evident.
The engaged student should already be thinking, “So how do you choose which ones?”
Exercise 14.2).
The example is a real one—for my blog. It’s important for students to know that each blog post
has an individual URL and therefore can be tracked as a web page.
I’ve tried to mimic some of my use of Google Analytics in Figure 14.11. I tried to tell an
interesting story and keep it as simple as I could. If you are using your own analytics or having
someone come in to demonstrate live, think about the story aspect. It’s more meaningful to
students that just showing the pages that exist. Their eyes glaze over after just a few pages.
At this point some student will invariably ask if they can see Google Analytics reports for sites
they are interested in. The answer is emphatically “NO.” Using Google’s Blogger platform and
their analytics is pretty easy. Google, however, is good about making sure the user owns a piece
of content before allowing many types of actions, including placing Google Analytics on it.
The example of segmentation analysis in Figure 14.12a was chosen because it was analyzing an
email campaign, not a website. The link takes you to the product PDF. It is hard to reproduce
these captures so they are readable, so you could go to it live. Figure 14.12b goes back to a web
page, showing click-through overlays. It’s nice, after looking at all the numbers to look at a
graph. Most platforms allow you to toggle back and forth between numbers, which the marketer
needs to study, and graphics, which managers are generally happier to view.
Keep reminding students that these are only a few of the reports available.
Metrics Based on Panel Data
This section reemphasizes the usefulness of panel data. The metrics based on server logs can
only produce data for the marketing programs of the business itself. Students need to be clear on
that. I keep being asked, “Can I put Google Analytics on a site I’m interested in?” NO! That can
only be done by the site owner.
Both Quantcast and Compete used to offer comparative data for free. Now they only offer data
for one site at a time. Both use panels to collect the data. Theirs seem to be convenience samples
while Nielsen and comScore both recruit statistically valid panels.
The topics of social media metrics and mobile metrics needed to be mentioned here for
completeness although they are discussed in their separate chapters. There is good news and bad
Marketers’ Confidence in Their Metrics
One thing that never seems to change is that marketing managers are not confident that they have
the data they need about the effectiveness of their Internet marketing activities. In Figure 14.16
few rate their capabilities as excellent or even good –20%. A lot of them are working on it, but it
doesn’t seem to be a destination with a clear goal. The Internet, its platforms and its metrics are
changing so rapidly that most marketers are severely challenged to keep up. eMarketer
frequently reports on studies that bear on the subject, or just try searching for recent reports. You
will usually find newer ones because of the importance and the difficulty of the measurement of
Internet marketing effectiveness.
Discussion Questions
1. The term “metrics” is commonly used by Internet marketers. Explain your
understanding of the meaning of the term.
The term metrics is a piece of marketing jargon, but it seems appropriate for its intended
meaning. Metrics are numbers, pure and simple.
2. True or False: Internet marketers need to decide whether they will conduct
measurement from a visitor usability perspective or from a traffic and audience
measurement perspective. Defend your answer.
The issue is not “either or.” It is “both.” This question tests the students’ understanding of
3. At what point in the site development or redevelopment process should usability testing
be done?
Usability testing should start in the prototype stage. There are many reasons to argue for many
4. The term “hit counter” implies that all this technology can do is to measure the number
of people who enter a site. Based on the discussion in this chapter, is that an accurate
perception? Why or why not?
Students may be familiar with hit counters, may have even used one on a personal site they have
5. Describe the similarities and differences between server request logs and panel data.
Which do you think is most useful?
• Server request logs, are the operational records kept by any file server. They provide
6. True or False: The web master and the marketer use the same metrics when assessing
website effectiveness.
The web master is tasked with site performance and needs that data. The marketer is tasked with
7. What are the uses of server log data, cookies, and tagged pages in the collection of
website effectiveness data?
8. What are some of the specific metrics that measure Internet traffic, audiences, and
campaigns? Which ones do you think are most important?
The text lists the following under each category:
• Traffic measures document site activity:
• Audience measures provide data about the people who visit the site:
9. What are some of the special advantages of user panel data?
The advantages are basically those we learned in marketing research. Panel data allows the
10. How should the marketer go about choosing the right metrics to measure website or
campaign effectiveness?
Another thing that cannot be said too often is that the only way is to match metrics to SMART
11. What art SMART objectives?
Kaushik uses DUMB:
Doable
12. What is the meaning and importance of the term “KPI”?
A Key Performance Indicator is a metric that has a direct relationship to business success.
13. What are some of the main challenges facing the web metrics discipline at present?
The challenge all alone—and it only gets worse—is the huge amount of data (now “big data”)
that is produced. It must be aggregated, analyzed, transformed into decision-relevant marketing
14. Do you believe that most commercial websites are using metrics to improve customer
experience and enhance marketing effectiveness? Do you have any personal experiences
that support your view?
The reality appears to me that most websites, even startups, usually put metrics on the site—
Google Analytics being the starter platform of choice. At the other end, large sites like Staples
Internet Exercises
1. Internet Career Builder Exercise.
2. Conduct a partial usability analysis of your school’s website. Describe two tasks that
you believe the target audience wants to perform on the site. Go to the site, and attempt
to perform both tasks. Record your experience for each, noting the time it took to
perform each task. What recommendations would you make to the web master of the
site based on your test?
This is a good activity to do live in the classroom or in a lab. Alternatively, you can assign
students to do it as an outside assignment. If so, they could be required to come up with a clear
set of instructions, just as if they were running a usability test.
3. Using one of the sites you are tracking, prepare a careful summary of the objectives you
believe that site has. In doing so, keep in mind that the objectives of the site may be a
subset of the marketing and business objectives of the organization. When you are
satisfied that the set of objectives is complete, list the corresponding metric or metrics
that would be needed to measure each objective. Present this information is a table that
shows objectives and the corresponding metrics.
This exercise can have two positive results—a better understanding of marketing objectives as
well as a more in-depth understanding of metrics. Encourage students to look at annual reports,
4. Using a website with which you are familiar, develop a scenario that tells the story of
how one user segment would behave on a visit to the site. When you are satisfied that
your scenario represents behavior of one segment adequately, identify the metrics that
would be useful in ascertaining if the segment’s needs are being met and if the customer
experience for this segment is satisfactory.
Sitting down and thinking about how a member of an identified target segment goes about using
benchmark an industry standard against which a business can measure its performance.
clickstream the complete data record, made up of mouse clicks, of consumer activity on the
Internet during a specified period, usually the duration of a visit to a single website.
DNS (Domain Name System) the process for converting the name of a website into its IP
address.
focus group the qualitative marketing research technique in which a group, usually six to twelve
people who fit a particular profile, are brought together to discuss an issue under the guidance of
a skilled moderator.
hit counter a piece of software inserted onto a website that measures the number of visits to the
site.
Pareto curve a plot of number of occurrences against percent of total; the source of the 80/20
rule.
pixel short for picture element; a single point on a display screen or in a graphic image.
prototype a preliminary model.
server a computer from which other computers request files.