PART ONE
Test Construction
and
Assessment
Anita L. Vangelisti
2 TEST BANK FOR THE ART OF PUBLIC SPEAKING
ew teachers regard constructing and grading exams as the most enjoyable aspect of their job.
Some see testing as a necessary but uninspiring part of teaching; others see it as boring, tedious,
or even oppressive. Yet most are unwilling to cast aside testing. Why? Because testing serves im-
portant functions for teachers and students alike.
Teachers use tests above all as a way of gauging what students have learned in a course. Although
some people may perform poorly on a given exam due to illness, personal problems, or other factors
not immediately related to course content, the accumulation of test scores over a full quarter or se-
mester usually provides a reliable index of what students have learned. In addition, tests provide in-
formation that instructors can use to improve their teaching. For example, if a test reveals that most
students are having difficulty with particular aspects of the course, the instructor should think about
ways she or he can present those aspects more effectively.
For students, well-constructed tests provide a stimulus for learning, an important form of feed-
back concerning their progress in a course, and a guide to help them refine their study habits. In an
ideal world, all students would keep up with reading assignments and review their class notes on a
regular basis. The reality, however, is that tests stimulate students to digest course material they might
otherwise overlook. Studying for exams is one way students learn. In addition, some very important
learning takes place after an exam, when students review the items they “missed.” It may become
clear, for instance, that a student understood the material presented in lecture and class discussion but
did not give sufficient attention to material in the textbook. Knowing this can help the student do bet-
ter on subsequent exams. Students can also use test results to uncover problems in their study habits.
They may discover that they need to devote more time to studying, that they need to study at a more
in-depth level, that they need to write out answers to sample questions as part of their studying, or
even that they need to meet with an academic counselor to review their study methods.
If exams are to function effectively for either teachers or students, however, they must be carefully
constructed. Designing a good exam takes considerable time, effort, and resourcefulness. Above all, it
requires a clear understanding of one’s educational objectives and of the kinds of exam questions one
can use to achieve those objectives. In this essay, we shall focus on both of these issues. In doing so,
we shall give special attention to the public speaking course and to the kinds of exam questions in the
Test Bank for The Art of Public Speaking.
Educational Objectives
One of the many challenges of constructing an exam is to design questions that reflect the full scope
of the instructor’s educational goals. Although most teachers want students to be able to recall im-
portant information from reading, lecture, and discussion, they typically have other instructional goals
in mind as well. In public speaking courses, for instance, teachers usually want students to identify
the kinds of evidence used by speakers to support their claims. But they also want students to know
which kind of evidence might work best in a given situation and to be able to evaluate the quality of
particular pieces of evidence in a speech.
Because instructors usually have multiple learning goals for their students, it is important to con-
struct tests that assess multiple levels of learning. There are several taxonomies of learning in the ed-
ucation literature. Teachers can use those taxonomies to help ensure that the educational goals of their
courses match the test questions they use to gauge what students have learned. The most widely used
taxonomy for this purpose is Bloom’s taxonomy of educational objectives. This taxonomy identifies
F
TEST CONSTRUCTION AND ASSESSMENT 3
six levels of learning—knowledge, comprehension, application, analysis, synthesis, and evaluation.
The taxonomy is hierarchically structured so that learning at each of the lower levels must occur be-
fore learning at the higher levels is possible. In other words, if learning is to occur at the comprehen-
sion level, students must first be secure in their knowledge of the subject matter. Similarly, if learning
is to occur at the analysis level, it must be preceded by learning at the knowledge, comprehension,
and application levels. The following definitions and examples of test questions will help illustrate
the progressive nature of Bloom’s taxonomy.
At the knowledge level, students are expected to recognize and recall key terms, facts, principles,
and concepts. A learning objective at the knowledge level might require that students be able to iden-
tify the first step in Monroe’s motivated sequence. To determine whether students have achieved this
objective, the following question might be included on a test:
What is the first step in Monroe’s motivated sequence?
a. action
b. need
c. visualization
d. satisfaction
e. attention
The comprehension level goes one step beyond the knowledge level and requires students to ex-
plain facts and principles, to interpret information, or to compare and contrast concepts. At this level, an
instructional objective might have students compare and contrast public speaking and conversation—as
in the following question:
One of the major differences between public speaking and ordinary conversation is that pub-
lic speaking usually requires
a. adapting to feedback from listeners.
b. a more formal manner of delivery.
c. organizing ideas for effective communication.
d. tailoring the message to the audience.
e. telling a story for maximum impact.
Learning at the application level requires that students utilize the facts and principles they have
learned. Educational objectives at this level might involve problem solving or applying a key concept
to a new situation. For instance, the following question assesses students’ ability to apply their
knowledge about specific purpose statements to a statement they have not previously seen:
“To inform my audience about the three basic steps in preventive medicine” is an example of
a __________ for an informative speech.
a. specific purpose
b. central idea
c. transition
d. general thesis
e. topic statement
4 TEST BANK FOR THE ART OF PUBLIC SPEAKING
At the analysis level, students must be able to understand the components of concepts, principles,
or arguments. Analysis involves the ability to break down a whole into its parts and to distinguish
among those parts. In a public speaking course, a multiple-choice test question at the analysis level
might ask students to distinguish among the kinds of situations for persuasive speeches:
At which of the following would you be most likely to hear a persuasive speech on a question
of fact?
a. a religious service
b. a classroom lecture
c. an awards ceremony
d. a jury trial
e. a retirement banquet
The synthesis level involves integration and creativity in working with principles and concepts.
An educational goal at this level might be for students to make generalizations about the reasons for
developing a limited number of main points in a speech. To test this objective, instructors might use
the following question:
What is the most important reason for limiting the number of main points in a speech?
a. It is difficult to organize supporting materials if there are too many main points.
b. There is usually not enough time to develop more than two to five main points.
c. It is hard to maintain parallel wording if you have too many main points.
d. It is difficult to balance the time devoted to each point when there are more than five
main points.
e. It is hard for the audience to keep track of too many main points.
Finally, learning at the evaluation level suggests that students are able to assess the soundness, ef-
ficacy, logic, or universality of a given principle or concept. An objective at this level might require
that students be able to assess the reliability of different sources of testimony in a speech—as in the
following question:
Which of the following would probably be the most reliable source of testimony in a speech
about the impact of automobile exhaust on air pollution?
a. the president of General Motors
b. a sociology professor at Yale University
c. the U.S. Environmental Protection Agency
d. a public relations officer at Ford Motors
e. the head of the Better Business Bureau
Using Bloom’s taxonomy in test construction does not require an equal number of questions for
each of the six levels of learning. It may be, for example, that the goals of a course are best represented
by the first three levels of the taxonomy. In such a situation, it would be inappropriate for instructors to
use a large number of exam questions that assess learning at levels four through six. The best way to
approach Bloom’s taxonomy is to use it as a guide to ensure that the types of questions designed for an
exam fit the teacher’s instructional objectives.
TEST CONSTRUCTION AND ASSESSMENT 5
Types of Examination Questions
There are four kinds of exam questions in the Test Bank for The Art of Public Speaking—multiple–
choice, true-false, short-answer, and essay. Let us look at each kind and at some factors to consider
when using them in an exam.
Multiple-Choice Questions
Multiple-choice questions require students to select one response from a set of alternatives. For in-
stance:
Speechmaking is a form of power and therefore carries with it heavy __________
responsibilities.
a. logical
b. ethical
c. emotional
d. psychological
e. sociological
A common misperception among teachers and students alike is that multiple-choice questions are
useful only for testing at the lower levels of Bloom’s taxonomy. In fact, multiple-choice questions
can also be used to assess students’ ability to analyze, synthesize, and evaluate course material. Con-
sider the following question, which requires students to synthesize concepts regarding audience anal-
ysis and apply them to a particular situation:
Ramona is preparing a persuasive speech on environmental issues to present to members of
the National Wildlife Federation. The most important factor Ramona should consider when
analyzing her audience is probably its
a. gender.
b. knowledge of the topic.
c. education.
d. group membership.
e. ethnic background.
Even more complex is a question such as the one below, which involves a combination of levels
in Bloom’s taxonomy:
In his persuasive speech, Jeremy argued that his school should dramatically increase the num-
ber of all-gender bathrooms on campus. Most of Jeremy’s classmates listened carefully to his
argument. Some were persuaded, while others continued to believe that the current system was
justified. Two audience members disagreed so strongly with Jeremy that instead of listening,
they wrote notes back and forth to each other throughout the speech. Which of the following
statements best describes the issues of ethical listening involved in this situation?
a. Everyone in the class was an ethical listener because no one interrupted Jeremy or pre-
vented him from speaking.
b. The people who listened carefully to Jeremy’s arguments were ethical listeners, re–
gardless of whether they were persuaded.
6 TEST BANK FOR THE ART OF PUBLIC SPEAKING
c. The two classmates who refused to listen to Jeremy’s speech and wrote notes back
and forth violated the guidelines for ethical listening.
d. all of the above
e. b and c only
Whether simple or complex, multiple-choice questions have a number of strengths. They allow in-
structors to assess a wide range of learning objectives as well as a broad base of course materials. They
allow for accurate, speedy grading and, when constructed well, result in highly reliable test scores.
Good multiple-choice questions, however, are difficult to construct. Because of this, instructors
may be tempted to devise relatively easy items that simply test students’ recall of information rather
than their abilities to comprehend, analyze, or synthesize course material. Further, if a multiple-choice
question is not well designed, students may misunderstand it or may easily guess the correct answer.
Teachers can take a number of precautions to avoid some of the more common difficulties asso-
ciated with multiple-choice questions. First, it is crucial that the question be written in clear, familiar
language. The objective is to test students’ understanding of course material rather than their ability to
decode a convoluted query. Second, to the extent possible, all of the potential answers should be par-
allel in structure and similar in length. If one of the potential answers differs dramatically from the
others in structure or length, students skilled in taking multiple-choice exams may be able to use the
difference in length or structure to help determine the correct response. Third, it is imperative that all
of the potential answers be plausible on first reading. If one or more is patently implausible, the ques-
tion may end up testing students’ ability to eliminate incorrect answers rather than their ability to
identify the correct one.
True-False Questions
True-false questions require students to assess the accuracy or inaccuracy of a given statement. For
instance:
T F Under normal circumstances, the introduction should constitute about 10–20 percent
of a speech.
Like multiple-choice questions, true-false questions provide highly reliable test scores as well as
efficient, accurate scoring. Moreover, because of their brevity, they can be used to test a broad base of
course material in a short period of time. True-false questions are usually employed to gauge stu-
dents’ recall of information, but they can be designed to assess learning at the upper levels of Bloom’s
taxonomy—as in the following questions:
T F Research indicates that evidence is usually more persuasive when it is stated in gen-
eral rather than specific terms.
T F If you were giving a speech with the specific purpose “To inform my audience of
the major geographical regions of Mexico,” the most effective organizational pattern
for arranging your main points would probably be chronological order.
T F Whenever you use a hypothetical example in a speech, it is usually a good idea to fol-
low it with statistics or testimony to show that the example is realistic.
TEST CONSTRUCTION AND ASSESSMENT 7
The major limitation of true-false questions, of course, is that students who have not studied for
an exam have a 50-50 chance of guessing the correct answer. As a consequence, true-false questions
are less effective than multiple-choice questions in discriminating between students who have a firm
grasp of the subject and those who do not.
The ability of true-false items to measure the knowledge and understanding of students can be
enhanced if teachers follow a few basic suggestions. First, it is important that true-false questions be
written as simply as possible. Overly complex items can be ambiguous and confusing to students.
Second, extreme terms such as “always” and “never” should be avoided except in the rare instances
when something is invariably true or false. Third, instructors should avoid negatively worded state-
ments when constructing true-false questions. Consider, for example, the following question:
T F Graphs are not an effective way to simplify and clarify statistics.
This question is potentially confusing. By forcing students to think in terms of a double negative,
it tests their ability to disentangle word puzzles more than it tests their knowledge of course content.
A more effective question would be:
T F Graphs are an effective way to simplify and clarify statistics.
Contrary to popular wisdom, it is not necessary to have an equal number of true and false items
when constructing a true–false exam. Because students are more likely to respond “True” when guessing
at an answer, questions with “False” answers do a better job of discriminating between students who
genuinely understand the material and those who do not. Many experts in test construction recommend
having a slightly larger number of false items than true items when constructing true-false exams. A
good ratio to shoot for is 60 percent false items and 40 percent true items.
Short-Answer Questions
Short-answer questions can be simple or complex. Simple short-answer questions require students to
respond to a question or statement by filling in a blank with the appropriate word or phrase. For in-
stance:
The connotative meaning of a word includes all of the feelings and associations the
word touches off in different people.
Or:
The following set of main points is arranged in topical order.
I. The first major type of environmental pollution is land pollution.
II. The second major type of environmental pollution is air pollution.
III. The third major type of environmental pollution is water pollution.
Simple short-answer questions such as these test students at the knowledge and comprehension
levels of Bloom’s taxonomy. Complex short-answer questions are more elaborate and can be used to
assess students at the application level and beyond. For example:
8 TEST BANK FOR THE ART OF PUBLIC SPEAKING
Supply the general purpose, specific purpose, and main points for a speech with the following
central idea.
General Purpose: To inform
Specific Purpose: To inform my audience of the three major causes of decaying bridges in
the United States.
Central Idea: The three major causes of decaying bridges in the United States are
overloading, weather, and neglect.
Main Points: I. The first major cause of decaying bridges in the United States is
overloading.
II. The second major cause of decaying bridges in the United States
is weather.
III. The third major cause of decaying bridges in the United States is
neglect.
Some teachers favor short-answer questions over multiple-choice and true-false questions be-
cause they reduce the likelihood that students will be able to guess the correct answer. Creating effec-
tive short-answer questions, however, is a challenge for even the most experienced teachers. Simple
short-answer questions should be phrased so there is one, and only one, correct answer. They should
also be written so as to reduce the chances of students’ guessing the correct answer. Instructors should
make sure that all blanks for a given question are uniform in length and that the question does not
contain any grammatical clues such as the word “a” before a blank when the answer begins with a
consonant or the word “an” when the answer begins with a vowel. To avoid this situation, the words
“a” or “an” should be replaced with “a(n).”
Although complex short-answer questions often allow students greater latitude in wording their
answers than do simple short-answer questions, they also need to be constructed carefully so as to
elicit the appropriate response without either confusing students or providing undue hints about the
answer. Moreover, when grading complex short-answer questions, instructors must decide how they
will assess answers that are similar to, but not identical to, the ideal response.
Essay Questions
Essay questions require students to formulate, organize, and compose their own original responses.
Some essay questions ask for brief answers; others call for extended analyses. Whether brief or ex-
tended, essay questions allow instructors to test a wide range of skills and knowledge. Here, for ex-
ample, is an essay question geared to the knowledge and comprehension levels of Bloom’s taxonomy:
List and explain the four objectives of a speech introduction discussed in your textbook.
In contrast, the following question goes well beyond the knowledge and comprehension levels to
assess students’ skills of analysis, synthesis, and evaluation:
What are the requirements of a good speech introduction? Evaluate the following complete
introduction to a classroom speech in light of those requirements. Be specific in your answer.
TEST CONSTRUCTION AND ASSESSMENT 9
The most dangerous source of radiation in the United States might be right here in this
classroom. The threat is not nuclear power or medical X-rays. Yet according to the En-
vironmental Protection Agency, it threatens up to eight million American homes.
The threat I am talking about is radon, a colorless, odorless gas that may cause as
many as 20,000 deaths from lung cancer each year in the United States.
Today we will explore the threat caused by radon gas. We will begin by learning
more about radon itself—what it is, where it comes from, and why it is so dangerous.
Then we will look at some solutions to the problem—steps that can be taken by the
government and by individual citizens to control the dangers of radon in our homes.
To the delight of many teachers, essay questions can be quick and easy to construct. However,
grading essay questions can be very time-consuming. Whatever time is saved in constructing essay
questions is often lost in reading and assessing the answers. Moreover, the open-ended nature of
many essay questions can create problems for instructors and students alike. Instructors worry about
whether they will be able to provide consistent, reliable evaluations, while students worry about
whether they will be able to generate the “right” answer.
Teachers can take several steps to reduce these concerns. First, the task required by an essay
question should be made as clear and explicit as possible. For instance, rather than asking students to
“describe” several concepts, teachers can require students to “identify, define, and provide examples
of” those concepts. Similarly, instead of having students “discuss” an issue, instructors can ask them
to “evaluate the issue based on both its strengths and its weaknesses.” Second, the number of points
or the percentage of the total test grade allotted to each essay question should be clearly indicated.
This allows students to estimate how much time they should devote to each essay relative to other
questions on the exam.
Third, it is best to give students a limited choice of essay questions to answer. If, for example,
students are asked to respond to three out of seven questions (rather than, say, three out of four), they
may spend an inordinate amount of time reading each question and deciding which ones to select.
Finally, it is imperative that teachers write out their criteria for grading and/or an optimal response to
each essay question before they begin to read the students’ answers. Failure to do so will produce in-
consistent grading and substantially reduce the reliability of the exam.
Other Issues to Consider
In addition to designing questions, there are several other issues teachers must face when they prepare
examinations. These issues include the material that should be covered on the exam, the length of the
exam, the kind and amount of advance information students should receive about the exam, and the
appropriate level of difficulty of the exam. Let us look at each of these in turn.
Exam Content
When selecting what to include on an exam, instructors should be guided by one fundamental principle—
test what you teach. This does not mean material from the textbook should be included on the exam on–
ly if it is explicitly treated in lecture or discussion. Students can legitimately be held responsible for any
assigned reading or exercise—whether or not it is discussed in class—as long as they are informed in
advance that it will be covered on the exam.
10 TEST BANK FOR THE ART OF PUBLIC SPEAKING
In fact, students are often frustrated by exams that do not deal fairly comprehensively with the as-
signed material. For example, if an instructor, without forewarning, includes several questions from
one chapter of the textbook and no questions from another chapter, students who have studied all the
required chapters may feel as if they wasted part of their study time. In addition, a test that neglects a
substantial portion of the course content is likely to slight some of the teacher’s instructional goals.
Length of Exams
Unless the goals of a course include teaching students how to deal with intense time pressure, stu-
dents should have adequate time to complete examinations. If even one-third of the students in a class
run out of time to finish an exam, the exam is probably too long. As a general rule, teachers should
allow 1 minute for each multiple-choice question, 5 to 10 minutes for a half-page essay, and 10 to 15
minutes for a full-page essay. One note of caution: When calculating the length of an exam, be sure to
allow sufficient time for explaining any instructions and for distributing test booklets, score sheets,
and the like.
Reviewing for Exams
On the one hand, many novice teachers worry that their exams will be too easy if students receive a
great deal of information about the test beforehand. On the other hand, experienced instructors often
note that regardless of how much information they provide about the test, some students don’t take
advantage of that information. As a general rule, instructors should let students know the topics that
will be covered on the test and the kinds of questions that will be asked (essay, multiple-choice,
etc.).
Many teachers give students a review sheet roughly one week before the test. Even if the review
sheet consists only of a general topic outline of the class lectures, it can help students organize and
synthesize the material they need to study. Another way to help students prepare for an exam is to set
aside class time the session before the exam to answer questions about the course material. It is also
helpful to distribute a few sample questions that clarify the format of the exam and the kinds of materi-
als it will cover.
Difficulty of Exams
The difficulty of an exam should be commensurate with the objectives of the course. If the course
objectives involve only the first three levels of Bloom’s taxonomy (knowledge, comprehension, ap-
plication), then exam questions should not go beyond these three levels. In contrast, if one of the
course objectives is to encourage students to process material at higher levels of the taxonomy (analy-
sis, synthesis, evaluation), then that objective should be reflected in the test questions.
Exams should be challenging but not unreasonable. They should distinguish among students who
know the course material inside and out, those who “crammed” the night before the exam, and those
who never cracked the textbook. If the test questions are well constructed and truly reflect what was
taught in the course, most students will acknowledge that the test provided a fair measurement of
course material.