127
CHAPTER 11 TEST BANK
Evaluation Research and Problem Analysis
TRUE/FALSE
1. Like survey designs and field studies, evaluation research refers to a research method, not
a specific research purpose.
2. Policy analysis focuses upon whether the intended result was produced.
3. The policy process, like the research process in general, is fluid and does not always
“start at the beginning and conclude at the end.”
4. Outputs refer to the basic question of what a policy action is trying to achieve.
5. Goal–oriented public policies cannot be viewed as “if–then” statements: if some policy
action is taken, then we expect some result to be produced.
6. Applied studies such as program evaluation is that they do not require the same degree of
planning as do survey designs.
7. Evaluation apprehension occurs when officials fear that their own job performance is not
being rated.
8. In evaluation research, outcome measures serve as the independent variables.
128
9. Not only is it necessary to measure dependent variables in impact assessment, the
researcher also needs to measure the context within which the program is conducted.
10. Quasi-experiments differ from true experiments in that they lack random assignment to
control and experimental groups.
MULTIPLE CHOICE
1. Which statement is false with regard to the use of evaluation research?
a.
it is gaining in popularity among researchers that really want to make a difference
b.
federal requirements for program evaluations often accompany the implementation
of new programs
c.
funding is available for program evaluations
d.
Evaluation research may only use experimental design.
2. Which action does not represent a step in the policy analysis process?
a.
there is a demand for some new course of action or strong opposition to an existing
policy
b.
formulate a hypothesis that will allow for exploration
c.
identify the ultimate goals and the different actions for achieving those goals
d.
consider the impact of the policy outputs
3. Which of the following is not a variable-oriented strategy for research?
a.
realist approach
b.
experimental approach
c.
case-study approach
d.
all of these
4. Program evaluation is designed to link the intended actions and goals of criminal justice
policy to ____.
129
a.
empirical evidence that supports them having the desired effects
b.
policies that are made to benefit communities
c.
implementation of policies
d.
randomized field experiments
5. Problem analysis takes place in the earlier stages of the policy process. In contrast,
program evaluation studies are conducted ____.
a.
in the interim stages
b.
In the later stages
c.
In the preliminary stages
d.
In the middle
6. Which statement is inaccurate?
a.
impact assessments evaluate whether or not specified goals are attained
b.
process evaluations focus on whether programs are carried out according to plans
c.
problem analysis usually considers a limited range of alternative choices
d.
ideally, impact assessments and process evaluations are conducted separately
7. Which of the following does not represent potential problems in doing evaluation
research?
a.
conflicting goals
b.
vague goals
c.
when the intended actions and goals are linked through empirical evidence.
d.
difficulty in expressing policy goals in the “if-then” form that can be empirically
tested
8. Which step is of least concern in the evaluability assessment?
a.
determine the degree of support for your project
b.
determine what the general program goals are as well as the specific objectives
c.
determine how the goals are translated into the program components
d.
determine whether people in the organization have some sort of direct or indirect
stake in the program
130
9. One of the most important parts of conducting evaluation research is _____.
a.
selecting a sample that will support the program goals
b.
formulating goal statements that are empirically testable
c.
gathering official documents since that is where the researcher can find complete
descriptions of goals
d.
making sure that the staff of the agency where you conduct research like you by
doing anything to get them on your side
10. All of the following would be examples of goal statements that must be clarified before
empirical testing can follow except for ____.
a.
participants will accept the philosophy of a drug-free life
b.
participants will be equipped with the life skills necessary to succeed
c.
participants will be able to obtain their GED upon completion of the program
d.
provides participants with problem-solving skills
11. Program goals represent ____.
a.
empirical indicators
b.
desired outcomes
c.
inputs
d.
standards
12. With regard to evaluation designs, which of the following statements is not accurate?
a.
variation in the levels of treatment delivered by a program can be a major threat to
the validity of even randomized evaluation studies
b.
uncontrolled variation in treatment is equivalent to unreliable measurement of the
independent variable
c.
controlled variation in treatment represents a threat to generalizability
d.
varying degrees of participation by clients in a program will impact the outcome
131
13. Randomization may not be appropriate for assignment of people to treatment or programs
for all of the following reasons except ____.
a.
practical reasons
b.
agency support
c.
ethical reasons
d.
legal reasons
14. Relating to randomization, which statement is inaccurate?
a.
as the number of exceptions to random assignment increases, the statistical
equivalence of the experimental and control groups decreases
b.
when exceptions to random assignment are made, no bias enters the selection
process
c.
randomization in experiments for evaluation purposes are best suited for programs
where exceptions are at a minimum
d.
Random assignment of people to receive some especially desirable or punitive
treatment may not be possible for legal, ethical, and practical reasons.
15. If the number of subjects in a program is small, statistical tests can detect only very large
program effects of differences in outcome measures between the two groups. This
illustrates the problem of ____.
a.
generalizability
b.
statistical conclusion validity
c.
maturation
d.
reliability
16. Treatment integrity is ____.
a.
roughly equivalent to measurement reliability
b.
roughly equivalent to measurement validity
c.
the difference between the experimental intervention is delivered as intended
d.
both reliable and valid
17. If a randomized experiment is used for evaluation purposes, ____.
a.
case flow must be adequate to produce enough subjects in both the experimental
132
and the control groups
b.
program staff may minimize the exceptions to randomization
c.
special interventions must be consistently applied
d.
maintaining the desired level of control over experimental conditions
18. In the home detention studies used in your text as examples of randomized studies, the
authors concluded that ____.
a.
randomization allows generalization
b.
randomization allows for reliability
c.
randomization does not control for variation in treatment integrity and program
delivery
d.
randomization does control for variation in treatment integrity and program
delivery
19. What can be said about the relationship between process evaluations and impact
assessments?
a.
process evaluations can be used in interpreting results from an impact assessment
b.
process evaluations monitor program implementation but impact assessments do
not answer questions about a program’s effects
c.
process evaluation cannot be used to interpret outcomes
d.
there is no relationship between impact assessments and process evaluations
20. Which of the following is not correct with respect to process evaluations?
a.
information about program implementation can be linked to outcome measures,
even when accompanied by a process evaluation
b.
process evaluation aids in interpreting results from impact assessments
c.
process evaluations are useful when a researcher is interested in the performance of
specific tasks within a program
d.
process evaluations assume that tasks within a program are linked to program
outcomes
21. Policy analysis is used to ____.
a.
help design alternative courses of action
133
b.
select a policy
c.
seek explanations
d.
select right employees
22. Policy analysis is ____.
a.
infeasible courses of action
b.
the generation of information and collection of evidence of the benefits
c.
the consequences that would follow the implementation of various courses of
action
d.
fluid and does not always “start at the beginning and conclude at the end.”
23. Which of the following is not accurate with respect to ethics and the political aspects of
applied research?
a.
There is no set of political norms that are agreed upon by all criminal justice
researchers.
b.
Just as there are codes for ethical conduct for researchers, there are formal codes of
accepted political conduct.
c.
Ethics of criminal justice research deals more with the methods used while
political issues are more concerned with the substance and use of the research.
d.
There is no formal code of accepted political conduct.
24. Program evaluation differs from policy analysis with respect to ____.
a.
when it occurs in the policy process
b.
the cost
c.
the time
d.
the role of the stakeholder
25. Scientific realism is rooted in the principle that:
a.
similar interventions can naturally be expected to have different outcomes in
different contexts
b.
the success of evaluations depends on the assignment of subjects to experimental
and control groups
c.
there be equivalence between experimental and control groups before an
intervention
134
d.
treatment integrity must be rigorously guarded
26. Dr. Smith is interested in implementing a program that is designed to reduce the
recidivism rates of property offenders. He selects two state prisons and, with the blessing
of the Department of Corrections, begins the programs in the main state prison for males
and a female institution that is nearby. Which design would be best for Dr. Smith to use?
a.
non-equivalent groups
b.
randomized study
c.
interrupted times-series
d.
none of these would work
27. Problem solving is a fundamental tool in problem-oriented policing and involves which
step?
a.
Carefully defined specific problems.
b.
Conduct in-depth analysis to understand their causes.
c.
Undertake broad searcher for solutions to remove these causes and bring about
lasting reductions in problems.
d.
All of these answers are correct.
COMPLETION
1. A(n) _______________ is an action taken for the purpose of producing some intended
result.
2. Policy _______________ refer to what is actually produced.
3. The term _______________ refer to the basic question of what a policy action is trying to
achieve.
135
4. Problem analysis takes place in the policymaking stage while ________ studies are
conducted in later stages.
5. A form of “pre–evaluation” called a(n) _______________ is designed to allow the
researcher to determine whether conditions necessary for conducting an evaluation are
present.
6. A researcher should complete a(n) _______________ that reflects their own
understanding of program goals, elements, and operations of the organization under
study.
7. _______________ are those that have a vested interest in the organization and the
outcome of evaluation research.
8. In evaluation studies, the group to whom the program is appropriate is referred to as the
program’s _______________ population.
ESSAY
1. Identify the differences between randomized evaluation designs and quasi-experimental
designs. Give an example of the appropriate use of each.
2. What are the strengths and weaknesses of quasi-experimental designs? Explain how the
use of quasi-experimental designs may be more useful than classical experiments for
conducting criminal justice research.
136
3. You are a member of student government. Your group has been asked by the President of
the University to conduct a study to determine if the newly implemented anti-drinking
program all new freshmen are required to take is effective in reducing underage drinking
on campus. Identify the goals and outcomes of the program. What type of design would
you suggest for such a study and why?
4. Your criminal justice department has been requiring graduating seniors to take an exit
exam as a measure of departmental outcomes. How could that tool be used to determine
if the criminal justice major is effective?
5. What does “treatment integrity” mean? Why is this an important issue in evaluation
studies?