Chapter_13 Evaluation_Research_and_Problem_Analysis
1. Like survey designs and field studies, evaluation research refers to the research purpose not a specific research
method.
a. True
b. False
2. Problem analysis focuses upon whether the intended result was produced.
a. True
b. False
3. The policy process, like the research process in general, is fluid and does not always “start at the beginning and
conclude at the end”.
a. True
b. False
4. Outputs refer to the basic question of what a policy action is trying to achieve while impacts refer to the means of
achieving desired policy goals.
a. True
b. False
5. Goal-oriented public policies can be viewed as “ifthen” statements: if some policy action is taken, then we expect
some result to be produced.
a. True
b. False
6. The good thing about applied studies such as program evaluation is that they do not require the same degree of
planning as do survey designs.
a. True
b. False
7. Evaluation apprehension occurs when people fear that their own job performance is being rated.
a. True
b. False
8. Stakeholders are persons and organizations with an indirect interest in a program.
a. True
b. False
9. Not only is it necessary to measure dependent variables in impact assessment, the researcher also needs to measure
the context within which the program is conducted.
a. True
b. False
10. Quasi-experiments differ from true experiments in that they lack random assignment to control and experimental
groups.
a. True
b. False
11. Which of the following is accurate concerning the use of evaluation research?
a. it is gaining in popularity among researchers but it will likely not make a difference
b. federal requirements for program evaluations often accompany the implementation of new programs
c. funding is usually not available for program evaluations
d. it is only of value to the agency you study
12. This type of analysis can help public officials plan and select an alternative action:
a. evaluation analysis
b. evidence-based analysis
c. policy analysis
d. problem analysis
13. Evaluation research is designed to:
a. be cost effective
b. determine if a program is effective in its implementation
c. determine what the public policy should be
d. be published
14. Which of the following does not represent steps in the policy analysis process?
a. there is a demand for some new course of action or strong opposition to an existing policy
b. formulate a hypothesis that will allow for exploration
c. identify the ultimate goals and the different actions for achieving those goals
d. consider the impact of the policy outputs
15. Which of the following would not be an example of problem analysis?
a. find the most cost effective manner to maintain prison capacity at established levels
b. determine the most efficient manner in assigning felony cases to circuit judges
c. determine the average sentence given for those convicted of armed robbery within the state of Georgia
d. what is the best manner to reduce the response time by police officers in calls for service
16. Which of the following statements is accurate concerning program evaluations?
a. program evaluation is designed to link the intended actions and goals of policy to empirical evidence that
supports them having the desired effects
b. program evaluation is concerned only with whether policies are popular
c. program evaluation is concerned with whether policies are ethical
d. all of the above
17. Which of the following is a type of program evaluation?
a. exploratory studies
b. content assessment
c. impact assessment
d. content analysis
18. At what stage does problem analysis take place?
a. policy implementing
b. later study
c. program evaluation
d. policy making
19. Which of the following represents potential problems in doing evaluation research?
a. conflicting goals
b. clearly stated goals
c. ability to express policy goals in the “ifthen” form that can be empirically tested
d. all of the above
20. Which of the following is not something of concern in the evaluability assessment?
a. determine the degree of support for your project
b. determine what the general program goals are as well as the specific objectives
c. determine how the goals are translated into the program components
d. whether the people in the organization have some sort of direct or indirect stake in the program
21. When a researcher prepares their understanding of a program’s goals, elements, and operations, she is preparing a
program:
a. evaluation
b. description
c. analysis
d. hypothesis
22. All of the following would be examples of goal statements that must be clarified before empirical testing can follow
except for:
a. encouraging participants to accept the philosophy of a drug-free life
b. encouraging participants to be equipped with the life skills necessary to succeed
c. encouraging participants to obtain their GED upon completion of the program
d. providing participants with problem-solving skills
23. In order to measure a program’s success in meeting goals, the researcher should be able to do which of the
following in advance of the inception of the research?
a. involve measureable performance dimensions
b. all important terms must have non–controversial definitions
c. relevant standards or criteria must be specified
d. all of the above
24. Program goals represent:
a. empirical indicators
b. desired outcomes
c. inputs
d. standards
25. Which of the following would a researcher completing an impact assessment be concerned with?
a. understanding how participants are selected
b. measuring the dependent variable
c. measuring the success of the program
d. measuring the independent variable
26. With regard to evaluation designs, which of the following statements is not accurate?
a. variation in the levels of treatment delivered by a program can be a major threat to the validity of even
randomized evaluation studies
b. uncontrolled variation in treatment is equivalent to unreliable measurement of the independent variable
c. controlled variation in treatment represents a threat to generalizability
d. varying degrees of participation by clients in a program will impact the outcome
27. Randomization may not be appropriate for assignment of people to treatment or programs for all of the following
reasons except:
a. practical reasons
b. agency support
c. ethical reasons
d. legal reasons
28. Relating to randomization, which of the following statements is not accurate?
a. as the number of exceptions to random assignment increases, the statistical equivalence of the experimental
and control groups decreases
b. when exceptions to random assignment are made, bias enters the selection process
c. randomization in experiments for evaluation purposes are best suited for programs where exceptions are at a
minimum
d. randomization in experiments for evaluation purposes are best suited for programs where there are many
exceptions
29. If the number of subjects in a program is small, statistical tests can detect only very large program effects of
differences in outcome measures between the two groups. This illustrates the problem of:
a. generalizability
b. statistical conclusion validity
c. maturation
d. reliability
30. Treatment integrity is roughly equivalent to:
a. reliability
b. validity
c. generalizability
d. the accuracy of the data
31. Randomized experiments for evaluation purposes require which of the following?
a. case flow must be adequate to produce enough subjects in both the experimental and the control groups
b. program staff must accept random assignment and minimize the exceptions to randomization
c. experimental interventions must be consistently applied
d. all of the above
32. In the home detention studies used in your text as examples of randomized studies, the authors concluded that:
a. randomization allows generalization
b. randomization allows for reliability
c. randomization does not control for variation in treatment integrity and program delivery
d. randomization does control for variation in treatment integrity and program delivery
33. What can be said about the relationship between process evaluations and impact assessments?
a. process evaluations can be used in interpreting results from an impact assessment
b. process evaluations monitor program implementation and questions about a program’s effects
c. process evaluations are a method of assessing the accuracy of the data collected
d. there is no relationship between impact assessments and process evaluations
34. Which of the following is not correct with respect to process evaluations?
a. information about program implementation can be linked to outcome measures, even when accompanied by a
process evaluation
b. process evaluation aids in interpreting results from impact assessments
c. process evaluations are useful when a researcher is interested in the performance of specific tasks within a
program
d. process evaluations assume that tasks within a program are linked to program outcomes
35. Policy analysis is used to:
a. help gather data
b. select an alternative course of action
c. produce an explanation
d. all of the above
36. Policy analysis searches for:
a. actions that will be accepted by funding agencies
b. operational definitions of policy terms
c. the consequences that would follow the implementation of various courses of action
d. whether the policy should be implemented
37. The relationship between policy analysis and program evaluation is best characterized as:
a. both rely upon clearly identical goals and objectives
b. program evaluation is used to test the achievement of goals and objectives worked out through policy analysis
c. program evaluation must precede policy analysis
d. there is no relationship between policy analysis and program evaluation
38. Which of the following is in contrast to incident-oriented policing?
a. problem-oriented policing
b. reactive-oriented policing
c. call for service policing
d. report-driven policing
39. Any public policy discussion involves politics. Which of the following is not accurate with respect to ethics and the
political aspects of applied research?
a. there is no set of political norms that are agreed upon by all criminal justice researchers
b. just as there are codes for ethical conduct for researchers, there are formal codes of accepted political
conduct
c. ethics of criminal justice research deals more with the methods used while political issues are more concerned
with the substance and use of the research
d. there is no formal code of accepted political conduct
40. When dealing with stakeholders a researcher should be aware of all of the following except:
a. be able to identify all the stakeholders
b. get particularly friendly with the stakeholders so they will not interfere with your research
c. find out the stakeholders’ perspectives on the program
d. find out the stakeholders’ perspectives on the evaluation
41. Which of the following embody the means to achieve desired policy goals?
a. outputs
b. inputs
c. impacts
d. achievements
42. Which of the following would a researcher be least likely to use randomized designs?
a. program evaluation studies
b. ex–post evaluations
c. victim centered studies
d. time-series designs
43. Scientific realism is rooted in the principle that:
a. similar interventions can naturally be expected to have different outcomes in different contexts
b. the success of evaluations depends on the assignment of subjects to experimental and control groups
c. there be equivalence between experimental and control groups before an intervention
d. treatment integrity must be rigorously guarded
44. Dr. Smith is interested in implementing a program that is designed to reduce the recidivism rates of property
offenders. He selects two state prisons and with the blessing of the Department of Corrections begins the programs
in the main state prison for males and a female institution that is nearby. Which design would be best for Dr. Smith
to use?
a. non-equivalent groups
b. randomized study
c. interrupted times-series
d. none of these would work
45. When evaluating full-coverage programs, it is typically best to use:
a. interrupted time-series
b. quasi-experimental design
c. ex post evaluation
d. randomized design
46. Which of the following would not be an example of an evidence-based policy?
a. the use of CCTV in public entertainment districts to reduce property crime
b. acquaintance rape of college students
c. underage drinking
d. Compstat
47. True or “classical” experiments differ from quasi-experiments with respect to the fact that
a. classical experiments use random assignment to control and experimental groups while quasi-experimental
designs do not
b. classical experiments use experimental groups while quasi-experimental designs use experimental and control
groups
c. quasi-experimental designs afford more control than do classical designs
d. only one independent variable can be used in a classical experiment, while there are no limits to the number of
independent variables in a quasi-experimental design
48. There are two jails in your city, one houses state detainees waiting trial while the other houses federal prisoners
awaiting trial. You have been asked to assess why there are differences in the institutional behavior between these
two institutions. Which design would be appropriate?
a. non-equivalent groups
b. true experiment
c. classical experiment
d. interrupted time-series
49. Nesting a quasi-experimental design within a randomized experiment refers to:
a. a way to study specific parts of a town
b. a way of backing up a true experiment
c. a substitute for a classical experiment
d. a way to examine the impact of the experiment on the subjects
50. A community policing program began in an entertainment area in a large metropolitan area. City leaders are
interested to see if the change in policing strategy has reduced the number of crimes and has asked faculty from a
local university to complete a study. What type of evaluation would be best?
a. interrupted time-series
b. ex–post evaluation
c. non-equivalent groups
d. class experiment
51. Specifying program goals and objectives in operational terms produces:
a. empirical indicators of program outcomes
b. conceptual definitions
c. program outcomes
d. program integrity
52. Many law enforcement agencies are using a program to detect emerging patterns in crime, this program is:
a. problem-oriented policing
b. conceptualization
c. reactive analysis
d. computerized crime maps
53. According to your text, “theory driven” evaluations that consider how each element of a program should affect its
targets should:
a. develop a micro model of the program production process
b. develop a vision statement that will be later used by the program staff
c. develop rigid measure of validity
d. define all inputs and outcomes as early as possible
54. Your state prison system is determined to reduce the recidivism rates of adult offenders leaving the system. To
accomplish this goal, a pre-release program is implemented. The Director of the department of corrections suggested
an ideal design involving:
a. randomization
b. interrupted time-series
c. non-equivalent groups
d. quasi-experimental design
55. Process evaluations:
a. assess the extent to which policies achieve their intended goals
b. are essentially the same as impact assessments
c. focus on policy demands and agenda
d. assess the extent to which policies are being implemented as intended
56. Which of the following is not a concern when a researcher uses an interrupted time-series design?
a. reliability of experimental treatment
b. history
c. instrumentation
d. construct validity
57. In its most elementary form, evaluation research is a process of:
a. evaluating whether a piece of research is valid
b. determining the reliability of research findings
c. determining whether the intended result, of a given policy intervention was produced
d. evaluating whether a given piece of legislation is effective
58. Which of the following can color research in many ways?
a. evaluation
b. politics
c. results
d. analyses
59. A scouting report is a product of:
a. a process evaluation
b. an outcome evaluation
c. an evaluability assessment
d. defining policy objectives
60. The relationship between process evaluations and impact assessments can be stated as:
a. impact assessments must precede process evaluations
b. impact assessments answer questions about program effects while process evaluations monitor a program’s
implementation
c. only process evaluations can be used to determine if a program is effective
d. all of the above
61. This is considered sort of a “preevaluation”?
a. Ppocess evaluation
b. outcome evaluation
c. evaluability assessment
d. impact assessment
62. What is the fundamental first step in conducting evaluation studies?
a. measuring program goals
b. obtaining evaluable statements
c. specifying outcomes
d. clearly specifying program goals
63. Your text discusses a concept called problem guides. Which of the following would be an example of this concept?
a. the analysis of specific problems such as crimes against the elderly
b. the analysis of broad issues such as race and crime
c. the analysis of specific crime databases such as UCR
d. the analysis of criminal justice agencies
64. Which of the following best describes the example of the use of evidence-based policy?
a. agencies using evaluation to decide what evidence backs up current funding
b. justice agencies are linked to evidence used for planning and evaluation
c. agencies are linked to other agencies
d. all of the above
65. Which of the following is accurate with respect to the relationship between impacts and outputs?
a. outputs are policy actions while impacts are goals
b. impacts are related to what the policy goals attempt to achieve while outputs are related to the means of
achieving the goals
c. impacts are the goals and outputs are what is achieved
d. outputs are policy goals while impacts are how the policies are applied
66. Evaluation in the criminal justice system seeks to link what?
a. the intended actions and goals of the policy to evidence that the policies are carried out
b. the policies to the people who work within the agency
c. the people within the agency to their specific responsibilities
d. the written policies and the written goals
67. In process evaluation, the focus is on:
a. the inputs to the program
b. the people who work in the process
c. the program outputs
d. the program process
68. It is often said that in evaluation research, the research questions may already be formulated. Which of the following
is an example of that?
a. in problem analysis usually there is a wide range of alternative choices
b. in process evaluation the focus is on whether specified goals are attained
c. in impact assessments the evaluation is on whether specified goals are attained
d. with the questions already formulated it decreases the amount of involvement for the researcher
69. Which of the following would not be an example of a program’s goal statement?
a. to encourage participants to take responsibility for their actions
b. to study why some offenders were allowed in the program and others were not
c. to provide a safer community through the use of Neighborhood Watch programs
d. to provide life skills to inmates to participate in the pre-release program
70. What is the relationship between desired outcomes and empirical indicators?
a. desired outcomes represent the program goals while empirical indicators determine whether the desired
outcomes are achieved
b. desired outcomes are the empirical indicators
c. desired outcomes represent the results of a program while empirical indicators are used to identify the goals
d. desired outcomes are what an agency hopes to get while empirical indicators identify what must be done to
get them
71. research is appropriate whenever some policy intervention occurs or is planned.
72. A policy is an action taken for the purpose of producing some intended result.
73. Policy refer to what is actually produced.
74. The refer to the basic question of what a policy action is trying to achieve.
75. takes place in the policymaking stage while program evaluation studies are conducted in
later stages.
76. When doing research, the first key step is to learn policy goals.
77. A form of “preevaluation” called a(n) assessment is designed to allow the researcher to
determine whether conditions necessary for conducting an evaluation are present.
78. A researcher should complete a program that reflects their own understanding of
program goals, elements and operations of the organization under study.
79. A(n) refers to researchers and program personnel together considering how each
program element affects the targets.
80. are those that have a vested interest in the organization and the outcome of evaluation
research.
81. Program goals represent desired outcomes, while outcome measures are indicators of
whether or not those desired outcomes are achieved.
82. In addition to the measurement of the outcomes of a program, it is also necessary to measure the program’s
, the experimental stimulus, or independent variable.
83. Uncontrolled variation in treatment is equivalent to measurement of the independent
variable.
84. In evaluation studies, the group to whom the program is appropriate is referred to as the program’s
____________________ population.
85. represents the process through which subjects are accumulated in experimental and
control groups.
86. Identify the differences between randomized evaluation designs and quasi-experimental designs. Give an example of
the appropriate use of each.
87. What are the strengths and weaknesses of quasi-experimental designs? Explain how the use of quasi-experimental
designs may be more useful than classical experiments for conducting criminal justice research.
88. You are a member of student government. Your group has been asked by the President of the University to conduct
a study to determine if the newly implemented anti-drinking program all new freshmen are required to take is
effective in reducing underage drinking on campus. Identify the goals and outcomes of the program. What type of
design would you suggest for such a study and why?
89. Your criminal justice department has been requiring graduating seniors to take an exit exam as a measure of
departmental outcomes. How could that tool be used to determine if the criminal justice major is effective?
90. What does “treatment integrity” mean? Why is this an important issue in evaluation studies?