CHAPTER 5
ANALYZING AND EVALUATING REPORTS OF QUANTITATIVE RESEARCH
STUDIES
TEACHING ACTIVITIES FOR CHAPTER 5
Activity 1. How to read a research report.
If your students have no prior experience with research methodology, they probably will find it difficult at first to
comprehend reports of quantitative research studies. If this is the case, you can have them study this chapter and
then, in class, walk them through one of the reprinted research articles in Chapters 9–12. You can have them read a
section at a time, asking them questions such as, “What are the research questions?,” “What are the variables?” and
“What are the statistical findings?”
Next, you can give them a homework assignment in which they analyze another of the reprinted research
articles. You might have them complete an outline in which they state the research study’s:
1. research questions/hypotheses/objectives.
2. main conclusions drawn from the article’s literature review.
3. constructs and their definition.
4. measures and the variables that they measure.
5. sampling procedures.
particular quantitative research study in class.
Activity 2. How to evaluate a research report.
If your students already are able to comprehend research reports, you can focus on developing their skill in
evaluating the quality of a research study. You can use the form in Appendix 3 for this purpose. You can select one
MULTIPLE-CHOICE TEST ITEMS FOR CHAPTER 5
1. Quantitative research studies are usually characterized by researchers’
a. belief in an objective reality.
b. analysis of reality into measurable variables.
c. study of samples that represent a defined population.
d. All of the above.
2. Constructs
a. are statistically equivalent to variables.
b. are statistically equivalent to constants.
c. are structures and processes inferred from observed behavior.
d. are used in qualitative research, but not quantitative research.
3. An example of a construct in quantitative research is
a. self-concept.
b. the duration of a classroom lesson.
c. the percentage of students responding correctly to a teacher question.
d. a measure of motor coordination.
4. Variables
a. represent a construct in quantitative form.
b. can take the form of test scores, but not categories.
c. are inferences about behaviors that cannot be directly observed.
d. are almost always derived from a specific theory.
5. Hypotheses
a. are explanations of observed research results.
b. involve predictions about how variables are related to each other.
c. cannot be derived from theories.
d. are used when researchers wish to explore phenomena without preconceived constructs.
6. A target population consists of the entire group of individuals
a. who have the characteristic of interest to a researcher.
b. who are studied by the researcher.
c. who are of interest to policy makers.
d. for whom statistics are calculated.
7. Sampling errors
a. are the result of errors in measuring research variables.
b. occur in nonrandom sampling, but not in random sampling.
c. are the difference between a sample statistic and the corresponding population parameter.
d. are the difference between statistics for different subgroups within a sample.
8. Stratified random sampling is used to ensure that all individuals in the population
a. have an equal chance of participating in the research study.
b. have an independent chance of participating in the research study.
c. who share certain characteristics are represented in the sample.
d. have been classified into groups based on characteristics such as ethnicity and gender.
9. If a research study has population validity, this means that
a. the sample statistic and the corresponding population variable are identical.
b. the research measures are appropriate for the target population.
c. the research design is appropriate for the target population.
d. the sample is representative of the population from which it was selected.
10. Performance measures
a. are easier to administer than paper-and-pencil tests.
b. evaluate individuals as they carry out complex tests.
c. are generally not used to measure constructs.
d. All of the above.
11. The measurement approach that can be adapted most easily to the situation in which data will be
collected is:
a. interviews.
b. performance measures.
c. content analysis.
d. direct observation.
12. When low-inference variables rather than high-inference variables are observed in research,
a. observers need less training to achieve satisfactory inter-rater reliability.
b a standard observation form is unnecessary.
c. observers must be concealed while making their observations.
d. the validity of the data tends to be higher.
13. A test is valid to the extent to which
a. the scores of the research sample are supported by evidence and theory.
b. the interpretation of the test scores is supported by evidence and theory.
c. it has withstood the scrutiny of measurement experts.
d. it satisfies the requirements of a ratio scale.
14. Match each of the following examples with the type of validity evidence to which it is most relevant.
___ 1. The researcher examines the degree of relationship between 10 items, all of which
presumably measure the same attitude.
___ 2. The researcher determines whether a measure with 10 items drawn from a 50-item
measure is an adequate substitute for the 50-item measure.
___ 3. The researcher selects a test to determine how well students have mastered a new
state-adopted mathematics textbook.
___ 4. The researcher is searching for a test that can be given to high school freshmen
to determine which of them are most likely to leave school before graduating.
___ 5. The researcher wants to select a test that measures a particular construct, but does
not stigmatize students based on their test scores.
a. Concurrent evidence
b. Evidence from the internal structure of the test
c. Consequential evidence
d. Predictive evidence
e. Content-related evidence
15. A test is reliable to the extent to which
a. students respond to its items consistently.
b. students receive the same score on the test the second time it is administered.
c. different individuals score the test the same way.
d. All of the above.
16. The standard error of measurement
a. specifies the exact amount of discrepancy between an individual’s true score and obtained score.
b. quantifies the amount of discrepancy between the scores of any two raters.
c. estimates the probable range within which the individual’s true score falls.
d. estimates the extent of change between an individual’s test score from one administration of the test
to the next.
17. In a test that has been developed using item-response theory (IRT),
a. the items are ordered by their difficulty relative to each other.
b. a student’s responses to each subset of items can be used to determine whether the test is reliable
for that student.
c. the amount of time that it takes the student to respond to each item is measured and used as a
component of the student’s total score.
d. it is possible for the test to have high validity or high reliability, but not both.
18. The discussion section of a research report typically presents
a. the statistical results of the study.
b. the researchers’ interpretation of the statistical results.
c. the research hypotheses and questions.
d. a rationale for the research design that was chosen.
SHORT-ANSWER TEST ITEMS FOR CHAPTER 5
1. Describe the three aspects of a quantitative research study that are usually covered in the methods section
of the research report.
2. A researcher developed a teacher preparation program designed to increase teachers’ empathy for ethnic-
minority students. Sixty teachers volunteered to take the program, for which they would receive university
credit. The researcher randomly assigned the teachers to two groups of 30 each and prepared one group in the
program. The researcher then interviewed all 60 teachers to determine whether the teachers who completed the
program had more empathy with ethnic-minority students. What are two possible sources of bias in this study?
3. What is the main drawback in using volunteer samples in research?
4. State two advantages and two limitations of the use of paper-and-pencil tests and scales in educational research.
5. What are (a) two advantages and (b) two limitations of the interview as a data-gathering tool in
educational research?
CHAPTER 5
ANALYZING AND EVALUATING REPORTS OF QUANTITATIVE RESEARCH
STUDIES
Multiple-Choice Test Items
Short-Answer Test Items
