365
Chapter 22 Evaluating a Research Report
Suggested Activities
1. For chapter 22 content, group students by program of study and have each group read an
article and critique it with the criteria provided in the text. Placing students who are from
the same program works well with this exercise because students erroneously believe students in
2. Share examples of manuscript reviews you have received and given. Students are not often privy
to this information and it is quite helpful in showing them how the review process works. You
can spend a little time talking about categories of review and feedback and describe the process.
Students really like this activity, especially if you share painful reviews. One helpful experience
for them is for you to include before and after manuscripts. This can also provide you the
opportunity to illustrate appropriate documentation of revisions and modifications you have made
that illustrate how you have modified the report to address reviewers’ concerns.
3. Almost every year at national conferences there are sessions dedicated to journal editor
presentations. These sessions provide a forum for journal editors to discuss their journals and
also to share experiences regarding some of the common mistakes that researchers make when
4. If you are on editorial boards another strategy is to request to the editor that your class
review a manuscript. If the class is small all can be listed as reviewers. If not, usually the
class is noted in a feedback letter from the editor. I have done this several times and the
students really like this activity. It works best if you have them all independently review
the article first and then work through it in class. This activity can also work with a
review you have done in the past. This allows you better control of the quality of the
manuscript that is reviewed. As the Editor-in-Chief of the Journal of Experimental Education, I
can support this strategy for your implementation.
5. With permission, if you are working on a project with a student, have them share their work
before and after feedback. If the student is a slightly more able peer it is quite well
366
6. Similarly, I have had some of my, and sometimes colleagues’, recent masters and doctoral
students come and talk about the process with my class. I usually ‘host’ it as a social. The
students come to a portion of my class session and talk about their recent experiences. They
367
7. Chapter 22 presents a checklist for critically reviewing research. There are others available on
8. As this chapter comes at the end of the class and addresses the preparation of a research report,
this is an excellent time to encourage the students to become VERY familiar with the APA
manual. If yours looks anything like mine—I would encourage you to share all your dog ears and
9. A peer review process in class works well as an out of class assignment. I have done it both so it
was a blind review process and have allowed students to swap with peers with whom they feel
most comfortable. By critiquing and reviewing another’s ‘submission’, the student is better able
to attend to important details in their own writing. In addition to the criteria found in Chapter 22,
the checklist in the manual can facilitate this peer review activity.
10. Students often submit versions of their research that are slightly different from a traditional
research proposal or a report of completed research. For example, my students often apply for
dissertation support, for participation in a professional conference, or for selection into a special
doctoral seminar hosted by a professional organization such as APA or AERA. These proposals
or reports often require a research summary longer than an abstract but much shorter than that my
students have written for any other purpose. These types of proposals and reports are challenging
11. At the end of the semester I host a poster fair for the last class where students can practice
presenting their proposals or completed projects. My colleague Peggy Van Meter started this in
one of her classes and I thought it was a great idea and adopted it for my classes. This activity is
368
Chapter 22 Test Items
1. The portion of a research report where the educational significance of the
problem is presented is the
a. introduction.
b. review of related literature.
c. hypotheses.
d. method.
• 2. When reading a recent journal article Juan was confused. It seems the author
used the term self-determination in a way that was not consistent with Juan’s
previous understanding of self-determination but Juan wasn’t quite sure because
the author did not operationalize his variables. The author should have included
operational definitions of his variables in the ____________ of his article.
a. introduction
b. method
c. results
d. discussion
• 3. Ya–ling was reading a research report on the challenges learners’ face when
reading diagrams in science textbooks. She was really interested in the research but
questioned the instructional materials that the participants were given. It appears that
they may have been much too hard for the participants to read. Since the author did not
adequately describe the materials, Ya-ling has identified a potential flaw in the
__________ of this research report.
a. introduction
b. references
c. methodology
d. results
• 4. Sam read a recent publication and shared it with his class. One of the other
students asked how many participants were in the study and Sam told her he
wasn’t quite sure because the article said that 900 surveys were sent out but never
stated how many were returned. This is a weakness in the ______________
section of the report.
a. design and procedure
b. participants
c. results
d. discussion
• 5. Judy forgot to mention that the survey she used in her dissertation employed a
seven-point Likert Scale. This omission is a weakness in which section of her
report?
a. Participants
b. Design and procedures
c. Instruments
d. Results
369
• 6. At a recent conference, Frank questioned the authors of a research report who
had used a test designed for middle school learners with a second-grade sample.
Frank has addressed a flaw in which aspect of the study?
a. instruments
b. participants
c. results
d. introduction
• 7. Milton recently reported on a survey study that he conducted with High School
students. In his work, he was particularly interested in how many high carbohydrate items
athletes consume during a day while they are training. In his report he does not convey
why the number of high carbohydrate items is of relevance to the related literature on
student athletes. This omission is a flaw primarily in which of the following components
of a research report.
a. Method
b. Results
c. Discussion
d. Introduction
8. In reading a recent article, you note that there is no research topic of study
presented. This is a flaw in the _____________ of the report.
a. introduction
b. method
c. results
d. discussion
9. Of the following questions one might ask when critiquing a research study,
which is a concern for the instruments section?
a. Is the relevancy of the references explained?
b. Are the size and characteristics of the sample described?
c. Are generalizations consistent with the result?
d. Are subtest reliabilities provided?
10. Of the following, which question is relevant when critiquing a narrative study?
a. Are the characteristics that differentiate the groups clearly defined or
described?
b. Are sources of invalidity associated with the design identified and discussed?
c. Are generalizations consistent with the results?
d. Is there a rationale for the choice of individual to study the chose phenomenon?
11. Janet is interested in response to intervention in a sixth-grade child with autism. She
employs a case study methodology. Of the following, which is something that is critical
for her to include in her research report?
a. Tables of the quantitative research results.
b. A rationale for selecting this case for study.
c. A description of how Janet is passionate about the study.
d. The psychometric properties of the instruments used.
370
• 12. Dave is reviewing a manuscript submitted for publication. The paper describes a
study that assessed the attitudes of pre-service teachers about their student
teaching experience. Which of the following questions is important for Dave to
ask himself regarding the study?
a. Was an appropriate experimental design selected?
b. Were control procedures applied to equate groups on extraneous variables?
c. Was the questionnaire pretested?
d. Were efforts made to overcome observer bias and observer effect?
• 13. Marci is reviewing a manuscript submitted for publication. The study is
somewhat interesting but the authors don’t ‘make a case’ that it is relevant to
research or practice. Given this flaw, in her review, Marci indicated that the
authors should focus on revisions to the
a. introduction.
b. method.
c. results.
d. procedures.
• 14. Recently, Sam was reading a study that addressed the benefits of the structure
strategy in text comprehension. The study was well designed but Sam was
surprised that Meyer’s relevant 1998 article was not mentioned in the study. Sam
has indicated a flaw in the _______ of the article.
a. problem statement
b. review of related literature
c. hypotheses statements
d. design and procedure
• 15. A recent publication in the Journal of Interesting Research stated that based upon
their experimental findings, children should only go to school 5 hours a day.
While there were tables and inferential statistics presented that illustrated a
comparison between 5 hour a day children and 7 hour a day children, there was
no discussion of how the study was conducted. This omission illustrates a flaw
in the
a. review of related literature.
b. interpretation of the findings.
c. design and procedure.
d. conclusions and recommendations.
• 16. Cindy conducted a path analysis for her thesis. She is most concerned about
addressing which of the following questions in her report?
a. Is a rationale given for selection of predictor variables?
b. Are the qualifications and special training of the interviewers described?
c. Was the level of inter-observer reliability obtained from at least two
independent raters?
d. Were groups randomly formed and the use of existing groups avoided?
371
17. The section where the application of the qualitative method chosen is described
in the report is the
a. introduction.
b. design and procedure.
c. conclusion and implications.
d. results.
• 18. In reviewing a manuscript relevant to your thesis you find that it is very similar to a study
conducted two years prior to the publication date. The authors of the two studies are
different and the sample populations are from different Universities, but both studies
address the transition of migrant students to large land grant Universities. You are
concerned that the authors of the second publication did not reference the first publication
in their work. This can be most directly attributed to a flaw in which section of the
second publication’s report.
a. Introduction
b. Method
c. Results
d. Discussion
19. The section of a report where the generalizations are critiqued is the
a. results
b. discussion.
c. method.
d. introduction.
• 20. Brandon is confused. He just read a study where the research questions
examined gender differences and the analysis used was Spearman’s rho. It
appears that there is a flaw in which section of the research report?
a. results
b. discussion
c. method
d. introduction
• 21. Nick, a qualitative researcher, is finalizing his research report. He needs to add a
piece about his purposive sampling technique. Where should Nick place this information
in his report?
a. results
b. instrument
c. participants
d. design and procedures
• 22. Won recently read the abstract of a research report of a study that investigated academic
procrastination. Of the following, which is information she should absolutely know after
reading the abstract?
a. Whether the study was qualitative or quantitative
b. The psychometric properties of the instrument used in the study.
c. The most relevant recent publication related to the study.
d. The demographic information and ethnicity of the participants in the study.
372
• 23. Sal is worried that he ahs not addressed validity enough in his research report and
adds a section on triangulation. Where should Sal add this information?
a. results
b. instruments
c. participants
d. design and procedures
24. Which is NOT a concern for inclusion in an abstract?
a. Are the procedures described?
b. Is the problem restated?
c. Is the design used identified?
d. Are they hypothesis clearly stated?
25. Which is a concern for inclusion in the method section of a quantitative research report?
a. The purposive sampling strategy used.
b. The previous existing related research.
c. Recommendations for future research.
d. The control procedures used in the study.
• 26. Marv needs to add a section indicating his ‘full cycle’ in the field to his research
report. What type of research did Marv likely conduct?
a. Narrative
b. Ethnographic
c. Experimental
d. Correlational
• 27. Hilary needs to include in her study a statement about response rate. What type
of research did Hilary likely conduct?
a. Questionnaire
b. Ethnography
c. Experimental
d. Correlational
• 28. Susan examined differences in students’ sustained independent reading time
between classrooms with and without a paraprofessional in the room and is
writing her report. She is likely concerned with which of the following questions
related to her causal-comparative study?
a. Is a rationale given for selection of predictor variables?
b. Were the sources of data related to the problem mostly primary?
c. Was the resulting prediction equation validated with another group?
d. Is the independent variable clearly defined or described?
• 29. Kristin is writing up her research and has provided rationale for why she
elected to use a within-subjects design. Most of the previous research had used a
between-subjects design. What type of research did Kristin likely conduct?
a. Experimental
b. Correlational
c. Narrative
d. Mixed methods
373
• 30. Mo conducted an interview study and is writing up her report. Which of the
following is of importance to Mo in her research report?
a. Is the criterion variable well defined?
b. Was an appropriate experimental design selected?
c. Is the method used to record responses described?
d. Were possible reactive arrangements controlled for?
• 31. Taryn has conducted a study on pre-kindergarten children’s displays of cultural
awareness. She observes three individual students within a class of fifteen preschoolers.
In the report of her study, it is unclear if the target of her investigation is the children or
the classroom. This is a flaw most directly related to the report of her
__________research.
a. Causal-comparative
b. Survey
c. Case-study
d. Experiemental
• 32. Amber recently read a research report that described a study that investigated the
benefits of a mnemonic strategy for special education learners. She would like to
replicate the study but there was little information about the mnemonic strategy
included. This lack of information illustrates a concern with the ____________
section of the report?
a. introduction
b. discussion
c. results
d. method
• 33.-34. Charles read a study that examined the effects of animation on learning science
text content. There were three groups in the study. Students were randomly
selected from a 10th grade science class and randomly assigned to a condition. One
group received a text only, the other group received a text and an animation, the
third group received only the animation. Five students were in each condition.
ANOVA indicated significant differences between groups and post hoc analysis
revealed the animation only group performed better than the other two
conditions.
• 33. The study Charles read is an example of a(n) _______________ study.
a. survey
b. correlational
c. experimental
d. prediction
• 34. One critique of the study is the number of students in each condition. This is
primarily a critique for which section of the research report?
a. participants
b. instruments
c. procedures
d. discussion
374
• 35. Vince recently critiqued a study that asked nonnative speaking parents about
their educational aspirations for their children. The study did not ascertain,
however, that the target population, the parents, could read the questionnaire, which was
written in English. This illustrates a flaw in the ____________ section of the
research report.
a. design
b. hypothesis
c. instruments
d. participants
• 36. Aaron conducted a study that examined teachers’ views about a new test mandated by the
State. He provides a questionnaire and then conducts follow up interviews. It is unclear
in his report, however, whether the data from the interviews or the questionnaire form the
basis for his conclusions. Of the following, this lack of clarity most directly demonstrates
a problem with Aaron’s ____________.
a. Introduction
b. Abstract
c. Methods
d. Discussion
• 37. In reviewing a study about the benefits of daily silent reading for third grade students’
comprehension skills, Sandra noted that the authors did not provide the amount of time
the students read daily or for how long the study was conducted. The authors should
have reported this information in the ________________ section of their report.
a. Introduction
b. Methods
c. Results
d. Discussion
• 38. Yuki read a study that compared pre- and post-test performance between two
different in-service training methods for teaching emergency first aid to teachers.
She really found the study interesting but there was no description of how the
two treatment groups were formed. This is a flaw in this _______________
research report.
a. survey
b. observational
c. experimental
d. correlational
• 39. Tim read a study that examined whether teacher-provided verbal signals helped
students retain information from lectures. Tim noticed that the researchers had
made sure that an equal number of both high and low ability learners were in
each condition. This is a strength in this ______________ researcher report.
a. observational
b. correlational
c. survey
d. causal-comparative
375
• 40. When Michelle got to the method section of the study she reviewed for a class
assignment she found that the researchers had clearly described the questionnaire
and the pilot procedures used to test the items. This is a strength of this
__________ research report.
a. observational
b. correlational
c. survey
d. causal-comparative
• 41. Donna, a reading specialist, compared reading motivation between two fifth
grade classes. For one class she provided a free selection of books of their
choice. For the other class she pre-selected the books for the students. Her
hypothesis was that students given more choice in the library would have higher
motivation to read. Her hypothesis was supported. Lisa, when reading her report,
wondered if there were pre-existing differences between the classes that were not
considered or if there were extraneous variables not addressed. These oversights
in Donna’s work are primarily a problem with her
a. participants.
b. hypothesis.
c. design and procedure.
d. Instruments.
• 42. Fred reported on his qualitative study that investigated perceptions of
higher-education by high-achieving but low SES students. In his report he does
not provide any information about the context of the study. This is primarily a
concern regarding his
a. instruments.
b. hypothesis.
c. analysis.
d. design and procedure.
43. In several studies in a recent edition of a peer-reviewed journal, there is no
reference in the articles about the ethnicity or race of the samples. There is a flaw
in the _______________ portion of the articles.
a. instruments
b. participants
c. analysis
d. design and procedures
• 44. Diane read a research report of a study that used a translated version of a measure
used in French Canadian schools in an English-speaking population in Nebraska.
There was little information about the approach the researchers took to
translation and no mention of a pilot test. While this might be a flaw in the
instrumentation of the study it is also a flaw in the
a. hypothesis
b. analysis.
c. design and procedure.
d. problem statement.
376
• 45. Monique read a study that included the use of eye-movement technology to
collect data from young learning disabled readers. She would really like to
replicate the study with an older population but there was no information about
the technology used in the study. This is considered a flaw in the
a. participants.
b. design and procedure.
c. analysis.
d. discussion.
• 46. Samantha was disappointed when she finished reading a survey study that
assessed principals’ hiring practices. The authors had drawn conclusions but had
not indicated any frequency distributions for items so Samantha was unsure how
the conclusions were supported. This is primarily a problem with the
______________ section of the research report.
a. problem statement
b. hypothesis
c. method
d. results
• 47. Marjorie conducted a study of students’ views of race relations in America. She
did not, however, share information about her participants. Marjorie has demonstrated
a. a threat to internal validity.
b. a threat to external validity.
c. a threat to statistical conclusion validity.
d. a threat to reliability.
48. The question of whether data are time constrained is most relevant to which of the
following types of research?
a. Experimental
b. Causal-comparative
c. Mixed methods
d. Single subject
• 49. Quidera conducted a study in her classroom. She has been frustrated by the lack
of time management skills of her graduating seniors. She implemented a required
assignment notebook and planner program for two months and observed the effects on
students’ homework completion. Quidera’s study can best be described as
a. action research.
b. narrative research.
c. experimental research.
d. ethnographical research.
377
• 50. Tobin conducted a study in which he randomly assigned participants to two
treatment conditions to test if there were differences by treatment on the number of
classroom referrals. He then interviewed selected students regarding their experiences
with the two treatments. In the results section of his study it is unclear at times from
which data he has drawn his conclusions. This flaw is most directly related to reporting
of ____________ research study.
a. Experimental
b. Mixed methods
c. Narrative
d. Corrleational
• 51. Chyleen owns a child care center for young children. She conducted a study of
parents’ perceptions of the need for structure in early learning environments. She was
careful to report how her findings would benefit her practice as an early childhood
learning professional. Chyleen’s research would best be described as
a. ethnographic.
b. experimental.
c. action.
d. correlational.
• 52. Ntozake conducted a study in which she observed bullying in a middle school
physical education environment. She conducted the study over a three-week time period.
This may not have been long enough to combat observer effects. This illustrates which
of the following concerns with conclusions from Ntozake’s study?
a. An instrumentation threat
b. An internal validity threat
c. A history threat
d. An external validity threat
53. Of the following, which is a concern for correlational studies in particular?
a. Is the method used to record responses described?
b. Is the rationale for variable selection described?
c. Is a rationale for the design selection given?
d. Is the method for group formation described?
• 54. Tom conducted a study that examined gender differences in ability to learn
physics concepts from computer games. In his study he found men learned better
from games than the women. He hadn’t, however, controlled for amount of game
playing or interest in physics. These lack of controls are a flaw in Tom’s
a. survey study.
b. causal-comparative study.
c. correlational study.
d. experimental study.
• 55. When reviewing a research report that examined on-task behavior of ADHD
children in various settings, Sue was confused because the data did not support
the recommendations for practice provided by the researchers. This
overgeneralization is a flaw in the ______________ in the report.
a. statement of hypothesis
378
b. conclusions and recommendations
c. design and procedures
d. review of related literature
• 56. Jill reported in her research that there were significant differences between males
and females on several items of a personality inventory. She conducted many t-
tests on a relatively small sample, without correcting for error rate problems. This
best illustrates a flaw in Jill’s
a. instruments.
b. design and procedures.
c. results.
d. conclusions and recommendations.
• 57. Willa conducted a study on the effects of violent television viewing on choice of
playground activity. Her data indicated no support for her research hypothesis
that those children that watched violent television played more aggressively on
the playground. Interestingly, however, although not related to her hypothesis,
she found that children who watched violent television played on the playground
for a shorter amount of time when compared to children who did not watch
violent television. How should Willa address this finding?
a. Change her hypothesis statements to include a hypothesis about duration of
playground activity.
b. Add the analysis to the results section and discuss the finding in the
conclusions and recommendations section.
c. Rewrite the research problem to include duration of play as a variable of
interest in the study.
d. Address the finding in the conclusions and recommendations section and
suggest future work that addresses duration of play as a variable.
• 58. Ron is reviewing a qualitative study. Which of the following should be of
concern to his review?
a. Is the deductive logic applied to the data to produce results stated in detail?
b. Is the rationale for variable selection described?
c. Were any control procedures used to equate the groups on extraneous variables?
d. Are sources of invalidity associated with the design identified and discussed?
• 59. Maria has turned in her honor’s thesis that considers the prevalence of ability
grouping in local intermediate grades. One of her readers questions the prompts that
might have been provided to the participants and how the participants’ responses were
coded. Given this information, what type of study did Maria conduct for her thesis?
a. Causal-Comparative
b. Ethnographic
c. Mixed Methods
d. Experimental
379
• 60. Jackie wants to examine teachers’ efficacy for her dissertation project. She has
consulted with her committee and they have suggested that Jackie must make it clear how
her qualitative and quantitative data relate. Which of the following types of research
study does Jackie propose?
a. Ethnographic
b. Mixed methods
c. Experimental
d. Correlational
• 61. Yaseen’s research has addressed middle school students’ body image. He is sure
to include information about how he selected the scale that he used. One of the
drawbacks of his study is that the scale had previously only been used with older
populations. Yaseen should include this information in which of the following sections
of his research report?
a. Literature review
b. Instruments
c. Results
d. Participants
62. A guiding hypothesis is likely to be found in which of the following type of research
study?
a. Qualitative
b. Causal-comparative
c. Correlational
d. Experimental
• 63. Ar–Mu’min has conducted a study that examines teachers’ understandings of
child development. He is certain to present how he gained entry into the research site and
how he coded his data. Which of the following is the most likely type of research that
Ar-Mu’min has conducted?
a. Ethnographic
b. Experimental
c. Correlational
d. Causal-comparative
• 64.-65. Eliza takes care to illustrate in her research report how long she was in the field
and how she has minimized observer effects.
• 64. Which of the following types of research did Eliza most likely conduct?
a. Experimental
b. Action
c. Correlational
d. Ethnography
• 65. In her efforts to address these concerns, Eliza is most directly addressing
a. threats to internal validity.
b. threats to external validity.
c. ethics concerns
d. reliability
380
• 66. Ai–mu conducted a study on recent immigrants’ opinions of schools in America.
She is careful to report her training and abilities to conduct the research in her
manuscript. By so doing, Ai-mu is specifically addressing
a. threats to internal validity.
b. threats to external validity.
c. reliability.
d. ethics concerns
• 67. Damik recently finished his dissertation on metacognition. At his defense one of
his committee members suggested that he had not clearly defined the relationship
between self-regulation and metacognition. This concern with Damik’s work is most
directly related to the __________ of the study.
a. external validity
b. internal validity
c. reliability
d. ethical standards
• 68. LuAnn, a child development researcher, recently conducted a study in a hospital
nursery to examine the feeding habits of newborn babies. She did not tell parents her
identity or the purpose of her work when she asked them questions. This concern with
LuAnn’s study is most directly related to a concern with
a. external validity.
b. internal validity.
c. reliability.
d. ethical concerns.
• 69. Jen recently conducted an observational study of athletes’ motivation for a class she took
at the University. Unknown to her at the beginning of the study, one of the participants
in her study was the professor’s daughter. Jen will need to report this ethical concern in
which section of her qualitative research report?
a. Results
b. Discussion
c. Method
d. Abstract
70. Stephen recently read a manuscript on students’ anxiety in classrooms. There are several
citations in the manuscript that are not discussed or tied to the research question. This is
a concern with which section of the manuscript Stephen read?
a. Introduction
b. Methodology
c. Results
d. Discussion
381
Chapter 22 Answers