2010:112
MASTER’S THESIS
Behavioral Detection of Cheating
in Online Examination
Matus Korman
Luleå University of Technology
D Master thesis
Computer and Systems Sciences
Department of Business Administration and Social Sciences
Division of Information Systems Sciences
2010:112 – ISSN: 1402-1552 – ISRN: LTU-DUPP–10/112–SE
Acknowledgements
I would like to thank everyone, who contributed in, opposed to, assisted with, or
otherwise helped me carrying out the study as well as writing this thesis – a result
of the study.
My thanks go to Dan Harnesk, PhD. (supervisor), S¨oren Samuelsson, PhD., and
John Lindstr¨om, PhD., for the valuable advice and research guidance I was given;
to Hugo Quisbert, PhD., Artjom Vassiljev and Viola Veiderpass for constructive op-
position; to Lars Furberg for the ideas, which helped me to navigate to the research
problem chosen and the interesting discussions we had; to Neil Costigan, PhD., for
his inspiring work and presentations; to professor Ann H¨agerfors for managing is-
sues also related to my study; and to my family for their mental support and advice.
My further thanks go to Amir Molavi, Onur Yirmibesoglu, Marko Niemimaa, Elina
Laaksonen, Nebojsa Mihajlovski, Vladimir Kichatov, Ali Fakhr, Darya Plankina,
Anna Selischeva, Sana Rouis, Svante Edz´en, Peter Anttu, and others, who con-
tributed to my thoughtflow through discussions, or supported me in different other
ways.
Special thanks go to Behaviometrics AB and the people, efforts of whom relate
to the study.
Also thanks to the contributions of all of you, the study has been done the way
it has, and I feel having learned valuable knowledge and gained practice, for which
there is use in the future.
Abstract
This thesis relates to studying possibilities of detecting online examination cheating
through the measures of human-computer interaction dynamics.
The need for and use of online or computer-based examination seems to be growing,
while this form of examination gives students a broader spectrum of opportunities
including those for cheating, as compared to non-computerized ways of examination.
The times are changing, there are many different reasons for examination dishonesty,
many ways of performing it, and many ways of coping with it. Given an equilib-
rium at this level, new ways of violation deserve new ways of prevention, or at least
detection.
The study focuses on a method of computer-based examination cheating detec-
tion based on measures of behavior and machine learning, and tries to link it to
a broadly taken concept of academic dishonesty. The detection potential of this
method is mainly indicated by cue leakage theory, subjects of which can be han-
dled with use of pattern recognition and anomaly detection theory, all through a
behavioral biometrics approach.
Contents
1 Introduction 1
1.1 Topic……………………………… 3
1.2 Research goals and delimitation . . . . . . . . . . . . . . . . . . . . . 3
1.3 Significance of the study . . . . . . . . . . . . . . . . . . . . . . . . . 4
1.4 Documentstructure ……………………… 5
2 Background 7
2.1 Examination cheating . . . . . . . . . . . . . . . . . . . . . . . . . . 7
2.1.1 What’s wrong with cheating? . . . . . . . . . . . . . . . . . . 8
2.1.2 Why do students cheat? . . . . . . . . . . . . . . . . . . . . . 9
2.1.3 The mission: preventing cheating . . . . . . . . . . . . . . . . 16
2.1.4 How do students cheat? . . . . . . . . . . . . . . . . . . . . . 19
2.1.5 Detecting cheating as a means of prevention . . . . . . . . . . 21
2.1.6 Cheating review summary . . . . . . . . . . . . . . . . . . . . 22
2.2 Specifics of distance operation . . . . . . . . . . . . . . . . . . . . . . 26
3 Conceptual framework 29
3.1 Cueleakagetheory………………………. 29
3.2 Pattern recognition theory . . . . . . . . . . . . . . . . . . . . . . . . 30
3.3 Anomalydetection………………………. 31
3.4 Behaviometrics ………………………… 32
3.4.1 Biometrics in general . . . . . . . . . . . . . . . . . . . . . . . 33
3.4.2 Specifics of behaviometrics . . . . . . . . . . . . . . . . . . . 38
3.4.3 Keystroke dynamics . . . . . . . . . . . . . . . . . . . . . . . 41
3.4.4 Mousedynamics ……………………. 43
3.4.5 Linguistic dynamics . . . . . . . . . . . . . . . . . . . . . . . 44
3.4.6 ‘Special purpose’ behaviometrics . . . . . . . . . . . . . . . . 44
3.5 Vision of a behavioral cheating detection approach . . . . . . . . . . 48
3.5.1 The angle of attack . . . . . . . . . . . . . . . . . . . . . . . . 49
3.5.2 Behavioral characteristics as the cheating detection unifier . . 50
3.5.3 The detection mechanism . . . . . . . . . . . . . . . . . . . . 50
4 Methodology 53
4.1 My setting and the research method . . . . . . . . . . . . . . . . . . 53
4.2 Validity of a research design . . . . . . . . . . . . . . . . . . . . . . . 55
4.3 Reliability and validity of a measure . . . . . . . . . . . . . . . . . . 56
4.4 Research design and research process . . . . . . . . . . . . . . . . . . 57
4.4.1 Empirical inputs . . . . . . . . . . . . . . . . . . . . . . . . . 58
4.4.2 Observations ……………………… 59
4.4.3 Questionnaire……………………… 62
4.4.4 Analysis………………………… 63
5 Analysis and observations 65
5.1 Analysis……………………………. 65
5.1.1 Quantitative molecular level . . . . . . . . . . . . . . . . . . . 65
5.1.2 Qualitative molecular level . . . . . . . . . . . . . . . . . . . 66
5.1.3 Qualitative molar level . . . . . . . . . . . . . . . . . . . . . . 66
5.2 Observations …………………………. 67
5.3 Observation1…………………………. 67
5.3.1 General highlights . . . . . . . . . . . . . . . . . . . . . . . . 67
5.3.2 Session-specific highlights . . . . . . . . . . . . . . . . . . . . 68
5.4 Observation2…………………………. 70
5.4.1 General highlights . . . . . . . . . . . . . . . . . . . . . . . . 70
5.4.2 Session-specific highlights . . . . . . . . . . . . . . . . . . . . 70
5.5 Observation3…………………………. 73
5.5.1 General highlights . . . . . . . . . . . . . . . . . . . . . . . . 73
5.5.2 Session-specific highlights . . . . . . . . . . . . . . . . . . . . 73
5.6 Triangulative analysis remarks . . . . . . . . . . . . . . . . . . . . . 75
6 Results and findings 77
6.1 Behavioral anomaly indication . . . . . . . . . . . . . . . . . . . . . 77
6.2 Indicatingcheating………………………. 77
6.3 Indication difficulties . . . . . . . . . . . . . . . . . . . . . . . . . . . 77
7 Conclusion and discussions 79
7.1 Conclusion ………………………….. 79
7.2 Cheating detection and prevention approach discussion . . . . . . . . 80
7.2.1 Behaviometric aspects . . . . . . . . . . . . . . . . . . . . . . 80
7.2.2 Cheatingaspects……………………. 82
7.2.3 Psychological aspects . . . . . . . . . . . . . . . . . . . . . . 83
7.3 Research approach discussion . . . . . . . . . . . . . . . . . . . . . . 84
7.4 Outlooks for further research . . . . . . . . . . . . . . . . . . . . . . 84
Appendices 97
A Subjects of automated observation 99
A.1 Basic structure of the analytics . . . . . . . . . . . . . . . . . . . . . 99
A.2 Keystroke dynamics features . . . . . . . . . . . . . . . . . . . . . . 100
A.3 Mouse dynamics features . . . . . . . . . . . . . . . . . . . . . . . . 100
A.4 Silence dynamics features . . . . . . . . . . . . . . . . . . . . . . . . 100
A.5 Linguistic dynamics features . . . . . . . . . . . . . . . . . . . . . . . 101
B Subjects of manual observation 105
C Questionnaire and observation task content 107
C.1 Questionnaire…………………………. 107
C.2 Authentic writing and formulating . . . . . . . . . . . . . . . . . . . 108
C.3 Verbatim copying by reading . . . . . . . . . . . . . . . . . . . . . . 109
C.4 Verbatim copying by listening . . . . . . . . . . . . . . . . . . . . . . 109
C.5 Copying by reading and reformulating . . . . . . . . . . . . . . . . . 110
List of Figures
2.1 A cheating-extended model of Ajzen’s theory of planned behavior . . 10
2.2 Model of student cheating decision based on internal (personal) and
externalfactors………………………… 11
2.3 Model of cheating causation . . . . . . . . . . . . . . . . . . . . . . . 15
2.4 Graphical overview of cheating and counter-cheating relations . . . . 24
2.5 Overview of a cheating and counter-cheating process . . . . . . . . . 25
3.1 A classification example . . . . . . . . . . . . . . . . . . . . . . . . . 30
3.2 Biometric system error rates . . . . . . . . . . . . . . . . . . . . . . . 36
3.3 A typical architecture of a biometric system . . . . . . . . . . . . . . 36
3.4 Fusion of biometric systems . . . . . . . . . . . . . . . . . . . . . . . 37
3.5 The biometric menagerie . . . . . . . . . . . . . . . . . . . . . . . . . 39
3.6 An example process of mouse dynamics analysis . . . . . . . . . . . 43
3.7 Deterrence mechanism of cheating detection . . . . . . . . . . . . . . 49
3.8 Model of the cheating detection approach . . . . . . . . . . . . . . . 52
4.1 Research process overview . . . . . . . . . . . . . . . . . . . . . . . . 58
4.2 The observation design used in the study . . . . . . . . . . . . . . . 60
4.3 The observation process (including questionnaire) . . . . . . . . . . . 62
4.4 Data flow and control relations of the data gathering and analysis
processes …………………………… 63
7.1 The cheating prevention approach . . . . . . . . . . . . . . . . . . . 82
A.1 Analyticsstructure………………………. 99
A.2 Context and process of the automated analysis part . . . . . . . . . 100
C.1 Example free diagram . . . . . . . . . . . . . . . . . . . . . . . . . . 109
C.2 Diagram to copy (redraw) . . . . . . . . . . . . . . . . . . . . . . . . 110
List of Tables
2.1 Factors correlated to plagiarism behavior 1 . . . . . . . . . . . . . . 13
2.2 Factors correlated to plagiarism behavior 2 . . . . . . . . . . . . . . 14
iv
3.1 Meta-functions of a computer mediated communication text analysis
framework…………………………… 45
3.2 Text analysis linguistic features 1 . . . . . . . . . . . . . . . . . . . . 45
3.3 Text analysis linguistic features 2 . . . . . . . . . . . . . . . . . . . . 46
4.1 Levels of the predictor variable (PV) . . . . . . . . . . . . . . . . . . 61
7.1 Biometric properties of the approach . . . . . . . . . . . . . . . . . . 81
7.2 Discussion of measure validity . . . . . . . . . . . . . . . . . . . . . . 85
A.1 Explanation of terms used in the description of features . . . . . . . 101
A.2 Keystroke dynamics features . . . . . . . . . . . . . . . . . . . . . . 102
A.3 Mouse dynamics features . . . . . . . . . . . . . . . . . . . . . . . . 103
A.4 Silence dynamics features . . . . . . . . . . . . . . . . . . . . . . . . 104
A.5 Linguistic dynamics features . . . . . . . . . . . . . . . . . . . . . . . 104
Chapter 1
Introduction
Cheating in online examination is an educational problem similarly as it is in conven-
tional examination. Because of its lower detectability, however, universal reputation
of distance degrees suffers. This study strives to explore and verify possibilities of
detecting specific types of cheating based on behavioral measures of computer inter-
action taken during an online examination. Detecting cheating is seen as a way to
preventing it (lowering its extent).
Distance education became an educational field in 1970s and since then it is
gaining popularity in diverse parts of the world (Keegan,1996;Allen & Seaman,
2005,2007;Howell et al.,2003). According to Allen & Seaman (2008), nearly 22%
of all higher education enrollments in the United States in year 2007 were online
enrollments. The number was around 4 million and there is still a growing ten-
dency. Moreover, online education is dominantly perceived as critical to long-term
institutional strategy by educational institutions at least across the United States
(Allen & Seaman,2008). Based on different trends and factors, the interest for
distance education is increasing and expected to increase further (Hawkridge,1995;
Irele,2005;Allen & Seaman,2003). The trends have varied characters, among other
motivational (Parker,2003;Maguire,2005;Allen & Seaman,2008), social, political
and technological (Bates,1995;Howell et al.,2003). Relatively high future growth
of distance education is expected in developing countries (Koul,1995).
Distance education is a form of education, in which teachers and class audi-
ence are separated by physical distance and/or by time (Moore & Kearsley,1996,
chap. 1) as compared to conventional (on-site) education, which is based on face-
to-face meetings and time-synchronous physical presence of students, required by
the technology predominantly employed (Keegan,1996, chap. 1,2). Following Kee-
gan (1996), distance education and conventional education differ at least in physical
centralization and time-synchronization (accessibility), economics, market, and also
didactics (Reushle & McDonald,2004;Reushle et al.,1999), administration and
evaluation. Nowadays, not only different universities around the world offer courses
and programs for distance studies, there are whole universities often called ‘open
universities’, which are built on the concept of, and provide solely distance educa-
tion.
As to the process of education, the concept of physical decentralization and
time-asynchronization is also applicable to the process of assessment including ex-
amination (Mason,1995), since the two are often employed at the same time, or in
1
a mutually successive manner. Distance examination is in different forms used to
validate the level of knowledge, skills or abilities of students/examinees. The most
common distance examination method seems to be online examination, which uses
a network-enabled computer environment (e.g., the Internet) to set up a two-way
communication.
Although dependent on specific environment, while major concerns in distance
education compared to on-site education are mostly related to finding, achieving and
maintaining effective means of teaching/tutoring, learning, student support and ad-
ministration (Holmberg,1995;Keegan,1996;Bates,2005;Kim et al.,2008), the
problems of fairness assurance and trust seem to be often more challenging in on-
line examination compared to traditional/conventional examination means (Rowe,
2004). Public trust and fairness in education including examination is an important
attribute (Rumyantseva,2005;Heyneman,2002), yet seemingly tricky to achieve and
maintain (Herberling,2002). The technology, which on one hand enables distributed
and asynchronous education, opens up a broad range of cheating possibilities within
an examination process on the other hand. Controlling or at least perceiving largely
unknown and distant examination environments as a way to detect and prevent ex-
amination dishonesty seems to be non-trivial. Also as a matter of this fact, distance
education often renders less accepted than conventional (on-site) education (Colum-
baro & Monaghan,2009;Bourne et al.,2005). In a more general context, Allen &
Seaman (2003) shows that online education is perceived inferior to conventional ed-
ucation, however, near future beliefs (for three years later) show an optimistic turn
in the balance. Around six years later, Columbaro & Monaghan (2009) show that
such beliefs have been and might tend to be too optimistic, since more than 95%
of employers would prefer to accept a traditional degree to an online one in several
different fields according to their study.
Examination cheating and academic dishonesty in general seem to have been an
educational problem since a long time ago (Cizek,1999). According to UC Berke-
ley (2009), cheating can be defined as “fraud, deceit, or dishonesty in an academic
assignment, or using or attempting to use materials, or assisting others in using
materials, that are prohibited or inappropriate in the context of the academic as-
signment in question” (no page numbering). Students often tend to shortcut achiev-
ing their grades and maintaining their sense of personal integrity otherwise than
through investing adequate amount of effort and time (Diekhoff et al.,1996). Aca-
demic cheating is prevalent and at the same time, it seems to have growing tendency
(Cizek,1999;Dick et al.,2003;McCabe et al.,2006;Wehman,2009;Howell et al.,
2009). A study in McCabe et al. (2006) shows that cheating was reported by 56%
business students and 47% non-business students. An earlier McCabe’s study (also
mentioned in the paper) shows that 66% of all students reported at least one serious
cheating incident in the past year, while among engineering students the number
was 72% and business students led with 84%. According to a survey carried out in
the United States, around 94% students reported cheating in any form, around 65%
students reported test cheating and more than 50% students reported plagiarism.
According to Stumber-McEwen et al. (2009), there is a wealth of studies on preva-
lence of cheating available, however, their quantitative results vary greatly based
on the type of survey and specific survey conditions. As to on-site examination,
cheating also applies to distance examination (Underwood,2006;Wehman,2009).
2
Different sources perceive the cheating prevalence among on-site and online exam-
ined students differently (Stumber-McEwen et al.,2009;Herberling,2002;Watson
& Sottile,2010). Assuming that an online examination environment tends to be less
cheat-constraining and less perceivable by examiners than an on-site one, students
may generally tend to cheat more from distance as also believed by Rowe (2004).
Following an information security approach (Whitman & Mattord,2008), the
occurrence of online examination cheating as an undesirable activity is a form of
risk, and the higher the cheating severity and probability, the greater the risk con-
trol importance. The ultimate goal of risk control, in this context applied to the
educational field, is to effectively reduce risk related to the educational process.
Effectively reducing risk of online examination cheating is a problem.
There are multiple approaches to controlling online examination cheating (Olt,
2002), many of them suitable in one way or another. The primary approach usable
with the thesis concerns is the ‘police approach’ – monitoring for and reacting on
suspicion or detection, along with deterrence-based cheating demotivation. This ap-
proach is somewhat analogous to a feedback control system (˚
Astr¨om & Murray,2008,
chap. 1) and within such one needs to first perceive the examination environment
and detect anomalies in order to be able to make effective control actions. Perceiv-
ing a distant online examination environment and effectively detecting cheating is a
problem.
1.1 Topic
The topic of this thesis is to explore and verify possibilities of detecting specific types
of online examination cheating based on behavioral measures of human-computer
interaction. More specifically, the focus lies on utilizing behaviometrics (behavioral
biometrics) for the analysis of keystroke, mouse and linguistic dynamics.
The primary motivation for this study is to enable or help faculties to both
(1) fight the prevalent and rather invisible online examination cheating, and to (2)
indirectly increase the acceptance of online grades.
By content, this study is focusing on the use of behaviometrics (work with
keystroke, mouse and linguistic dynamics) based on information technology and
machine learning (software, pattern recognition, anomaly detection, visualization)
for detecting examination cheating (an educational concern).
1.2 Research goals and delimitation
The goal of the research endeavor is not to enable one to exactly tell whether a
student cheats or not. According to the character of a probabilistic analysis process
and the variety of input data to it in context and time frame of this thesis, such a goal
would render extremely difficult to achieve to me. Instead, I consider the following
information to be both useful and realistic to indicate based on the measures of
human-computer interaction (keystroke and mouse events with their timings, and
linguistic features from keystrokes):
1. Histogrammatically displayed extent of behavioral anomaly compared to a be-
havioral baseline
3
2. Histogrammatically displayed amount of stress
3. Histogrammatically displayed probability of cheating together with the type of
possible cheating activity (e.g. copying by reading, listening, etc.) for each
suspicious segment of behavior during the examination session time
Being able to effectively and in a highly automated way provide the above about the
target population (described below) is the research vision (in a longer term). The
goal of this study, however, is to approach this vision with focus on the first and the
third point.
The target population to which the research goal relates are distance students, a
great part of whose might be employed adults (Paulsen & Rekkedal,2001), mostly
aged between 25 to 40 years. The rest of the target group might be graduate students
aged mostly between 20 and 30 years. The age ranges used are assumptive and they
constitute a part of the study’s delimitation.
The following are research questions, answering which I expect to contribute to
achieving the research goal:
RQ-1 What are the behavioral signs of tasks carried out when cheating that
manifest themselves on keystroke, mouse and linguistic dynamics of the
user’s computer interaction during a computer-based examination?
RQ-2 How distinct is normal behavior from a cheating behavior and how dis-
tinct are different types of cheating behavior from each other?
