» E-mentor nr 4 (26) / 2008

This study looks at two closely connected issues in online learning - cheating in non-proctored but tightly timed testing environments and formative assessment strategies designed to alleviate stress in taking timed exams. Several courses in Biology - Histology and Human Biology, serve as models for analysis of these issues. The results show similar grade distributions for exams and final course grades in proctored and non-proctored courses and that formative assessment has a positive effect on student learning as measured in summative assessments. It was concluded that similar online, non-proctored formative assessment strategies could also be beneficial to students taking blended, hybrid or on-ground courses in disciplines other than Biology.

Cheating under non-proctored online assessment conditions is problematic since most investigations of cheating in the United States, even in proctored conditions, indicate that 2/3 or more of students have admitted to cheating in high school¹ and university classes². The idea of letting students "cheat" by testing open book³ but under tightly timed testing conditions was employed as a way of "fighting fire with fire" to decrease the possibility and reason to cheat by students being tested online without a proctor. Student test anxiety and stress, however, rise when taking timed exams and to help ameliorate stress, timed practice exams can be employed which also served as a type of formative assessment (assessment for learning)⁴. The number of times that students take the practice tests can be viewed as an estimate of "time on task" which can then be compared to student success on summative assessments (assessment of learning)⁵.

It is important to understand how timing of online exams can be utilized as a reliable way to combat cheating in non-proctored exams while at the same time being fair to students taking the open book timed exams. Keeping track of grade distributions for exams and final course grades will be followed in this analysis to ascertain if the time chosen for exams is appropriate. Comparison of non-proctored lecture exams and final course grade distributions with those from proctored courses also provides some evidence of the success of timed exams as a strategy to combat cheating during non-proctored open book testing.

Methods

This study will focus on two online courses offered during the 2007 - 2008 academic year in the Department of Biological Sciences at The University of Texas at El Paso. Histology is a Junior-Senior-Graduate level course which attracts many pre-medical and pre-dental students. The second course is Human Biology and is a lower division course designed for non-Biology majors to satisfy their general education science requirement. Both online courses were offered through The University of Texas Telecampus⁶ which is the central support unit for online education for the University of Texas System. The Histology course was chosen as one of the courses included as a sample course⁷ on the University of Texas Telecampus web site.

Both courses were offered using the Blackboard course management system (CMS). In this analysis, statistics were obtained describing the number of times students took lecture chapter practice exams (quizlets) and time taken to complete exams which were then linked to student grades. For Histology, students taking lecture exams 1 and 2 (100 question exams) were given 45 minutes and a 30 seconds grace period to complete the exam averaging to 27 seconds per T/F or multiple choice question. For the Human Biology lecture course, a time of 55 minutes followed by a grace period of 30 seconds was chosen for all the 100 question lecture exams which is 33 seconds per T/F or multiple choice question. A longer time per question was utilized in the Human Biology course because it was assumed that students entered the non-majors Human Biology course with less pre-knowledge about the subject matter than did the student majors taking the Histology course. Students in both courses were penalized one point for each 30 seconds increment past the exam deadline and 30 seconds grace period.

The times chosen for the Histology and Human Biology lecture exams were analyzed by comparing the actual time each student in each course took to complete the first lecture exam with the grade they received in the exam. Students who took more time than the 45 or 55 minutes plus 30 seconds grace period are indicated in Figures 1 and 2 below. Note that in the Histology course, a student would begin to lose points if they took longer than 2730 seconds (45 minutes plus 30 seconds grace period) and for the Human Biology course, students would lose points if they took longer than 3330 seconds (55 minutes plus 30 seconds grace period).

Figure 1. Student test times were graphed from low to high time for each grade category. No "A" student lost any points, one "B", "C" and "D" student went over the time limit and eight "F" students went over the time limit of 2730 seconds. 30.6% of students went over the time limit of 2730 seconds




Figure 2. Student test times were graphed from low to high time for each grade category. No "A" student lost any points, one "B" student went over time limit, no "C" student lost points, three "D" students went over the time limit and four "F" students went over the time limit of 3330 seconds. 27.6% of students went over the time limit of 3330 seconds

As can be seen from the data in Figures 1 and 2, the 45 minutes plus 30 seconds grace period for Histology and the 55 minutes plus 30 seconds grace period for the Human Biology course produced acceptable grade distributions and percentage of students who took the exam within the time periods and grace period provided. It is not surprising to find that some students complain that they did not have enough time in completing these exams. But, when shown that a majority of students in each course were able to take the exam and get passing grades in the time assigned for each exam, their initial criticisms were met so that now discussion over their study habits could proceed with suggestions made as to how to complete timed exams within the assigned time limits. Some students likely underestimate how much they must study for open book exams⁸ and the suggestion of a formative assessment strategy which encourages time-on-task becomes a valuable advising tool to assist students who need to increase their exam taking speed.

Formative Assessment Strategy

Formative assessment can take the form of practice exams⁹ which when used in two dental education courses showed that students who took the practice exams performed better on the summative exams than did students who didn't take the formative assessments. In this study, the chapter practice exams were called "quizlets" and the questions were drawn from pools of T/F and multiple choice questions for each chapter. Each quizlet consisted of 12 questions to be taken in 5 minutes amounting to 25 seconds per question.

The important point about using the quizlets as a type of formative assessment is to not count them for credit. If quizlets count for credit, students will only take the quizlets until they get the credit they desire. At that point, quizlets stop being formative assessments and become summative assessments defeating the purpose of the formative assessment strategy for learning.

Results

Grade distribution: One approach to gathering evidence that a non-proctored open book but tightly timed exam is similar in outcome to exams offered in proctored non-open book conditions is to compare grade distributions. Grade distributions can also provide information about cheating in that if similar grade distributions are observed for traditional exams and open book exams, then it can be conjectured that if cheating is occurring, it is happening at the same rate in traditional and open book testing conditions. Figure 3 below compares grade distributions for Lecture Exam 1 in a Histology course offered during the fall semester of 2007 with grade distributions of Lecture Exam 1 exams from Histology courses offered during the spring, 2000 semester and also during the summer one-month semester of 2005. The Fall 2007 semester grade distribution was for a non-proctored, open book but tightly timed exam offering 100 T/F and multiple choice questions at 27 seconds per question. The other two Histology courses offered in 2000 and 2005 used proctored exams and essay questions written in blue books.

Figure 3. Spring, 2000 - N = 18, Summer 2005 - N = 19,
Fall, 2007 - N = 38

Although differences exist amongst the three grade distributions, acceptable levels of grade comparisons were observed for the three classes for Lecture Exam 1 despite the Fall 2007 exam offered using the non-proctored, open book, tightly timed testing conditions.
Final course grade distributions for a number of courses taught prior to the course using non-proctored, open book tightly timed testing conditions can be seen in Figure 4 for Histology and Figure 5 for the Human Biology course.













Figures 4 and 5 show close correlation of final grade distribution for both the Histology and Human Biology courses offered under non-proctored or proctored conditions. The idea of "fighting fire with fire" which is based on letting students "cheat" by permitting open book exams appears to work well as indicated by the close similarity of grade distributions in Figures 4 and 5. The tightly timed approach also results in an acceptable grade distribution between the non-proctored, open book courses and the courses taught in previous years in which exams were proctored and taken without the help of open book.

A formative assessment strategy was introduced consisting of practice exams termed "quizlets" to give students an opportunity to see how well they knew the lecture material and also to provide them the chance to practice taking tightly timed exams. Figure 6 shows how closely this formative assessment strategy relates to the grade received in the summative assessment which in this case is Lecture Exam 1 in both the Histology course and the Human Biology class.






Figure 6. N refers to the number of students in a particular grade category and the Quizlet mean is the average number of quizlets taken by students in a particular grade category. Remember also that both of these exams were taken under non-proctored, open book but tightly timed conditions. The mean value of 20.8 +/- 15.4 quizlets taken by the "A" students for the Histology course Lecture Exam 1 was found to be significantly different from the mean value of 4.2 +/- 1.55 for the students in the "F" category with a p value of 0.001. The Quizlet mean for Human Biology "A" category students compared to the Quizlet mean for "F" students comes close to the accepted significance level with p = 0.065.

The number of quizlets taken compared to grade received for Lecture Exam 2 and the Final Comprehensive Exam for the Histology course can be seen in Figure 7.






Figure 7. Final Exam was a comprehensive exam covering all lecture material for the course and consisted of 150 T/F and multiple choice questions. Students had 65 minutes and 30 seconds grace period to complete the final exam without penalty point deduction. Note that Quizlet mean values with asterisks indicate that only one student was in that grade category. The mean value of 32.08 +/- 20.2 quizlets taken by the "A" students for Lecture Exam 2 was found to be significantly different from the mean value of 4.29 +/- 5.15 for the students in the "F" category with a p value of 0.0012. The mean value of 40.7 +/- 43.3 Quizlets taken by the "A" students for the comprehensive final exam was found to be significantly different from the mean value of 7.4 +/- 13.2 for the students in the "F" category with a p value of 0.029.

The number of quizlets taken compared to grade received for Lecture Exams 2, 3 and 4 for the Human Biology course can be seen in Figure 8.






Figure 8. All lecture exams were composed of 100 questions of the T/F and multiple choice type. Students had 55 minutes and a 30 seconds grace period to complete each exam before penalty points were applied. For Lecture Exam 2, there was a significant difference between the Quizlet mean for "A" students compared to "D" students with p = 0.0201. For Lecture Exam 4, there was a significant difference between the Quizlet mean for "A" students compared to the combined Quizlet number for "D" plus "F" students with p = 0.0189.

Discussion

Does the "fighting fire with fire" strategy for combating certain types of cheating really work? Evidence in this study obtained by comparing grade distributions from lecture exams and course final grades from Histology and Human Biology courses appears to support the idea that if cheating is occurring in non-proctored, open book exams taken under tightly timed conditions, it is happening at a rate similar to proctored courses offered in previous years. The types of cheating eliminated or at least that become of lesser concern when open book exams are given fall into the category of students carrying into exams cheat sheets, crib notes, answers written on their skin or clothing or more modern electronic cheating methods such as sharing answers using mobile phone technologies. The observation of similar as well as acceptable grade distributions for lecture exams and final course grade distributions for the two courses studied compared to proctored courses offered in prior years supports the idea that a strategy can be developed in which non-proctored testing is employed as the primary method of summative assessment.

Another important feature of this strategy is to be sure the timing aspect of the tightly timed exam is accurate enough to provide students who know the material best to show that on the exam. Timed exams are certainly not novel to this study but evidence obtained by comparing how long each student took to complete the exams in Histology and Human Biology to the grade received is a step forward in developing a sound basis for choosing the amount of time in seconds that ought to be available for each exam. The closer look at the timing data is useful to convince some students that others were able to complete the exam and receive passing grades so that they are now more receptive to advice about how to improve their study habits and speed of test taking. Evidence of the fairness of a tightly timed exam is also useful from an institutional policy perspective. Some students may complain to administrators that not enough time was given for a particular exam. Data obtained from the course management system's statistics package can be an important source of information to demonstrate that fair standards were applied to a tightly timed exam as indicated by the number of students who did pass the exam with acceptable grades in the time provided. If a student has a disability requiring more time on exams, that student must first present supporting documentation to the institution's office overseeing special accommodations and then the instructor is informed that more time can be provided.

As mentioned earlier, timed exams can be the cause of student anxiety or stress and to assist students in alleviating stress led to the idea of providing timed practice exams called quizlets. Again, the statistics package of the course management system, in this case, Blackboard, provided a way to estimate time-on-task by keeping track of the number of times a student took the quizlets. Results in this study showed a consistent linkage of the number of times students took the quizlets and the grade they received in the lecture exam relating to those quizlets. The quizlets proved to be a strong formative assessment strategy for both upper division Biology majors taking Histology as well as non-science majors taking the Human Biology lecture course. This type of formative assessment, i.e, assessment for learning, would also likely be helpful to students taking online courses in disciplines other than Biology as well as in blended, hybrid or on-ground courses as long as the formative assessment was non-graded and able to be taken by students as many times as they wished.

The nature of the lecture exams in both courses was at the lower levels of Bloom's taxonomy¹⁰ with more focus on knowledge recall and an understanding of what was read, i.e., comprehension. Exam questions required answers relying on an understanding of the definitions of terms and their use in sentences assessing a student's understanding of basic concepts in Histology and Human Biology. The data collected in both courses linking quizlets to better learning can be viewed as baseline data upon which future exams can be constructed employing higher level questions on the Bloom's taxonomic scale.

References

• F. Schab, Schooling Without Learning: Thirty Years of Cheating in High School, Adolescence, 23: 839-47, 1991.
  
• D. McCabe, Why Students Do It and How We Can Help Them Stop, American Educator, Winter Edition, 2001; http://www.aft.org/p... [22.06.2008].
  
• G.C. Rakes, Open Book Testing in Online Learning Environments, Journal of Interactive Online Learning, 7(1): 1-9, 2008.
  
• D, McCabe, L.Trevino, K. Butterfield, Cheating in Academic Institutions: A Decade of Research, Ethics and Behavior, 11(3): 219-232, 2001.
  
• J.C. Cassady, B.E. Gridley, The effects of online formative and summative assessment on test anxiety and performance, Journal of Technology, Learning and Assessment, 4(1), 2005; http://escholarship...., [25.06.2008].
  
• G.S. Wellman, H. Marcinkiewicz, Online Learning and Time-on-task: Impact of Proctored vs. Un-proctored Testing, Journal of Asynchronous learning Networks, 8(4), 2004; http://www.sloan-c.o..., [23.06.2008].
  
• UT Telecampus, http://www.telecampu... [27.06.2008]
  
• UTTC sample course, 2008; http://www.telecampu...
  
• C. Theophilides, M. Koutselini, Study behavior in the closed-book and the open-book examination: A comparative analysis, Educational Research and Evaluation, 6(4), 379-393, 2000
  
• B. L. Olson, J. L. McDonald, Influence of Online Formative Assessment Upon Student Learning in Biomedical Science Courses, Journal of Dental Education, 68(6), 656-659, 2004.
  
• B. Bloom, M. Englehart, E. Furst, W. Hill, & D. Krathwohl, Taxonomy of educational objectives: The classification of educational goals. Handbook I: Cognitive domain. New York, Toronto: Longmans, Green, 1956.