Some people used virtual machines, others made direct phone calls, and six cheaters escaped unscathed! Is the online proctoring software just a decoration?

Produced by Big Data Digest

Author: Caleb

When it comes to cheating, it can be said that it is a typical example of someone who has a high level of knowledge and a high level of magic.

The 2017 movie "The Gunslinger" showed a new twist on the seemingly "common" student behavior of cheating. Some netizens even said that "cheating is the theme of student days."

Previously affected by many factors such as the epidemic, online exams around the world were also forced to be put on the agenda, followed closely by various anti-cheating measures. .

But after so long, have these anti-cheat software really played their due role?

A recent investigation found that Proctorio, the anti-cheating software used by schools around the world, was not effective at detecting cheaters in their environment, despite their claims to have a high degree of confidence in doing so.

Research paper: https://ris.utwente.nl/ws/portalfiles/portal/275927505/3e2a9e5b2fad237a3d35f36fa2c5f44552f2.pdf

University of Twente, Netherlands In a very vivid metaphor, researchers at , said the software was like taking a placebo: it had some positive effect, not because it worked, but because people believed it worked, or that it might work.

How unreliable is Proctorio?

Before that, we still need to understand Proctorio.

As an AI system, one thing that must be acknowledged is that Proctorio frees up the human resources of proctors. Instead, it mainly uses software to monitor students, detect and report any suspicious behavior during exams.

This requires students to install a browser extension or an app on their computers to better track their eye movements and body language. Students must also upload their student photo and ID card so that the system can better perform facial recognition.

On the other hand, Proctorio mainly records student computer video and audio clips, captures student keystrokes, records computer screen data, marks when students move their heads, eyes, or bodies, and tracks their movements. Invigilate the exam by categorizing the websites visited during the exam.

This study was first presented at the 13th International Conference on Computer-Assisted Education in April 2021, in an attempt to prove whether the software is effective.

To do this, the researchers found 30 student volunteers from the university's computer science classes and had six of them cheat on an exam proctored by Proctorio. Five others were told not to cheat but to Act suspiciously and the rest take the exam honestly.

As for how to cheat, the researchers gave this power back to the students, allowing them to decide how best to cheat the system.

Judging from the results, Proctorio is not good at detecting cheaters, and the system did not mark any cheaters as cheaters.

Someone is using a virtual machine and this is a known vulnerability in the Proctorio system. The software did flag these students as "infractions," but it also flagged other honest students for the same infractions, the study wrote. Meanwhile, some cheaters used audio calls, but Proctorio didn't flag their audio as anomalous, but did flag audio of students taking the test in a noisy environment.

An independent manual review of the data and footage found only one of the six cheaters, primarily because they had no way of knowing the student's chest below due to the angle of the camera and the optional "room scan" feature. What exactly the stuff is, and how students behave.

Proctorio spokesman David Lux ​​described three other studies on the "effectiveness of online proctoring" that were "multidisciplinary and had more robust sample size”. The authors acknowledge that the University of Twente study has flaws. However, none of these three studies were controlled experiments. Instead, they either use surveys or regression models to estimate cheating rates and student attitudes toward online proctoring.

These studies were not only different research designs, but also addressed fundamentally different questions. After all, none of them attempted to prove whether online proctoring software was effective in catching cheaters.

Tsinghua University launches AI Eagle Eye system

In addition to Proctorio, domestic online proctoring technology is also steadily following up.

In March this year, Tsinghua University developed the "AI Eagle Eye" digital proctoring and anti-cheating system, which can detect abnormal conditions such as face deviation and gaze aversion and issue warnings without the need to download an APP in advance. It is expected to start next year use.

Zhou Xiuzhuan, director of the Teaching Development Center of Tsinghua University, said that the AI ​​​​Eagle Eye digital proctoring system is currently the only system that does not require downloading software in advance, candidates do not need to hand over computer control, supports mobile phone answering, and supports all browsing It is an anti-fraud system suitable for invigilating exams in large classes with more than 100 people.

The core of the AI ​​​​Eagle Eye digital proctoring system is the application of face recognition and eye tracking technology, so that the video lens of a general home computer or the lens of a mobile phone has the function of scanning the trajectory of eye gaze, combined with the mouse AI artificial intelligence deep learning systems such as trajectories and information can accurately determine whether candidates have cheated.

In this regard, the AI ​​​​Eagle Eye system divides cheating behavior into 4 types of abnormal phenomena such as "face deviation", "face leaving", "eyes away" and "opening paging", and counts the number of exceptions to remind candidates Note to the invigilator that the invigilator can also ask the candidate to turn on the call function to determine the on-site situation.

Tsinghua stated that the AI ​​​​Eagle Eye system will be piloted on a small scale this year to collect more samples and data, continue to improve and improve the accuracy of the system, and aim to officially use it next year .

How should I do online proctoring?

In fact, in China, online proctoring has already been practiced.

In May of this year, the “One Model” exam for junior high schools in Fengtai District, Beijing, was conducted using a combination of online and offline methods.

The first is the requirements for the examination environment. Students participating in the online exam must ensure that the desktop is clear. There is a laptop in front of the student and a mobile phone 45 degrees behind the student. The cameras of the two devices capture the student's answering status from different angles to ensure that the invigilator's perspective is complete and clear for full monitoring. .

Shen Lei, the academic director of Zhao Dengyu School in Beijing, said that in addition to one invigilator in each examination room, school teaching cadres, grade leaders, and class teachers all entered the examination room online. Conduct inspections, "There will be at least 5 to 6 inspections in each session to check whether the students' camera angles are standardized and whether each student is serious about answering questions."

After the first day of the "One Model" exam, the school also put forward some optimization plans for this online invigilation. For example, one screen cannot fully display the real-time status of all candidates, and pages need to be turned. In this regard, Fengtai No. 2 Middle School proposed an optimization: in addition to the invigilator, the class teacher also entered the venue, and two devices were used to invigilate the exam at the same time, so that the answers of all students could be seen at a glance.

Most of the current online proctoring systems are more or less in line with the core point of the University of Twente study, that is, the software will detect some too blatant cheating behaviors, such as students Taking exams together and discussing them in real time has a deterrent effect.

But after all, the school may spend a lot of money on the invigilation system for each exam, and there is also the risk of endangering student privacy. In other words, should an invigilator be allowed to come in when everyone has their cameras turned on? Taking the test seems to add a few more variables. ​

The above is the detailed content of Some people used virtual machines, others made direct phone calls, and six cheaters escaped unscathed! Is the online proctoring software just a decoration?. For more information, please follow other related articles on the PHP Chinese website!