For professors that may be worried about students using AI like ChatGPT on their assignments, Stanford Online High School junior Benjamin Klieger has a solution.
Over winter break, Klieger launched CheckforAi, a platform that attempts to use machine learning to detect whether AI was used to generate text in anything from essays to emails.
“What I found important was the detection of AI Content, because I think without the transparency right now to know if content is made by a human or AI, generative AI is actually, or at least perceived to be, a threat to industries and to integrity,” Klieger said.
With over 13 million unique visitors a day, ChatGPT has been hailed as a revolutionary tool, one capable of threatening Google’s search engine. An informal poll conducted by The Daily on the anonymous social media platform Fizz suggested that many Stanford students may have used ChatGPT on final exams.
“I think that it can be sort of game-changing in education,” said Eric Mitchell, a fourth-year computer science PhD student, who created DetectGPT, another AI detector.
“I’m worried about cheating but our work should not be read as there’s no role for language models in education” Mitchell said. “I think there almost certainly will be scenarios where it’s not appropriate to use them, and perhaps tools that do detection can be one part of the solution to making that work.”
CheckforAi is based on OpenAI’s existing Roberta-base model for GPT detection, as well as custom models built by Klieger. Klieger believes that his platform is more effective than OpenAI’s GPT detector.
“If there’s small changes in the text, the original model is really good at detecting that it came from ChatGPT,” Klieger said. “But if you mix it up a bit, it says, ‘okay, that didn’t come from ChatGPT exactly.’ Those things edited by a human, but made by AI originally, that’s where the additional model on top is able to identify those middle grounds.”
However, Mitchell says there are many concerns with placing so much trust in these AI checkers because they are so recently developed.
“What is the level of performance that is necessary to make these actually useful, to have them not do more harm than good?” Mitchell said. “If you have a model that has a false positive rate, you’re going to be accusing a lot of students of plagiarizing that aren’t plagiarizing, and that can be catastrophic for their learning and for their trust in the system. We still don’t really have a rigorous methodology to determine ‘is this good enough to use in the classroom?’”
ChatGPT has meant that teachers and schools must find new ways to adapt in the classroom.
“We’re trying to find innovative ways that we can use ChatGPT and tools like that,” said Eric Guise, who is a technology specialist at Hopewell Valley Central High School that reviewed Klieger’s CheckforAi. One way they are trying to implement AI in the classroom is by having students distinguish between a human-written essay and a ChatGPT-written essay, then having CheckforAi verify their guesses.
CheckforAi is not the only AI checker that has been released since the emergence of ChatGPT. Stanford researchers, including Mitchell, created DetectGPT in February and a Princeton Student released another AI detector, GPTZero, in January.
“I tried the [AI checker] that was made by the Princeton student and then I tried Ben’s CheckforAi,” Guise said. “Ben’s has a great interface. It’s super simple to use. Ben’s looks a little bit cleaner but they both do the same thing, in my opinion.”
However, Klieger’s platform takes a different approach than many detectors. Most other detectors look at sentence structure and overall randomness of words, whereas CheckforAi detects individual words. Through the use of neural networks, CheckforAi looks at specific word output to check for the use of AI, leading to a more thorough inspection of text.
Klieger says he has received lots of feedback that will allow him to continue to grow this platform to be a tool.
“I want it to be a resource for educators, professionals and just personal use across the nation, across the world,” Klieger said. “I’m hoping that I continue to improve the model accuracy as well as how easy it is to use.”