DetectGPT, GPTZero… Tools to detect cheating with ChatGPT

A great cheat tool, ChatGPT has teachers worried a lot. And to fight against trickery, many researchers are trying to develop tools capable of knowing if a text was written by AI or by a human.

ChatGPT, the conversational AI of OpenAI, never ceases to panic the Internet sphere. but not only ! While many fields such as journalism, medicine, cybersecurity and psychiatry are studying the possibilities offered to them by this artificial intelligence capable of generating precise and detailed texts in a natural language via a simple request, they are also very attentive to the excesses that it can generate. The teaching environment in particular is taken by surprise, because ChatGPT, already widely used by students since its appearance, proves to be so effective for writing homework and summaries that it is sometimes impossible to differentiate its creations from the production of a human: a real problem, and not only in terms of ethics and transparency. Indeed, students see it as a valuable time saver as well as the possibility of obtaining a good grade without too much effort. One problem among many others, which pushes its developers to find a way to set up a signature within the text in order to differentiate those produced by the AI ​​from those resulting from a human.

The fact remains that in the meantime, teachers find themselves helpless in the face of a technology that is beyond them and that upsets uses. AI has also been banned in New York universities and Sciences Po – where any use for homework may result in the student’s exclusion from school, or even higher education, as in the case of classic cheat. But it’s fine to prohibit, but you still have to be able to enforce the rules. And students have always, since the dawn of time, taken pleasure in circumventing and breaking the rules. Several researchers and developers have already implemented several tools to detect which of the artificial intelligence or the student has produced a text.

DetectGPT: an anti-ChatGPT tool in development

A team of Stanford researchers has tackled the problem that many teachers are currently facing and have developed a method called DetectGPT. It’s a barometer that determines whether or not text is machine-generated, without the need for AI or “the collection of a data set of actual or generated passages” to compare the text. To achieve this, the tool detects samples of pretrained language models “using the local curvature of the model’s logarithmic probability function”, that is to say that it manages to recognize content structure models generated by an AI and to report them when it detects them. The researchers did not give more information, except that DetectGPT is currently only at the prototype stage and that it will be necessary to wait in order to benefit from a public version.

ChatGPT detector – GPT3: a Franco-Canadian start-up on the spot

But they are not the only ones to be on the spot. Thus, the Franco-Canadian start-up Draft & Goal has just created a detector available free online capable, in just a few seconds, of telling whether the text was written by an artificial intelligence or a real person, and with a reliability rate of 93%. Simply copy the text of at least 400 characters that you wish to submit, paste it into the interface and click on the “Analyze” button. The tool then delivers a score between 0 and 100. Vincent Terrasi, the co-founder of Draft & Goal, explains to France Live that “When the result is above 60, there is a very high probability that the content is from the artificial intelligence of ChatGPT. If the result is below 40, then it is probably the result of human work. “ For the gray zone which is between 40 and 60, it is usually a mixture of the two, a technique often used by students to decrease the risk of getting caught. The start-up’s tool is currently in beta and only works with English texts – French should be supported within a few days. On our side, we managed to fool the tool in beauty from our first attempt, which concerned an explanation of “deep learning”, but it was able to detect the deception afterwards.

According to Vincent Terrasi, 80% of their users are teachers. “To Truth be told, this isn’t the first time teachers have feared being tricked by technology. This has long been the case with some English teachers who have refrained from giving homework since the advent of Google Translate, for example. But ChatGPT undoubtedly marks a new stage”, he explains. However, he does not fully reveal his method for detecting texts produced by AIs. It only indicates that the detector also uses an artificial intelligence based on machine learning. Therefore, the more it will be used, the more efficient it will become.

To determine the score of a text, it will cross a whole series of “fingerprints”, either “sequences of words that can only be written by artificial intelligence. Quite simply because a human would not use these words, these formulations or these ideas. […] This is the case, for example, of ready-made expressions which are not ‘natural’. ChatGPT will thus use a formulation like ‘play’ tennis or piano. A human is more likely to say that he ‘plays’ tennis or that he ‘plays’ the piano.” The detector is therefore very attentive to highly technical jargon, grammatical errors, spelling errors, repetitions, miscalculations, exaggerations, contradictions and “hallucinations” – when the AI ​​is convinced that something completely false is true. For the moment, the use of the detector is limited to ten text tests per day, but the start-up is considering increasing this limit.

GPTZero: a practical but perfectible “anti-plagiarism” tool

Edward Tian, ​​a computer science and journalism student at Princeton University in the United States, has developed an application whose algorithm can identify whether a text was produced by the chatbot or whether it was written by a human. Called GPTZero, it analyzes the text in order to evaluate its complexity, its random character compared to a text model as well as its uniformity. Combined together, these elements make it possible – well, in theory – to determine whether the content was created by a human or by an AI.

GPTZero can be accessed for free through any web browser from gptzero.me Where this address. Be careful, the tool is still in beta version, so it often bugs and is not very fast. You have to wait a few minutes between each step for the page to load and do not hesitate to refresh it if an error message appears. You don’t have to be in a hurry! The site is in English, but GPTZero is quite capable of analyzing a text in French. To use it, simply copy/paste the text in the corresponding field then press Ctrl + Enter for the tool to analyze it. You must then wait a few minutes for all the results to be displayed and, at the end, click on “Get GPTZero Result”.

To determine whether the text was written by an artificial intelligence or a human, GPTZero measures its Perplexity, which designates “the randomness of the text”. This is’“a measure of the ability of a language model like ChatGPT to predict a sample of text. In other words, it measures how well the computer model likes the text.” The higher the perplexity of a text, the more likely it is to have been written by a human.

In our test, ChatGPT generated a text from the following instruction: “write an exhaustive text on the place of philosophy in Harry Potter.” His perplexity score was 38. However, it does not take into account many factors, such as the length of the text. However, longer texts are less random and generally have lower perplexities. This is why it is necessary to take into account the average perplexity, that is to say over the set of sentences. This time it was 72.3. Finally, by analyzing the complexity of the text sentence by sentence, GPTZero came to the conclusion that our text on Harry Potter was written by an AI. On our article on the Apple Pencil, our total puzzlement was 190, and the tool concluded that the text was indeed written by a human. To note that some sentences written by the human may have low perplexity, but there are bound to be peaks of perplexity as he continues to write – hence the need to analyze sentence by sentence. In contrast, perplexity is evenly distributed and consistently low for machine-generated texts.

ChatGPT anti-cheat: what future for AI and education?

If the development of anti-cheating tools is good and necessary news, the place to be given to AIs like ChatGPT in the field of education is debated. Of course, it seems logical that teachers prohibit their students from taking the “merits” of the work provided by an AI. Thus, in a letter addressed to all students and teachers, the management of Sciences Po announced that “the use, without explicit mention, of ChatGPT at Sciences Po, or of any other tool using AI, is, with the exception of pedagogical use supervised by a teacher, for the moment strictly prohibited during the production of written or oral work by the students”. Same story on the other side of the Atlantic, in the district of New York, whose spokesperson Jenna Lyle believes that “the tool is certainly capable of providing quick and easy answers to questions, but it does not develop critical thinking or problem-solving skills, which are essential for academic success and lifelong success. of life”. An argument that is far from unanimous within the educational community.

Indeed, as reported EducationWeekvice president of research and development at the Educational Testing Service Andreas Oranje considers the ban on AI to be “a bad idea because ChatGPT is a fact of life. And we want to prepare students for life”. And artificial intelligences seem to be well on their way to becoming full-fledged assistants in our daily lives. On the other hand, she thinks it is necessary to incorporate technological advances in the way of teaching, such as “create assignments that teach the skills you want to teach, but in a way that works with ChatGPT.”