The 'magic tool for seducing girls' doesn't work anymore! OpenAI releases official 'ChatGPT detector'

The public’s crusade came one after another, and OpenAI finally couldn’t stand it anymore.

It has been a long journey from university professor to Stack Overflow. Now, whether a teacher sees a paper submitted by a student or a coder sees the code on the Internet, they are not sure whether the author is a human or AI.

Under the criticism, OpenAI couldn’t stand it anymore and quickly released an AI detector.

Fighting left and right: OpenAI releases ChatGPT detector

Just now, OpenAI officially announced its own AI text classifier.

It is a fine-tuned GPT model that can infer the likelihood that a piece of text was generated by AI.

Interestingly, ChatGPT is also based on the GPT model. Using this classifier to detect ChatGPT can be called a fight between left and right.

In training, this model uses human handwriting and AI-generated text pairs on the same topic.

The materials used come from the Wikipedia data set, the WebText data set collected in 2019, and a set of human demonstrations collected while training InstructGPT.

##Experience address: https://platform.openai.com/ai-text-classifier

But, this accuracy rate is really not high...

When evaluating the English text in the "Challenge Set", the classifier only classified 26% of the AI The generated text was correctly classified as “probably written by an AI” (true positive).

Is the "Magic Tool for Flirting Girls" still working?

There is no doubt that ChatGPT is creating a hurricane around the world.

Students are enjoying using it, and the media exclaimed that the "education system" has been subverted. Recent surveys show that ​​has been 89% of college students use ChatGPT to write homework​​.

In addition, it has passed the U.S. Medical Licensing Examination, the Wharton MBA exam and 4 law school exams. Its ability is simply extraordinary; the US version of "Headline" BuzzFeed announced that it will After the news that ChatGPT was used to write articles, the stock price skyrocketed by 119%.

And the wonderful literary talent has also made ChatGPT regarded as a "magic tool for flirting with girls" by many guys.

Although it is not very good at doing math problems, it is so easy to let it write a brilliant and affectionate love letter.

No, this guy from abroad is addicted to using ChatGPT to write love letters to his wife Monika.

However, Monika has already seen through his tricks and warned him not to be addicted to ChatGPT anymore.

There are also friends who have played a lot and sent the love letters generated by ChatGPT to their good friends. The good friends immediately looked like black question marks on their faces.

Although, it can be seen that ChatGPT’s literary talent is quite good. Essays, love letters, novels, ChatGPT is no problem. No wonder everyone is crazily using ChatGPT to "create articles".

So how effective is the official AI detector released by OpenAI?

Let’s test it.

First try the love poem that Monika’s husband sent her.

Unfortunately, OpenAI stated that it can only detect articles with more than 1,000 characters, and there is nothing it can do to help...

Translation: At least 1000 characters are required, which is 150-250 English words. For Chinese, it can only be a "thousand-character long article."

Indeed, OpenAI’s official announcement also said that “the classifier is very unreliable for detection of short texts (less than 1,000 characters). Even longer texts are sometimes classified The detector incorrectly labeled it."

In addition, OpenAI's detector has several "limitations":

• The classifier does not Not always accurate, it may incorrectly label both AI-generated and human-written text, and the tone is assertive
• It is recommended to use the classifier only for English text. It performs much worse on other languages ​​and is unreliable on code
• It cannot reliably recognize very predictable text. For example, it is impossible to predict whether a list of the first 1,000 prime numbers was written by an AI or a human because the correct answer is always the same
• AI-generated text, after editing, is likely to evade detection by the classifier
• If the input is very different from the text in the training set, the classifier will make a wrong judgment

Since the detector needs to be long, let’s give it a long text of. The editor has put together an excerpt from Wang Xiaobo's love letter, with a total of 1,027 words.

After inputting into the detector, it gave a judgment: this love letter was written by a human. Bingo!

Then, the editor asked ChatGPT to imitate Wang Xiaobo and generate a new love letter.

As you can see, the love letters written by ChatGPT are not bad in literary style, but compared to the words of great writers, they really have no "soul".

The detector also accurately detected that this love letter was most likely generated by AI.

The division of the results is super fine

Specifically, the classifier will divide the input text into Category 5:

• "Very unlikely to be AI-generated"

The threshold of the classifier is

• 「Unlikely to be AI-generated」

Classifier The threshold is between 0.1 and 0.45. In the "challenge set", about 15% of handwritten text and 10% of AI-generated text have this annotation.

• 「Unclear if it is AI written」

Classifier The threshold is between 0.45 and 0.9. In the "challenge set", about 50% of handwritten text and 34% of AI-generated text have this annotation.

• "Possibly AI-generated"

The threshold of the classifier is between 0.9 and 0.98. In the "challenge set", about 21% of handwritten text and 28% of AI-generated text have this annotation.

• "Likely AI-generated"

The threshold of the classifier is >0.98 . In the "challenge set", about 9% of handwritten text and 26% of AI-generated text have this annotation.

Reliability needs to be strengthened

Although the classifier can identify most AI-generated text, but OpenAI cautions that it is not "completely reliable."

In the area under the curve (AUC) test, the classifier achieved a score of 0.97 in the validation set and a score of 0.66 in the challenge set, while the previously published classifier achieved a score of 0.97 in the validation set. It scored 0.95 points and reached 0.43 points in the challenge set.

In addition, as the size of the language model increases, the performance of the classifier decreases.

But the good news is that this means that the text generated by large language models is more like what humans would write.

At the same time, OpenAI also specifically pointed out that this classifier has not been evaluated on rumors, student compositions or chat records; it has not been tested on texts co-written by AI and humans. And this is one of the most common usage scenarios now.

It seems that GPTZero developed by the 22-year-old Chinese guy from Princeton University is more user-friendly.

Not only requires only 250 characters, but it can also clearly indicate which paragraph in a piece of text was generated by AI and which paragraph was written by humans.

Actual test of AIGC’s self-developed “AI detector”

Before OpenAI released the official “AI detector”, the market There are many AI detectors available today.

David Gewirtz, a writer and computer scientist from the United States, personally tested three online AI detectors.

The most interesting thing here is that two of them are made by the website that sells AI-generated text.

Note: Humans write in normal fonts, AI generates italics. After each text, the detector results are displayed. (The actual text input to the detector is all English text, and this article is the translated result).

By the way, this David Gewirtz is also a member of FBI InfraGard and one of the important cybersecurity experts in the United States. He previously received a Bachelor of Science degree with honors from Worcester Polytechnic Institute, but dropped out of his Ph.D. program at the University of California, Berkeley.

The test begins!

1. Explain why plagiarism caused by ChatGPT will cause trouble to teachers and editors

##ChatGPT It is a variant of the GPT (Generative Pre-trained Transformer) language model developed by OpenAI. It is designed to generate human-like text by predicting the next word or phrase in a given input sequence. While ChatGPT can be a useful tool for generating text and responding to prompts, it can also cause problems for teachers and editors when it comes to plagiarism.

One of the main reasons ChatGPT can cause problems for teachers and editors is that it can generate text that closely resembles real humans. This means that it is possible for people to use ChatGPT to generate text and then submit it as their own work, thereby committing plagiarism.

For teachers and editors, detecting whether a specific piece of text was generated by ChatGPT or written by a human can be very difficult because the generated text may Contains no obvious errors or discrepancies that would allow one to identify its source. This can make it difficult for teachers and editors to identify and prevent plagiarism, as they may not be able to tell whether a passage is original or copied from another source.

Overall, ChatGPT and other similar language models can become a problem for teachers and editors when it comes to plagiarism because they can generate difficult Differentiated text makes it easier for people to plagiarize without being caught.

GPT-2 Output Detector: 99.69% true.

Writer AI Content Detector: 97% is human-generated content, which is amazing!

Content at Scale AI Content Detection: 87% human score, looks good!

2. Tell me at least three ways to detect whether an article was generated using ChatGPT

There are several ways to detect whether ChatGPT or other language models were used to generate a piece of text. Here are three potential methods you can use.

1. Look for repeated or unusual patterns in text: One way to detect whether a language model is used to generate a piece of text is to look for patterns in the writing. Repeating or unusual patterns. For example, a language model might repeat certain phrases or word combinations, or use unusual or inconsistent sentence structures.

#2. Check for lack of originality: Another way to detect whether a language model was used to generate a piece of text is to check for a lack of originality in the writing. . If a piece of text appears to have been copied from another source, or if it contains information that is widely available on the Internet or other sources, this may be a sign that it was generated by a language model.

3. Use a plagiarism checker: Finally, you can use a plagiarism checker to check whether a piece of text has been plagiarized from another source. While a plagiarism checker may not be able to definitively determine whether a language model was used in a paper, it can help you identify instances of plagiarism or paraphrasing from other sources.

#Keep in mind that it is difficult to unambiguously determine whether a language model was used to generate a piece of text, and these methods do not always work. However, they can be a useful tool in helping you identify potential instances of language model-generated text.

GPT-2 Output Detector: 99.98% fake.

Writer AI Content Detector: "No result".

Content at Scale AI Content Detection: 0% human score, obviously AI.

3. List three plagiarism checkers that can detect ChatGPT text

Notable Yes, there are currently no dedicated plagiarism checkers for detecting text generated by ChatGPT or other language models. While some plagiarism checkers may be able to identify certain characteristics of text generated by a language model, thereby proving that the text came from AI, there is no guarantee that they can unambiguously determine whether a piece of text was generated by a language model.

So, here are three plagiarism checkers you might consider using to help identify potential instances of language model-generated text:

1. Turnitin: Turnitin is a popular plagiarism checker widely used by schools and universities. It compares a text to a database of other texts to determine whether the text is plagiarized or paraphrased. While it may not be able to definitively determine whether a piece of text was generated by a language model, it may be possible to identify certain characteristics of language model-generated text that would prove it was generated by AI.

2. PlagScan: PlagScan is another plagiarism checker that compares a piece of text to a database of other texts to identify instances of plagiarism or paraphrasing. Like Turnitin, it may be able to identify certain characteristics of language model-generated text that may indicate that the text was AI-generated, but it is not a tool specifically designed to detect language model-generated text.

3. Copyleaks: Copyleaks is a plagiarism checker that uses advanced machine learning algorithms to scan a text and compare it with a database of other texts Compare to identify instances of plagiarism or paraphrasing. While it may be able to identify certain features of language model-generated text, it is not specifically designed to detect language model-generated text.

#It is worth noting that no plagiarism checker is completely foolproof, and there will always be text generated by some language models that can escape detection. Therefore, it is important to use a plagiarism checker only as a complementary means.

GPT-2 Output Detector: 99.62% true.

Writer AI Content Detector: "No result".

Content at Scale AI Content Detection: 86% human score, looks good!

4. Online artificial intelligence plagiarism checker

The principle of most plagiarism detectors is to compare text with Other corpora for comparison. For example, when a student turns in an essay, a product like Turnitin checks it against the vast library of essays in its database, as well as other texts on the Internet, to determine whether the submitted essay contains content that has already been written.

But AI writing tools will produce original content, at least in theory. Yes, they build content from a training data set, but the words they create are unique in each article.

Therefore, the plagiarism checker mentioned above may not work because the AI-generated content is not likely to exist in another student's paper.

So I searched on Google for detectors specifically designed to find AI-generated content. I found three. For the test in the screenshot below, I asked ChatGPT this question: "Is Star Trek better than Star Wars?" The answer was not bad at all, and I fed that answer back to three detectors.

GPT-2 Output Detector: 99.98% true.

Writer AI Content Detector: 100% human-generated content, awesome!

Content at Scale AI Content Detection: 100% human score, looks good!

Result evaluation

Provincial flow:

• OpenAI has previously targeted GPT-2 The detectors built by the new model are sometimes fooled by the content generated by the new model.
• The "detectors" created by companies that sell AI-generated content are basically unreliable.

1. GPT-2 Output Detector (66% accuracy)

GPT-2 Output Detector is a “duplication checking” tool developed by OpenAI itself.

Although it was originally built for GPT-2, it can now be used to detect various AI-generated texts and can also achieve good results. It got four of the six tests correct.

2. Writer AI Content Detector (33% accuracy)

Writer.com Mainly Providing generative AI writing services to enterprise teams, a tool called "AI Content Detector" can be used to determine whether the content is generated by AI.

But unfortunately, this tool is very unreliable. It only ran successfully three times out of a total of six tests...and of those only three results, two were correct and one was incorrect.

##3. Content at Scale AI Content Detection (50% accuracy)

The third tool is also produced by an AI content generation company. Judging from the results, it is much more reliable than the previous one, but the overall effect is still average.

Content at Scale’s marketing slogan is: “Just upload a keyword list and get articles that bypass AI content detection—all without human intervention!” It was accurate three times out of a total of six tests.

Some thoughts (AI is also involved lol)

Plagiarism caused by AI has become increasingly serious, because like Notion The text generated by AI and AI like ChatGPT is almost indistinguishable from that written by humans.

There are several ways to detect whether a piece of text was generated by AI, such as looking for repeated or unusual patterns, checking for a lack of originality, or the use of plagiarism Checker.

Plagiarism checkers like Turnitin, PlagScan, and Copyleaks may be able to identify certain characteristics of language model-generated text, but they are not foolproof.

At this point, we may not be ready to pit AI against AI. On the contrary, we still need to rely on human perception to identify whether a certain work is generated by AI.

What do you think about this?

Have you ever used these AI tools? And is it possible to tell whether an article was generated by AI? Do you think they help or hinder the industry?

Welcome everyone to leave a message at the end of the article.

The following is ChatGPT’s answer:

As a language model, I can’t see to, so I can’t tell whether an article is AI-generated.

#However, I can tell you that there are several AI writing tools now that can be used to generate a variety of written content, including news articles, Novels, poetry, etc.

#I have no personal experience using these tools as I'm just a computer program and don't have the ability to browse the internet or use the physical tools. However, I can tell you that some people find AI writing tools helpful in education and journalism because they can save people time and energy by automating certain tasks. Others have raised concerns that the tools could be used to create misleading or biased content, or to replace human writers.

Ultimately, whether AI writing tools help or hinder depends on how and in what context they are used. So, yes, that’s it.

In the process, David discovered a very interesting point: don’t just read something stupid or incorrect when you see it online. Immediately thought it was written by AI.

David said that as we all know, humans are fully capable of writing garbage content on their own.

After all, regardless of whether the content is generated by humans or machines, stupidity has always been a common trait.

The above is the detailed content of The 'magic tool for seducing girls' doesn't work anymore! OpenAI releases official 'ChatGPT detector'. For more information, please follow other related articles on the PHP Chinese website!