It only costs $0.003 a time, which is 20 times cheaper than humans! ChatGPT puts data annotators in danger

Recently, a research team from the University of Zurich found that ChatGPT outperformed crowdsourcing workers on multiple NLP annotation tasks, with high consistency, and the cost of each annotation was only about US$0.003, which is 20 times cheaper than MTurk.

Currently, many natural language processing (NLP) applications require high-quality annotated data to support, especially when these data are used for tasks such as training classifiers or evaluating the performance of unsupervised models.

For example, artificial intelligence researchers often want to filter noisy social media data for relevance, assign text to different topic or conceptual categories, or measure its sentiment or stance.

Moreover, no matter what specific method (supervised, semi-supervised or unsupervised) is used for these tasks, labeled data is needed to establish a training set or gold standard.

However, in most cases, to complete high-quality data annotation work, it is still inseparable from crowdsourcing workers on the data annotation platform or well-trained annotators such as research assistants. You can do it manually.

Typically, trained annotators first create a relatively small gold standard data set, and then hire crowd workers to increase the amount of annotated data and perform repetitive work. Depending on the size and complexity, data annotation tasks can sometimes be very time-consuming and labor-intensive. Not only do they require a certain amount of labor costs, but the quality of data annotation cannot be guaranteed.

So, can machines help humans complete this basic task?

In the past, machines were not good at this kind of "slow work and careful work" tasks, but unexpectedly, the "data annotation" matter has been completed by ChatGPT, and it is better than Most people do better.

In a new study published today, a research team from the University of Zurich used a sample of 2,382 tweets to demonstrate that ChatGPT performs better on relevance, topic, and Outperforms crowdsourcing workers on multiple annotation tasks such as frame detection.

The related research paper is titled "ChatGPT Outperforms Crowd-Workers for Text-Annotation Tasks" and has been published on the preprint website arXiv.

Specifically, ChatGPT’s zero-shot accuracy exceeded crowdsourcing workers in four of the five tasks; it demonstrated intercoder consistency in all tasks In terms of agreement), ChatGPT not only surpasses crowdsourcing workers, but also surpasses trained annotators. ​

ChatGPT zero-sample text data annotation performance

It is worth mentioning that the cost of each annotation of ChatGPT is less than 0.003 US dollars, which is about 20 times cheaper than the data annotation platform.

The research team believes that while further research is needed to better understand how ChatGPT and other LLMs perform in a broader context, the findings suggest that they have the potential to change the way researchers annotate data. , greatly improving the efficiency of text classification and destroying some business models of data annotation platforms.

At least for now, these findings demonstrate the importance of delving deeper into the text annotation properties and capabilities of LLMs.

In the future, the research team will study the performance of ChatGPT in multiple languages, the performance of ChatGPT in multiple types of texts (social media, news media, legislation, speeches, etc.), and use Chain of Thoughts (CoT) Work continues on hints and other strategies to improve the performance of zero-shot inference.

It is worth mentioning that when the research team was conducting this work, OpenAI had not yet released GPT-4. What would be the result if GPT-4 was used to complete the data annotation task?

Reference:​​​https://arxiv.org/abs/2303.15056​

The above is the detailed content of It only costs $0.003 a time, which is 20 times cheaper than humans! ChatGPT puts data annotators in danger. For more information, please follow other related articles on the PHP Chinese website!