The reviewer at the AI ​​Summit is actually an undergraduate? A female physicist posted a complaint on Twitter, and big guys in the industry liked it one after another.

Do you think that reviewers for top academic conference journals are all experts in their respective fields?

In fact, the reviewers of the research paper you have worked hard for several months are most likely to be a group of undergraduates or graduate students?

Recently, Lenka Zdeborova, a female physicist from École polytechnique fédérale de Lausanne (EPFL), complained about the reviewer mechanism of the top conference on Twitter.

Applicants to graduate schools already have experience in reviewing manuscripts? And there are quite a few.

I am reviewing applications for the EPFL Computer Science Graduate School and I see that many of the applicants have experience as reviewers for top ML conferences and I am wondering if the peer review is large Why should we bother doing peer review when some of it is done by undergraduates! ?

Lenka Zdeborová is Professor of Physics and Computer Science at EPFL, where she leads the Statistical Physics section of the School of Computing.

She is an editorial board member of many academic journals such as Journal of Physics A, Physical Review E, Physical Review X, SIMODS, Machine Learning: Science and Technology, Information and Inference, etc.

Lenka specializes in applying concepts from statistical physics (such as advanced mean field methods, replication methods, and related message passing algorithms) to machine learning, signal processing, inference, and optimization The problem.

Some people in the circle also agree that first-year graduate students can already review manuscripts at the top conference, and no one thinks this is a problem .

The reviewers you think are these academic leaders are actually the undergraduates or graduate students they teach...

Zhihu answerer "Kakakakabi" believes that this phenomenon has become commonplace. From ICLR to AAAI, reviewers The levels vary, and if you fail to ensure that your paper falls into the hands of an undergraduate student.

After complaining about the reviewers’ level, the key is to think about how to write a work that makes everyone feel sense and can take into account the review taste of most people.

Source: Kaka Kakabi

Answered by another The author "Hysterical i" also believes that reviewers rarely understand your field, so you have to present the results of the paper clearly and clearly, so that it is your problem if others cannot understand it.

Source: Hysterical i

How to review manuscripts at the top meeting

So how does the Dinghui review the manuscript?

Peer-review is the cornerstone of modern science, and almost all top machine learning (ML) conferences (such as NeurIPS, ICML, AAAI) rely on it to decide the papers to be submitted. Is it relevant to the community and original enough to be published there.

Unfortunately, as the number of submissions has grown exponentially over the past decade, the quality of review has declined just as fast.

If you've ever submitted a paper to one of these conferences, after working hard for months on what you thought was a brilliant idea, what you got was crappy, useless , and (worse) sarcastic comments mean you'll have to go through the submission process all over again without any hint as to what's wrong with your paper.

Turing Award winner Geoffrey Hinton gave one of the reasons for this situation in an interview with Wired magazine in 2018:

Now if you submit a paper with a completely new idea, it has no chance of being accepted because these young junior reviewers simply won’t understand it.

Or it will meet a senior reviewer who has reviewed too many papers and did not understand them the first time, and Think it must be bullshit. I think this is really bad.

Zhihu answerer Electric Light Phantom Alchemy made an assumption about the rejection of the review. If you find that your paper was rejected because of some conceptual issues that the reviewer did not understand. There are three possibilities:

## (1) The reviewer is an undergraduate/lower year student PhD student, he doesn’t understand these concepts.

(2) The reviewer is a super boss. He attaches great importance to basic conceptual issues, and he has not been in this field for many years.

(3) The reviewer is a super boss, but he has no time, so he leaves it to his junior doctoral students to review the manuscript. Therefore, the academic circle is a cycle, and undergraduate students behave most like super bosses.

• What do you think of the ideal reviewer:

Open the paper on Monday, look at the abstract, and read it quickly , evaluate the novelty. On Tuesday, look closely at methods and find faults. Take a look at the experiment on Wednesday, read the relevant papers of the baseline, and check the experimental details. Start writing review comments on Thursday to determine the review score. On Friday, I will add some review comments and submit the review scores.

• Actual reviewer:

#I opened the review paper at eight o'clock on Monday morning, read the abstract, and felt Nothing novelty. After looking at the pictures, I feel like I can’t understand it without reading the text. I turned to the back and looked at the experimental numbers, and I felt that the improvement was not big. Take a look at the methods, there are too many pages to read. I still have something to do in the afternoon, so I’ll give you 5 points and leave. The review work ends at 11:30 on Monday morning.

Let’s take a look at the review rules of ICML2022:

• Significance of topic selection:

The purpose of this part of the review is to show you to MR (Meta-reviewer) and the author How well you understand the paper and what do you think about it.

• Writing Innovation, Relevance, Importance:

Are they relevant to our community? Are they new? If the answer is no (or partial no, e.g. citing precise results from an earlier paper), a precise reason is needed so that the author knows how to fix the paper.

• Soundness:

Ideally, a paper should make claims that are well supported by theoretical arguments or experimental results.

• Quality of Writing:

Is the paper well organized and clearly written? Does it explain the novelty and results well? Does the paper contain enough information to support its claims?

• References:

Is the paper appropriately placed within contemporary literature? If not, please specify what is missing. Note that because papers have page limits, judgment is often required as to whether results should be mentioned.

These are the review standards for reviewers in the first stage. Those who meet the review standards can enter the second stage.

Papers that pass the first stage will be reviewed by the meta-reviewer (MR) and the project chair. Afterwards, there will be additional reviewers to ensure the quality of the paper review.

And why the level of reviewers at the top reviewers now is uneven, and there are even a group of undergraduates and graduate students. The reason is mainly due to the industry It is developing too fast, and the number of submissions is increasing much faster than the rate of cultivating qualified reviewers.

This not only results in a smaller proportion of people with the ability to review manuscripts, but also less time that capable reviewers can devote to each article.

The above is the detailed content of The reviewer at the AI ​​Summit is actually an undergraduate? A female physicist posted a complaint on Twitter, and big guys in the industry liked it one after another.. For more information, please follow other related articles on the PHP Chinese website!