Reviewing Policy for EMNLP
Every year sees the number of papers submitted to EMNLP growing sharply. Accordingly the work is also growing for the programme committee (PC), who need to provide a careful and reasoned assessment of every paper, and decide which of these should appear in the conference. Although recent global affairs will have an effect on submissions, we still anticipate another record year in terms of numbers of submissions, and thus need an ever larger troupe of qualified, engaged and committed individuals to serve as reviewers on the PC to handle these papers. COVID-19 adds a further spanner in the works by generally increasing the stress and workload on many in the field, further complicating the reviewing effort.
For this reason, in EMNLP 2020 we are introducing a new policy:
In order to submit paper(s) to EMNLP, you must nominate at least one author to serve as a reviewer, (usually the most senior author) and for that author to take on a full load (of up to 6 reviews).
This applies to every paper submitted to the conference, and everyone in the field, even those with significant roles in the ACL community (with a few limited exceptions, listed below). Papers where no authors have volunteered to review, will be rejected without review.
This is related to, but different from, the ACL 2020 policy, which required all authors to sign up as reviewers. The key differences are that for EMNLP we ask for only one reviewer, the reviewer will be required to take on a higher number of papers (hence the need for an experienced author), and we will be enforcing penalties for non-compliance.
The reviewing process needs to run to a tight deadline, with a limited time period for review, reviewer discussion and author response. All reviewers must ensure they meet these deadlines. We reserve the right to reject the papers of reviewers who do not meet their review deadlines.
While we expect most authors to adhere to the above policy, we recognise that this is not always possible. For this reason we will allow some exceptions to the above rule, including
- Papers with junior authors only, or other first time authors who are not qualified to review; and
- Papers with senior persons taking on an organising role in EMNLP, or a significant current role in the ACL community.
This list is not exhaustive, however we really would like for everyone to contribute. This new requirement is necessary to get a sufficiently large cohort of experienced reviewers for the PC to function during this difficult time. Excuses such as "reviewing burnout", having served as an AC for a recent conference, and similar are not sufficient.
Senior persons covered by option 2 above will be asked to suggest some well-qualified alternative reviewers, or better, agree to review anyway, which they facilitated through sub-reviewing mentoring junior researchers or colleagues in their lab/organisation (as detailed below).
Multiple submissions. For those submitting multiple papers to the conference, we will not insist on a unique author per submission to be nominated as a reviewer. But we would ask that you put forward as many reviewers as possible to adequately handle the additional reviewing load your submissions incur. Reviewers may ask for an increased load beyond 6 papers, by emailing emnlp2020programmechairs@gmail.com
Which author(s) to nominate? While this is up to you, we ask that you nominate the author(s) with the most relevant track record to the EMNLP community, which will most often be the senior author. These individuals will be the most able to provide high quality reviews, of the standard expected by EMNLP. We understand that these senior authors tend to be the busiest, and may already be involved in other service roles for EMNLP or the ACL community, raising the question of:
How to handle the reviewing effort? We encourage senior authors to use "secondary reviewers", who are typically younger and less experienced members of your research group, to perform the reviews. For example, these might be students in your research group, for whom reviewing is a novel and exciting experience. Your role would then be as a mentor, to guide them to produce good and detailed reviews, and to calibrate their expectations against the questions in the reviewing form. Assessing research quality and providing constructive and detailed feedback is an important learning experience for all researchers, and we would invite you to grow the skills in your group by mentoring your team.
What makes a good review? Stay tuned to this blog, where we'll tackle this question in a later post. We'll provide guidance on what to strive for in your reviews, to best serve the goals of the conference, in accepting only the very best work, and the authors, in providing detailed and constructive feedback on their submissions. On top of this we will recognise the efforts of excellent reviewers, more details to follow.
Registering as a reviewer. To indicate if you can serve as a reviewer for EMNLP 2020, please visit the Softconf EMNLP site to fill in reviewer survey form at the local user profile page.