EMNLP Episode I: A New Hope – a.k.a. “Findings of EMNLP”

An innovation for EMNLP 2020 is a new acceptance category, which will allow for more high quality papers (short and long) to be accepted than we can accomodate in the conference itself. We are creating a new sister publication, “Findings of ACL: EMNLP 2020” (hereafter Findings), which will serve as an online companion publication for papers that are not accepted for publication in the main conference, but nonetheless have been assessed by the programme committee as solid work with sufficient substance, quality and novelty to warrant publication.

These papers will be invited to be published with the stamp of peer review in a new Findings online companion, which will be indexed as part of the ACL anthology.

Papers appearing in Findings will follow the same timetable as for papers accepted into the EMNLP 2020 conference proceedings. That is, they will have the same camera ready deadline, and be available online by the time of the conference. These papers differ from papers accepted to the EMNLP conference in that Findings papers will not be given a presentation slot at the conference (neither talk or poster). Instead Findings should be considered as a form of online open access publication: papers appearing in Findings will count as full publications, and therefore cannot be submitted to other conferences or journals. Long and short papers will be distinguished in separate sections of Findings.

What kinds of papers will be accepted to Findings?

In addition to the existing review criteria of novelty, rigour, experimental soundness and so forth, reviewers will be asked to assess papers for “publishability” as an additional field in the review form. Reviewers (and chairs) will be asked whether the paper is of a suitable standard to be publishable in its current form, possibly with minor revisions based on reviewer feedback.

The idea here is to separate the paper ranking process that is used to select papers to accept for the main conference, from the classification of whether a paper has sufficient substance, quality and novelty to warrant publication. It is based on the assumption that there are significant numbers of rejected papers that are of a publishable standard, but for various reasons, could not be accepted into the conference.

Beyond simply ranking, certain kinds of papers are more appropriate for appearing in Findings, for instance:

• Papers that extend the state of the art on a particular focused task, but have few novel insights or Findings of broader applicability to the wider EMNLP community;
• Papers that have well-executed, novel experiments and present thorough analyses and Findings, but using methods that are not thought to be sufficiently “novel”; and

One common requirement is that the reviewers must agree that the paper is well written, makes an original contribution, has sound methodology, and includes appropriate analysis and conclusions. What sets Findings apart from the main conference papers is that there is no requirement for high perceived impact, and accordingly solid work in untrendy areas and other more niche works will be eligible. These requirements are based on the criteria for publication at PLoS One, which you can read about in more detail.

Why might you want your paper to appear in Findings?

Those with papers accepted to Findings will have the option of having the work accepted largely as-is, and being made available online promptly. This will be an attractive option for work that is particularly time-critical, e.g., the novelty of the work would be diminished if publication were delayed by being rejected from EMNLP. This will also help for papers caught by overlapping anonymity periods between conferences, which can preclude public release for a long period. In this case, Findings can offer the option of a quick publication. Note that authors whose papers are accepted for publication in Findings can, of course, opt out, by withdrawing or simply not submitting a camera-ready copy of the paper.

Many papers languish on arXiv without ever being published in a peer-reviewed venue because they were overtaken by the accelerating pace of NLP research. This new publication option will give more authors the chance to publish papers that are good and decent contributions at a given point in time, even if their long-term impact may be modest. This will also reduce the amount of unreviewed work in our field, and thereby increase quality.

Is this a one-off event?

Should this new avenue for publication be warmly received by the EMNLP community, with sufficient uptake, we anticipate that Findings will continue to operate alongside future iterations of EMNLP, and potentially other conferences in the ACL stable. Any future iterations would appear as different volumes of Findings.

We hope that having a paper appearing in Findings will be a mark of pride, and provide an important new avenue to have your work published in a timely manner.

– EMNLP Programme Chairs (based on ideas from Tim Baldwin & the ACL Reviewing Committee)

Addendum

We’ve been receiving considerable feedback on this initiative, and questions about the scheme. Please see our answers to several common questions and concerns:

Will papers that meet the criteria for Findings be excluded from the main conference?

No, many such papers make fine additions to the conference. Findings will take those borderline papers which would otherwise have been rejected, but that PC have assessed as being solid work. Reviewers will be asked some additional questions (exact wording yet to be decided) to aid the senior PC to make these decisions, and additional guidance will provided to educate reviewers, and to ensure reviewers don’t use this as an invitation to down-vote work they see as non-trendy, or other counterproductive behaviours.

Will this scheme exacerbate biases in reviewing and paper acceptance?

The reviewing process isn’t perfect, with several inherent biases. See for example, Ken Church’s papers on this topic, and broader issues with reviewing that led to the creation of EMNLP in the first place. We will do our best to mitigate these biases (e.g., overly conservative reviewing biased towards well established tasks, reviewers overly biased towards “trendy” tasks, or their own sub-fields) and ensure that the reviewing process is as fair as possible. This applies equally to papers for the EMNLP conference and Findings. Our primary focus will be on EMNLP conference papers, and only after these decisions are made, will we look at the remaining papers to identify those that warrant acceptance into Findings.

How will Findings papers be reviewed?

The same as for all other EMNLP papers, it’s just that in the final decision stages by ACs, SACs and the PCs we will have a new third category of decision, alongside accept and reject, which is to accept into Findings. Findings decisions will be made after we have decided the papers to be accepted into the main conference. The notification date will be the same for EMNLP conference papers and Findings papers.

How do I submit to Findings?

Just submit to EMNLP, and your paper will be assessed for both the EMNLP main conference and for Findings.

Should Findings be called a “journal”?

We updated the post to use the term “companion” instead of journal, as a neutral term as it’s neither a journal nor a conference proceedings (thanks to Yoav Artzi for the suggestion).

Will I get a presentation or poster presentation at the main conference for my Findings paper?

No, but we will be providing a means for match-making with workshops, where you might be able to present your work.

Will Findings be a recognised “journal”?

Not yet, but we will take steps to ensure it has its own section in the ACL anthology, with all papers accessible, and with its own ISBN. We expect that Findings will grow with future conferences (e.g., having a new volume associated with each conference), and become a recognised and respected publication venue within the field. Initially, we will not be seeking to have Findings indexed in Scopus and Web-of-Science etc - this is a complex process, and TACL is only now applying after 7 years of operation - but this is a natural option we might consider in future years.

Is there any advantage of publishing in Findings versus an EMNLP workshop?

This depends on the publication prestige, which we would expect to be higher for findings than for most workshops. Workshops offer a presentation slot, and Findings does not, however we expect that many Findings papers will be invited to be presented at specific workshops. Added to this, Findings papers will not require further review, and consequent delay in publication or extensions to the anonymity period.

Why should the authors agree to publish in Findings if they know the paper was only narrowly rejected from EMNLP, and so might get into another conference?

This is a judgement call to be made by the authors. Some authors may want to take the acceptance, rather than risk another rejection, especially as the paper was borderline. This might be more pertinent if the paper has already been rejected from another publication venue, or if the author needs to fill out their CV, e.g., a PhD student wanting to secure a postdoc or faculty position. Other reasons might be that the importance of the work may diminish with time, and the work needs to be published quickly to have an impact.

Will there be an option for papers to choose to go to Findings instead of the main conference, if they cannot travel to the conference?

We’ll duck this one, as it’s not relevant for an online conference. It’s an interesting idea, but stay tuned for the discussion of novel reviewing models in the ACL 2020 business meeting, which include similar features (which will include Findings.) Please bear in mind that this is the first trial of a new method for publication in the field, and there may be some shortcomings or risks that arise - as with any change. The proof will be in the pudding, and we discuss Findings in the ACL and EMNLP business meetings, where you’ll also have a chance to air your thoughts about Findings, and other topics relating to reviewing.