Findings of EMNLP: Responding to ACL feedback
The ACL 2020 reviewing meeting featured a lively discussion of the proposed changes to reviewing for ACL affiliated conferences. Thanks to all who participated for your questions and comments!
Part of the discussion centred around “Findings”, a new publication outlet being trialed in EMNLP 2020, which is based on a proposal developed by the ACL review committee (a.k.a. “ACL archives”). This post is an attempt to close the loop, by responding to criticisms raised by the community.
Common concerns raised about Findings include:
Will the presence of Findings empower reviewers to dismiss papers from the main conference, which otherwise would have been accepted?
We don’t expect this to happen, but acknowledge the risk. Based on this feedback, we have decided NOT to involve reviewers in the Findings decisions, i.e., there has been no change to the questions in the review form. Instead, this decision will lie solely with the senior programme committee. We will take care to ensure decisions for main conference acceptance are not influenced by the decisions about Findings.
Can we define more precisely what separates a Findings paper from a main conference paper?
While we would love to, it is very difficult to pin down. We have some broad guidelines in our earlier blog post, and a clearer definition might be possible in future. For the time being, we hope that the differences will emerge organically, similar to the difference between talks and posters at a standard conference, which also lacks a concrete definition.
How will we measure the success or otherwise of the Findings experiment?
We plan to do this in two ways. Firstly, we will measure how many papers offered publication in Findings take up the offer. If the uptake is high, then we would judge the community to value this new publication venue. Delving deeper, we could seek to characterise aspects of submissions and their reviews that predict the authors’ decisions. Secondly, we will survey authors to assess their reactions to Findings, including those with Findings offers, those with main conference accepts, and those with rejections. We would be seeking to understand author preferences for Findings versus another chance at a big conference, a lower tier conference or a workshop, among other questions. Community suggestions would be most welcome.
What will the effect of Findings be on the workshops and other conferences?
People with a paper accepted into Findings may prefer to send it instead to a workshop or a subsequent conference. Many workshops are moving towards non-archival proceedings, with authors being allowed to present their work published elsewhere. Accordingly, EMNLP workshops will be encouraged to invite authors with Findings papers to present their work. Be warned that not every paper will have a good match to a workshop running at EMNLP, so there’s no guarantee that this will apply to all Findings papers.
Overall, Findings should not detract from the quality of work presented at the workshops. We acknowledge that Findings may impact conferences later in the conference cycle, but we feel that the benefits to authors and reviewers make this a worthwhile tradeoff.
Could Findings papers have some form of presentation / poster session in the main conference?
We are not planning to do so. If the schedule & technology permits, we might be able to allow for some form of short presentation, like the “poster boaster” and “one minute madness” sessions run in previous conferences. But we’re not committing to anything just now.
ACL has just run a survey on reviewing, which included questions on Findings/Archives. This feedback wasn’t referenced in the above post. The ACL reviewing committee is in the process of working through this feedback, and will await the outcomes of the EMNLP trial before taking further actions on Findings/ACL Archive.
After the EMNLP trial is complete, we will have a better idea of the community’s view. As it stands, we are not sure if this will be a one-off, or continue as part of future conferences, as originally planned.
- EMNLP Programme Chairs, with input from Tim Baldwin