Reproducibility track

Reproducibility track

Similar to ECIR 2018 there will be a reproducibility track at ECIR 2019. While the general goal of reproducibility is well accepted, namely, to establish reliable, referenceable, and extensible research, ECIR 2019 wants to address this moving target more specifically by asking: How to assess the performance of technology when being applied in a different context (e.g., different time, location, access device, or task)? In addition to ``classical'' reproducibility research that analyzes the efforts and the success when reproducing (similarly: replicating) an experiment, we are soliciting contributions to develop a general theory for performance prediction. Such a theory may address the following problem areas: Measures (toward a better understanding of the assumptions and user perceptions underlying different metrics), Performance analysis (toward evaluation protocols that can explain performance), Application determinants (toward a theory to reduce the gap between test collections and real-world applications), and, Performance models (toward predictive models that simulate the transfer of a technology into a new context).

At ECIR 2019, the theme of the Reproducibility Track will be focused on assessing how well technology performs in new contexts (e.g., different time, location, access device, task, etc.), while also, soliciting ``classical’’ reproducibility papers that repeat and analyze prior work. We particular encourage papers that analyse and predict the performance and behaviour of algorithms.