Title: Errors and secret data in the Italian research assessment exercise. A comment to a reply,
Abstract: Italy adopted a performance-based system for funding universities that is
centered on the results of a national research assessment exercise, realized by
a governmental agency (ANVUR). ANVUR evaluated papers by using 'a dual system
of evaluation', that is by informed peer review or by bibliometrics. In view of
validating that system, ANVUR performed an experiment for estimating the
agreement between informed review and bibliometrics. Ancaiani et al. (2015)
presents the main results of the experiment. Baccini and De Nicolao (2017)
documented in a letter, among other critical issues, that the statistical
analysis was not realized on a random sample of articles. A reply to the letter
has been published by Research Evaluation (Benedetto et al. 2017). This note
highlights that in the reply there are (1) errors in data, (2) problems with
'representativeness' of the sample, (3) unverifiable claims about weights used
for calculating kappas, (4) undisclosed averaging procedures; (5) a statement
about 'same protocol in all areas' contradicted by official reports. Last but
not least: the data used by the authors continue to be undisclosed. A general
warning concludes: many recently published papers use data originating from
Italian research assessment exercise. These data are not accessible to the
scientific community and consequently these papers are not reproducible. They
can be hardly considered as containing sound evidence at least until authors or
ANVUR disclose the data necessary for replication.