As a signatory of Publish Your Reviews, I have committed to publish my peer reviews alongside the preprint version of an article. For more information, see http://publishyourreviews.org.
I very much enjoyed reading this thoughtful paper in which the author discusses pros and cons of preprinting, with a special focus on the field of AI ethics. As a strong advocate for preprinting, I am glad to see that the author has decided to preprint his paper. This has enabled me to publish this review report.
I appreciate the detailed and careful overview that the paper offers of (potential) advantages and disadvantages of preprinting, and I believe the guidelines proposed by the author can serve as a good starting point for further discussion. However, I also have an important concern. In my view, the dichotomous way in which the author distinguishes between preprints on the one hand and articles in peer-reviewed journals on the other hand is artificial. There are two reasons for this.
Firstly, an increasingly large number of services offer peer review for preprints. Prominent examples include eLife, Peer Community In, Review Commons, and PREreview. F1000Research and Open Research Europe may also be seen as examples of peer review services for preprints. While most of these services focus primarily on research in the life sciences, similar services could be set up for AI ethics research. This shows that we do not necessarily need journals to organize peer review processes. Instead, these processes can be organized around preprints, and this could be a way to address many of the objections often raised against preprints. For a further discussion of developments in the area of preprint peer review, please see https://doi.org/10.31219/osf.io/cht8p.
Secondly, I believe it is important to recognize the huge heterogeneity in the rigor of the peer review processes offered by journals. Some journals offer highly rigorous peer review processes, while many other journals offer peer review processes that are much more superficial, and so-called predatory journals even perform no peer review at all. When a journal offers only a relatively superficial peer review process, it is likely that many flaws in research articles will not be noticed during peer review, and therefore publishing articles in these journals may have disadvantages similar to the disadvantages of preprints discussed in Section 3. In fact, I would even argue that the disadvantages of publishing in these journals may be bigger than the disadvantages of preprints. In the case of a preprint, readers will typically be aware that the research has not been peer reviewed (unless peer review has been performed by a preprint peer review service like those mentioned above). In contrast, in the case of a journal article, readers may be under the impression that the article has gone through a rigorous peer review process and that the research can therefore be trusted, while this may in fact not be the case. The large numbers of retracted journal articles clearly show that we cannot trust that journals perform rigorous peer review (https://doi.org/10.1038/d41586-023-03974-8). Research published in a journal might be as unreliable as research published on a preprint server.
Because of the above two reasons, I think the idea of a sharp dichotomy between preprints and journal articles is questionable. A more nuanced perspective is needed. The publishing landscape offers a large variety of publication platforms (both preprint servers and journals). These platforms differ in the level of rigor of their quality assurance process, ranging from no quality assurance at all to highly rigorous in-depth peer review. They also differ in the level of transparency they offer about their quality assurance process. Many platforms offer a description of their quality assurance process that is very general (“we perform rigorous peer review”) and not necessarily very accurate (e.g., journals may boast about the rigor of their peer review process). Other platforms are more transparent, for instance by publishing review reports openly. Discussing the pros and cons of preprinting relative to publishing in peer-reviewed journals doesn’t do justice to the complexity of the publishing landscape. I would prefer to see a discussion of the pros and cons of the different levels of quality assurance that publication platforms provide. For instance, what are the pros and cons of publishing research in a relatively rapid way (either on a preprint server or in a journal), but with only a (very) limited level of quality assurance? And what are the pros and cons of opting for more rigorous quality assurance, and would it be preferable for this quality assurance to be performed before the research is published (the journal model) or after the research is published (the preprint peer review model)? And what level of transparency should be provided, both by publication platforms and by authors, about the level of quality assurance that an article has undergone?
In addition to my main concern discussed above, I also have a few minor comments on the paper:
“All the benefits of preprinting apply to the AI ethics discipline, but it is likely that technically focused researchers and research have greater benefits than the more philosophically oriented research. The need for speed – for both author and audience – is arguably greater when new methods, algorithms, etc. are researched, than when a new way to apply the philosophy of John Rawls is proposed for the nth time.”: I am not sure if this argument is fully convincing. Couldn’t one just as well make the opposite argument? Suppose we have a philosophical paper pointing out why the use of ChatGPT is deeply questionable from an ethical point of view and a technical paper proposing a minor adjustment to one of the many technical definitions of the notion of fairness. In this case, wouldn’t the need for speed be greater for the philosophical paper than for the technical one?
“If your ideas are of the kind that requires a particularly extensive process of review and discussion, preprinting might allow for a broad “review” process before you finalize the research. However, this can also be achieved by using blogs or other outlets”: Indeed this can be achieved either using preprints or using blogs, but does this constitute an argument against preprinting and in favor of blogging? Perhaps it offers an argument in the opposite direction. Preprinting has the benefit that an article can be expected to remain available in the long term, that an article has a persistent identifier (usually a DOI), that an article can be found through scientific search engines, that an article can be updated without earlier versions of the article getting lost, etc. Blogging doesn’t have these benefits. All these benefits can be of value to support an ‘extensive process of review and discussion’.
“you should know that your research will always have social implications, so be sure to consider the potential for your non-reviewed research to misdirect policy or practice in various fields.”: I agree that researchers should consider the social implications of their research, but I think it is problematic to relate this specially to preprints. This point is equally important for research published in peer-reviewed journals. In fact, as already mentioned above, publishing in a journal with superficial peer review might be more risky than publishing on a preprint server, since readers may not be aware that the journal performs only superficial peer review and may therefore put too much trust in the research.
“One issue of great importance is how we can all help each other navigate and make use of a sea of increasingly non-reviewed research”: Again, I think this is a bit one-sided. An equally important issue is how we can navigate and make use of a sea of articles that may seem to have undergone rigorous peer review, while the peer review process was actually quite superficial.
Creative Commons Attribution 4.0 International License (CC-BY 4.0)