Nicolas LIMARE / pro / notes / 2013 / Upgrading Journal Policies via Peer-Review Influence

we need an upgrade of the computational research communication standard

In all the recent meetings about "reproducible computational research etc."^2, I have come to the personal conclusion that good practices in computational research (be it publishing and citing software, adding a reproducibility statement to every article, etc.) won't be adopted out of the small group of believers unless they are implanted at the journal level.

ICERM Workshop on Reproducibility in Computational and Experimental Mathematics (Providence, USA, 2012-12) and the SINTEF eVita Winter School on Reproducible Science And Modern Scientific Software (Geilo, Norway, 2013-01), both great creative and motivating events.

Unless journals define and enforce good practices, researchers won't shift their standards because it's a prisoner's dilemma. As a reader, everyone is interested in reliable, proved, verifiable and reusable articles, but as an author it is less time and effort consuming to follow the current practices. It will always be easier and quicker to write a half-baked paper with hand-picked examples, doctored illustrations, misleading benchmarks and secret and buggy implementations, while you can always benefit from reading the works of others who would go by great length to try to provide something better.

The bio-medical research community is reviewing its standards after a few catastrophic cases of fabricated data and erroneous protocols. In bio-medical research, people can die if you do it wrong. Do we need to wait for a nuclear meltdown or bridge collapse to review the computational research communications standard?

but publishing policies are very slow to evolve

The scholarly publishing community is an old lady, attached to her habits. The traditions, values and procedures of multi-centenary scientific societies and multinational publishing companies don't change unless they are forced to.

Just looking at the publishing format is interesting: we have been in the age of digital communication for 20 years and a research communication still is a "paper", usually a PDF file, not much more than the electronic rendering of a physical paper. While the web has been reinventing itself every 3 or 4 years for two decades, for better or for worse, the publishing society is just starting to think about the future of research communications. And this is just for the "thinking about" step, because it will take decades before a new medium is adopted. We can also think about how big publishers had to be kicked by aggressive new players like PLoS to start seriously considering Open Access options.

Policies are changing for good, but sooo slowly. SIAM is reviewing its definition and management of supplementary materials, ACM updating the copyright status of software attached to articles. Nice, positive changes, but so epsilon-esque when compared to the revolutions I am waiting in the editorial policies.

it could be done by you/me/us, the peer-review crowd

The inertia is one of the motivations behind IPOL: instead of waiting forever for journals and papers to improve, let's make our own journal based on the values we defend, experiment, refine our vision by a dive into the reality, build the tools we couldn't guess we would need until they are lacking, and finally show that new standards are possible.

But this is not enough. We need to spread the word, and have every author understand that some colleagues are expecting more. So, here comes the main idea:

Enhanced publication quality standards do not need to be adopted by editorial boards first. Any reviewer sensible to the need for quality in computational research can right now mention it in their peer-review reports. Without hindering the acceptance of articles, asking for minor revisions can include the need for the addition of a (non-)reproducibility statement, the communication of an essential piece of code or data, or more details about the computing environment, will gradually raise the standards for published papers.

Young researchers and post-docs are often put to use for peer-reviews and it seems from the profile of reproducible research meeting that they are more sensible to these issues than the average academic. Peer-review could be the lever we can access in the big machine of scholarly publishing.