As for R&R (which was not a paper), it should be ground for dismissing any such paper until data and calculations are made available. They were hardly trade secrets (which should not be a valid excuse anyway): they were national statistics...

Earth provides enough to satisfy every man's need, but not every man's greed. Gandhi

"Proper" peer review would require replicating such calculations.

Peer review should verify that the methodology used is not insane, that the paper properly references its data, that the author has performed adequate robustness and specification tests, and that the data is available to other investigators who wish to replicate the analysis.

It is possible to imagine cases where the analysis is based on data that cannot be made available to the general public for ethical reasons, or because doing so would be an unreasonable commercial loss for the source of said data. However, in those cases I would argue that journals should demand full independent replication rather than the much more cursory process of peer review.

The above is already a higher standard than current academic peer review observes, and I don't think going beyond this is realistic - or necessarily a desirable use of the reviewers' time.

Now, there's a whole issue of replication not receiving the recognition it ought to. But that is a slightly different matter, and one I think can be solved with standard governance methods, like formalized KPIs for researchers requiring them to publish two replications for each original result.

- Jake

Friends come and go. Enemies accumulate.