The code, script and documentation of the 16 March report, which is available on Github, was subject to an independent review led by Dr Stephen Eglen, from Cambridge’s Department of Applied Mathematics and Theoretical Physics.

Eglen co-founded Codecheck last year to help evaluate the computer programs behind scientific studies. Researchers provide their code and data to Codecheck, who run the code independently to ensure the work can be reproduced.

Last week, Codecheck certified the reproducibility of arguably the most talked-about computational model of the COVID-19 pandemic, that of the Imperial College group led by Professor Neil Ferguson. The model suggested that there could be up to half a million deaths in the UK if no measures were taken to slow the spread of the virus, and has been cited as one of the main reasons that lockdown went into effect soon after. However, the Imperial group did not immediately make their code publicly available.

Codecheck.org.uk provided an independent review of the replication of key findings from Report 9 using CovidSim reimplementation. The process matches domain expertise and technical skills, taking place as an open peer review. The reviewer conducts the codecheck and submits the resulting certificate as part of their review.

The results confirm that the key finding of Report 9 - on the impact of non-pharmaceutical interventions (NPIs) to reduce COVID-19 mortality and healthcare demand - are reproducible. Eglen did not review the epidemiology that went into the Imperial model, however.

In his analysis, Dr Eglen said: “Each run generated a tab-delimited file in the output folder. Two R scripts provided by Prof Ferguson were used to summarise these runs into two summary files... These files were compared against the values generated by Prof Ferguson...The results were found to be identical. Inserting my results into his Excel spreadsheet generated the same pivot tables.”

The codecheck found that: “Small variations (mostly under 5%) in the numbers were observed between Report 9 and our runs.” The codecheck confirmed the trends and findings of the original report.

Building in part on code originally developed, published and peer-reviewed in 2005 and 2006, the code used for Report 9 continues to be actively developed to allow examination of the wider range of control policies now being deployed as countries relax lockdown. The Imperial team is sharing the code to enhance transparency and to allow others to contribute and make use of the simulation.

Refactoring the code has allowed changes to be made more quickly and reliably, including incorporating new data that has become available as the pandemic has progressed.

In addition to the features presented in Imperial Report 9, further strategies can now be examined such as testing and contact tracing, which was not a UK policy option in March.

Users also now have the ability to vary intensity of interventions over time and to calibrate the model to country-specific epidemic data.

Adapted from a piece originally published on the Imperial College London website

Donate to support COVID-19 research at Cambridge