Two major papers on repurposed drugs for COVID-19 by researchers from Brigham and Women’s Hospital face intense skepticism from the research community, prompting the editorial boards of The New England Journal of Medicine and The Lancet to publish Expressions of Concern on the validity of the underlying data. Such expressions are often the first step toward an outright retraction.*
There is comfort to the fact that the scientific method is doing what it is designed to do – self-correcting.
And in a pandemic, the speedy dissemination of even preliminary information is of the essence. The World Health Organization’s daily situation report for June 3, 2020, listed more than 3,600 deaths in the preceding 24 hours, and roughly 380,000 total deaths since the beginning of the outbreak.
But the papers are disturbing because in their rush to publication, the journals violated standards that make scientific self-correction possible in the first place.
Open letters signed by hundreds of scientists pointed out multiple issues with the papers, ranging from a lack of ethical oversight, to claims about the data used in the analyses that are mathematically all but impossible.
Among the concerns noted in the open letter about The Lancet paper was the fact that the “the authors have not adhered to standard practices in the machine learning and statistics community. They have not released their code or data. There is no data/code sharing and availability statement in the paper.” The same is true for the paper published by the same research team in the NEJM.
The 2016 Wellcome statement was drafted in response to the Zika and Ebola public health crises of the time, and addresses the central tension of emergency science – emergencies require rapid responses, while good science takes time.
Publication of the paper led to interruptions in multiple randomized controlled trials, including WHO’s Solidarity trial, which paused enrollment in its hydroxychloroquine arm in response to The Lancet publication. The WHO announced on June 3 that it was reopening the arm, effective immediately.
In emergencies, observational trials and data mining of existing trials are a reasonable method to rapidly get a sense of whether an intervention might be useful. But they are not a substitute for ultimately conducting randomized controlled trials.
The Lancet paper, published online on May 22, 2020, claimed worse outcomes for COVID-19 patients treated with chloroquine or hydroxychloroquine. The NEJM paper, published online on May 1, 2020, asserted that the use of angiotensin-converting enzyme (ACE) inhibitors and angiotensin receptor blockers (ARBs) did not worsen outcomes for COVID-19 patients.
Both papers were co-authored by the founder of Surgisphere Corp., a company that bills itself as “a leader in health care data analytics and medical education” on its Linkedin page. The data used in the papers were contributed by Surgisphere.
And when they are compared to publicly available data, they appear to be questionable.
For example, in their open letter about the NEJM paper, the signatories note that “Mehra et al report electronic patient data from 706 patients hospitalized with PCR confirmed COVID-19 in just 7 of the UK’s 1,257 [National Health Service] hospitals. A high proportion of patients hospitalized in the UK on March 15th were in London, and yet no London borough, let alone hospital, had more than 100 PCR positive confirmed cases by this date,” making it mathematically highly implausible that any seven U.K. hospitals could report 706 PCR-confirmed cases.
Co-author and Surgisphere CEO Sapan Desai has defended the data, and said he is working on nondisclosure agreements that will allow the NEJM editors access to the data, which are governed by confidentiality agreements.
And Mandeep Mehra, medical director of the Brigham and Women’s Hospital (BWH) Heart and Vascular Center and the corresponding author on both papers, released a statement announcing an independent audit.
“Upon completion of the reviews, the auditor will simultaneously provide its findings directly to the editors of the journals and to the co-authors, independent of Surgisphere,” Mehra said.
However, an independent audit still violates the terms of the “Statement on data sharing in public health emergencies,” published in 2016 by the Wellcome Trust and signed by both The Lancet and the NEJM, to “require researchers undertaking work relevant to public health emergencies to set in place mechanisms to share quality-assured interim and final data as rapidly and widely as possible, including with public health and research communities and the World Health Organization.”
The Lancet’s Expression of Concern acknowledged that “important scientific questions have been raised about data reported in the paper.”
Meanwhile, as of 11 am on June 3, the paper itself was still an “Editor’s Pick” on The Lancet home page. Later in the day, the paper appeared to have been removed, but as of press time no link to the Expression of Concern could be found on the home page.
The NEJM was more forthcoming, with its Expression of Concern displayed on the journal’s home page.
*Researchers later retracted both papers on June 4, 2020.