Two studies seek to answer the most pressing question for physicians examining a patient with COVID-19: What's this person's risk of death? Mount Sinai researchers presented their clinical prediction model in The Lancet Digital Health and a team from Johns Hopkins published their risk calculator in the Annals of Internal Medicine. Both use just a handful of readily available patient information to determine the risk of rapid progression and death with a high degree of accuracy.
"Predicting mortality among patients with COVID-19 who present with a spectrum of complications is very difficult, hindering the prognostication and management of the disease," said lead author Gaurav Pandey, assistant professor of Genetics and Genomic Sciences at Mount Sinai School of Medicine in New York. The trajectory of the disease varies widely, with some patients presenting with acute shortness of breath who recover quickly and “happy hypoxics” who require immediate ventilation. Other patients are first diagnosed after suffering heart attacks or strokes. Some patients never develop symptoms; others die within hours of diagnosis; many take weeks to recover.
“Notably, accurate prediction of clinical outcomes for patients across this spectrum of clinical presentations can be difficult. This problem presents an enormous challenge to the prognostication and management of patients with COVID-19, especially within disease epicenters that need to triage a high volume of patients,” the authors said.
The Mount Sinai model is breathtakingly simple. While the researchers initially evaluated 17 factors, the final model uses just three clinical features – patient age, minimum oxygen saturation over the course of the medical encounter, and type of encounter (inpatient, outpatient, telehealth) – and a systematic machine learning framework to predict mortality.
The three “variables were individually pretty discriminative between the deceased and alive classes of patients,” Pandey told BioWorld. “We were excited that a rigorous machine learning framework and large dataset objectively determined that a combination of these three variables could discriminate between these classes even more accurately.”
The team suggested that the model could function as an additional “vital sign” that could be regularly assessed for patients with SARS-CoV-2 infections. The researchers were keenly aware of the need for a quick and reliable model having witnessed the massive surge in patients and the extraordinary death rate from COVID-19 in New York this spring. Rapidly identifying patients at high risk of death would allow clinicians to focus treatment and resources appropriately and minimize mortality.
“We used several established data analysis and machine learning techniques, specifically for (i) imputing missing values using available data, (ii) automatically selecting the most predictive subset of features, and (iii) developing prediction models from the selected features,” Pandey explained. “The resultant model generates the probability of a patient’s mortality due to COVID-19. Using this rigorous machine learning framework allowed us to objectively identify a parsimonious and accurate prediction model of this outcome directly from routinely collected clinical data.”
To develop the XGBoost algorithm, the team analyzed patient-level data from 5,051 patients treated in the Mount Sinai system of eight hospitals and 400 ambulatory practices between March 9 and April 6, 2020. The researchers used 3,841 records for the development dataset to create the mortality prediction model using machine learning. A test dataset included 961 patient records and the algorithm was prospectively evaluated on 249 patient records. The model demonstrated high accuracy, with an AUC of 0.91.
The study had the advantage of using the largest dataset and beginning with a wide range of clinical features, leading the authors to conclude that “the ones that we identified to be most strongly associated with mortality are more objective and accurate” than previous models.
The Johns Hopkins model
The team at Johns Hopkins sought to provide greater gradations from their model, which predicts how likely a patient’s disease is to worsen and when progression is most likely to occur. They found that a larger number of factors, all available on admission, told the likely story of a patient’s trajectory better.
The Johns Hopkins model uses age, nursing home residence, comorbid conditions, obesity, respiratory symptoms, respiratory rate, fever, absolute lymphocyte count, hypoalbuminemia, troponin level, and C-reactive protein level and the interactions among these factors. Together these factors predict in-hospital disease progression with an AUC of 0.85 at day two and 0.79 at days four and seven.
The Johns Hopkins researchers developed the model using data from 827 consecutive COVID-19 admissions to five Maryland and Washington, D.C., area hospitals between March 4 and April 24, 2020, with follow up until June 27, 2020.
Like the Mount Sinai researchers, the team at Johns Hopkins was acutely aware that time was of the essence when dealing with a patient with COVID-19. The authors noted that just 45 patients had severe COVID-19 at admission, but 120 developed severe disease or died within 12 hours of being admitted. Of the 302 patients in their study who died, the median time to progression was 1.1 days.
"Rapid progression of disease following admission [to the hospital] provides a narrow window to intervene," lead author Brian Garibaldi and colleagues wrote. "Different combinations of risk factors appear to predict severe disease or death, with probabilities ranging from over 90% to as little as 5%."
"This is some of what we've learned in the months since we started seeing patients with COVID-19 at our hospitals," says Garibaldi. "As we continue to grapple with high numbers of COVID-19 infections across the United States, it's important to share knowledge with our colleagues at other hospitals." The model, called the COVID Inpatient Risk Calculator (CIRC), is available online.