Data flaws sink current AI models for COVID-19 diagnosis
Researchers at the University of Cambridge have analyzed more than 60 AI-based models for the detection of COVID-19 based on chest X-rays or CT scans that were published, in peer-reviewed journals or as preprint, in the first nine months of 2020, and reached the sobering conclusion that none of them were appropriate for clinical use, due to the universal presence of “methodological flaws and/or underlying biases.” In their paper, the team listed a number of such issues, including the use of inappropriate comparison groups. One frequently used dataset, for example, was from children ages 1 to 5 with pneumonia, making it likely that the models were learning to distinguish adults from children rather than pneumonia from COVID-19. Other datasets were what the authors called “Frankenstein datasets,” where the reuse and repackaging of publicly used datasets leads to training and testing of an algorithm on datasets that are believed to be distinct, even though the raw data within them overlap. They also pointed out that publicly available images are likely not representative of COVID-19 cases in general, as “it is likely that more interesting, unusual or severe cases of COVID-19 appear in publications.” The authors included several recommendations for data curators, machine learning researchers, manuscript authors, and reviewers to improve data and publication quality. The reported performance of current models, they wrote, is “highly optimistic... In their current reported form, none of the machine learning models included in this review are likely candidates for clinical translation for the diagnosis/prognosis of COVID-19.” The review appeared in the March 15, 2021, issue of Nature Machine Learning.
Smartphone-based gaze may provide a scalable, digital biomarker of mental fatigue
Mental fatigue can affect alertness and wellbeing, but existing fatigue tests are often time-consuming and subjective. Using validated fatigue metrics such as Brief Fatigue Inventory (BFI) and NASA Task Load Index (NASA TLX), a team at Cornell University and Google Research leveraged advances in smartphone-based eye tracking to see if smartphone-based gaze can detect mental fatigue. To test their hypothesis, subjects performed a series of tasks over the course of about an hour, starting and finishing with time-consuming gold standard tests to measure their level of alertness, focused attention and mental fatigue. Each block of tasks was followed by a brief fatigue questionnaire to measure the progression of mental fatigue during the study. Two types of fatigue-causing tasks were employed: a language-independent, object-tracking task and a language-dependent, reading task. “Analysis of gaze behavior shows significant gaze impairments with mental fatigue. Gaze features such as entropy, mean and standard deviation in gaze error (computed as the difference between gaze vs. actual target position) increase significantly with mental fatigue,” the authors wrote. Given the ubiquitous nature of smartphones, “a smartphone-based digital biomarker could provide a scalable and quick alternative for detecting mental fatigue.” Their findings appeared online March 11, 2021, in npj | digital medicine.
AI developed to predict disease based on genetic mutations
Tokyo-based Fujitsu Ltd. and a team of researchers at Kyoto University have developed an artificial intelligence (AI) verification system for estimating the disease-causing potential of genetic mutations, including mutations with unknown pathogenicity. Knowing whether a genetic mutation can cause disease is essential, but only a fraction of the myriad possible genetic mutations have been linked to disease. When the genetic mutation information is entered into the AI verification system, called MGeND Intelligence, its disease-causing capacity is estimated by pathogenicity estimation AI using Deep Tensor machine learning technology, and text explaining the basis of the finding is displayed with the estimated result. “The degree of fit of each explanation pattern is calculated form the estimation factor obtained through the ‘Deep Tensor,’ and a sentence explaining the basis of the finding is generated based on multiple patterns with a high degree of fit,” the research team said. The system also includes a literature search for retrieving related articles. MGeND Intelligence draws from the Integrated Database of Clinical and Genomic Information program managed by the Japan Agency for Medical Research and Development (JAMRD), and beginning in April, Kyoto University plans to offer the researchers and institutions participating in JAMRD. Future plans include enhancing the system’s explanatory functions and developing new functions to support genomic medicine in hospital information systems, primarily through use of electronic medical records. Fujitsu intends to introduce the functions in hospitals and medical institutions throughout Japan.
High blood fats, sugars alter antigen processing
Researchers at the University of Massachusetts Medical School and Weill Cornell Medical College have identified a link between hyperglycemia and hyperlipidemia, or high levels of blood sugars and blood fats, and antigen processing in type 2 diabetes. One consequence of hyperglycemia and hyperlipidemia is that individual sugar and lipid molecules are more likely to react with each other even in the absence of enzymes to catalyze those reactions. The authors took a closer look at those reactions and their consequences, and showed that oxidation affected “key components of the major histocompatibility complex (MHC) class II molecule antigen processing and presentation machinery in vivo under conditions of hyperglycemia-induced metabolic stress.” In particular, processing and presentation of the antigen APOB100, which has been linked to diabetic and metabolic syndrome complications, was altered. They concluded that “these findings highlight a link between glycation reactions and altered MHC class II antigen presentation that may contribute to T2D complications.” Their work appeared in the March 15, 2021, online issue of Immunology.