Medical Device Daily Washington Editor

GAITHERSBURG, Maryland – Device makers know that there is no such thing as a simple PMA. And Wednesday’s meeting of the Radiological Devices Advisory committee did nothing to change that assessment as the panel communicated clearly that the physician community wants industry to hew to an increasingly higher standard for PMA data submissions.

The hearing also demonstrated that FDA is increasingly mandating that firms test their devices for off-label use as a condition for approval, even though such a requirement is not legal (see sidebar, p. 7).

In an introductory briefing, Sophie Paquerault, PhD, of FDA’s Office of Device Evaluation, gave a thumbnail sketch of the agency’s concerns regarding CAD systems in detecting neoplasms of the lungs.

Paquerault said that more than 250 million chest X-rays are performed each year in the U.S. and computerized tomography (CT) chest exams number 10 million a year. Clinicians then must sort out the important from the unimportant images because most patients who exhibit solitary nodules in the lung “rarely have symptoms related to the nodule.” And then “a positive CT typically results in additional radiation dose,” which carries risks as well.

Interpretations of chest scans is no simple matter because while a CT scan yields more data, these scans can be “overly burdensome because [producing] large number of individual images,” Paquerault said, and that accuracy varies “greatly varies with physician experience.”

Unfortunately, even a computer-assisted detection (CAD) system does not bring the diagnostician into a perfect world, according to Paquerault, who noted that the literature shows that CAD systems are associated with “up to five false positives per view on chest X-ray” and “up to 10 false positives per patient on chest CT.”

FDA’s concerns regarding physician interpretation hinge partly on the fact that the typical human lung exhibits a range of abnormalities other than lung cancers. Another problem in the case of standard chest X-rays is that the large amount of data and the prompts provided by CAD software can cause the radiologist to stop reading a set of scans before completing an exhaustive review.

Paquerault said a consideration for CADs used to process CT scans is that they can blunt the appetite for examining findings “unrelated to the findings the CAD [procedure] was designed to detect.”

The validity of the stand-alone test for CAD applications assumes that the areas highlighted by the CAD software represent all areas of potential concern but the rate of false positives is still an issue. As a result, PMAs for such systems often incorporate a sequential read, consisting of an examination of the raw image followed by one of the CAD image.

Paquerault said that the results of such a trial “are highly dependent on nodule size and location,” as well as whether the original image was digital or film. Co-morbidities, of course, can also affect the accuracy of a CAD analysis.

Paquerault said that in FDA’s opinion, “it may be necessary that the testing encompass all reading paradigms when using a chest X-ray CAD” because of lack of uniformity in the use of sequential versus concurrent readings of scans and because “chest CAD indicated for use as a second reader may not be adequately controlled at the software level.”

The first question FDA posed to the panel was whether a definitive diagnosis of lung cancer and the specifics of its spread (known as “ground truth”) have to be established in order for the study of the CAD to be valid.

Several members of the panel reiterated concerns over the difference in performance between human readers in establishing a diagnosis and how those differences might affect a study.

Georgia Tourassi, PhD, associate professor of radiology at Duke University Medical Center (Durham, North Carolina), said she saw “a large degree of variance between radiologists” even when several are looking at a given set of scans together. “The number of radiologists needed to do such a ‘truthing’ needs to be more than two.”

Brian Garra, MD, vice chairman of the department of radiology at Fletcher Allen Healthcare (Burlington, Vermont), said “you can establish ground truth by just using imaging modality and being careful.”

The panel chairman, Leonard Glassman, MD, a radiologist with the Washington Radiology Associates (Washington, DC), summed up the panel’s discussions by saying that ground truth “will be defined by an expert panel for detection and diagnosis using the full knowledge they have, including pathology, follow-up CT for an abnormal X-ray, and PET for an abnormal CT.”

On the question of stand-alone performance testing in the PMA trial, Glassman said, “[W]e all agree that stand-alone testing is necessary” in broad terms, but that it should also be used in determining whether the software can distinguish between lung nodules and other co-morbidities, including emphysema, congestive heart failure and interstitial lung disease.

Because the rate of diagnosis of lung cancer in chest X-rays and chest CT scans is low, FDA inquired as to whether the scans used in a PMA study should consist of enriched data sets for evaluation of both stand-alone and reader testing. Enriched data would consist of a study population with higher disease prevalence than the population at large.

The panel discussed whether such a study group would cause the study to underestimate the rate of false positives, but this concern got little traction.

The size of the nodules was also discussed because of questions about whether the threshold of size, set at 4 mm at the low end, was appropriate. The panel concluded that enrichment was appropriate and that enriched cases should include several pathologies, including sarcoidosis and lung scarring, inasmuch as scars are sometimes confused with nodules.