Diagnostics & Imaging Week Washington Editor

GAITHERSBURG, Maryland – The role of clinical trials seems universally paramount and important for FDA device advisory panel members, and Tuesday’s meeting of the radiological devices committee for clinical trials for computer-aided detection (CAD) software for mammography equipment was no different.

FDA convened the meeting in pursuit of developing a guidance on the question, but Joyce Whang, FDA’s deputy director of the Division of Reproductive, Abdominal, and Radiological Devices (DRARD), cautioned that the agency may need all of two years to pound out a guidance on the matter.

Much of the discussion hinged on the role of CAD analysis without human interpretation – known as stand-alone CAD – in determining the viability of CAD systems. The current thinking of stand-alone CAD is that it generates a number of false positives, but that it is as accurate as human “readers” at detecting cancerous lesions when the focus is on specific areas of the breast.

Robert Smith, MD, a DRARD medical officer, reminded the panel that CAD is intended to “reduce errors when interpreting screening mammograms” by aiding the detection and description of abnormal tissues and that the description part of the task “directly drives diagnosis.”

“The primary purpose [of CAD] is to reduce errors” in diagnosis, Smith said, the potential errors including ones that are “perceptual” – those based on a failure to notice an image that conveys a tissue abnormality – and those that are “cognitive” — the failure to properly categorize an image.

Thomas Gwise, PhD, an FDA biostatistician, said that the two prime considerations for diagnostics are sensitivity and specificity, and that they are “not comparable if estimated in separate studies.” And he said that most submissions for CADs have designated software use as a second reader behind the radiologist.

Gwise said that while a prospective study using controlled, randomized trial would establish whether a CAD system enhances sensitivity and specificity, such a study “would result in large amounts of time and large enrollments.”

A retrospective study, on the other hand, would require feeding the existing scans through a CAD system and comparing the outcomes to the established diagnoses.

But he said that such a study might be biased if the readers were less attentive to the task based on the understanding that “their readings do not matter to the patient,” Gwise said.

A study based on “sequential reading is susceptible to potential reader bias” as well, Gwise said, because though such a study would have the same clinician read the raw scans first and then the CAD-analyzed scans, “a reader may subconsciously want to compete with the CAD device and search more aggressively for lesions.” Such a study could include a washout period between readings to cut down on this effect.

A trial involving human readers “should represent the real population of readers,” Gwise said, and that a small sample of readers might not sufficiently capture this range.

Robyn Birdwell, MD, speaking on behalf of the Society for Breast Imaging (Reston, Virginia), also weighed in, saying that regardless of any apparent uncertainty over the value of the data in the published literature, “in my opinion, CAD does reduce false positives.” But she said that despite intensive training of radiologists, “we miss cancers.”

Any tool that increases sensitivity “is a good idea,” said Birdwell, and that CAD can help with observational oversights that occur despite more sensitive equipment.

CAD “reminds us to look again,” Birdwell said, to catch the tumors that are visible but often overlooked without the computer’s diagnostic boost. She asked rhetorically whether CAD prompted more recalls of patients that end up being negative findings all the same.

“Yes, [but] if you want to find more, you have to do more,” she said.

Birdwell, the section head for breast imaging at Brigham and Women’s Hospital (Boston), said that the data show an increase in cancer detection of more than 19% and an improvement in sensitivity of 11%. “The preponderance of evidence shows that the current FDA-approved devices ... increase detection” and that the current “FDA review process works for women.”

The first question FDA had for the panel was to provide some clarity on the role of stand-alone performance testing in the clinical evaluation of CAD devices.

The replies to this ranged from general agreement on “the importance of stand-alone testing” to comments that the scoring methodology for such trials should be standardized.

Acting panel chairman Leonard Glassman, MD, a radiologist with the Washington Radiology Associates (Washington), posed the question of whether “stand-alone [is] good enough for minor changes” to CAD software, and then responding that “the answer is probably yes.”

But he was reluctant to set a threshold of change before comparisons with human readers are required. “Its more of a judgment call based on what FDA knows about the algorithm,” he said.

After further discussion, Glassman said that the consensus of the panel seemed to be that “[s]tand-alone testing is necessary but not sufficient,” for approval of any new CAD system, describing it as unbiased because it cuts out human variability.

“While all the endpoints were useful, the sentiment was that the per-lesion and per-view were most useful” in determining a CAD system’s viability.

Another question raised by FDA was whether CAD software written for a specific FFDM configuration would need extensive testing to evaluate its use in other FFDMs.

Glassman said that the earlier in the process at which CAD software intervenes, the more weight the question bears.

After discussion, he said that “the sense of the committee is that we don’t know enough about the vagaries of image processing and the link between CAD and FFDM. And if the link is such that CAD is late in the process, a stand-alone processing might be enough” to validate the use of the software on a different machine.

“However, if it is early in the digital process, reader studies may be needed,” Glassman said.