Science Editor

High hopes are riding on biomarkers. By stratifying patient populations and bringing the right drug to the right patient at the right time, the idea goes, biomarkers will be critical to the success of personalized medicine. As a result, there is a veritable parade of biomarkers in the scientific literature.

But a new paper rains on that parade. According to a report in the June 1, 2011, edition of the Journal of the American Medical Association, widely cited papers routinely overstate the connection between biomarkers and the diseases or outcomes they can supposedly predict.

In their paper, John Ioannidis and Orestis Panagiotou compared a total of 35 highly cited papers to meta-analyses investigating the same connection. In almost all cases – 29 out of 35 – the highly cited papers reported a greater relative risk of disease for those with a certain biomarker vs. those without it than the meta-analyses.

The papers that Ioannidis and Panagiotou analyzed were for more general markers, such as blood levels of certain proteins, and not for the precise mutations that have been instrumental to the success of drugs such as Gleevec (imatinib, Novartis AG) and PLX4032 (vemurafenib, Plexxikon Inc.).

Still, the problem is, if not universal, then not restricted to just a few indications, either. The papers the team analyzed included ones that looked at cancer, heart disease, diabetes and infectious diseases.

Senior author Ioannidis, who is professor of medicine at Stanford University, told BioWorld Today that the overstatement is due to "a constellation and combination of different factors.

"First of all, there are just too many scientists looking for biomarkers." Ioannidis estimated the total number of papers on biomarkers most likely exceeds 100,000.

With so many scientists scouring the body for biomarkers, some studies will find strong associations by sheer chance. And once they do, the idea that an association is stronger than it really is can become self-perpetuating.

"Associations that are stronger have a higher chance of being published in a better journal" and tend to be cited more because they report a strong effect in a good journal.

In addition, he said, among scientists "there is a tendency to focus on the most promising results, because it justifies what they are doing; it is research worth doing, and worth exploring further, and worth getting grants for."

Ioannidis said the problem was not unique to biomarkers, but biomarkers do tend to suffer from it more than some other research areas.

"There are some commonalities of this exaggeration of large effects," he said. At the same time, "biomarkers may be a special case because of the sheer size of the literature." The crowdedness of the field, in other words, "adds an extra layer to the distortion."

The authors of a paper reporting such a strong connection, of course, might argue that the reason their association is strong, and their paper is widely cited, is because the underlying research is excellent. Ioannidis acknowledged that such as assertion is "a fair argument, and it really has to be looked at on a case-by-case basis. In other cases it could be the opposite – it could be that the studies that find large effects are the worse ones," riddled with bias and errors.

Overall, his team found no evidence for the idea that studies that identified large effects don't look to be either better or substantially worse in their design.

But the real problem, he said, is that quite often, despite the fact that there has been "some effort" to standardize both methods and how they are reported, both methods and reporting are often "just not very transparent" in the final paper.

Here too, biomarkers are hardly alone – Ioannidis, in fact, was one of the driving forces behind a recent checklist of 25 items recommended for strengthening the reporting of genetic risk prediction studies that was published simultaneously in 10 scientific journals.

But, he noted, biomarker research is not terribly complicated from an experimental design or statistical point of view, which is often a straightforward comparison of the frequency of disease, or one particular disease outcome, between patients with and without the biomarker.

"You really need to do something stupid to get [those analyses] wrong," he said.

Nevertheless, he said, it is important to standardize experimental procedures and reporting, if for no other reason than that until such improvements are implemented on a broad scale "there will be no end to that sort of argument."

Ioannidis said his paper shows the necessity for prudence in using biomarkers in the clinic.

"We want to see some of them translated," he said. But before translation, it is important to make sure that what is being translated is a meaningful piece of information, not a false association that somehow got lodged in the general consciousness without solid science backing it up.

To be valuable, a biomarker has to be validated by "large-scale studies showing consistently that these markers have a strong effect – which is very rarely the case," he added.

One of the reasons for such prudence is medical: Depending on the details, bad biomarkers could lead to overtreatment or to denial of treatment to people who could benefit.

But costs are also a concern.

Biomarkers "could be used by millions, if not billions, of people," he said. "Healthcare costs could really escalate very rapidly to amazing figures."