Want Better Data? Align Professional Incentives

April 2, 2012 – 1:57 pm | By Anette Breindl | No comments yet

This week, researchers are presenting their latest and greatest at the annual meeting of the American Association of Cancer Research. The excitement of all those novel findings, though, received something of an advance puncturing last week by a commentary in Nature that reported a successful replication rate of just over 10 percent for roughly 50 landmark studies in cancer research.

From my perspective, the study makes it even more difficult to figure out which of the many research findings that (briefly) come to my attention every day to write about, and which to ignore. Is this study more interesting, or that one? Well, uh . . . can either of them be replicated?

Authors Glenn Begley and Lee Ellis stressed that they were not out to pillory individual scientists. Indeed, the sheer magnitude of the problem indicates that this is a systemic issue that needs a systemic fix – because journals and funders are valuing perfect stories over accurate data.

In the meantime, though, one of the implications of the study is that far from being riddled with conflicts of interest, preclinical results reported by industry, may be more reliable than those coming out of academia.

This has nothing to do with who’s the better person. Or the better scientist. It has to do with timing. For an academic scientist, a paper in a high-impact journal is a happy ending, the culmination of a lot of effort. For an industry scientist, that paper is also a success – but it is the successful beginning of what could turn into a billion-dollar development effort.

This is not a new view among industry scientists. Several years ago, at a meeting, a friend from a large biotech company and I sat and listened to a speaker hold forth on recent findings that the rate of positive results was higher in papers that come out of industry.

The speaker’s point was that if industry’s rate of positive findings was higher, that was indicative of data massaging. My friend wasn’t buying, though. Instead, she rolled her eyes and snapped, “Well, maybe we run better studies!”

Before industry starts patting itself on the back too hard, let me point out that I am talking specifically about preclinical data. In industry, the problem of selective reporting gets shifted to later in the process – where you can find plenty of stories of whole arms of clinical trials suddenly disappearing from clinical trial reporting and the like.

And so, my point is not that it’s academia’s turn to wear the dunce cap. Companies want to develop good medicines, but they also want to – and need to – be economically viable. Academics want to do good research, but they also want to – and need to – be professionally viable by publishing in good journals and getting grants. Everybody has multiple goals, and the better our system of research manages to align those goals, the better the science that our drugs are based on will be – no matter where that science is coming from.


Post a Comment

You must be logged in to post a comment.

Oops! We could not locate your form.