In God we trust. All others must bring data.
Yesterday, our local paper reprinted a piece from the Associated Press with the title, “Alzheimer’s drug promising.” To me, this was a typical example of what I call “press release medical reporting.” The question I asked of myself and of the article was, “how much is being promised.” In my opinion, the answer is, “not very much,” but to the investment community to which this news release was being reported, it was enough to raise the stock prices of the manufacturer, Eli Lilly. As a physician, I am not jumping up and down. As an individual whose family was savaged for over three decades by Alzheimer’s disease, I am annoyed. Let’s take a look at what is behind this commercial enthusiasm.
For some weeks or months now, the financial news media have been abuzz about the impending results of drug studies by Eli Lilly of their new drug, Solanezumab, for the treatment of mild to moderate Alzheimer’s disease. The drug is actually a man-made antibody to a protein that accumulates in the brains of people with Alzheimer’s which may or may not actually cause the disease. An earlier version of such an antibody made Alzheimer’s patients worse. The drug has been on the list of potential “blockbusters” which refers to the amount of money to be made and not necessarily to the amount of good a drug might do. These antibody drugs are proteins themselves and must be given intravenously. They are in the class of drugs costing multiple thousands of dollars per infusion, which generate big-time income for individual physicians, and which are playing a major role in bankrupting our healthcare system. We need to be sure they work as claimed.
The meat of the press release actually stated that the results of the two large studies of 1000 patients each were negative: there was no evidence that the drug did what it was supposed to. When the results of the two studies were merged together, a claim is being made that there was a “statistically significant result” for one of the presumably many things being measured. Additionally, another “statistically significant result” is claimed for a subgroup of patients with mild disease only. Of course, Eli Lilly has every reason to want to make lemonade from this unsweetened fruit, and we have certainly not heard the last of this salvage effort. Indeed, Lilly has already received FDA approval for a diagnostic test to detect early Alzheimer’s disease, so if approved, we can expect aggressive co-marketing. Allow me to suggest a few issues and concerns that are important to allow independent judgments.
Secondary Analysis of Scientific Data.
Scientists have long known that trying to extract more conclusions from an experiment than the experiment was designed to detect is fraught with hazard. The two studies mentioned above were presumably designed to have enough patients to be able to detect a clinically meaningful effect of the drug and yet they were not able do so. I do not have access to the details of the actual studies and do not even know if the two were identical. (If not, there is also the problem of comparing apples and oranges.) The reason that statistical science is used in the design and analysis of such studies is to make sure the primary results are not due to chance alone. If large numbers of different measurements are made on a large number of patients, there will always be something that will give the illusion of being significant. The more subgroups you look for, the more likely you will find something sticking out, even if it is an artifact. For example, it might be the case that only young male vegetarians appeared to benefit from the drug. Unless, however, the study was designed to measure a potential difference in vegetarians, no claim of clinical effectiveness can yet be made. This is not to say that such secondary analysis is uninteresting or of no value, but you would not go to the bank over such a conclusion, nor should you inject it into your patient. This is statistics 101. It is also the basis for the statistics joke:
If you torture the data long enough, it will eventually tell you what you want to hear.
What is being measured and does it make sense?
I have no idea at this time. I saw allusion in one business Internet site (where these press releases are appearing) to the “Mini Mental Status Exam (MMSE).” This is a simple screening exam widely used by clinicians in everyday practice to detect a decrease in cognitive (mental) functioning. It includes some 8 questions or tasks such as “what day is it,” “where are we,” counting by 7s, naming a pencil and watch, repeating phrases, and the like. It is very useful for this screening purpose but I have no personal knowledge of its use in research. It is possible that blood tests or other physical studies were performed as “proxies” for what the real desired result of a patient and their family would be: better or independent function at home, remembering the names of your spouse or family longer, or perhaps delay of entry into a nursing home. Note that some traditional endpoints of other drug studies such as longer life are irrelevant here. Management of Alzheimer’s disease is all about quality of life for the individual and for their family and family caregivers. These things are much harder to measure and even more difficult to attach a value to.
Is the effect meaningful?
Even if the drug were shown to have a “statistically significant” effect on the entire group of patients who participated in the study, that is a far cry from having a clinically meaningful effect. Even if there are no side effects (and there will be) the magnitude or duration of the effect may not be worth it. Statistics can detect differences that are invisible to the human eye!
As an example for this article, I generated a hypothetical clinical study of the effect a new drug on Alzheimer’s disease. I wanted to see if my new pill (I’ll call it Wonder Drug- Extended Relief) could keep people with Alzheimer’s disease out of nursing homes longer. Based on a study of the natural history of Alzheimer’s by Jost and Grossberg, I “enrolled” 100 patients with random ages between 70 and 79 when most patients receive their diagnosis. In real life, the average time between diagnosis and institutionalization is 24 months with very wide variation. Therefor, as an example of how things might have turned out without treatment, I generated a random pre-institutionalization interval for each of my patients of between 14 and 34 months. Not surprisingly, with no treatment, my patients stayed home for an average of 23 months. I then assumed that Wonder Drug would add a single day at home to each and every one of my study subjects. When I performed a statistical two-tailed paired t-Test on the hypothetical before-and-after results, I found that the one-day difference was highly statistically significant at the P=.05 level! Even the most desperate or optimistic person would have to admit that this “significant” improvement was absolutely worthless. If my Wonder Drug pill were one of the $100-Wonders, the patient and his family would better put the money to use enjoying their lives together while they could, or saving some of it for the education of the children or future Alzheimer care. Of course, any statistician worthy of the name can and will rip my example to shreds, but its conclusion, that some statistically significant scientific results are without meaningful social value, is unassailable.
Why is all this important?
I once had a friend in the drug business brag to me that they could make a drug study turn out any way they wanted to by carefully selecting the patients to be studied and the things to be measured. He also claimed that the development cost of a new drug could be captured in the first few months after release. While there was a bit of hyperbole there, the core of the claims has some truth. There is tremendous pressure from all sides to get new drugs on the market, even if they do not work very well. Certainly the many scandals involving drug companies, withdrawn drugs, judicial penalties, examples of misleading marketing, and other kinds of pharmaceutical malfeasance give us abundant reasons to closely examine the drug development and approval process with an independent eye. Some would go so far as to say that even the Food and Drug Administration (FDA) that approves new drugs, but is funded by the drug companies, is not fully independent any more. We already know the FDA is not politically independent and therefore is vulnerable to the lobbying process to which so much money is devoted. Alas, even my beloved academic enterprise can no longer be assumed to be independent of pharmaceutical largess and influence.
Evidence-based skepticism.
Let me be very clear that I have absolutely no evidence personally that Eli Lilly has done anything improper in the development of Solanezumab. I chose this press release to write about solely because it crossed my desk by chance. However, even this King of American pharmaceutical companies has had its days of ignobility in the development and marketing of drugs. In 2009 it paid a fine of $1.4 Billion for illegal off-label marketing of Zyprexia, including to people with Alzheimer’s disease. (Off-label marketing means promoting a drug for a medical indication for which it has not been shown by research to be effective. That is to say, there is no statistical significance at all!) By total coincidence as I write this today, I am reminded that my own father suffered an adverse effect of receiving Zyprexia for off-label purposes, and which in my medical opinion robbed him of one of the few things in his life that still gave him pleasure. In 2006 Eli Lillly agreed once more to plead guilty and pay $36 Million for its illegal promotion of the drug Evista used for osteoporosis. I did not look further back, but other drug companies have recently paid even larger fines for their illegal activities. The drug company Abbot just paid a $1.6 Billion fine related to its marketing of the antiepileptic drug Depakote to Alzheimer’s patients. Are such fines now just part of the cost of doing business? Drug companies are desperately looking for some way to cash in on this common and terrible disease. Obviously, bringing drugs to market in the right way is vitally important to us all and demonstrably cannot be left to unregulated market forces alone.
We of the public have delegated supervision of drug development to organizations such as the US Government, but also to many other public and private watchdogs. Even so, all of us have some responsibility to learn some of the language used in medical research and how the balance of adverse vs. beneficial effects of drugs is weighed. Otherwise we are collectively at the mercy of both legitimate and illegitimate purveyors of medical therapies.
Enough of this for now. We will be writing more about how to protect ourselves from being over-managed by the health care system. Remember Hasselbacher’s Law: It is just as bad to be over-treated as under-treated!
Peter Hasselbacher, MD
President, KHPI
Emeritus Professor, UofL
August 26, 2012