The Evidence of Enough
Samueli Institute does research: meticulous and rigorous research in the laboratory; pragmatic and personal research in the clinic and with the public. One of the questions we wrestle with all the time with our scientific partners, clinicians and patients is “When and what type of research is enough?”
Is it enough to have three randomized controlled trials and a meta-analysis? Do we require placebo controls for everything? Is it essential to isolate the specific effects of a treatment to be considered scientific? How do we determine what’s really important for patients and the public? Where is the role of stories and narrative in creating evidence?
The Cycle of Confusion
In a recent story in the Washington Post summarizing dietary and nutritional supplements, a variety of types of evidence were used. The evidence used for making statements about supplements was incomplete, inconsistent, even irrational.
Data from multiple rigorous clinical studies was sometimes dismissed as only minor evidence; whereas anecdotes from patients about supplements were given out in recommendations to the readers.
No wonder the public is confused about what works and what they should be doing with diet, nutrition and supplements. Over 50% of the public takes dietary supplements regularly. What are they to do when prominent media outlets have no consistent framework or process for making their conclusions evidence based?
The Role of Interpretation
The methods of science are well established and rational in order to get to the truth. Science has developed a process that, even if it takes time, works to separate the wheat from the chaff. But the certainty of this evidence varies.
Often, even after the most rigorous of studies, the confidence we have in the results may be questionable. In most science then, we need to interpret the meaning of the results.
Personal judgment then comes into play.
This judgment is in many ways no different than what happens in a court room when the evidence may also be often sketchy and unclear.
Unfortunately, the judgment process in science is often left to the drug company, whose eyes are on product sales, or the media industry, whose high-selling product is a controversial headline. Whether misinterpretation is due to a lack of understanding the science, subconscious prejudices, or more malicious intent, this kind of judgment can undermine even the most rigorous science and confuse not only the public but scientists themselves.
A more rigorous management of this judgment processes for evidence is needed. Lives, money and the mitigation of suffering depend on it.
Evaluating Good Science
At Samueli Institute we are working closely with the RAND Corporation to develop such a method and incorporate it into the judgment process in medical science. We call it the SEaRCHTM process. After an objective summary of the evidence occurs, the careful management of the judgment process by expert panels is then done and the degree of opinion difference is quantified in a rigorous manner. We then apply a balanced and quantified process for controlling judgment, which reduces the likelihood that good science is subverted by personal bias and vested interests.
This process is the next step in clinical decision-making and should be incorporated into the development of all guidelines and approval processes and also by funding agencies to determine what they pay for. It should also be used by the media to communicate science to the public. In fact, this process can also be used to find out what the public really thinks about the value of evidence.
SEaRCHTM is a process by which health care can be both evidence-based and become more “person-centered” and so better serve the relevant needs of the public.
Without better management of our biases and conflicts (both monetary and mental) our investment of even billions of dollars in high-quality science can be rendered useless. Without this, the public risks being left to whims and vagaries of opinion after all.