That’s the view of NICE Chairman Michael Rawlins, delivering the annual Office of Health Economics lecture at the Royal Society of Physicians in London on June 18.
It’s a view worth noting – and exploring – given that Rawlins helms what’s arguably the most influential Health Technology Assessment body in the world. His conclusion – that decision-makers including NICE should consider a far broader evidence base, including observational studies – may comfort pharma, but regulators have a long way to go.
Most drugs are approved on the basis of RCTs – trials that randomize patients to either a treatment arm, or a control arm, assume no difference between them, and then assess the probability that observed differences in outcomes between the two groups are significant or occur by chance.
RCTs have well-known advantages, Rawlins conceded. They include minimal bias (each patient has the same chance of getting the treatment, or not), minimized chance of random error, and simple, widely accepted statistical methodology and criteria for significance.
But Rawlins outlined how RCTs are only poorly generalizable to the wider population, in terms of patient numbers, patient profile and treatment duration. He declared them outrageously expensive. And he outlined their “serious statistical shortcomings” too. “They’re clumsy. Starting from the basis that there’s no difference at all between the treatment arms [the ‘null hypothesis’] is clumsy, and ignores the results of previous studies, Phase IIs, for instance.” What’s more, he continued, the p-value cut-off (usually less than 0.05) for rejecting that chance alone could have produced the result, is arbitrary, and doesn’t distinguish truth from falsity.
RCTs’ weaknesses are compounded when investigators start doing interim analyses, as is often the case in oncology drug trials. Indeed, only a third of cancer trials go through to the end, according to some estimates. The trouble is, “if you examine [the data] early enough, the likelihood of getting a [significant] p value by chance increases,” Rawlins explained. As more cancer treatments enter development, joining an increasingly crowded landscape, interim analyses will increase.
In sum, “to regard RCTs as the gold-standard is unsustainable.”
So what kinds of trials should pharma be doing? Rawlins outlined the benefits of the Bayesian approach – which looks for the probability that a particular intervention does cause a particular outcome (the reverse of the null hypothesis), thus combining new evidence with a prior belief. He noted that it’s widely used outside medicine, for example by bookmakers (“and they expect a 10% profit on turnover”).
The Bayesian method has limitations — the statistical nerds among you will know them, any others can click here. But they’re likely to be less expensive than RCTs, and thus should have their place. So, indeed, should various other kinds of observational studies, such as historical-controlled trials, and case-control studies. These aren’t perfect either: selection bias is the most obvious problem. But they have value, especially when there’s no reasonable comparator for a particular drug, and when there’s a strong effect size. Novartis’ Gleevec was approved on the basis of historical controlled trials.
Statistics Should Support Judgment, Not The Other Way Round
Rawlins isn’t recommending the end of RCTs. For one thing, he noted, regulators still find it hard to imagine a drug being approved without RCTs (though FDA appears more open than European regulators). But observational studies should legitimately be combined with RCTs where appropriate. And there’s a middle-ground, too — for instance in adaptive RCTs (highlighted in JAMA on June 13) that help improve patients’ outcomes by adjusting trial enrollment rules as data on treatment benefit emerges. Ideally, says Rawlins, “decision-makers should not be single-minded. They should take a pluralistic approach to evidence, instead of sticking to [RCT-dominated] hierarchies.”
In the end, it’s about judgment. “Judgment, conditioned by the totality of the evidence base, lies at the heart of decision-making,” says Rawlins.
That NICE’s chairman is declaring he’s open to evidence beyond the RCT is a bad news, good news story for pharma. On the one hand, judgment is by definition subjective: since there are no rules attached, pharma will be taking risks as it defines what real world evidence is acceptable–and what isn’t . On the other hand, statements like Rawlins are a signal that pharma may dare to experiment, as healthcare reimbursement faces its biggest shake-up to date. There’s not a lot more to lose.