Potential and limitations of a Bayesian approach to the analysis and synthesis of evidence from multiple, heterogeneous sources. An inquiry into its usefulness in supporting policy decisions on drugs.

Policy decisions concerning reimbursement of drugs have crucially important implications for access to medical treatments for patients. They also have implications for physicians, pharmacists, insurers, and manufacturers. Ultimately, they may affect patients’ health expectations and overall health care costs: drug acquisition costs may be substantial, but this may also hold for the savings that are achieved in other areas. For reasons such as these, the available evidence on clinical and cost effectiveness of drugs should be carefully synthesized when reimbursement decisions are made. This task is frequently hampered, due to the fact that the available evidence is incomplete or inconsistent. Also, the evidence may originate from multiple, heterogeneous sources, including randomized controlled trials, cohort studies, record reviews, registries, laboratory studies, and clinical and patients’ experiences. For a variety of reasons, the standard, frequentist approach is not well geared to analyze and synthesize such an amalgam of evidence to assist evidence-based policy making. Firstly, frequentist approaches result in an estimate of the probability of obtaining an observed outcome (or one more extreme than this) under the null hypothesis of no effect. When this probability exceeds a pre-specified threshold (usually 0.05), the null hypothesis is rejected. It is important to note that from this result, it cannot be inferred how unlikely the null hypothesis in fact is, nor how likely the alternative hypothesis. Arguably, however, this is precisely the sort of inference that we would like to make: what is the probability that an intervention produces a particular outcome, given the observed results? Secondly, the frequentist approach is quite unflexible, being constrained by its error rates, alpha and beta. Thirdly, the frequentist approach has no means of weighing evidence from heterogeneous sources in a systematic and transparent way. A Bayesian approach to analysis and synthesis of data may solve each of these problems: it results in an estimate of the probability that an intervention will, in fact, produce a particular outcome, it is highly flexible since it is not error-based, and it allows for differentially weighing various types of evidence in a systematic and transparent way through its use of prior probability estimates. In spite of these potential advantages, the Bayesian approach is relatively unfamiliar and relatively little used in the context of synthesizing evidence to support policy decisions. This may be due to the contentious nature of the Bayesian concept of probability and to the lack of standardization of specific components of its methodology, e.g., for eliciting prior probability estimates. Moreover, its alleged advantages have been rarely tested in practice. In the current project, the potential and limitations of the Bayesian approach will be explored in the context of supporting policy decisions on reimbursement of drugs. For this purpose, specific subjects will be selected where the National Health Insurance Board of the Netherlands (College voor Zorgverzekeringen, CVZ) has recommended the Ministry of Health on reimbursement of drugs. The (then) available evidence will be reanalyzed and synthesized, using Bayesian methodology. For this purpose, prior probability estimates will be solicited from a variety of stakeholders, including physicians, medical advisers of insurers, of manufacturers, etc. Prior probability estimates will be combined with likelihood ratios, derived from the relevant literature, using Bayes’ theorem. The result will consist of probabilities that specific reimbursement decisions will, in fact, have particular health and economic outcomes. The results will be presented to CVZ staff, and serve as a basis for drafting a policy recommendation which is entirely based on Bayesian methodology. This approach allows for a comparison to be made between the original, frequentist-based policy recommendations and the newly drafted recommendations, based on Bayesian analysis. Comparisons will be made in terms of outcomes, judgments of various stakeholders of comprehensiveness of the evidence base, interpretability of outcomes, transparency of how evidence was used in formulating policy recommendations, etc. Stakeholders will include CVZ staff, members of the national CVZ appraisal committee, and representatives of patients’ and professional organizations. This research project will be conducted in close collaboration with CVZ, which provides a unique opportunity to optimize knowledge utilization in policy making. By using different modes of eliciting prior probabilities, the project will also contribute to further standardization of Bayesian methodology.