The impact of incorporating Bayesian network meta-analysis in cost-effectiveness analysis - a case study of pharmacotherapies for moderate to severe COPD
© Thorlund et al.; licensee BioMed Central Ltd. 2014
Received: 13 July 2013
Accepted: 26 February 2014
Published: 13 March 2014
To evaluate the impact of using network meta-analysis (NMA) versus pair wise meta-analyses (PMA) for evidence synthesis on key outputs of cost-effectiveness analysis (CEA).
We conducted Bayesian NMA of randomized clinical trials providing head-to-head and placebo comparisons of the effect of pharmacotherapies on the exacerbation rate in chronic obstructive pulmonary disease (COPD). Separately, the subset of placebo–comparison trials was used in a Bayesian PMA. The pooled rate ratios (RR) were used to populate a decision-analytic model of COPD treatment to predict 10-year outcomes.
Efficacy estimates from the NMA and PMA were similar, but the NMA provided estimates with higher precision. This resulted in similar incremental cost-effectiveness ratios (ICER). Probabilities of being cost-effective at willingness-to-pay thresholds (WTPs) between $25,000 and $100,000 per quality adjusted life year (QALY) varied considerably between the PMA- and NMA-based approaches. The largest difference in the probabilities of being cost-effective was observed at a WTP of approximately $40,000/QALY. At this threshold, with the PMA-based analysis, ICS, LAMA and placebo had a 43%, 30, and 18% probability of being the most cost-effective. By contrast, with the NMA based approach, ICS, LAMA, and placebo had a 56%, 19%, and 21% probability of being cost-effective. For larger WTP thresholds the probability of LAMA being the most cost-effective became higher than that of ICS. Under the PMA-based analyses the cross-over occurred at a WTP threshold between $60,000/QALY-$65,000/QALY, whereas under the NMA-based approach, the cross-over occurred between $85,000/QALY-$90,000/QALY.
Use of NMAs in CEAs is feasible and, as our case study showed, can decrease uncertainty around key cost-effectiveness measures compared with the use of PMAs. The approval process of health technologies in many jurisdictions requires estimates of comparative efficacy and cost-effectiveness. NMAs play an increasingly important role in providing estimates of comparative efficacy. Their use in the CEAs therefore results in methodological consistency and reduced uncertainty.
Network meta-analysis (NMA) (also known as multiple or mixed treatment comparisons) are becoming widely accepted for establishing comparative efficacy between competing health technologies [1–4]. In contrast with conventional pair wise meta-analysis (PMA), NMAs allow for comparisons between interventions that have not been compared head-to-head in randomized clinical trials (RCTs), and offer additional precision by ‘borrowing strength’ from indirect evidence [1, 2, 5–7]. In medical decision-making, NMAs are commonly used in health technology assessments produced by government agencies or pharmaceutical companies in connection with technology approval submissions [8–10]. In this context, NMAs can provide reliable and consistent evidence on the efficacy and safety of the considered interventions. The contemporary technology approval process in many jurisdictions is informed by evaluating comparative efficacy as well as cost-effectiveness analysis (CEA) comparing the new technology with the alternative choices. NMAs are increasingly popular frameworks for synthesizing evidence on comparative efficacy [2, 3]. Despite the merits of NMAs, it is still common that evidence synthesis for the CEA is based on conventional PMA meta-analysis. While some integration of NMAs and CEAs are beginning to take place in commercially prepared health technology assessment (HTA) reports, we are not aware of any published applications intended to inform decision-making.
In addition, it is well accepted that CEAs should be comprehensive . That is, the analysis should include all available treatment options; and the evidence synthesis should be based on all the available evidence . A CEA based on PMA meta-analyses may however fall short in these two aims. First, evidence on comparative efficacy and safety may not be available for all treatments via PMA meta-analysis because not all options have been compared head-to-head or with a common control intervention. Second, when more than two options are compared, the evidence synthesis for a PMA is often based on taking one technology as the ‘reference’ and looking for comparative studies of other technologies with that reference. In this vein, head-to-head comparisons between the considered interventions, as well as relevant comparisons with older interventions might be discarded, and so the full evidence-base is not utilized in the CEA. NMAs on the other hand can produce estimates of comparative efficacy for all considered options, and allow for inclusion of all relevant randomized evidence (i.e., both direct and indirect evidence). Therefore NMAs are likely to more optimally and rationally utilize the available evidence, and the resulting added precision and accuracy may translate into a more confident adoption decision.
The use of PMAs rather than NMAs for evidence synthesis in economic evaluations therefore represents a missed opportunity for optimizing decision-making . To provide insights on the benefit of using NMAs, rather than PMAs, in CEAs we use an illustrative case of pharmacotherapies for chronic obstructive pulmonary disease (COPD). We demonstrate how the precision gained on efficacy estimated via the NMA, as opposed to PMA, can reduce the uncertainty around CEA outputs and can result in more confident adoption decisions. We also provide practical guidance on the step-wise processes needed to incorporate the NMA analysis into the CEA process.
Methods and material
We use a motivating example of pharmacotherapies for the treatment of moderate to severe of COPD. COPD is a chronic disease of the airways that is responsible for a substantial economic and humanistic burden . Exacerbations (lung attacks) are hallmarks of COPD, and are associated with significant costs, impaired quality of life, and risk of mortality . There are multiple pharmacotherapies available for COPD and there is considerable debate on which pharmacotherapy should be used as first line treatment in COPD . There is inconsistent evidence as to whether pharmacotherapies can change the course of COPD. Nevertheless, pharmacotherapies have a proven impact on reducing the exacerbation rate in COPD . There are several RCTs comparing such therapies with placebo (i.e., no treatment), as well as a large number of RCTs providing head-to-head comparisons between such therapies .
NMA model and data
Efficacy data was taken from a recent NMA on the effect of pharmacotherapies in reducing the exacerbation rates in patients with COPD . In particular, five interventions were considered: no treatment (placebo), inhaled corticosteroids (ICS), long-acting beta-agonists (LABA), long-acting muscarinic agents (LAMA), and the combination of ICS and LABA (ICS + LABA). Several agents are available within each of these three drug classes (e.g., salmeterol, formoterol, and indacaterol are all LABAs) but they were considered equally effective in this analysis. While some may challenge this assumption, there are a number of reasons for employing this assumption in our study. First, our study is predominantly of an educational nature, and thus, simplicity in assumptions is key. Second, the NMA on which this study is based also assumed class-effects . Third, other NMA that have distinguished between therapies within classes have failed to demonstrate statistically significant differences within classes . Lastly, the assumption of ‘class effect’ for medications within the same class has long been an accepted paradigm in COPD .
Economic model and data
A decision-analytic model of COPD was created that translated the measures of treatment effect , combined with parameters representing the epidemiology [13, 19] and natural history [20, 21] of COPD, into the costs [22, 23], exacerbation rates and quality-adjusted life years (QALYs) associated with each treatment [16, 20, 21]. The time-horizon was 10 years with one-year time cycles. A constant yearly rate of exacerbations was assumed, thus allowing for the NMA RR estimate to be employed for determining transition probabilities for each of the ten cycles. Yearly mortality rates were taken from American life Tables . The yearly discount rate was set to 3% for both health and cost outcomes. The analysis adopted a third-party payer perspective. All costs were converted and presented as annual costs in year 2011 US dollars ($).
Each state of COPD was associated with an annual exacerbation rate for each treatment, which was calculated as the product of a baseline (no treatment) rate multiplied by the RR of the treatment versus no treatment. Exacerbations were categorized as either minor or major. The impact of treatment was assumed to be independent of the severity of the exacerbation.
Parameter estimates and their probability distributions used to populate the model
Assumed input value at GOLD stages
Assumed probability distribution at GOLD stages
Annual COPD mortality 
Utility and disutilities 
Exacerbations rates and probabilities 
Indirect maintenance cost 
Direct exacerbations costs ($) 
General practitioner visit
Direct medication costs ($) [ 
Inhaled corticosteroids (ICS)
Long-acting beta-agonists (LABA)
ICS + LABA
Long-acting muscarinic agents
The Bayesian NMA model was run in WinBUGS v.1.4.3 , and the economic model was run in R v2.14 . WinBUGS and R code is available from the authors upon request. The step-wise implementation of the PMA and NMA analyses and the CEA is described further in the Additional file 1. A total of 10,000 posterior distribution samples were used for the CEA, separately for the NMA and PMA meta-analyses. The model outputs on costs and QALYs were used to calculate the ICERs and incremental net monetary benefits (INMB), with no treatment as the reference group, and to draw the cost-effectiveness planes and cost-effectiveness acceptability curves (CEACs). Treatments were also ranked according to their INMB at WTP of $50,000/QALY, separately for PMA- and NMA-based analyses.
Incidence rate ratio estimates for the considered interventions based on pair-wise meta-analysis (PMA) and network meta-analysis (NMA)
Rate ratios (95% CrI)
ICS + LABA
10-year average cost, number of exacerbations, and quality adjusted life-years for each intervention using both pairwise meta-analysis (PMA) and network meta-analysis (NMA)
Number of exacerbations
Quality adjusted life years (QALYs)
25 458 (19 927, 32 312)
12.6 (12.5, 12.8)
5.67 (5.36, 5.99)
25 316 (19 950, 31 897)
12.6 (12.5, 12.7)
5.67 (5.34, 5.99)
27 116 (22 194, 33 307)
10.3 (8.62, 11.9)
5.73 (5.40, 6.07)
26 979 (21 991, 33 420)
10.2 (9.10, 11.4)
5.73 (5.38, 6.06)
28 304 (23 081, 34 897)
11.0 (9.31, 13.2)
5.71 (5.39, 6.04)
28 116 (23 002, 34 604)
10.9 (9.86, 12.2)
5.71 (5.37, 6.05)
ICS + LABA
30 849 (25 947, 36 741)
9.13 (7.69, 11.2)
5.76 (5.42, 6.10)
30 496 (25 902, 36 133)
8.85 (7.83, 10.0)
5.75 (5.40, 6.11)
28 840 (23 928, 35 073)
9.33 (7.47, 11.4)
5.76 (5.42, 6.10)
28 816 (24 237, 34 548)
9.39 (8.37, 10.4)
5.77 (5.41, 6.12)
Incremental cost-effectiveness ratios (ICERs) and probabilities of each intervention being the most cost effective at various willingness-to-pay thresholds
Probability of being cost-effective by willingness-to-pay threshold
Placebo as reference
ICS + LABA
ICS as reference
ICS + LABA
At WTP of $50,000/QALY, the ranking of the first three treatments (ICS, LAMA, and no treatment) remained the same between PMA- and NMA-based analyses. The only difference in the results was that the treatment with the lowest INMB for the PMA-based analysis was ICS + LABA whereas for the NMA-based analysis it was LABA.
In the present work we elaborated on the theoretical advantages of using NMAs over PMAs in economic evaluations of health technologies, and used a case study to demonstrate the practical aspects of the use of NMAs as well as the empirical differences in the outcomes of the economic evaluation when NMA instead of PMA is used for evidence synthesis. The results demonstrate how the CEA can benefit from the gain in precision from using the entire network of evidence rather than the results of pair wise comparisons alone. In our case study, while the added precision did not result in major changes in the choice of the optimal treatment across a wide range of WTP, it prevented the counter-intuitive situation of the optimal treatment not having the maximum probability of cost-effectiveness .
The network of evidence underlying the case study was a well-connected treatment network including large studies and head-to-head RCTs for almost all comparisons. As such, the use of the entire available evidence base synthesized through NMA resulted in similar point estimate for the effect size but with an increased precision. However, situations may occur where NMA estimates are not close to their PMA counterparts; and where the combination of indirect and direct evidence does little to increase the precision . However, the theoretical justifications underpinning the use of NMA instead of PMA are unrelated to the empirical gains in certainty and stand valid regardless of any particular results.
The performed analyses come with some limitations. We used a simple decision-analytic model of COPD for the case study, mainly based on the modeling assumptions used by previous authors [20, 21]. The simplicity of this model allowed us to focus on the practical aspects and illustration of the results; but we acknowledge that to inform policy, a deeper analysis, including a detailed set of sensitivity and alternative analyses will be required. For example, our model did not account for the potential impact of treatments on disease progression , a controversial aspect of the treatment that needs to be considered in a sensitivity analysis. Our model also did not account for potential long-term adverse events associated with corticosteroid treatment and their associated costs. However, the complexity of building a decision-model is not intensified by the use of NMA versus PMA for evidence synthesis.
The implications of the results are rather straightforward: the potential theoretical and practical gains in using NMAs as opposed to PMAs in cost-effectiveness analysis are too significant to be ignored. However, this does not mean that CEAs should only ever rely on efficacy estimates from NMAs. NMA is a method of inference and as such is based on certain statistical assumptions that are generally more restrictive than the assumptions underlying PMA [2, 3]. For example, there are situations where NMA estimates may be more biased than their PMA counterparts estimated only from placebo comparisons [28, 29]. If in a particular context where there are misgivings about the suitability of such assumptions, the investigator might deliberately choose PMA. Overall, a thorough assessment of the potential biases and confounders in both the NMA and the PMA is necessary before deciding which data and type of research synthesis method should be used for informing the cost-effectiveness analysis.
In summary, incorporating NMA in CEA offers consistency and added certainty in comparison with CEA informed by conventional PMA. As the role of NMAs in informing comparative efficacy in the evaluation of new health technologies is growing, NMAs could and should be considered for informing the evidence used in CEA.
- Hoaglin DC, Hawkins N, Jansen JP, Scott DA, Itzler R, Cappelleri JC, Boersma C, Thompson D, Larholt KM, Diaz M, Barrett A: Conducting indirect-treatment-comparison and network-meta-analysis studies: report of the ISPOR Task Force on Indirect Treatment Comparisons Good Research Practices: part 2. Value Health 2011,14(4):429–37. 10.1016/j.jval.2011.01.011PubMedView ArticleGoogle Scholar
- Mills EJ, Ioannidis JP, Thorlund K, Schunemann HJ, Puhan MA, Guyatt GH: How to use an article reporting a multiple treatment comparison meta-analysis. JAMA 2012,308(12):1246–53. 10.1001/2012.jama.11228PubMedView ArticleGoogle Scholar
- Jansen JP, Fleurence R, Devine B, Itzler R, Barrett A, Hawkins N, Lee K, Boersma C, Annemans L, Cappelleri J: Interpreting indirect treatment comparisons and network meta-analysis for health-care decision making: report of the ISPOR Task Force on Indirect Treatment Comparisons Good Research Practices: part 1. Value Health 2011,14(4):417–28. 10.1016/j.jval.2011.04.002PubMedView ArticleGoogle Scholar
- Mills EJ, Bansback N, Ghement I, Thorlund K, Kelly S, Puhan MA, Wright J: Multiple treatment comparison meta-analyses: a step forward into complexity. Clin Epidemiol 2011, 3: 193–202.PubMed CentralPubMedView ArticleGoogle Scholar
- Cooper NJ, Peters J, Lai MC, Juni P, Wandel S, Palmer S, Paulden M, Conti S, Welton NJ, Abrams KR, Bujkiewicz S, Spiegelhalter D, Sutton AJ: How valuable are multiple treatment comparison methods in evidence-based health-care evaluation? Value Health 2011,14(2):371–80. 10.1016/j.jval.2010.09.001PubMedView ArticleGoogle Scholar
- Higgins JP, Whitehead A: Borrowing strength from external trials in a meta-analysis. Stat Med 1996,15(24):2733–49. 10.1002/(SICI)1097-0258(19961230)15:24<2733::AID-SIM562>3.0.CO;2-0PubMedView ArticleGoogle Scholar
- Thorlund K, Mills EJ: Sample size and power considerations in network meta-analysis. Syst Rev 2012,1(1):41. 10.1186/2046-4053-1-41PubMed CentralPubMedView ArticleGoogle Scholar
- Sutton AJ, Ades AE, Cooper NJ, Abrams KR: Use of indirect and mixed treatment comparisons for technology assessment. Pharmacoecon 2008,26(9):15.View ArticleGoogle Scholar
- Wells GA, Sultan SA, Chen L, Khan M, Coyle D: Indirect evidence: indirect treatment comparison in meta-analysis. Ottawaa: Canadian Agencies for Drugs and Technology in Health (CADTH); 2009.Google Scholar
- Dias S, Welton N, Sutton A, Ades A: NICE DSU Technical Support Document 2 - A generalised linear modelling framework fro pairwise and network meta-analysis of randomised controlled trial. 2011.Google Scholar
- Cooper NJ, Sutton A, Abrams KR, Turner D, Wailoo A: Comprehensive decision analytic modelling in economic evaluation: a Bayesian approach. Health Econ 2003,13(3):24.Google Scholar
- Sculpher MJ, Claxton K, Drummond M, McCabe C: Whither trial-based economic evaluation for health care decision making? Health Econ 2006,15(7):677–87. 10.1002/hec.1093PubMedView ArticleGoogle Scholar
- Mannino DM, Duist AS: Global burden of COPD: risk factors, prevalence, and future trends. Lancet 2007,370(9589):9.View ArticleGoogle Scholar
- Connors AF Jr, Dawson NV, Thomas C, Harrell FE Jr, Desbiens N, Fulkerson WJ, Kussin P, Bellamy P, Goldman L, Knaus WA: Outcomes following acute exacerbation of severe chronic obstructive lung disease. The SUPPORT Study. Respir and Crit Care Med 1996,154(4):9.View ArticleGoogle Scholar
- Celli BR, MacNee W, ATS/ERS Task Force: Standards for the diagnosis and treatment of patients with COPD: a summary of the ATS/ERS position paper. Eur Respir J 2004,23(6):932–946. 10.1183/09031936.04.00014304PubMedView ArticleGoogle Scholar
- Mills EJ, Druyts E, Ghement I, Puhan MA: Pharmacotherapies for chronic obstructive pulmonary disease: a multiple treatment comparison meta-analysis. Clin Epidemiol 2011, 3: 107–29.PubMed CentralPubMedView ArticleGoogle Scholar
- Cope S, Capkun-Niggli G, Gale R, Jardim JR, Jansen JP: Comparative efficacy of indacaterol 150 mug and 300 mug versus fixed-dose combinations of formoterol + budesonide or salmeterol + fluticasone for the treatment of chronic obstructive pulmonary disease--a network meta-analysi. Int J Chron Obstruct Pulmon Dis 2011, 6: 329–44.PubMed CentralPubMedView ArticleGoogle Scholar
- McAlister FA, Laupacis A, Wells GA, Sackett DL: Users’ Guide to the Medical Literature: XIX. Applying clinical trial results B. Guidelines for determining whether a drug is exerting (more than) a class effect. JAMA 1999,282(14):7.View ArticleGoogle Scholar
- Donaldson GC, Wedzicha JA: COPD exacerbations 1: epidemiology. Thorax 2011,61(2):5.Google Scholar
- Spencer M, Briggs AH, Grossman RF, Rance L: Development of an economic model to assess the cost-effectiveness of treatment interventions for chronic obstructive pulmonary disease. Pharmacoeconomics 2005, 23: 19.View ArticleGoogle Scholar
- Buist AS, Vollmer WM, Sullivan SD, Weiss KB, Lee TA, Menezes AM, Crapo RO, Jensen RL, Burney PG: The Burden of Obstructive Lung Disease Initiative (BOLD): rationale and design. COPD 2005, 2: 7. 10.1081/COPD-200050680View ArticleGoogle Scholar
- Dalai AA, Shah M, D’Souza AO, Rana P: Costs of COPD exacerbations in the emergency department and inpatient setting. Respir Med 2011,105(3):7.Google Scholar
- MIMS: MIMS: prescription drug database and drug prescription guide. 2012., 59: Google Scholar
- Aries E: United States Life Tables. London, UK: CDC; 2007.Google Scholar
- Lunn D, Spiegelhalter D, Thomas A, Best N: The BUGS project: evolution, critique and future directions. Stat Med 2009,28(25):3049–67. 10.1002/sim.3680PubMedView ArticleGoogle Scholar
- The R, Core T: R: A Language and Environment for Statistical Computing. Vienna, Austria: R Foundation for Statistical Computing; 2005.Google Scholar
- Sadatsafavi M, Najafzadeh M, Marra C: Technical note: acceptability curves could be misleading when correlated strategies are compared. Med Decis Mak 2008,28(3):2.View ArticleGoogle Scholar
- Song F, Harvey I, Lilford R: Adjusted indirect comparison may be less biased than direct comparison for evaluating new pharmaceutical interventions. J Clin Epidemiol 2008,61(5):455–63. 10.1016/j.jclinepi.2007.06.006PubMedView ArticleGoogle Scholar
- Song F, Clark A, Bachmann MO, Maas J: Simulation evaluation of statistical properties of methods for indirect and mixed treatment comparisons. BMC Med Res Methodol 2012, 12: 138. 10.1186/1471-2288-12-138PubMed CentralPubMedView ArticleGoogle Scholar
This article is published under license to BioMed Central Ltd. This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly credited.