- Open Access
Incorporating MCDA into HTA: challenges and potential solutions, with a focus on lower income settings
Cost Effectiveness and Resource Allocation volume 16, Article number: 43 (2018)
Multicriteria decision analysis (MCDA) has the potential to bring more structure and transparency to health technology assessment (HTA). The objective of this paper is to highlight key methodological and practical challenges facing the use of MCDA for HTA, with a particular focus on lower and middle-income countries (LMICs), and to highlight potential solutions to these challenges.
Key lessons from existing applications of MCDA to HTA are summarized, including: that the socio-technical design of the MCDA reflect the local decision problem; the criteria set properties of additive models are understood and applied; and the alternative approaches for estimating opportunity cost, and the challenges with these approaches are understood.
Existing efforts to implement HTA in LMICs suggest a number of lessons that can help overcome the practical challenges facing the implementation of MCDA in LMICs, including: adapting inputs from other settings and from expert opinion; investing in technical capacity; embedding the MCDA in the decision-making process; and ensuring that the MCDA design reflects local cultural and social factors.
MCDA has the potential to improve decision making in LMICs. For this potential to be achieved, it is important that the lessons from existing applications of MCDA are learned.
Health care decision making bodies across the world face the challenge of choosing which technologies to fund with scarce resources. This is supported by health technology assessment (HTA), to estimate the value for money of technologies. While factors other than cost and health benefit are acknowledged by HTA agencies , these are not always precisely measured, and their value is not formally assessed . This has led to calls for methods capable of capturing the broader set of value of interest to payers, including consideration of, for instance, equity issues, burden of disease and a broader set of social benefits.
The past few years have witnessed a surge of interest in the use of multi-criteria decision analysis (MCDA) in HTA . MCDA is a collection of approaches that support decision making by taking explicit account of multiple criteria. They guide decision makers through the process of agreeing what factors are relevant to a decision, measuring performance of options against these criteria, and understanding the trade-offs between values that may be conflicting. Without such structure, priority setting processes can be ad hoc, and not include all relevant stakeholders . This is particularly the case in lower and middle income countries (LMICs), where priority setting tends to be more complex due to there being limited evidence to inform decisions, countries’ fragile institutional capacity and the dominant influence of policy makers’ opinions and international donor agencies [4, 5]. In these circumstances, MCDA can support the quality of decision making and increase transparency and consistency .
A growing number of decision making bodies and HTA agencies in high income countries (HICs) are either using or starting to explore these approaches to improve their transparency and accountability, including: Germany (Institute for Quality and Efficiency in Health Care [IQWiG];  Italy (Lombardy);  South Korea;  Hungary;  UK (National Institute for Health and Care Excellence [NICE] highly specialised technology ). While a lot of attention is given to HIC’s use of MCDA, an unusually large proportion of published MCDAs for HTA are undertaken in LMICs. Two recent reviews of MCDAs in health care [3, 12] identified 10 studies of MCDAs used to inform HTA in LMICs, including: formulary management in Malaysia  and Cote d’Ivoire , and priority setting in Brazil , Thailand [16, 17], Ghana [18, 19], Nepal , Morocco and Tunisia  and South Africa . In one of these reviews, nine out of a total of 23 published MCDAs for HTA were undertaken in LMICs .
The objective of this paper is to provide an overview of the methodological and practical challenges facing the use of MCDA for HTA, with a particular focus on LMICs, and to highlight potential solutions to these challenges. The next section outlines methodological challenges facing the application of MCDA for HTA. The following section then considers some of the practical challenges to implementing MCDA, specific to LMICs. Four case studies of the application of MCDA in LMICs are referred to throughout these sections.
The use of MCDA for HTA is still in its infancy, and guidelines on good practice have only recently been published . As guidelines are developed, it is becoming clear that many examples of MCDA for HTA, whether in LMICs or otherwise, are not applying good practice. It is important that those implementing MCDA in LMICs learn the lessons from this experience, and are aware of the current, more general methodological debates in the application of MCDA for HTA. This section identifies some of these challenges and points towards good practice that can help practitioners address them.
Positioning on the socio-technical continuum
Perhaps the first challenge facing the design of an MCDA is to determine where on the socio-technical continuum the MCDA design should sit, ranging from purely deliberative to fully quantified/algorithmic . The technical element of MCDA addresses the analytical questions: how to ensure criteria properties and criteria set properties comply with good practice, how to measure performance against these, how the criteria are weighted, and how performance and weights are aggregated. The social element of MCDA is concerned with which stakeholders are involved in the MCDA, and when and how they contribute. There is no a priori optimal position on this spectrum. Technical and social elements need to work in concert to achieve the aims of the MCDA, and the appropriate combination of elements will depend on the decision problem.
It has been argued that HTA is an ethical problem, and that MCDA can support HTA by facilitating the deliberation required for stakeholders with diverse perspectives to learn from one another and achieve agreement . That is, the appropriate MCDA design for HTA emphasises fair process, argumentation, iteration and systematic thinking, alongside the technicalities of preference elicitation and aggregation methods. However, whether this is the case will depend on the decision problem facing HTA agencies. For instance, the MCDA required by IQWiG to support economic evaluation would probably be considered towards the more technical end of the spectrum, involving the use of a discrete choice experiment to elicit the preferences of large samples of patients .
The case studies highlighted in this paper illustrate a range of socio-technical approaches to MCDA for HTA. All involved significant stakeholder engagement in the design, implementation, and interpretation of the results of the MCDA. The stakeholders involved varied; for instance, criteria weights were elicited from policy makers in Thailand (Table 1), Indonesia (Table 2) and Ghana (Table 3), but from the general population in Colombia (Table 4). In Ghana, Indonesia and Thailand, the results of the MCDA were discussed and validated by decision makers before they used the MCDA to support their decision making. In Colombia, the MCDA had a more direct impact on decisions, with the technologies that performed best in the MCDA being included in benefits package until the point at which the available budget was spent.
It is recommended that those designing an MCDA are familiar with the decision problem, including the relevant stakeholders, and the alternative MCDA methods available to support this decision problem. Decision makers should be engaged in defining the decision and the appropriate MCDA solution. Alternative approaches should be presented to decision makers, with an assessment of their relative merits, so that decision makers can input into the process of methods design. Regardless of where on the continuum the appropriate MCDA design is located, it is important to follow good practice recommendations relating to both the technical and social elements of MCDA [24, 27].
Quantifying the benefits of technologies
By far the most prevalent aggregation function adopted by MCDAs for HTA is the additive one. This has the advantage of being analytically simple. However, such simplicity requires that the criteria set have certain properties. In particular, they need to be non-overlapping (i.e., avoiding the double counting of the value generated by an alternative) and preferentially independent (i.e., the weight attached to one criterion should not depend on the performance on other criteria).
Unfortunately, many applications of the additive model to HTA violate these requirements. Preferential dependence—when the weight attached to one criterion is dependent on the performance on another—is a common problem in MCDAs for HTA. Perhaps the most prevalent example of this is the additive combination of health gain and severity of disease, when we might reasonably expect the value of health gain to be dependent on disease severity . Three of the four case studies presented in this paper are subject to this potential type of preferential dependence, including both severity of disease and effectiveness [Thailand (Table 1), Colombia (Table 4)] or cost-effectiveness [Ghana (Table 3)]. In the presence of preferential dependence, the additive model is invalid and two options are available: adopt a different model structure that reflects the non-additive relationship between criteria, or update the criteria to ensure they are independent .
Measuring opportunity cost
HTA requires not only that the benefits of technologies are estimated, but also that opportunity cost is estimated using the same measure. This is the case whether HTA is supported by MCDA or other methods, such as cost-effectiveness analysis. Where HTA involves a technology-by-technology evaluation, we can distinguish three approaches to measure willing to pay (WTP) for gains measured on the MCDA-benefit scale. First, stated preference methods can be used to elicit stakeholders WTP for the benefits of technologies. Many MCDAs for HTA include cost or budget impact as a criterion. While not explicitly acknowledged by these studies, this is a stated preference approach. The weight elicitation stage of the MCDA essentially elicits stakeholder WTP for benefits. Two of the case studies presented in this paper include cost-effectiveness as a criterion [Ghana (Table 3) and Indonesia (Table 2)].
This approach, however, poses significant challenges. Oftentimes these MCDAs elicit weights separate from scales of performance. For instance, asking stakeholders to estimate the relative importance of ‘budget impact’ and ‘health gain’. However, without more precise definitions of these criteria, including precise scale ranges, it is almost impossible to provide a precise WTP estimate. Even with more precise definitions, it is questionable whether stakeholders would have the necessary knowledge of the returns on current expenditure, or the cognitive capacities to translate this into reliable responses to preference elicitation exercises. For instance, the relative weight attached to cost and benefits need to reflect the willingness to pay for different benefits and how this changes in different circumstances. This challenge is exacerbated when the criterion is cost-effectiveness, as many different changes in effectiveness and cost could be reflected in a single cost-effectiveness ratio.
These concerns form the basis of the recommendation that MCDAs should not include cost or budget impact as a criterion, and instead that MCDA should be used to estimate the benefits of technologies. This is the approach adopted in the other two case studies [Thailand (Table 1) and Colombia (Table 4)]. This broader benefit measure could then be compared against the incremental cost of the technology to form a new incremental cost-effectiveness ratio (ICER). Then, akin to the current application of the cost-effectiveness analysis, the efficiency of a technology is assessed by comparing an ICER against an opportunity cost threshold. The broader benefit measure could be seen as replacing the quality-adjusted life-year (QALY)—though the QALY could still be one of the criteria making up this broader measure.
Two methods have been adopted to estimate such an opportunity cost threshold in the context of cost-effectiveness analysis . Both could be applied in the context of MCDA, but doing so would pose challenges. First, an analysis of historic decisions to reveal decision makers’ preferences . However, that no decisions would have been made using MCDA at the point of designing an HTA approach makes this unfeasible. Second, empirical estimates of the changes in benefits resulting from changes in expenditure . This too faces practical challenges. Efforts to apply this approach to estimate the cost-effectiveness threshold have been criticised for the number of assumptions required to make up for the limitations in the available data . More challenges would be faced applying this approach to MCDA.
Faced with similar challenges when implementing cost-effectiveness analysis, it has been proposed that LMICs adopt a strategy of extrapolating opportunity cost estimates from wealthier settings  or estimating opportunity cost based on local income levels , especially in the absence of WHO guidance on the matter. The same approach cannot, however, be applied in the context of MCDA until these wealthier settings identify estimates of their own MCDA-based opportunity cost.
Alternatively, those designing HTA systems might avoid a technology-by-technology assessment process. If HTA involves the assessment of all technologies simultaneously, the challenge of estimating opportunity cost is removed. In that instance, a decision rule could take the form of investing in technologies in order of their ICERs until the budget is expended (see for instance Airoldi et al. 2011 ). In this instance, MCDA can be applied as part of programme budgeting marginal analysis . This is the approach adopted in three of the four case studies presented in this paper—Thailand (Table 1), Indonesia (Table 2), and Ghana (Table 3). However, in none of these instances was the whole health care budget or the whole benefits package considered, something which would be practically impossible. Instead, this application of MCDA is practically limited to individual care pathways or vertical programmes, such as HIV. As a consequence, opportunity costs issue remains which are not considered by the analysis.
Linked to the methodological challenges presented above, there are a number of practical challenges related to the application of MCDA as part of an HTA process. Many of these apply to HTA generally, whereas some are specific to MCDA. Most are also more acute in resource constrained settings, whether in low income settings or middle income economies with budding HTA mechanisms in the context of insurance agencies.
Sourcing inputs in systems with less reliable information systems, including limitations in context-specific information, is a major challenge with HTA. Regardless of whether MCDA is employed, HTA will require data on (unit) costs, resource use, epidemiology, comparative effectiveness specific to the setting, and outcomes data including patient-reported outcomes. Distribution of health and access to health care is another type of information required by HTA, and while not specific to LMICs, it is particularly important in LMICs.
The literature on the development of HTA systems in LMICs identifies potential solutions to gathering this data [40, 41]. Promising avenues for the collection of these data is collating and adapting evidence on effectiveness from other settings; and the use of expert opinion to help translate evidence into the local setting and fill gaps in the evidence [42,43,44]. In the Thai case study (Table 1) there was a lack of local scientific information on one of the topic selection criteria ‘targeting the poor and those with rare diseases’. Performance against this criterion was determined on the basis of the Thai experts’ opinion and international guidelines. Though caution is advised, as there are also examples of the application of external data to local settings that have not been successful, such as in the case of the Filipino health insurance scheme .
Faced with limited information, MCDA has a number of benefits. First, it is able to accommodate criteria defined and measured using expert opinion. Second, it allows for the formal consideration of the impact of uncertainty on the analysis. Relying on expert opinion, or generalising from evidence collected in other jurisdictions, introduces uncertainty into HTA. By breaking up HTA into its components parts—criteria, performance and valuation—MCDA enables the impact of uncertainty in any of these components on the results of a HTA to be assessed . This will allow a formal assessment of whether this uncertainty impacts on the conclusion of the MCDA.
MCDA also involves information requirements of its own, such as the need to attach appropriate weights/scores reflecting social preferences. Indeed, weight elicitation and how transparent and understandable those are to decision makers and the broader public is a general practical challenge for MCDA. Some criteria used in MCDA, like equity, are difficult for non-experts to understand. The elicitation of weights is a specialist activity, prone to bias, pointing to the need for appropriate technical capacity—the topic of the next section.
Technical capacity obstacles—different types of capacity shortfalls
Related to the informational/evidential challenges raised above, the relative lack of well-trained people who can initiate, manage and apply the whole process of HTA/MCDA is a practical obstacle to its routine introduction, especially in LMIC settings. Technical capacity required to deliver an MCDA includes: knowledge of good MCDA practice, such as the properties of criteria required to implement an additive model (see above); modelling skills, though most MCDA models are not technically demanding; and preference elicitation expertise. The latter may include techniques such as decision conferencing, which encourage stakeholders to express their views, facilitate the generation of consensus and help inform the model as it develops on-the-spot with continuous display of the model’s results through a highly ‘interactive and iterative group process’ .
The ability to commission and quality assure MCDAs from the right groups on behalf of decision makers, and of translating the findings into policy language and specific and actionable recommendations, is an important one which is often lacking in LMICs. This said, there are examples of countries that within a 5- to 10-year period have built a significant capacity, technical, clinical, political and administrative, in support of HTA including MCDA, such as Thailand with the Health Intervention and Technology Assessment Program (HITAP) [16, 23, 48, 49]. Also, regional initiatives such as HTAsiaLink can help countries work together, oftentimes pooling experiences as well as expertise, to support local decision making efforts [41, 50]. Finally, while training programmes at local universities are often in short supply, they are available. Mahidol’s HTA postgraduate programmes in Southeast Asia is one such example . Its programmes were established in 2016, partly with funding from iDSI , to build up social and intellectual capital for, ultimately, strengthening and sustaining capacity for HTA research and implementation, and support UHC in LMICs.
Issues of governance, institutions and consistency in decision making
The MCDA literature tends to emphasise the technical aspects of implementation, such as those describe above. Relatively little attention is paid to the extent to which the MCDA is embedded in the institutional policy-making process, something that is necessary to ensure that its results will actually be used . Five of the nine examples of MCDAs for HTA in LMICs identified by Marsh et al.  were applied in an institutional setting; three being used to support decisions [16, 18, 22] and two being exploratory studies involving officials [14, 17].
The embedding of MCDA in relevant decision-making processes requires established or mature enough institutions. First, if such institutions do not exist, the ability to engage relevant stakeholders to elicit their input into the implementation of the MCDA, as well as to communicate the results of the MCDA, can be undermined by limited trust in public institutions. Second, institutions are required that are able to carry out or at least influence budgetary allocation (or de-allocation). Obstacles to implementing decisions include: weak regulation; unreliable payment mechanisms; or misaligned ones such as ‘Fee For Service’ which can encourage supply-induced demand, and a lack of effective performance-based contracts. Additional practical challenges include the fragmentation of decision making structures and of financing cash flows, including oftentimes between central and federal or public and private sectors, and between decision makers at the Ministry of Health and various insurance schemes as well as significant out of pocket spending.
However, where policy makers develop an interest in HTA, there are significant opportunities for progress in terms of institutionalising HTA-type approaches. This can be facilitated by creating positions within ministries and insurance funds for supporting institutions, and signalling to academic institutions, the health care products’ industry and international players that the government is determined to adopt a more explicit and evidence informed approach to priority setting. India, through the Indian Council of Medical Research and the Department for Health Research at the Union level, and China, under the leadership of the National Health and Family Planning Commission, both recently announced the launch of HTA institutions and networks, respectively, to support the transition to UHC [53,54,55]. Another encouraging experience can be found in Thailand. The Thai experience highlights the need for senior figures from powerful bodies to champion MCDA. The introduction of MCDA/HTA in Thailand was supported by the chair of the sub-committee of UHC benefit package development and the chair of the committee of National List of Essential Medicines, both of whom realised the importance of evidence-informed decision making [16, 48, 56].
The role of international donors in LMICs creates both challenges to and opportunities for the embedding of MCDA in decision making. To the extent to which donors use MCDA to support their own decision making, their demands for related evidence can drive the adoption of MCDA in LMICs. Further, donors or funding conduits such as the Global Fund , can contribute to the building of in-country capacity, technical and institutional, for commissioning, carrying out and applying MCDAs to decisions about countries’ own spending, which is increasingly important as countries transition away from aid. The potential for applying MCDA in the context of specific diseases and conditions, with its structured approach to stakeholder engagement and explicitness about consideration of issues such as equity and human rights, ought to make its application attractive to funding channels such as Gavi, which funds vaccines, and the Global Fund, which is responsible for funding interventions targeting HIV/AIDS, TB and malaria in developing countries.
On the other hand, donor’s influence may have a detrimental impact on the adoption of MCDA. Being explicit about opportunity costs or costs in general can often seem controversial in the global development world, acknowledging budgetary constraints is often deemed unethical and economists demonised [58, 59]. Further, when funding pots are limited to vaccines or a certain disease such as TB or HIV, issues of allocative efficiency become less relevant in the short term, at least for the foreign budget holders making investment choices in poorer countries. We hope however, that as the value for money rhetoric  finds its way into government policies in rich and poorer countries, MCDA will offer a means for considering in a systematic fashion most of the things that matter when allocation decisions are made, including costs and distribution.
Cultural and social factors
MCDA presents the possibility of formally and transparently incorporating local cultural and social factors into decision making. It is not only the health and economic impacts of interventions that are of interest in HTA, but also the social, ethical, and institutional implications of a technology. MCDA can help HTA consider these factors, by incorporating them into the criteria list and weighting them to reflect their relative importance to stakeholders. For instance, the example of the use of MCDA to inform UHC in Thailand demonstrates the influence of social values throughout the MCDA, from the topic selection stage, through the inclusion of health technologies that are not equally accessible throughout the country or across different health facilities, to the criteria reflecting equity and social implications, with preference given to health problems inflicting the poor or minority groups with rare diseases . Another example is the Indonesian case study (Table 2), which includes a criterion measuring the impact of interventions on the stigma associated with being HIV positive.
Furthermore, by structuring stakeholder input, and by explicitly incorporating cultural and social factors into decision criteria, MCDA reduces the risk that decision making is unduly influenced by local power structures. However, adopting MCDA does not guarantee stakeholders’ representation in decision making. In LMICS, the technical nature of MCDA may result in deliberations being dominated by highly educated people with the views of the majority less well reflected . It is important that this is considered when designing the MCDA process, so that the appropriate support is provided to allow everyone to participate.
MCDA is increasingly being considered as a means to address some of the limitations with existing HTA methods. In particular, MCDA offers a means to more formally and transparently capture the multiple factors relevant to HTA. Despite the attention given to debates in high income countries, an unusually large proportion of published MCDAs for HTA are undertaken in LMICs. This may be explained by the lack of established HTA methodologies in LMICs, allowing recent innovations to take hold. This literature, and broader debates about the application of MCDA in health care, point to a number of lessons for those implementing MCDA in LMICs:
It is important that new efforts to implement MCDA in LMICs don’t replicate the technical limitations of existing applications, such as ignoring the preferential dependence between criteria (e.g. severity of disease and health gain). Those implementing MCDAs should familiarize themselves with recent good practice guidance.
The estimation of opportunity cost for use in an MCDA-framework that undertakes technology-by-technology evaluations is a significant challenge. This can be somewhat addressed if the decision problem can be defined to be prioritizing which of many interventions to fund from a fixed budget.
A lack of evidence in LMICs can partly be addressed through generalizing from evidence generated in other settings and the use of expert opinion, in combination with an analysis of the impact of uncertainty on the conclusions of the MCDA.
Investment in technical capacity is necessary to ensure a rigorous implementation of the MCDA, in particular the design of the overall MCDA approach, and the elicitation of weights.
If it is to impact decision making, it is important that the design of the MCDA reflects the local decision problem, including being embedded in decision making institutions, and engaging relevant stakeholders in design, implementation, and interpretation.
Decision Support Unit
high income country
Health Intervention and Technology Assessment Program
health technology assessment
incremental cost-effectiveness ratio
Instituto de Evaluación Tecnológica en Salud
Institute for Quality and Efficiency in Health Care
lower and middle income country
multi-criteria decision analysis
National Health Security Office’s
National Institute for Health and Care Excellence
sub-committee for the development of benefit package and service delivery
Universal Health Coverage
willingness to pay
National Institute for Health and Care Excellence (NICE). Guide to the methods of technology appraisal 2013. Process and methods [PMG9] 2013. https://www.nice.org.uk/process/pmg9/chapter/foreword. Accessed July 2017.
Baltussen R, Niessen L. Priority setting of health interventions: the need for multi-criteria decision analysis. Cost Eff Resour Alloc. 2006;4:14.
Marsh K, Lanitis T, Neasham D, Orfanos P, Caro J. Assessing the value of healthcare interventions using multi-criteria decision analysis: a review of the literature. Pharmacoeconomics. 2014;32:345–65.
Chalkidou K, Marten R, Cutler D, et al. Health technology assessment in universal health coverage. Lancet. 2013;382:e48–9.
Oliver K, Innvar S, Lorenc T, Woodman J, Thomas J. A systematic review of barriers to and facilitators of the use of evidence by policymakers. BMC Health Serv Res. 2014;14:2.
Baltussen R. Question is not whether but how to use MCDA. Value and Outcomes Spotlight; 2015.
Mühlbacher A, Bridges J, Bethge S, et al. Choice-based conjoint analysis—pilot project to identify, weight, and prioritize multiple attributes in the indication “hepatitis C”. IQWiG Report; 2013.
Radaelli G, Lettieri E, Masella C, et al. Implementation of EUnetHTA core Model(R) in Lombardia: the VTS framework. Int J Technol Assess Health Care. 2014;30:105–12.
Bae EY, Hong JM, Kwon HY, et al. Eight-year experience of using HTA in drug reimbursement: South Korea. Health Policy. 2016;120:612–20.
Endrei D, Molics B, Agoston I. Multicriteria decision analysis in the reimbursement of new medical technologies: real-world experiences from Hungary. Value Health. 2014;17:487–9.
National Institute for Health and Care Excellence (NICE). Interim process and methods of the highly specialised technologies programme 2013. https://www.nice.org.uk/Media/Default/About/what-we-do/NICE-guidance/NICE-highly-specialised-technologies-guidance/Highly-Specialised-Technologies-Interim-methods-and-process-statements.pdf. Accessed July 2017.
Adunlin G, Diaby V, Xiao H. Application of multicriteria decision analysis in health care: a systematic review and bibliometric analysis. Health Expect. 2015;18:1894–905.
Ramli A, Aljunid SM, Sulong S, Md Yusof FA. National drug formulary review of statin therapeutic group using the multiattribute scoring tool. Ther Clin Risk Manag. 2013;9:491–504.
Diaby V, Lachaine J. An application of a proposed framework for formulary listing in low-income countries: the case of Cote d’Ivoire. Appl Health Econ Health Policy. 2011;9:389–402.
Nobre FF, Trotta LT, Gomes LF. Multi-criteria decision making—an approach to setting priorities in health care. Stat Med. 1999;18:3345–54.
Youngkong S, Baltussen R, Tantivess S, Mohara A, Teerawattananon Y. Multicriteria decision analysis for including health interventions in the universal health coverage benefit package in Thailand. Value Health. 2012;15:961–70.
Youngkong S, Teerawattananon Y, Tantivess S, Baltussen R. Multi-criteria decision analysis for setting priorities on HIV/AIDS interventions in Thailand. Health Res Policy Syst. 2012;10:6.
Jehu-Appiah C, Baltussen R, Acquah C, et al. Balancing equity and efficiency in health priorities in Ghana: the use of multicriteria decision analysis. Value Health. 2008;11:1081–7.
Baltussen R, Stolk E, Chisholm D, Aikins M. Towards a multi-criteria approach for priority setting: an application to Ghana. Health Econ. 2006;15:689–96.
Baltussen R, ten Asbroek AH, Koolman X, et al. Priority setting using multiple criteria: should a lung health programme be implemented in Nepal? Health Policy Plan. 2007;22:178–85.
Holdsworth M, El Ati J, Bour A, et al. Developing national obesity policy in middle-income countries: a case study from North Africa. Health Policy Plan. 2013;28:858–70.
Miot J, Wagner M, Khoury H, Rindress D, Goetghebeur MM. Field testing of a multicriteria decision analysis (MCDA) framework for coverage of a screening test for cervical cancer in South Africa. Cost Eff Resour Alloc. 2012;10:2.
Mohara A, Youngkong S, Velasco RP, et al. Using health technology assessment for informing coverage decisions in Thailand. J Comp Eff Res. 2012;1:137–46.
Marsh K, IJzerman M, Thokala P, et al. Multiple criteria decision analysis for health care decision making-emerging good practices: report 2 of the ISPOR MCDA emerging good practices task force. Value Health. 2016;19:125–37.
Dodgson J, Spackman M, Pearman A, Phillips L. Multi-criteria analysis: a manual. 2009. http://eprints.lse.ac.uk/12761/1/Multi-criteria_Analysis.pdf. Accessed July 2017.
Baltussen R. MCDA. Do not provide a mathematical solution to what really is an ethical problem. In: Priority Setting in global health symposium Oct 5–6, 2016. Boston: Harvard T.H. Chan School of Public Health; 2016.
Thokala P, Devlin N, Marsh K, et al. Multiple criteria decision analysis for health care decision making—an introduction: report 1 of the ISPOR MCDA emerging good practices task force. Value Health. 2016;19:1–13.
Morton A. Treacle and smallpox: two tests for multicriteria decision analysis models in health technology assessment. Value Health. 2017;20:512–5.
Tromp N, Mikkelsen E, Viergever R, Baltussen R. Embedding MCDA in priority-setting policy processes: experiences from low- and middle-income countries. In: Marsh K, Goetghebeur M, Thokala P, Baltussen R, editors. Multi-criteria decision analysis to support healthcare decisions. New York: Springer International Publishing; 2017.
Thokala P, Ochalek J, Leech AA, Tong T. Cost-effectiveness thresholds: the past, the present and the future. Pharmacoeconomics. 2018;36:509–22.
Raftery J. NICE: faster access to modern treatments? Analysis of guidance on health technologies. BMJ. 2001;323:1300–3.
Claxton K, Martin S, Soares M, et al. Methods for the estimation of the NICE cost effectiveness threshold. CHE research paper 81; 2013. https://www.york.ac.uk/media/che/documents/papers/researchpapers/CHERP81_methods_estimation_NICE_costeffectiveness_threshold_(Nov2013).pdf. Accessed July 2017.
Barnsley P, Towse A, Karlsberg Schaffer S, Sussex J. Critique of CHE research paper 81: methods for the estimation of the NICE cost effectiveness threshold. London: Office of Health Economics; 2013.
Woods B, Revill P, Sculpher M, Claxton K. Determining value from healthcare spending: the role of cost-effectiveness thresholds; 2015. http://www.idsihealth.org/knowledge_base/cost-effectiveness-thresholds/. Accessed Mar 2018.
Bertram MY, Lauer JA, De Joncheere K, et al. Cost-effectiveness thresholds: pros and cons. Bull World Health Organ. 2016;94:925–30.
Airoldi M, Morton A, Smith J, Bevan G. Healthcare prioritisation at the local level: a socio-technical approach. Priority setting for population health, working paper series. London School of Economics and Political Science. Working paper no. 7; 2011.
Wilson EC, Peacock SJ, Ruta D. Priority setting in practice: what is the best way to compare costs and benefits? Health Econ. 2009;18:467–78.
Caro J, editor. The use of MCDA in Colombian HTA. HTAi: Tokyo; 2016.
Personal communication with Aurelio Mejia Mejia, senior economist IETS.
Tantivess S, Teerawattananon Y, Mills A. Strengthening cost-effectiveness analysis in Thailand through the establishment of the health intervention and technology assessment program. Pharmacoeconomics. 2009;27:931–45.
Asia Pacific Observatory on Health Systems and Policies. Policy brief: conducive factors to HTA development in Asia; 2013. http://www.wpro.who.int/asia_pacific_observatory/resources/policy_briefs/hta/en/. Accessed Mar 2018.
Disease Control Priorities. Economic evaluation for health. http://dcp-3.org/. Accessed Apr 2018.
World Health Organization (WHO). CHOosing Interventions that are Cost Effective (WHO-CHOICE). WHO CHOICE Tools. http://www.who.int/choice/toolkit/en/. Accessed Apr 2018.
World Health Organization (WHO). Cost effectiveness and strategic planning (WHO-CHOICE). http://www.who.int/choice/en/. Accessed Apr 2018.
Wong JQ, Haw NJ, Uy J, Bayani DB. Reflections on the use of the World Health Organization’s (WHO) OneHealth Tool: implications for health planning in low and middle income countries (LMICs). F1000Res. 2018;7:157.
Groothuis-Oudshoorn CGM, Broekhuizen H, van Til J. Dealing with uncertainty in the analysis and reporting of MCDA. In: Marsh K, Goetghebeur M, Thokala P, Baltussen R, editors. Multi-criteria decision analysis to support healthcare decisions. Cham: Springer; 2017.
Phillips LD. Chapter 19: decision conferencing; 2006. http://eprints.lse.ac.uk/22712/1/06085.pdf. Accessed July 2017.
Teerawattananon Y, Tantivess S, Yothasamut J, Kingkaew P, Chaisiri K. Historical development of health technology assessment in Thailand. Int J Technol Assess Health Care. 2009;25(Suppl 1):241–52.
Culyer AJ, Podhisita C, Santatiwongchai B. A star in the East. A short history of HITAP; 2016. http://www.idsihealth.org/wp-content/uploads/2016/02/A-STAR-IN-THE-EAST_resize.pdf. Accessed July 2017.
HTAsiaLink. http://www.htasialink.org/. Accessed July 2017.
Li R. Mahidol University and NICE commit to strengthening HTA capacity; 2015. http://www.idsihealth.org/blog/mahidol-nice-commit-to-strengthening-hta-capacity/. Accessed Mar 2017.
iDSI. Scholarship opportunity for health policy and technology assessment HePTA program; 2016. http://www.idsihealth.org/blog/hepta-scholarship-mahidol/hepta-flyer/. Accessed July 2017.
Downey L. Institutionalizing HTA in India—a joint iDSI-DHR event; 2016. http://www.idsihealth.org/blog/institutionalizing-hta-in-india-a-joint-idsi-dhr-event/. Accessed Mar 2017.
Chalkidou K. Launch of China health policy and technology assessment network under auspices of NHFPC; 2016. http://www.idsihealth.org/blog/launch-of-china-health-policy-and-technology-assessment-network-under-auspices-of-national-health-and-family-planning-committee/. Accessed Mar 2017.
Downey LE, Mehndiratta A, Grover A, et al. Institutionalising health technology assessment: establishing the medical technology assessment board in India. BMJ Glob Health. 2017;2:e000259.
Teerawattananon Y, Tritasavit N, Suchonwanich N, Kingkaew P. The use of economic evaluation for guiding the pharmaceutical reimbursement list in Thailand. Z Evid Fortbild Qual Gesundhwes. 2014;108:397–404.
Kanpirom K, Luz ACG, Chalkidou K, Teerawattananon Y. How should global fund use value-for-money information to sustain its investments in graduating countries? Int J Health Policy Manag. 2017;6:1–5.
Farmer P. Who Lives and who dies. London Rev Books. 2015;37:17–20.
Parkin D, Appleby J, Maynard A. Economics: the biggest fraud ever perpetrated on the world? Lancet. 2013;382:e11–5.
GOV.UK. Performance agreement: UK and the Global Fund to fight aids, tuberculosis and malaria; 2016. https://www.gov.uk/government/publications/performance-agreement-uk-and-the-global-fund-to-fight-aids-tuberculosis-and-malaria. Accessed July 2017.
Chootipongchaivat S, Tritasavit N, Luz A, Teerawattananon Y, Tantivess S. Factors conducive to the development of health technology assessment in Asia: impacts and policy options, vol. 4. Manila: WHO Policy Brief; 2015.
Authors’ contributions All authors contributed to conceptualizing and writing this manuscript. All authors read and approved the final manuscript.
The authors would like to thank the team at the Harvard T.H. Chan School of Public Health who organized the workshop during which the ideas in this manuscript were first shared and discussed.
The authors declare that they have no competing interests.
Availability of data and materials
Consent for publication
Ethics approval and consent to participate
The publication costs for this article were funded by Mark O’Friel, the Brinson Foundation, and the Payne Family Foundation.
About this supplement
This article has been published as part of Cost Effectiveness and Resource Allocation Volume 16 Supplement 1, 2018: Priority Setting in Global Health. The full contents of the supplement are available online at https://resource-allocation.biomedcentral.com/articles/supplements/volume-16-supplement-1.
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.