Skip to main content

Volume 16 Supplement 1

Priority Setting in Global Health

Stakeholder involvement in Multi-Criteria Decision Analysis


This brief perspective highlights the importance of decision maker buy-in and ownership through stakeholder engagement in the co-construction of the multi-criteria decision analysis (MCDA) model. A brief historical overview of MCDA is presented before outlining the importance of bridging the gap (and to gain trust) between the tool developers and users. The issues with the current MCDA tool development and testing efforts are highlighted, and the ownership and routine adoption of the MCDA process is discussed.

The past few years have witnessed a surge of interest in the use of multi criteria decision analysis (MCDA) for priority setting. Many different tools and techniques are available under the general heading of MCDA, and examples include the use of ‘Program budgeting and marginal analysis’ (PBMA) tools [1,2,3] for resource allocation decisions by local health care budget holders; use of discrete choice experiments to inform priority setting [4, 5]; use of the Evidem framework in making decisions about value for money of new health care technologies [6, 7]. However, the use of formal MCDA tools for health care priority setting is still limited, with most of the published MCDA studies being either pilots or hypothetical in nature [8]. This section outlines our reflections on why MCDA is not more widely used for priority setting and highlights key requirements for successful application of MCDA tools.

‘Buy-in’ from the key decision makers is imperative for the implementation of a structured priority setting process. In other words, if there is no buy-in from the decision makers, the results of the MCDA studies will not be implemented. Unfortunately, many of the MCDA studies seen in the literature on health care priority setting seem to ignore this fundamental requirement, with the MCDA researchers and practitioners working independently from the key stakeholders when conducting the MCDA. Examples of these include determining the criteria, performing the scoring and weighting independently from the stakeholders/decision makers. Whilst these studies may be methodologically sound, the outputs of the MCDA are not implemented as they are not trusted by the decision makers, given they had little or no involvement in the MCDA process.

There may be several potential reasons for the lack of ‘buy-in’ from the decision makers. One could be the reluctance to move away from cost-effectiveness analysis and its legacy, even if it does not include many other factors that affect real decisions [9]. This issue is not specific to MCDA; health care field has always been slow to adopt new techniques/methodologies [10, 11]. MCDA might be seen as too objective, transparent and explicit to allow the flexibility for making exceptions to individual cases. There may also be resistance to follow a process like MCDA as the decision makers may not want to reveal a set of priorities, over concerns that they may be challenged if the priorities shift over time. Furthermore, there could be other cultural/political factors that may affect the ‘buy-in’ from the decision makers.

The emphasis of MCDA researchers should be on educating the decision makers about the MCDA process and its benefits for supporting priority setting—for example, by conducting training courses or developing manuals to support MCDA process for priority setting. It should be noted that MCDA can help support the implementation of ‘accountability for reasonableness’ framework [12], put forward by Norman Daniels and other ethicists, which argues for the inclusion of due and fair process in setting and agreeing priorities.

‘Ownership’ of the MCDA process and results can help with achieving the ‘buy-in’ from the key decision makers/stakeholders. This involves engaging the stakeholders/decision makers in each step of the MCDA process [13], i.e. defining the decision problem, selecting the criteria, developing performance matrix, scoring alternatives and weighting criteria, aggregation, uncertainty analysis, and interpretation of results. The researchers should validate each step to ensure that MCDA design, input, and outputs are plausible and consistent with the decision maker objectives and stakeholder preferences [14]. Wherever needed, the researchers should help interpret the meaning of the different components of MCDA to ensure confirmation from the stakeholders.

The aim of this ‘co-production’ between the decision makers and MCDA researchers is to ensure that the stakeholders/decision makers feel that the decision is ‘their’ decision, not the tool’s decision. MCDA is a mechanism for making explicit the criteria and judgements that feed into the complex process of resource allocation. If the decision makers do not understand or feel like they were not of the process, they would be reluctant to use the MCDA results.

Many different tools and techniques are available under the general heading of MCDA. The technical complexity of MCDA, especially the weighted sum model, is very simple (each alternative’s scores on the criteria are multiplied by the weights and these weighted scores are then summed across the criteria to get a “total value” for each alternative) but the challenges are in the MCDA process (i.e. choosing the criteria, explaining the scoring/weighting to stakeholders, eliciting the preferences, presenting and interpreting the MCDA results). A key consideration is to ensure that the MCDA tools can support a committee or group of decision makers for use in real policy settings, as opposed to tools designed for single users [15, 16]. However, the MCDA tools cannot solve the problem on their own; they need engagement from the decision makers. Thus, rather than developing more tools, capacity building is more beneficial for the use of MCDA to support priority setting.

In summary, given MCDA is about eliciting the decision makers/stakeholders’ preferences, it is a fundamental limitation if they are not part of the whole MCDA process. The success of MCDA for priority setting relies on the buy-in and engagement from the health care bodies; and we should focus our efforts in supporting this rather than developing more MCDA tools.


  1. Mitton CR, Donaldson C, Waldner H, Eagle C. The evolution of PBMA: towards a macro-level priority setting framework for health regions. Health Care Manag Sci. 2003;6(4):263–9.

    Article  Google Scholar 

  2. Mitton CR, Donaldson C. Setting priorities and allocating resources in health regions: lessons from a project evaluating program budgeting and marginal analysis (PBMA). Health Policy. 2003;64(3):335–48.

    Article  Google Scholar 

  3. Peacock SJ, Richardson JR, Carter R, Edwards D. Priority setting in health care using multi-attribute utility theory and programme budgeting and marginal analysis (PBMA). Soc Sci Med. 2007;64(4):897–910.

    Article  Google Scholar 

  4. Marsh K, Dolan P, Kempster J, Lugon M. Prioritizing investments in public health: a multi-criteria decision analysis. J Public Health (Oxf). 2012;35(3):460–6.

    Article  Google Scholar 

  5. Youngkong S, Teerawattananon Y, Tantivess S, Baltussen R. Multi-criteria decision analysis for setting priorities on HIV/AIDS interventions in Thailand. Health Res Policy Syst. 2012;10:6.

    Article  Google Scholar 

  6. Goetghebeur MM, Wagner M, Khoury H, Levitt RJ, Erickson LJ, Rindress D. Evidence and value: impact on DEcisionMaking—the EVIDEM framework and potential applications. BMC Health Serv Res. 2008;8:270.

    Article  Google Scholar 

  7. Goetghebeur MM, Wagner M, Khoury H, Rindress D, Gregoire JP, Deal C. Combining multicriteria decision analysis, ethics and health technology assessment: applying the EVIDEM decision-making framework to growth hormone for Turner syndrome patients. Cost Eff Resour Alloc. 2010;8:4.

    Article  Google Scholar 

  8. Diaby V, Campbell K, Goeree R. Multi-criteria decision analysis (MCDA) in health care: a bibliometric analysis. Oper Res Health Care. 2013;2(1–2):20–4.

    Article  Google Scholar 

  9. Phelps C, Madhavan G, Rappuoli R, Colwell R, Fineberg H. Beyond cost-effectiveness using systems analysis for infectious disease preparedness. Vaccine. 2017;35(1):A46–9.

    Article  Google Scholar 

  10. Brailsford SC, et al. Overcoming the barriers: a qualitative study of simulation adoption in the NHS. J Oper Res Soc. 2013;64(2):157–68.

    Article  Google Scholar 

  11. Brailsford S. Overcoming the barriers to implementation of operations research simulation models in healthcare. Clin Invest Med. 2005;28(6):312.

    PubMed  Google Scholar 

  12. Daniels N. Accountability for reasonableness: establishing a fair process for priority setting is easier than agreeing on principles. BMJ. 2000;321(7272):1300.

    Article  CAS  Google Scholar 

  13. Thokala P, Devlin N, Marsh K, Baltussen R, Boysen M, Kalo Z, Ijzerman M. Multiple criteria decision analysis for health care decision making-an introduction: report 1 of the ISPOR MCDA emerging good practices task force. Value Health. 2015.

    Article  Google Scholar 

  14. Marsh K, IJzerman M, Thokala P, Baltussen R, Boysen M, Kaló Z, Devlin N. Multiple criteria decision analysis for health care decision making—emerging good practices: report 2 of the ISPOR MCDA emerging good practices task force. Value Health. 2016.

    Article  PubMed  Google Scholar 

  15. Phelps C, Madhavan G. Using multicriteria approaches to assess the value of health care. Value Health. 2017;20(2):251–5.

    Article  Google Scholar 

  16. Madhavan G, Phelps C, Rappuoli R. Compare voting systems to improve them. Nature. 2017;541:151–3.

    Article  CAS  Google Scholar 

Download references


Authors’ contributions Both authors read and approved the final manuscript.

Authors’ information

The views expressed in this article are those of the authors and not necessarily of the National Academies of Sciences, Engineering, and Medicine.


The authors would like to thank Dan Chisholm for his useful comments on an initial version of the manuscript.

Competing interests

The authors declare that they have no competing interests.

Availability of data and materials

Not applicable.

Consent for publication

Both authors consent for publication.

Ethics approval and consent to participate

Not applicable.

Publication funding

The publication costs for this article were funded by Mark O’Friel, the Brinson Foundation, and the Payne Family Foundation.

About this supplement

This article has been published as part of Cost Effectiveness and Resource Allocation Volume 16 Supplement 1, 2018: Priority Setting in Global Health. The full contents of the supplement are available online at

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Author information

Authors and Affiliations


Corresponding author

Correspondence to Praveen Thokala.

Rights and permissions

Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License (, which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver ( applies to the data made available in this article, unless otherwise stated.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Thokala, P., Madhavan, G. Stakeholder involvement in Multi-Criteria Decision Analysis. Cost Eff Resour Alloc 16 (Suppl 1), 0 (2018).

Download citation

  • Published:

  • DOI: