- Open Access
Stakeholder involvement in Multi-Criteria Decision Analysis
Cost Effectiveness and Resource Allocation volume 16, Article number: 0 (2018)
This brief perspective highlights the importance of decision maker buy-in and ownership through stakeholder engagement in the co-construction of the multi-criteria decision analysis (MCDA) model. A brief historical overview of MCDA is presented before outlining the importance of bridging the gap (and to gain trust) between the tool developers and users. The issues with the current MCDA tool development and testing efforts are highlighted, and the ownership and routine adoption of the MCDA process is discussed.
The past few years have witnessed a surge of interest in the use of multi criteria decision analysis (MCDA) for priority setting. Many different tools and techniques are available under the general heading of MCDA, and examples include the use of ‘Program budgeting and marginal analysis’ (PBMA) tools [1,2,3] for resource allocation decisions by local health care budget holders; use of discrete choice experiments to inform priority setting [4, 5]; use of the Evidem framework in making decisions about value for money of new health care technologies [6, 7]. However, the use of formal MCDA tools for health care priority setting is still limited, with most of the published MCDA studies being either pilots or hypothetical in nature . This section outlines our reflections on why MCDA is not more widely used for priority setting and highlights key requirements for successful application of MCDA tools.
‘Buy-in’ from the key decision makers is imperative for the implementation of a structured priority setting process. In other words, if there is no buy-in from the decision makers, the results of the MCDA studies will not be implemented. Unfortunately, many of the MCDA studies seen in the literature on health care priority setting seem to ignore this fundamental requirement, with the MCDA researchers and practitioners working independently from the key stakeholders when conducting the MCDA. Examples of these include determining the criteria, performing the scoring and weighting independently from the stakeholders/decision makers. Whilst these studies may be methodologically sound, the outputs of the MCDA are not implemented as they are not trusted by the decision makers, given they had little or no involvement in the MCDA process.
There may be several potential reasons for the lack of ‘buy-in’ from the decision makers. One could be the reluctance to move away from cost-effectiveness analysis and its legacy, even if it does not include many other factors that affect real decisions . This issue is not specific to MCDA; health care field has always been slow to adopt new techniques/methodologies [10, 11]. MCDA might be seen as too objective, transparent and explicit to allow the flexibility for making exceptions to individual cases. There may also be resistance to follow a process like MCDA as the decision makers may not want to reveal a set of priorities, over concerns that they may be challenged if the priorities shift over time. Furthermore, there could be other cultural/political factors that may affect the ‘buy-in’ from the decision makers.
The emphasis of MCDA researchers should be on educating the decision makers about the MCDA process and its benefits for supporting priority setting—for example, by conducting training courses or developing manuals to support MCDA process for priority setting. It should be noted that MCDA can help support the implementation of ‘accountability for reasonableness’ framework , put forward by Norman Daniels and other ethicists, which argues for the inclusion of due and fair process in setting and agreeing priorities.
‘Ownership’ of the MCDA process and results can help with achieving the ‘buy-in’ from the key decision makers/stakeholders. This involves engaging the stakeholders/decision makers in each step of the MCDA process , i.e. defining the decision problem, selecting the criteria, developing performance matrix, scoring alternatives and weighting criteria, aggregation, uncertainty analysis, and interpretation of results. The researchers should validate each step to ensure that MCDA design, input, and outputs are plausible and consistent with the decision maker objectives and stakeholder preferences . Wherever needed, the researchers should help interpret the meaning of the different components of MCDA to ensure confirmation from the stakeholders.
The aim of this ‘co-production’ between the decision makers and MCDA researchers is to ensure that the stakeholders/decision makers feel that the decision is ‘their’ decision, not the tool’s decision. MCDA is a mechanism for making explicit the criteria and judgements that feed into the complex process of resource allocation. If the decision makers do not understand or feel like they were not of the process, they would be reluctant to use the MCDA results.
Many different tools and techniques are available under the general heading of MCDA. The technical complexity of MCDA, especially the weighted sum model, is very simple (each alternative’s scores on the criteria are multiplied by the weights and these weighted scores are then summed across the criteria to get a “total value” for each alternative) but the challenges are in the MCDA process (i.e. choosing the criteria, explaining the scoring/weighting to stakeholders, eliciting the preferences, presenting and interpreting the MCDA results). A key consideration is to ensure that the MCDA tools can support a committee or group of decision makers for use in real policy settings, as opposed to tools designed for single users [15, 16]. However, the MCDA tools cannot solve the problem on their own; they need engagement from the decision makers. Thus, rather than developing more tools, capacity building is more beneficial for the use of MCDA to support priority setting.
In summary, given MCDA is about eliciting the decision makers/stakeholders’ preferences, it is a fundamental limitation if they are not part of the whole MCDA process. The success of MCDA for priority setting relies on the buy-in and engagement from the health care bodies; and we should focus our efforts in supporting this rather than developing more MCDA tools.
Mitton CR, Donaldson C, Waldner H, Eagle C. The evolution of PBMA: towards a macro-level priority setting framework for health regions. Health Care Manag Sci. 2003;6(4):263–9.
Mitton CR, Donaldson C. Setting priorities and allocating resources in health regions: lessons from a project evaluating program budgeting and marginal analysis (PBMA). Health Policy. 2003;64(3):335–48.
Peacock SJ, Richardson JR, Carter R, Edwards D. Priority setting in health care using multi-attribute utility theory and programme budgeting and marginal analysis (PBMA). Soc Sci Med. 2007;64(4):897–910.
Marsh K, Dolan P, Kempster J, Lugon M. Prioritizing investments in public health: a multi-criteria decision analysis. J Public Health (Oxf). 2012;35(3):460–6.
Youngkong S, Teerawattananon Y, Tantivess S, Baltussen R. Multi-criteria decision analysis for setting priorities on HIV/AIDS interventions in Thailand. Health Res Policy Syst. 2012;10:6.
Goetghebeur MM, Wagner M, Khoury H, Levitt RJ, Erickson LJ, Rindress D. Evidence and value: impact on DEcisionMaking—the EVIDEM framework and potential applications. BMC Health Serv Res. 2008;8:270.
Goetghebeur MM, Wagner M, Khoury H, Rindress D, Gregoire JP, Deal C. Combining multicriteria decision analysis, ethics and health technology assessment: applying the EVIDEM decision-making framework to growth hormone for Turner syndrome patients. Cost Eff Resour Alloc. 2010;8:4.
Diaby V, Campbell K, Goeree R. Multi-criteria decision analysis (MCDA) in health care: a bibliometric analysis. Oper Res Health Care. 2013;2(1–2):20–4.
Phelps C, Madhavan G, Rappuoli R, Colwell R, Fineberg H. Beyond cost-effectiveness using systems analysis for infectious disease preparedness. Vaccine. 2017;35(1):A46–9.
Brailsford SC, et al. Overcoming the barriers: a qualitative study of simulation adoption in the NHS. J Oper Res Soc. 2013;64(2):157–68.
Brailsford S. Overcoming the barriers to implementation of operations research simulation models in healthcare. Clin Invest Med. 2005;28(6):312.
Daniels N. Accountability for reasonableness: establishing a fair process for priority setting is easier than agreeing on principles. BMJ. 2000;321(7272):1300.
Thokala P, Devlin N, Marsh K, Baltussen R, Boysen M, Kalo Z, Ijzerman M. Multiple criteria decision analysis for health care decision making-an introduction: report 1 of the ISPOR MCDA emerging good practices task force. Value Health. 2015. https://doi.org/10.1016/j.jval.2015.12.003.
Marsh K, IJzerman M, Thokala P, Baltussen R, Boysen M, Kaló Z, Devlin N. Multiple criteria decision analysis for health care decision making—emerging good practices: report 2 of the ISPOR MCDA emerging good practices task force. Value Health. 2016. https://doi.org/10.1016/j.jval.2015.12.016.
Phelps C, Madhavan G. Using multicriteria approaches to assess the value of health care. Value Health. 2017;20(2):251–5.
Madhavan G, Phelps C, Rappuoli R. Compare voting systems to improve them. Nature. 2017;541:151–3.
Authors’ contributions Both authors read and approved the final manuscript.
The views expressed in this article are those of the authors and not necessarily of the National Academies of Sciences, Engineering, and Medicine.
The authors would like to thank Dan Chisholm for his useful comments on an initial version of the manuscript.
The authors declare that they have no competing interests.
Availability of data and materials
Consent for publication
Both authors consent for publication.
Ethics approval and consent to participate
The publication costs for this article were funded by Mark O’Friel, the Brinson Foundation, and the Payne Family Foundation.
About this supplement
This article has been published as part of Cost Effectiveness and Resource Allocation Volume 16 Supplement 1, 2018: Priority Setting in Global Health. The full contents of the supplement are available online at https://resource-allocation.biomedcentral.com/articles/supplements/volume-16-supplement-1.
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.