The framework is designed to support fair reasoning and deliberation to increase legitimacy of decisions. This is done by guiding decisionmakers through a fair process, using a generic interpretive frame rooted in the compassionate impetus of healthcare systems. Such frame, by providing a common road map is geared to facilitate communication across policy committee members, patients and physicians, and healthcare stakeholders at large.
The framework builds on four dimensions described below: (i) Universal impetus of healthcare systems, (ii) Reasoning, values and ethics, (iii) Evidence and knowledge on interventions, and (iv) Transformative process over time. It includes four tools for easy adaptation and operationalization: (a) concepts and operationalization (Additional file 1), (b) Adapt and pilot [step by step manual, Additional file 2), (c) Evidence matrix (step by step manual, Additional file 3), (d) Mathematical representation of reasoning (Excel calculator, Additional file 4).
EVIDEM 10th edition
The 10th edition is based on 10 years of open source development. Although EVIDEM does use some features of MCDA, its roots are not in this methodology but rather in real-life deliberation and decision (bottom-up approach). However, to facilitate its understanding, its visual representation reflects the standard steps in MCDA (Thokala) [14] [(1) goal, (2) criteria, (3) weights, (4) evidence, (5) scores, (6) visualization and uncertainty, (7) ranking and deliberation] (Fig. 1).
Universal impetus of healthcare systems
To ensure that the universal compassionate impetus of healthcare systems (i.e., achieve health for all) remains at the core of decisions, the framework includes generic criteria directly derived from this impetus. “Generic” is used in the sense of being applicable to all interventions, types of decision and region. Four universal aspects related to this impetus and 20 criteria, derived from informal and formal consultation and research [15, 16] are defined including (see part 1 and 2 of Fig. 1):
-
Three ethical imperatives (normative aspects):
-
1.
Alleviate/prevent suffering of patients (5 criteria: efficacy, safety, patient perspective, type of preventive benefit, type of therapeutic benefit).
-
2.
Prioritize those who are worst off while ensuring greatest good for greatest number (4 criteria: disease severity, unmet needs, size of population, country population priorities).
-
3.
Ensure sustainability (6 criteria: cost of intervention; other medical costs; non-medical costs; opportunity costs for the system; scope of healthcare system; environmental impact).
-
Contextual imperatives (feasibility aspect), defined as the wisdom of making decisions informed by knowledge and adapted to context (5 criteria: quality of evidence; expert consensus; system capacity; specific interests; political, historical and cultural context).
This set of generic criteria creates a generic interpretive frame which, by design, is a reminder of the compassionate impetus of healthcare systems. This should be borne in mind when adapting the framework by removing/adding generic criteria. Limiting the number of generic criteria for methodological reasons may constrain the reasoning and compromise the integrity of the comprehensive interpretive frame on which EVIDEM is built. Of note, for each generic criterion, several sub-criteria are proposed to reflect specificities of therapeutic areas or types of interventions. Details are available in the EVIDEM tool Concepts and definitions which provides a rationale on each aspect of the framework and a guidance for adaptation (Additional file 1).
Reasoning, values and ethics
Criteria are organized in an operationalizable framework, designed to structure and clarify individual reasoning in all its aspects:
-
1.
The general motivation of the individual: by stating the compassionate impetus clearly, the framework raises awareness on one’s individual motivation and its alignment with this impetus
-
2.
The individual reflection on:
-
(a)
the generic criteria considered: what matters most to me according to my own value system? what is the relative importance of criteria?; This can be elicited informally or formally through a value system elicitation method (weighting) combined with a narrative and face validity exercise to confirm that the weights reflect the value system of the individual (for details, see previous research [17] and options proposed in the Tool Adapt and pilot, Additional file 2).
-
(b)
the evidence considered: what is the evidence available (scientific, colloquial, imputed by logic, my own insights) for each criterion for the intervention appraised? Evidence is made available for each criterion using instructions described in the Tool Evidence matrix, Additional file 3); how the intervention performs on each of these criteria? This can be elicited informally or formally through an interpretive scoring scale capturing judgment (see Tool Adapt and pilot, Additional file 2).
-
3.
The conclusion of this individual reflection: the balancing act of these considerations is facilitated by the interpretive grid, which serves as a reminder us of the underlying ethical imperative associated with each criterion [18].
As illustrated in Fig. 1, the framework allows to elicit individual value systems (part 3), aligns criterion with associated evidence which give a visual cartography of available knowledge (part 4). This structure clarifies reasoning and allows to express this reasoning through interpretative scoring scales and or insights (part 5).
Evidence and knowledge on interventions
The way evidence and knowledge on interventions is conveyed to the decisionmaker has a critical impact on the decision. Therefore, detailed instructions based on good HTA practices [19] adapted to multicriteria approaches to research, synthesize and report evidence for each criterion are provided in the tool Evidence matrix (Additional file 3). These instructions aim at synthesising and presenting best available and most relevant evidence (scientific, colloquial, imputed by logic) for each criterion in a clear format. These instructions also ensure that the reflection is as unobstructed as possible by irrelevant or biased data. The tool also includes instruments to assess quality of evidence (clinical, patient reported outcomes, economic, epidemiologic), initially derived from a number of quality assessment tools [13] and updated over the years.
A transformative process
The framework is designed to promote a fair process by stimulating reflection and deliberation, fostering transparency and clarity, ensure accountability and relevance of decisions while facilitating communication, participation and collaboration as well as appeal and revision. It is geared to generates a transformation of current processes into processes that are more aligned with the principles of the A4R.
Individual reasoning made explicit through the interpretive grid of the framework becomes shareable into a group deliberation; each aspect of the grid is enriched by insights and reflection from other members of the group, transforming the individual perspective into a rich exchange though a deliberation leading to an equilibrium on which the group decision is based. Decisions made through such a process, with a constant reminder of the compassionate impetus, in a committee composed of members that represent the diverse perspective at stake, provides a good basis for legitimate decisions.
The framework implementation is meant to be phased in carefully depending on the culture of the institution and the region in which it is to be established (. e.g., perception on transparency, positioning towards diversity of perspectives). Of note, although mathematical aspects of the framework are designed to help clarify, express and share individual reasoning, this non-conventional use of numbers requires a cultural change and needs to be phased in slowly.
In a first step, the framework can be applied in a qualitative mode that uses the interpretive frame (multicriteria grid) to capture individual interpretations for each criterion in a narrative form (see Additional file 2). This supports the group deliberation, using implicit weights and scores to arrive at a decision. This piloting allows to experience the process, reveals to users its pros (efficiency gains, clarity, ease of use) and cons (changes required in current process), and initiates a reflection and transformation of activities surrounding decision processes.
A mixed qualitative/quantitative approach can be phased in if there is interest to further the transformation. When using the quantitative aspects of the framework, it is crucial to bear in mind that they are meant to help visualize and share the reasoning. They involve technical aspects such as weight and score elicitation and their aggregation, and face validity needs to be done at each step to ensure that the mathematical transformation truly reflects the reasoning (see tool Data analysis and visualisation of reasoning, Additional file 4). Such mathematical transformation allows to rank interventions. Ranking is modulated by qualitative considerations. Over time, this supports the transformation of the basket of healthcare services towards interventions well adapted to the context and contributing to more relevance, more equity and more sustainability of healthcare systems (concept of maximum value). The mathematical aspects are thus designed to help clarify, express and share the reasoning, not as a substitute of it; they increase the power of transformation towards healthcare systems geared to achieve the compassionate impetus on which they were created. For details see the important notice available in each EVIDEM tool (Additional files 1, 2, 3 and 4).
A reminder of the motivation: name and logo
The name EVIDEM (Evidence and Values Impact on DEcision Making) reflect its objective: support decision making, and the associated reasoning which requires consideration of evidence and values. The stylized V of the Logo (Fig. 2) represents the three basic ethical imperatives underlying decisions for healthcare systems: relevance at the patient level, equity at the population level and sustainability (top of V), which need to be based on a good understanding of the Context (bottom of V, contextual imperative) in which healthcare interventions are to be used. It serves as a reminder of the compassionate impetus of healthcare systems and the aspects to be kept in mind when making a decision. This “reminder” aspect is at the heart of EVIDEM.
Applications
Application is useful throughout all types of healthcare interventions, for all levels of decision, and across the globe. By design, the framework facilitates interpretations of any decision-making situation, both at the individual and a t the group level. This resulted in a variety of applications including coverage decisions (its initial object), but also for benefit-risk evaluations, shared decision-making, clinical research question prioritization, design of clinical trials. Applications are ongoing in several countries, most of these are not public domain. A few applications of EVIDEM have been published [18, 20,21,22,23,24,25,26,27].
For example, in Italy, the Lombardy Health Directorate developed a reflective multicriteria approach combined with the EUnetHTA core model to appraise health technologies; this has been in place to make coverage decisions since 2012 [28]. The reasoning of the committee members is made available openly on a web site in a format combining visual representations of the decision rationale and narratives. A key output is the facilitation of the interactions with stakeholders as a result of a transparent process. The process is evolving to further increase legitimacy by involving a wider array of stakeholders such as citizens or patients.
In Colombia, the Ministry of Health and Social Protection, carried out a consultation to adapt EVIDEM criteria to the Colombian context, as part of its exploration of methodologies for coverage decisions. In this pilot, criteria weights were elicited through participation of more than 200 stakeholders including experts and citizens [29]. A pilot testing of the adapted framework revealed that a reflective multicriteria approach for complemented by a budgeting exercise was well adapted to assess thoroughly healthcare interventions. It helped to structure and clarify reasoning and deliberation of committee members. Further developments are ongoing to implement participative approaches rooted in A4R enhanced by multicriteria approaches.
Recently, the EVIDEM framework was adapted to build the list of priority devices for cancer care published in 2017 by the WHO. It was used as a mean to collect experiential knowledge from a diversity of experts around the world and to support the reflection leading to the inclusion/exclusion from the list of priority devices. The generic tools developed for this application were also designed to be directly usable, with some contextual adaptation, by members states to make fair decisions regarding medical devices [30].
In 2017, to celebrate its 10 years of success and to facilitate its widespread use, the EVIDEM Collaboration resolved to make freely available the 10th edition of the EVIDEM Framework, with no legal binding. This represented the dissolution of the legal entity, the not-for-for profit EVIDEM Collaboration. This important step led to the transformation of the network of active members into an open international community who shares an interest in reflective multicriteria approaches.
