Skip to main content

Table 2 Overview of substantive criteria, sub-criteria, and decision aspects

From: Development of the WHO-INTEGRATE evidence-to-decision framework: an overview of systematic reviews of decision criteria for health decision-making

CriteriaSub-criteriaDecision aspects
Health related balance of benefits and harmsGeneral considerations surrounding benefit/effectBenefits/effect/efficacy/effectiveness/impact [1, 2, 4–10, 12–15, 17–27, 29, 33–36]
Health related Benefits/effect/efficacy/effectiveness/impact [1, 2, 4, 8, 11–15, 21, 23, 26, 31, 34, 35]
Uptake of intervention [15, 20, 34, 35]
Magnitude of benefit/effect/impact [2, 4, 11, 14, 18, 31, 33, 36]
Additional or indirect effects [2, 6, 33, 34]
Type and composition of effect/benefit/impact [2]Impact on mortality, survival, longevity and life expectancy [1, 2, 4, 11, 16, 19, 21, 24–26, 28, 34–36]
Last chance therapies [23, 24]
Impact on morbidity and disability [1, 2, 16, 35]
Potential changes in health consequences [24, 25]
Impact on (health-related) quality of life [2, 8, 11, 12, 14, 19, 20, 22, 25, 26, 28, 29, 31, 33, 35, 36]
Impact on patient-reported outcomes [2, 12, 16, 21, 26]
Valuation of health outcomes by patients and desirability of the effects [2]
Preventive benefits/effects or preventive approaches [1, 2, 4, 5, 16, 21, 25, 26, 31]
Character of benefit or effectOnset of effect and time until benefit [2, 11, 13]
Duration, sustainability and lasting effect [2, 11, 13, 15, 27, 31]
Individual and population level of benefitClinical benefits/effectiveness/impact [2, 3, 5, 11, 14, 17–22, 24, 25, 28, 31–33, 36]
Individual level benefit, effectiveness or impact [1, 2, 4, 18, 21, 22, 25, 26, 36]
Marginal benefits (for every patient) [20, 26]
Population level benefit, effectiveness or impact [1–5, 9, 12, 16, 18, 19, 24–27, 31, 34, 35]
Threshold effectiveness on populations (herd immunity) [15, 34]
Balance of benefits and harmsBalance of (health) benefits and harms [5, 14, 18, 19, 23, 24, 34]
General considerations surrounding harm/riskGeneral safety, risk and tolerability of intervention [2, 4, 5, 7–9, 11–17, 19–26, 29, 31, 33–36]
Magnitude and likelihood of adverse events [26, 33–35]
Valuation of health outcomes by patients and desirability of the effects regarding harms [34]
Short and long term risk and safety profile [26]
Over diagnosis and over treatment [16, 26]
Stigmatization [2, 26]
Risk of failure of intervention [15, 34, 35]
Burden of treatment [2, 26, 33]
Risk of inappropriate use [8, 16, 25, 26]
Impact on disease patterns and reduced long-term effectiveness [15, 34]
Other or additional adverse events [2]
Health-related need and priorityHealth-related needs: in general [1, 2, 11, 14, 22, 23, 24]
Burden and impact of disease: in general [1, 2, 4, 7–10, 13–16, 21, 23–26, 33, 34, 36]
Magnitude of the problem [4, 10, 25, 34]
Burden of disease measured through epidemiological indicators [1, 2, 4, 5, 8, 11, 13–15, 21, 23–26, 28, 33–36]
Size of affected population and number of potential beneficiaries [1, 2, 4, 5, 8, 10, 11, 13–17, 20–26, 28, 31–34, 36]
Maximum potential for disease burden reduction [10]
Severity of disease/condition: in general [1, 2, 4, 5, 9–12, 14, 16–19, 21–26, 28, 29, 31, 33–36]
Severity of disease/condition: long term outcomes [34, 35]
Severity of disease/condition:
life threatening disease/condition and prognosis without treatment [1, 5, 11, 16, 21, 28, 29, 34]
Severity of disease/condition:
late stage or end-of life status of disease/condition [5, 21, 24, 25, 28, 29, 36]
Outbreaks and epidemic potential [15, 34]
Urgency and emergencies [1, 2, 13, 25]
Human and
individual
rights
Human rights considerations [2, 11, 12, 20, 36]
Autonomy and informed consent [2, 11, 35]
Privacy and confidentiality [26, 35]
Intrusiveness of intervention [22]
Acceptability considerationsPerceived priority of the problemPublic perception of disease burden, disease risk or severity [15, 34, 35]
AcceptabilityAcceptability in general [2, 4, 6, 14, 15, 24, 26, 27, 34–36]
Acceptability of cost and financial outcomes [2, 14, 25, 26, 34]
Acceptability by beneficiariesAcceptability by beneficiaries: in general [2, 6, 8, 11, 15, 16, 26, 35]
Comfort, convenience and user experience [2, 11, 12, 14, 15, 20, 24–26, 31, 33, 35]
Acceptability by those providing interventionAcceptability by those providing intervention [15, 16, 34, 35]
Social and cultural acceptabilitySocial and cultural acceptability [2, 4, 8, 9, 11, 15, 18, 21, 22, 25, 26, 31–34, 36]
Ethical/moral acceptability [2, 6, 11, 12, 15, 18, 22, 25, 26, 32–34, 36]
Stakeholder demand, interests and pressuresAdvocacy and stakeholder (in general) interests and pressures [1, 2, 6, 14, 15, 21, 25, 33, 36]
Demands, interest and pressures of the public [1, 2, 8, 14, 24–26, 36]
Demands, interest and pressures by industry [2]
Pressures, demand and interest of beneficiaries and patient representatives [2, 17, 20, 22, 25, 32, 33]
Pressures, demand and interest of those providing intervention [2, 17, 20, 25, 26, 32, 33]
Media attention and coverage [1, 32]
Societal considerationsSocietal needs and prioritySocial burden of disease (individual/population) [8, 18, 21, 29, 34, 35]
Social needs [24]
Economic burden of disease on society (in general) [32]
Social and societal impactSocial impact or benefits [2, 3, 5, 6, 14, 21, 23–26, 31, 33, 35]
Impact on non-health outcomes (in general) [15, 18, 26, 34]
Impact on poverty [1, 2, 4, 14, 25, 26, 36]
Relevance to social development of the country [10]
Value of hope [11]
Raise profile of condition [14]
Impact on economyImpact on economy [2, 25, 26]
Impact on productivity and population in productive age [2, 12, 16, 25]
Relevance to economic development of the country
Innovativeness (potential to encourage innovation) [2, 9, 10, 14, 21, 23–25, 31]
Environmental impactEnvironmental and/or ecological impact of intervention (in general) [14, 16, 21, 29–31, 33]
Impact on future generationsImpact on future generations [19]
Considerations of equity, equality and fairnessEquity and equalityEquity/equality considerations:
in general [1–4, 9–11, 14, 15, 18, 19, 22–24, 26, 27, 29, 33–36]
Fairness considerations:
in general [2, 3, 6, 11, 16, 26, 36]
Impact on (health) (in-)equity/(in-)equality [2, 6, 10, 14, 26, 31, 34, 35]
Distribution of benefits and harms [10, 26]
AccessibilityAccessibility in general [2–4, 6, 9, 14–17, 19, 21, 24, 29, 31, 33–35]
Equity in accessibility [1, 2, 4, 6, 7, 11, 25, 26, 35]
Physical and spacial accessibility [2, 4, 6, 16, 26]
Timeliness of access (time spent waiting for treatment) [2, 3, 22, 36]
Informational accessibility [11]
Financial accessibility of intervention/Affordability [4, 9, 10, 12, 14, 19, 25, 26, 34–36]
Affordability: risk of catastrophic health expenditure [11, 16, 25]
Affordability: cost and Financial impact on beneficiaries [11, 13, 14, 16, 22, 25, 26, 31, 34, 35]
AvailabilityAvailability/lack of suitable alternatives [1, 2, 5, 11, 12, 14, 17, 19–25, 28, 32, 34–36]
Limitations of alternative interventions [2, 11, 14, 15, 25, 26, 34, 35]
Rare diseases/orphan disease [12, 14, 16, 21, 24, 28]
Unmet needs [11, 12, 16, 21, 24, 29]
ResponsibilityAbility to reduce own health risk and conditions arising from patient behavior [4, 5, 11, 16, 24, 25]
Non-discriminationNon-discrimination [11, 23, 24]
Consideration regarding specific populationsConsideration of high-risk populations [2, 11, 14, 26]
Consideration of vulnerable populations [1, 2, 4, 11, 21, 25, 26]
Consideration of Socio-economic status [4, 16, 25, 26, 36]
Consideration of sex, gender, and/or sexual orientation [2, 6, 16, 25, 26, 34]
Consideration of race/ethnicity [25, 36]
Consideration of care giver responsibilities [29]
Consideration of age-groups [1, 2, 4, 11, 14, 16, 26, 28]
Consideration of place of living [11, 16, 25]
Consideration of identity and ideology [25]
Other group related considerations [11]
Cost and financial considerationsFinancial burden of diseaseFinancial burden of disease or current intervention on health system [2, 15, 16, 25, 32, 34, 35]
Cost and budget impact of interventionCost/budget impact:
in general [1, 2, 4–6, 8, 9, 11–19, 21–23, 25–31, 33–36]
Cost per unit/usage [1, 2, 5, 11, 15, 17, 20, 23–26, 33, 34, 36]
Cost over time [20, 21, 25, 32, 34]
Long term cost/budget impact [25]
Overall cost/budget impact [1, 2, 11, 22, 26, 36]
Direct cost [11, 25, 32, 34, 35]
Indirect/additional/hidden cost [11, 34–36]
Marginal cost [11, 25]
Opportunity cost [2, 11, 21, 31]
Impact on other spending/investments [2, 6, 14, 25]
Investment/start-up cost [2, 22, 30]
Operating cost [2, 30]
Lifecycle cost of intervention/technology [30]
Economic/financial benefits and cost-minimization potential [1, 3, 8, 11, 13, 22, 25, 27, 28, 34–36]
Cost to/budget impact on government or society [1, 2, 5, 18, 25, 26, 29, 33, 36]
Relation of cost and benefitsRelation of cost and benefits [1–7, 9–16, 18–26, 28, 31–36]
Financial contextAppropriateness [2, 14]
Financial feasibilityAffordability to health system
Availability or lack of funds/funding [1–4, 6, 10–12, 14, 15, 21–25, 31, 34, 36]
Financial sustainabilityFinancial sustainability of intervention and consistency of funding [4, 11, 15, 31, 34]
Feasibility and health system considerationsHealth-system related needs and priorityBurden of disease on health system [2, 25, 26, 31, 33, 34]
Needs of the health-system
Feasibility and capacity to implementGeneral feasibility considerations [2–4, 7, 9, 10, 14–16, 22–27, 31, 33, 34, 36]
Technical feasibility considerations [6, 26, 33]
Practical feasibility considerations [4, 6, 26]
Capacity to implement [1, 2, 6, 14, 21, 25, 29]
Considerations of management and organization of health systemAvailability of, capacity of and need for management and organizational structure [1, 2, 16, 22, 26, 34]
Impact on management and organizational structure [2, 8, 11, 25]
Logistical considerations [2, 15, 34]
Availability of, capacity of and need for monitoring, surveillance and information system [34, 35]
Resource considerationsAvailability of, capacity of and need for resources (in general) [1, 2, 10, 14, 30, 36]
Impact on resources (in general) [2, 22–24]
Efficiency of resource use [18, 24, 31]
Considerations of human resources and their skillsAvailability of, capacity of and need for human resources [2, 10, 13, 14, 18, 22, 26, 30]
Availability of, capacity of and need for skill levels/knowledge of human resources [2, 3, 14, 16, 22, 25, 30, 35]
Impact on human resources and skill levels [2, 14, 18, 25, 31]
Considerations of Non-financial physical resources (equipment, infrastructure)Capacity of, availability of need for of physical resources and infrastructure [16, 18, 22, 25, 26, 34–36]
Impact on non-financial physical resources and infrastructure [18]
Interaction with and impact on health systemImpact on performance of health system
and impact on other services [1, 2, 4, 6, 8, 10, 11, 14, 16, 18, 20, 22, 26, 31, 33–35]
Interaction and compatibility with other health system components [1–3, 6, 14, 22, 25, 33–35]
Ease of use, application and burden of intervention [13, 15, 22, 26, 35]
Appropriateness within health systemAppropriateness (in general) [2, 3, 11, 21, 25, 26, 31, 36]
Appropriateness of intervention for specific context [2, 16, 30]
Legislative and regulatory considerationsAdherence to legal requirements, constrains and implications [2, 8, 10, 15, 16, 18, 25, 26, 31–35]
Adherence to other directives, standards and requirements [2, 6, 14, 22, 30, 34]
Political considerationsPolitical acceptability, interests and pressures [1–4, 6, 10, 11, 14, 18, 21, 25, 26, 33–36]
Donor and global interests and pressures [2, 3]
Political impact [2, 25, 33, 35]
Alignment with priorities [1–4, 6, 13–15, 21, 23, 24, 26, 29, 30, 34, 35]
Mission, mandate and goals of health system [2, 13, 14, 21, 25]
Strategic considerationsStrategic planning and considerations [1, 2, 13, 19, 25, 31]
Existing cooperation [2, 10, 14, 22]
Decisions and practice of other institutions or stakeholders [2, 11, 17, 24, 29, 34]
Alignment with recommendations, guidelines and standards [1–3, 7, 11, 14, 16, 21, 24, 26, 29, 34]
Historical context and past decisions [1–3, 11, 13, 14, 17, 19–22, 25]
Availability of incentives [1, 2]
Impact on future decisions [6, 14]
Keeping promises and commitments [2, 13]
Characteristic of intervention(technical) Complexity of intervention [2, 4, 6, 16, 21]
Scalability of intervention [9, 27]
Ability to evaluate intervention [2, 33–35]
Reversibility of intervention [2]
Flexibility of implementation [2]
Uniqueness [21]
Frequency of use and expected level of usage/activity [1, 2, 6, 8, 14, 25, 26, 32]
Dependence on maintenance [26, 30]
Sustainability of intervention utilization
(e.g. Supply of parts, vaccines) [2, 14, 15, 27, 29, 34]
Position of intervention in care pathway [5, 12, 20, 24]
Additional uses of intervention [5, 21, 23–25]
  1. 1 Youngkong et al. 2009 [10], 2 Guindo et al. 2012 [6]; 3 Waithaka et al. 2018 [36]; 4 Wiseman et al. 2016 [35], 5 Fischer 2012 [43], 6 Wahlster et al. 2015 [13], 7 Ricciardi et al. 2015 [47], 8 Specchia et al. 2015 [34], 9 Hayati 2018 [42], 10 McGregor et al. 2014 [41], 11 MacLeod et al. 2016 [7], 12 Angelis et al. 2018 [48], 13 Barasa et al. 2015 [57], 14 Cromwell et al. 2015 [5], 15 Burchett et al. 2012 [4], 16 Varela-Lema et al. 2016 [38], 17 Vuorenkoski et al. 2008 [12], 18 Cowles et al. 2017 [50], 19 Golan et al. 2011 [33], 20 Erntoft et al. 2011 [51], 21 Friedmann et al. 2018 [28], 22 Ølholm et al. 2015 [37], 23 Stafinski et al. 2011a [19], 24 Stafinski et al. 2011b [20], 25 Mobinizadeh et al. 2016 [32], 26 Marsh et al. 2014 [27], 27 Rudan et al. 2017 [39], 28 Ghijben et al. 2018 [31], 29 Drake et al. 2017 [23], 30 Diaconu et al. 2017 [52], 31 Polisena et al. 2013 [22], 32 Noorani et al. 2007 [8], 33 Johnson et al. 2009 [53], 34 González-Lorenzo et al. 2015 [54], 35 Piso et al. 2009 [21], 36 Niessen et al. 2012 [11]
  2. The publication by Niessen et al. (Ref. 36) is highlighted in italic, as it is the only study meeting all criteria of our critical appraisal tool
\