Skip to main content

A comprehensive framework for considering additional unintended consequences in economic evaluation



In recent years there has been a growth in economic evaluations that consider indirect health benefits to populations due to advances in mathematical modeling. In addition, economic evaluations guidelines have suggested the inclusion of impact inventories to include non-health direct and indirect consequences. We aim to bring together this literature, together with the broader literature on internalities and externalities to propose a comprehensive approach for analysts to identify and characterize all unintended consequences in economic evaluations.


We present a framework to assist analysts identify and characterize additional costs and effects beyond that of direct health impact primarily intended to be influenced by the intervention/technology. We build on previous checklists to provide analysts with a comprehensive framework to justify the inclusion or exclusion of effects, supporting the use of current guidelines, to ensure any unintended effects are considered. We illustrate this framework with examples from immunization. These were identified from a previous systematic review, PhD thesis work, and general search scoping in PubMed databases.


We present a comprehensive framework to consider additional consequences, exemplified by types and categories. We bring this and other guidance together to assist analysts identify possible unintended consequences whether taking a provider or societal perspective.


Although there are many challenges ahead to standardize the inclusion of additional consequences in economic evaluation, we hope by moving beyond generic statements to reporting against a comprehensive framework of additional effects we can support further consistency in this aspect of cost-effectiveness analysis going forward.


Guidelines for performing economic evaluations of healthcare interventions have emerged as a tool to support the quality and standardization of cost-effectiveness analysis [1,2,3]. Generally, these guidelines state that analysts should consider all relevant direct and indirect effects [4, 5], although these may be limited by the payer perspective, and often are contained to health effects. Some guidelines provide additional guidance on the inclusion of non-health effects [6, 7]. For example, the Second Panel on Cost-Effectiveness in Health and Medicine defines societal benefits as non-health effects and costs which are not accrued to health budgets, and provides a reporting checklist, known as an ‘impact inventory’ to report these. The checklist includes impacts on areas such as educational outcomes. The International Society for Pharmacoeconomics and Outcomes Research (ISPOR) has also highlighted the importance of mapping indirect health and non-health effects into economics frameworks for value assessment. The ISPOR Special Task Force report states that while indirect effects such as labor productivity and adherence improving factors are covered to some extent in current cost-effectiveness analyses, other categories such as scientific spillover effects are virtually absent [8]. Finally, various infectious disease-specific guidelines address the inclusion of non-direct effects, and some (for example, immunization) have made also attempts at scoping possible types of impact [9,10,11].

Despite this guidance, it can be challenging for analysts to comprehensively map out which non-direct health and non-health effects to include in analyses, in addition to the primary intended effect [12,13,14]. As many as two-thirds of cost-effectiveness studies do not consider how to include indirect health effects even when they may be substantial [15, 16]. We present here a framework to assist analysts identify and characterize additional costs and effects beyond that of direct health impact primarily intended to be influenced by the intervention/technology. Our aim is not to argue that all these effects should be included in every economic evaluation; this should depend on the perspective of the evaluation and strength of evidence supporting importance of the effect. Instead we aim to build on previous checklists to provide analysts with a comprehensive framework to justify the inclusion or exclusion of effects, supporting the use of current guidelines, to ensure any unintended effects are considered. We illustrate this framework with examples from immunization (see Table 1). These were identified from a previous systematic review [15], PhD thesis work [42], and general search scoping in PubMed databases.

Table 1 Examples of ‘internal’ and ‘external’ person consequences using immunization(s)


Search strategy used to illustrate the framework

The immunization examples used to illustrate the proposed framework were identified from a previous review on cost-effectiveness analyses of human vaccines. Nymark et al. [15] utilized the following search strategy which they limited to English language with free text and MeSH terms; vaccin, economic evaluat, humans.

Other examples of unintended consequences of immunization were identified from a PhD thesis cost-effectiveness analysis of Bacillus Calmette–Guérin (BCG) versus no BCG vaccination for prevention of atopic dermatitis in neonates with predisposition to atopic disease [42].

Finally, a search scoping strategy which consisted of search terms ‘cost–benefit analysis’ or ‘cost–benefit’ and ‘externality’ was utilized to identify any additional examples of unintended consequences of immunization to illustrate the framework.

Development and validation of the framework

The framework was developed following a review of current economic evaluations guidelines [1,2,3,4,5,6,7,8,9,10,11] and reviews of the economic evaluation for TB and immunization—in which relevant unintended consequences were identified and classified into different types; biology, demand and supply. Unintended consequences were structured based on existing frameworks such as the Second Panel on Cost-Effectiveness in Health and Medicine and ISPOR guidelines and then generalize commonalities among secondary effects from these.

Additional unintended consequences were also identified previous PhD thesis work on immunization [42].

Types and categories of ‘internal’ and ‘external’ person consequences

Interventions can have impact beyond the intended direct health consequences; such as ‘internal’ consequences (occurring ‘within’ in the individual) and ‘external’ consequences (when the reaction occurs ‘outside’ the individual). We refer to the former as an ‘internality’ and the latter is commonly referred to in economics as an externality; defined as a cost or benefit which is caused by one person’s action while impacting others who are not part of the action [17]. In non-economic literature the term externalities is often used interchangeably in reference to indirect effects, unintended consequences, spillover effects and non-direct effects.

Depending on perspective, analysts may wish to consider both additional health and non-health consequences. Within health impact, there are several types of ‘internalities’ and ‘externalities’ (see Fig. 1). These can be divided into biological effects and demand-side and supply-side behavioral consequences. The first type ‘biological’ includes the impact of the intervention on other diseases, infection and pathogens. Interventions can also impact individual, household and population health related consumption; ‘demand-side’ impact. Finally, interventions can also impact the supply side, by changing the behavior of health providers and impacting other health services or the provision of non-health services. Likewise, there may be internal and external non-health consequences demand and supply side impact. Figure 1 illustrates our framework and we provide further explanation and an example below.

Fig. 1

Conceptual framework ‘internal’ and’external’ consequences

Health sector perspective

Additional biological effects

Biology refers to any chemical, molecular, cellular, immune or physiologic process or mechanism which is activated inside the human body [18]. Central to the description of the type biology are the functions and mechanisms which work within the human body and relate to how organisms, organ systems, cells and molecules carry out their chemical and physical functions and how these interactions are regulated, including control mechanisms and communication between cells [19]. For the type ‘biology’ there are three categories of potential additional ‘internal’ and ‘external’ person (host) effects. These are: (1) Non-specific effects (NSE) (impacts on other diseases); (2) transmission effects (infection to others) and (3) pathogen response (impact on the pathogen, such as creating new resistant disease).

An example of the first category (NSE) is the case of the BCG vaccine is indicated for the prevention of tuberculosis (TB) [20]. NSE of immunization are beneficial effects which offer protection beyond specific pathogens [21]. A study of BCG immunization at birth in low-birth weight neonates (< 2500 g) in Guinea-Bissau found a reduced risk of sepsis and pneumonia (non-specific diseases which are unrelated to the direct proportion which BCG offers against tuberculosis) (see Table 1) [22]. Likewise, BCG vaccine has been observed to train the adaptive immune response to become more vigilant to subsequent vaccinations [23]. Other examples of internal additional health effects include the impact of anti-retrovirals for HIV on the development of active TB from infection. NSE can be divided into two sub-categories, those that have consequence for other diseases and those that impact the likelihood of subsequent infections.

The second category, transmission is defined as the passing of an infectious agent (pathogen) from an infected host individual to another individual. The externality of transmission is increasing being included in economic evaluation of infectious diseases. In addition, the transmission of disease from human to human is reduced by immunization. Herd immunity is an indirect or ‘external’ person effect associated with immunization which occurs when unvaccinated (susceptible) individuals avoid infectious diseases because a large enough percentage of the population who surround them are immunized [24]. Thereby providing a measure of indirect protection for individuals who are not immunized. An example of herd immunity is the HPV vaccination programme for females in Australia which is associated with declines in the diagnoses of genital warts in young indigenous men in Australia [25]. Hence, vaccinated females not only reduced their own direct susceptibility but also reduced the risk for unvaccinated males who benefited from a measure of indirect herd protection even though they were not the direct recipients of the HPV vaccine.

The third category of biology is ‘pathogen response’ which refers to the human body’s reaction when an infectious agent (pathogen) causes disease or illness to its host. The category is classified as an ‘internal’ additional effect. In the case of immunization, vaccination can result in changes in the underlying serotype of the pathogen (in the case of other diseases, incomplete intervention can also result in resistance). Serotype is the term used to refer to a group of organisms within a species that have the same type and number of surface antigens. Antigens are molecules that are capable of stimulating an immune response specifically activating the body’s infection-fighting white blood cells. Serotype replacement is a phenomenon which induces resistance to sub-types of serotypes if the frequency of a sub-type of serotype declines due to high levels of immunity, allowing other serotypes to replace it [26]. Widespread use of pneumococcal conjugate vaccines has been associated with serotype replacement or shifting. As demonstrated in Navarra, Spain, the incidence of invasive pneumococcal disease (IPD) from serotypes not included in the PCV7 vaccine increased due to resistance following the introduction of this vaccine, which was attributed to serotype replacement. However, the incidence of IPD declined, and the change in patterns of serotypes causing IPD produced marked declines in children and moderate declines in adults following the replacement of the PCV7 vaccine by the PCV13 vaccine [27].

Another potential additional pathogen response is cross-protection. Cross-protection is the protection conferred on a host (‘internal’) by infection with one strain of a virus that prevents infection by a closely related sub-strain of that virus. Serotype cross-over effects are an example of this. In Sweden, a direct comparison of the impact against IPD was performed in 21 counties who used either PCV10 or PCV13 vaccination. An increase in serotype 6C in the PCV10 counties was observed but not in the PCV13 counties. This was assumed to suggest that serotype 6A, which is included in PCV13 but not in PCV10, offers cross-protection against serotype 6C [28].

Finally, pathogens can induce side effects. A side effect refers to an effect, whether therapeutic or adverse, that is secondary to the one intended [29]. Although the term is predominantly employed to describe adverse effects, it can also apply to unintended consequences of the use of a drug. An example of this is the Hemophilus influenzae type B (Hib) vaccination that by preventing bacterial infections from occurring or spreading prevents infections and thus reduces the use of antibiotic treatment that would have been used to treat infections. Subsequently this reduces the development of secondary effects such as antimicrobial resistance [30].

Additional demand-side consequences

The law of supply and demand is a theory that explains how the market allocates resources based on the interaction between the sellers of a resource and the buyers of that resource [31]. The demand-side refers to the demand of goods and services which individuals want to buy or consume [32]. Previous frameworks have outlined the type of demand and supply constraints that may be considered in economic evaluations in health sector [33]. In terms of additional consequences, the type demand-side, there are two categories related to health: Changes in health influencing behavior and changes in health services consumption (both ‘internalities’).

Changes in health behavior and health services consumption

Health behaviour refers to a person’s beliefs and actions regarding their health and well-being. An example of a positive additional behaviour consequences was observed in Nepal, where changes in mothers’ behaviours and knowledge regarding hygiene and sanitation practices have been observed as a consequence of a universal immunisation programme which educates mothers about vaccination [34]. Health care consumption refers to the utilization of additional (or reduction in utilization of) services by persons (‘internal’ person) for the purpose of preventing and curing health problems, promoting maintenance of health and well-being, or obtaining information about one’s health status and prognosis. For example, following maternal tetanus vaccination in Bangladesh the consumption of health care services (‘internal’ person) was markedly reduced indicating a positive change in the need to seek health care services [35].

Additional supply-side consequences

The supply-side refers to the part of the economy which involves the production of goods or supplying of services which are available to consumers or individuals [32]. For the type supply-side, we identify one category that fall under the health sector perspective: health systems (‘external’).

Health systems

Providing an intervention can have many positive and negative spillovers in the health system. In addition, the direct opportunity cost of spending on interventions, the way they are delivered may influence the quality and delivery of other health services. For example, training provided as part of the scale-up of immunization programmes, may strengthen general clinical and non-clinical skills, that may have positive benefits for the delivery of other services. A further example the benefits of influenza vaccination of healthcare workers is important for protecting staff and patients, for example previously influenza vaccinated healthcare providers demonstrated positive behavioral changes in their willingness to be vaccinated in forthcoming seasons thus generating additional benefit to the initial vaccination [36].

Finally, in recent years there has been some attention to the potential, particularly of new interventions, to have spillover scientific benefits for the development of other health related technologies. For example, the knowledge of the mechanism of action associated with a new a drug or vaccine might offer value beyond itself and lead to later innovations in drug development or the treatment of very different diseases. The BCG vaccine exemplifies this by subsequent discovery resulting in treatment possibilities aside from those originally indicated (see Table 1).

Societal perspective

There is a wide literature of the possible non-health impacts of health interventions. These impacts can occur at the individual, household, and have follow-on consequences for the sectoral and macro-level.

Demand-side consequences

For the type demand-side, we highlight two broad categories: behavior/education/knowledge (‘internal’ person), and consumption of non-health goods.

Non-health changes in behavior and consumption

Health intervention can impact non-health related behaviors in many ways. One example is the impact on health on education. Education is defined as the process of learning and acquisition of knowledge, skills, values, beliefs, and habits. An illustration of how immunization enhances learning and the acquisition of knowledge (education) ‘external’ to the individual originates from South Africa. The pneumococcal conjugate vaccines have been associated with children improving their educational attainment, due largely to the prevention of pneumococcal pneumonia [14]. Changes in educational attainment may have follow-on consequences for a wide range of behaviors that impact social welfare.

Health intervention can also result in changes of consumption of non-health goods Households have reported negative changes in behaviours such as food consumption and school attendance in children when a parent in the household has cervical cancer. This resulted in negative impact on educational attainment for children in the household which could have been prevented with the HPV vaccination [37].

Supply-side consequences

For the type supply-side, we identify two categories that fall under the health consequences perspective: labor productivity (‘internal’ person) and provision of non-health services.


Productivity impact is commonly captured in a societal perspective (although typically considered as a cost, rather than on the impact side of the incremental cost-effectiveness ration). In respect of immunization, the Hib vaccination has been demonstrated to increase mothers’ productivity in the labor market [38]. Hib vaccination can reduce child mortality resulting in mothers of vaccinated children can achieve their target family size through fewer births. Having fewer children, may enable mothers to invest more resources in their own development which results in increased individual productivity (‘internal’ person).

There may also changes in productivity in respect of informal labour. Informal care is generally defined as the unpaid care provided to older and dependent persons by a person with whom they have a social relationship, such as a spouse, parent, child, other relative, neighbour, friend or other non-kin [39]. An illustration of this comes from a Belgium study in which the caregiver burden within households in which children were not vaccinated against rotavirus was markedly severe; especially if no medical care was sought [40].

Non-health sector service provision

Non-health sector public service provision (‘external’) include areas like (e.g. public safety, infrastructure and social support services). An example of how immunization impacts services external to the healthcare system is yearly influenza immunization among the elderly. This reduces infection and related complications in the elderly and therefore results in reduced use of long-term social support systems outside the health care sector, such as institutional care in residential care home [41]. A simple checklist addresses (Table 2) how these categories of the effects of health technologies can be measured (tangible) or not (intangible) and what may condition their classification and inclusion into the economic evaluations.

Table 2 Checklist ‘internal’ and ‘external’ person consequences

Discussion and conclusion

We present a comprehensive framework to consider additional consequences. While there is other guidance, we bring this guidance together to assist analysts identify possible unintended consequences whether taking a provider or societal perspective. There is currently a rigorous debate around capturing the full value of health interventions both within and beyond the health sector. The intention of this paper is not to support the inclusion of all additional consequences in every economic evaluation, but instead provide a framework to identify those that may be important. Moreover, any effects identified may not be additive in terms of overall value. Many of the effects listed interact (also with intended direct health effect) and care should be taken to also identify any interactions, at the individual, household, sectoral or macro-level.

The consideration of additional effects can influence the analytic choice of health outcome measures used, and non-health outcome measures included. For example, vaccine-induced improvements to immunity against non-targeted antigens (i.e. ‘internal’ person effects) manifest in non-specific epidemiological health outcomes, such as for example all-cause acute gastroenteritis related hospitalizations that may not typically measure in vaccine trials.

In addition, it may impact other methodological choices. For example, if the framework identifies a potentially major non-specific effect, it may be important to identify a comparator that also addresses that effect in the economic evaluation. For example, if the effect of the BCG vaccine extends beyond the protection against TB, and has substantial effects on auto-immune conditions (e.g. atopic dermatitis in high risk infants) [42], an economic evaluation could consider comparing interventions or strategies beyond TB. Furthermore, the analytical choice of target population and subgroups for the economic evaluation may also change when considering non-direct effects. The additional effects of the BCG vaccine, as described above, may result in the comparison of the existing strategy of BCG vaccination to a strategy in which subgroups or risk groups who are not recommended to receive the vaccine on grounds of risk of TB are included.

Applying this framework will not be without empirical challenges. Firstly, once potential additional consequences are identified, their importance will need to be assessed. Determining when a consequence is sufficiently important is often challenging a priori and will be based on a mixture of prior evidence of plausible effect. Moreover, even where there is evidence from other populations, the data to parameterize additional consequences may not be available for the population being evaluated. Further guidance is therefore required as to how to make the decision to include an additional consequence identified or when to simply report it qualitatively (as in the Second Panel impact inventory). While guidelines are clear that analyst should include all costs and effects relevant to the study perspective, practice on which are included varies substantially, which may lead to inconsistent decisions. Although there are many challenges ahead to standardize the inclusion of additional consequences in economic evaluation, we hope by moving beyond generic statements to reporting against a comprehensive framework of types and categories of additional effects we can support further consistency in this aspect cost-effectiveness analysis going forward.

Availability of data and materials

Secondary data and material only


  1. 1.

    Rudmik L, Drummond M. Health economic evaluation: important principles and methodology. Laryngoscope. 2013;123(6):1341–7.

    PubMed  Google Scholar 

  2. 2.

    Edwards RT, Charles JM, Lloyd-Williams H. Public health economics: a systematic review of guidance for the economic evaluation of public health interventions and discussion of key methodological issues. BMC Public Health. 2013;13:1001.

    PubMed  PubMed Central  Google Scholar 

  3. 3.

    Byford S, Raftery J. Perspectives in economic evaluation. BMJ. 1998;316(7143):1529–30.

    CAS  PubMed  PubMed Central  Google Scholar 

  4. 4.

    Drummond MF, Jefferson TO. Guidelines for authors and peer reviewers of economic submissions to the BMJ. The BMJ Economic Evaluation Working Party. BMJ. 1996;313(7052):275–83.

    CAS  PubMed  PubMed Central  Google Scholar 

  5. 5.

    Husereau D, Drummond M, Petrou S, et al. Consolidated Health Economic Evaluation Reporting Standards (CHEERS)–explanation and elaboration: a report of the ISPOR Health Economic Evaluation Publication Guidelines Good Reporting Practices Task Force. Value Health. 2013;16(2):231–50.

    PubMed  Google Scholar 

  6. 6.

    Claxton K, Revill P, Sculpher M, Wilkinson T, Cairns J, Briggs A. The gates reference case for economic evaluation. Seattle: The Bill and Melinda Gates Foundation; 2014.

    Google Scholar 

  7. 7.

    Sanders GD, Neumann PJ, Basu A, et al. Recommendations for conduct, methodological practices, and reporting of cost-effectiveness analyses: second panel on cost-effectiveness in health and medicine. JAMA. 2016;316(10):1093–103.

    PubMed  Google Scholar 

  8. 8.

    Lakdawalla DN, Doshi JA, Garrison LP Jr, Phelps CE, Basu A, Danzon PM. Defining elements of value in health care-a health economics approach: an ISPOR Special Task Force Report [3]. Value Health. 2018;21(2):131–9.

    PubMed  Google Scholar 

  9. 9.

    Ultsch B, Damm O, Beutels P, et al. Methods for health economic evaluation of vaccines and immunization decision frameworks: a consensus framework from a European Vaccine Economics Community. Pharmacoeconomics. 2016;34(3):227–44.

    PubMed  Google Scholar 

  10. 10.

    Pitman R, Fisman D, Zaric GS, et al. Dynamic transmission modeling: a report of the ISPOR-SMDM modeling good research practices task force-5. Value Health. 2012;15(6):828–34.

    PubMed  PubMed Central  Google Scholar 

  11. 11.

    Mauskopf J, Standaert B, Connolly MP, et al. Economic analysis of vaccination programs: an ISPOR good practices for outcomes research task force report. Value Health. 2018;21(10):1133–49.

    PubMed  Google Scholar 

  12. 12.

    Deogaonkar R, Hutubessy R, van der Putten I, Evers S, Jit M. Systematic review of studies evaluating the broader economic impact of vaccination in low and middle income countries. BMC Public Health. 2012;12:878.

    PubMed  PubMed Central  Google Scholar 

  13. 13.

    Ozawa S, Mirelman A, Stack ML, Walker DG, Levine OS. Cost-effectiveness and economic benefits of vaccines in low- and middle-income countries: a systematic review. Vaccine. 2012;31(1):96–108.

    PubMed  Google Scholar 

  14. 14.

    Barnighausen T, Bloom DE, Canning D, O’Brien J. Accounting for the full benefits of childhood vaccination in South Africa. S Afr Med J. 2008;98(11):842–6.

    PubMed  Google Scholar 

  15. 15.

    Nymark LS, Sharma T, Miller A, Enemark U, Griffiths UK. Inclusion of the value of herd immunity in economic evaluations of vaccines A systematic review of methods used. Vaccine. 2017;35(49 Pt B):6828–41.

    PubMed  Google Scholar 

  16. 16.

    Drake TL, Devine A, Yeung S, Day NP, White LJ, Lubell Y. Dynamic transmission economic evaluation of infectious disease interventions in low- and middle-income countries: a systematic literature review. Health Econ. 2016;25(Suppl 1):124–39.

    PubMed  PubMed Central  Google Scholar 

  17. 17.

    Bhattacharya J, Packalen M. The other ex ante moral hazard in health. J Health Econ. 2012;31(1):135–46.

    PubMed  Google Scholar 

  18. 18.

    Woese CR. A new biology for a new century. Microbiol Mol Biol Rev. 2004;68(2):173–86.

    CAS  PubMed  PubMed Central  Google Scholar 

  19. 19.

    Lemoine M, Pradeu T. Dissecting the meanings of “Physiology” to assess the vitality of the discipline. Physiology (Bethesda). 2018;33(4):236–45.

    PubMed  Google Scholar 

  20. 20.

    Benn CS, Netea MG, Selin LK, Aaby P. A small jab—a big effect: nonspecific immunomodulation by vaccines. Trends Immunol. 2013;34(9):431–9.

    CAS  PubMed  Google Scholar 

  21. 21.

    Higgins JP, Soares-Weiser K, Lopez-Lopez JA, et al. Association of BCG, DTP, and measles containing vaccines with childhood mortality: systematic review. BMJ. 2016;355:i5170.

    PubMed  PubMed Central  Google Scholar 

  22. 22.

    Biering-Sørensen S, Aaby P, Lund N. Early BCG-Denmark and neonatal mortality among infants weighing < 2500 g: a randomized controlled trial. Clin Infect Dis. 2017;65(7):1183–90.

    PubMed  PubMed Central  Google Scholar 

  23. 23.

    Arts RJW, Moorlag S, Novakovic B, et al. BCG Vaccination protects against experimental viral infection in humans through the induction of cytokines associated with trained immunity. Cell Host Microbe. 2018;23(1):89–100 e105.

    Google Scholar 

  24. 24.

    Barnighausen T, Bloom DE, Cafiero-Fonseca ET, O’Brien JC. Valuing vaccination. Proc Natl Acad Sci USA. 2014;111(34):12313–9.

    CAS  PubMed  Google Scholar 

  25. 25.

    Ali H, McManus H, O’Connor CC, et al. Human papillomavirus vaccination and genital warts in young Indigenous Australians: national sentinel surveillance data. Med J Aust. 2017;206(5):204–9.

    PubMed  Google Scholar 

  26. 26.

    Moraga-Llop FA. Pneumococcal disease and emergence of serotypes in the vaccine era. Transient trends and serotype replacement. Enferm Infecc Microbiol Clin. 2009;27(1):1–4.

    PubMed  Google Scholar 

  27. 27.

    Guevara M, Ezpeleta C, Gil-Setas A, et al. Reduced incidence of invasive pneumococcal disease after introduction of the 13-valent conjugate vaccine in Navarre, Spain, 2001–2013. Vaccine. 2014;32(22):2553–62.

    PubMed  Google Scholar 

  28. 28.

    Naucler P, Galanis I, Morfeldt E, Darenberg J, Ortqvist A, Henriques-Normark B. Comparison of the impact of pneumococcal conjugate vaccine 10 or pneumococcal conjugate vaccine 13 on invasive pneumococcal disease in equivalent populations. Clin Infect Dis. 2017;65(11):1780–9.

    PubMed  PubMed Central  Google Scholar 

  29. 29.

    Edwards IR, Aronson JK. Adverse drug reactions: definitions, diagnosis, and management. Lancet. 2000;356(9237):1255–9.

    CAS  PubMed  Google Scholar 

  30. 30.

    Saha SK, Darmstadt GL, Baqui AH, et al. Direct detection of the multidrug resistance genome of Haemophilus influenzae in cerebrospinal fluid of children: implications for treatment of meningitis. Pediatr Infect Dis J. 2008;27(1):49–53.

    PubMed  Google Scholar 

  31. 31.

    Painter PR. Supply-demand balance in outward-directed networks and Kleiber’s law. Theor Biol Med Model. 2005;2:45.

    PubMed  PubMed Central  Google Scholar 

  32. 32.

    Clay PG. MTM economics 101: supply vs. demand. J Am Pharm Assoc. 2018;58(2):228–9.

    Google Scholar 

  33. 33.

    Vassall A, Mangham-Jefferies L, Gomez GB, Pitt C, Foster N. Incorporating demand and supply constraints into economic evaluation in low-income and middle-income countries. Health Econ. 2016;25(Suppl 1):95–115.

    PubMed  PubMed Central  Google Scholar 

  34. 34.

    Curtis V, Cairncross S. Effect of washing hands with soap on diarrhoea risk in the community: a systematic review. Lancet Infect Dis. 2003;3(5):275–81.

    PubMed  Google Scholar 

  35. 35.

    Canning D, Razzaque A, Driessen J, Walker DG, Streatfield PK, Yunus M. The effect of maternal tetanus immunization on children’s schooling attainment in Matlab, Bangladesh: follow-up of a randomized trial. Soc Sci Med. 2011;72(9):1429–36.

    PubMed  Google Scholar 

  36. 36.

    Llupia A, Garcia-Basteiro AL, Mena G, et al. Vaccination behaviour influences self-report of influenza vaccination status: a cross-sectional study among health care workers. PLoS ONE. 2012;7(7):e39496.

    CAS  PubMed  PubMed Central  Google Scholar 

  37. 37.

    Arrossi S, Matos E, Zengarini N, Roth B, Sankaranayananan R, Parkin M. The socio-economic impact of cervical cancer on patients and their families in Argentina, and its influence on radiotherapy compliance. Results from a cross-sectional study. Gynecol Oncol. 2007;105(2):335–40.

    PubMed  Google Scholar 

  38. 38.

    Barnighausen T, Bloom DE, Canning D, et al. Rethinking the benefits and costs of childhood vaccination: the example of the Haemophilus influenzae type b vaccine. Vaccine. 2011;29(13):2371–80.

    PubMed  Google Scholar 

  39. 39.

    Mudrazija S. Work-related opportunity costs of providing unpaid family care in 2013 and 2050. Health Aff (Millwood). 2019;38(6):1003–10.

    PubMed  Google Scholar 

  40. 40.

    Bilcke J, Van Damme P, Beutels P. Cost-effectiveness of rotavirus vaccination: exploring caregiver(s) and “no medical care’’ disease impact in Belgium. Med Decis Making. 2009;29(1):33–50.

    PubMed  Google Scholar 

  41. 41.

    Cai S, Temkin-Greener H. Influenza vaccination and its impact on hospitalization events in nursing homes. J Am Med Dir Assoc. 2011;12(7):493–8.

    PubMed  Google Scholar 

  42. 42.

    Nymark LS. Economic evaluation of the indirect effects of vaccines. The herd immunity effects of vaccines against human transmissible diseases and the non-specific effects of the measles-mumps-rubella and Bacillus Calmette-Guérin vaccines, Aarhus University. Denmark; 2017.

Download references


Not applicable.


Funding was received from the Dutch Ministry of Foreign Affairs [Grant Number DSO/GA-264/15]

Author information




LN and AV developed the conceptual framework. The data and material collection and analyses were conducted by LN. LN prepared the manuscript. Both authors read and approved the final manuscript. LN is the guarantor of this work.

Corresponding author

Correspondence to Liv Nymark.

Ethics declarations

Ethics approval and consent to participate

Not applicable.

Consent for publication

Not applicable.

Competing interests

The authors declare that they have no competing interests.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit The Creative Commons Public Domain Dedication waiver ( applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Nymark, L., Vassall, A. A comprehensive framework for considering additional unintended consequences in economic evaluation. Cost Eff Resour Alloc 18, 27 (2020).

Download citation


  • Cost-effectiveness analysis
  • Indirect effects
  • Consequences
  • Externality
  • Internality
  • Guidelines