Methodological inconsistency

the heterogeneity of methods and outcome measures used in economic evaluations conducted by different investigators in different settings has complicated both the synthesis and the interpretation of cost-effectiveness results. For example, the measurement of costs may or may not have included assessment of informal care, travel and productivity losses so that the results of one study are not comparable with those of another, even if they were undertaken in the same setting.

Data unavailability

There remain considerable gaps in the cost-effectiveness evidence base, particularly for historically marginalized services and for currently under-served populations (e.g. mental health care in developing countries). This has limited the ability of policy-makers to address issues of allocative efficiency in the health sector.

Lack of generalizability

No country has yet been able to undertake all the studies necessary to compare the cost-effectiveness of all possible interventions in their own setting. They must borrow results from other settings. CEA findings, particularly costs, do not travel well due often to differences in health and economic systems. Because results have not been presented in ways that allow for transferability across settings, this has limited their use to the specific context in which they were derived.

Limited technical or implementation capacity

There is a shortage, particularly in lower-income countries, not only in terms of technical expertise to undertake economic evaluations in the first place, but also in terms of health service management capacity or political willingness to translate and implement findings into everyday health care practice.

Despite the limitations, this type of sectoral analysis is potentially important, although it is also clear that it can and should be only one input into the priority-setting process. As is shown in Figure 1, the health system framework developed by WHO is concerned not just with the generation of health itself, but also with meeting other key social goals and preferences, including being responsive to consumers and ensuring that the financial burden of paying for the health system is distributed fairly across households [15]. Figure 1 also shows that the health system seeks to reduce inequalities in health and responsiveness as well as increasing aggregate levels. Yet often health interventions do not adequately reach the poor despite being cost-effective and widely promoted. A benefit-incidence analysis of 44 countries across Africa, Asia and Latin America showed, for example, that interventions like oral dehydration and immunization – technologies developed with the needs of the poor particularly in mind – do not reach the target group. Only one-half of all cases of diarrhoea among children in the poorest 20% of families had been treated with some kind of oral liquid. Similarly, immunization programmes are not reaching the poor nearly so well as they are the better off. On average, immunization coverage in a developing country's poorest 20% of the population is around 35%–40%, slightly more than half the level achieved in the richest 20% [16, 17].

In short, cost-effectiveness analysis can show what combination of interventions would maximize the level of population health for the available resources. Since it is only one input – albeit an important one – to the decision-making process, the information it provides needs to be evaluated against the impact of different mixes of interventions on other social goals [18]. We return to this issue later in the paper.

Conceptual foundations

The advantage of using the counterfactual or null scenario as the basis of the analysis is that it can identify current allocative inefficiencies as well as the efficiency of opportunities presented by new interventions [10]. From the starting point of the situation that would exist in the absence of the interventions being analyzed, the costs and effects on population health of adding interventions singly (and in combination) can be estimated, to give the complete set of information required to evaluate the health maximizing combination of interventions for any given level of resource constraints.

Traditional cost-effectiveness analysis does not evaluate the efficiency of the current mix of interventions, but considers only the efficiency of small changes, usually increases, in resource use at the margin (i.e. the starting point for analysis is the current situation of usual care). This shows whether a new procedure is more cost-effective than the existing one but avoids the question of whether the current procedure was worth doing, implicitly taking it as given that something would have to be done in that particular area. Because the current mix of interventions varies substantially across countries, the costs and effects of small changes in resource use also vary substantially, which is one factor limiting the transferability of results across settings. Removal of this constraint by using the counterfactual of what would happen in the absence of the interventions means that the results not only allow assessment of the efficiency of current resource use, but are also more generalizable across populations sharing similar demographic or epidemiological characteristics.

One perceived disadvantage of using a counterfactual situation as a starting point for analysis is that policy-makers are more familiar with moving from the known, current situation. However, by incorporating currently implemented strategies (at specified levels of effective coverage) in the set of interventions for analysis, the ability to assess the incremental costs and effects of changes to the current allocation of resources is in fact preserved. In any case, Generalized CEA should not be viewed as a substitute to the acquisition of more context-specific economic evidence on the efficiency of adding new health technologies to the existing intervention mix. Both types of analysis are, in fact, complementary to each other. Generalized CEA is most useful to decision-makers in terms of broadly identifying within a sectoral assessment framework an efficient mix of interventions. Thus, as a first step in policy analysis using Generalized CEA, interventions are first classified into groups that interact in terms of costs or effects. Within each group, and at different levels of coverage, interventions are evaluated singly and then in combination, allowing for non-linear interactions in terms of effectiveness (multiplicative) as well as costs ([dis]economies of scope). As a result, the most efficient combination for a given resource constraint is identified. Efficient combinations are then compared across mutually exclusive groups in a single league table, ranked according to the cost per unit of health gain achieved. Subsequently, threshold values can be decided for classifying interventions into, say, those that are very cost-effective, those that are cost-ineffective and those in between.

Incremental analysis, which is constrained by the current mix of interventions, can subsequently be employed to provide more context-specific information on how this efficient mix of interventions can best be achieved in a particular setting.

Practical implementation

The WHO-CHOICE project, using Generalized CEA, has been established to provide key information to policy-makers wishing to implement sectoral CEA. WHO-CHOICE has assembled sub-regional databases on the cost-effectiveness of an extensive range of interventions for leading causes of disease burden, including analysis of the interactions inherent in many combined interventions http://www.who.int/evidence/cea. A recent analysis of the cost-effectiveness of interventions for reducing exposure to leading risk factors for disease appears in the World Health Report 2002 [19]. The generation of such databases, which removes an important impediment to the analysis of health sector-wide allocative efficiency, has been facilitated by a number of methodological strategies:

In order to facilitate more meaningful comparisons across regions, costs are expressed in international dollars (an international dollar has the same purchasing power as one US dollar has in the USA); effectiveness is measured in terms of disability-adjusted life years or DALYs averted (relative to the situation of no intervention for the disease or risk factor in question); and cost-effectiveness is described in terms of cost per DALY averted. One benefit of using the DALY as a primary measure of outcome is that it enables analysts to express population-level gain as a proportion of current disease burden (also measured in DALYs). In terms of thresholds for considering an intervention to be cost-effective, WHO-CHOICE has been using criteria suggested by the Commission on Macroeconomics and Health [25]: interventions that avert one DALY for less than average per capita income for a given country or region are considered very cost-effective; interventions that cost less than three times average per capita income per DALY averted are still considered cost-effective; and those that exceed this level are considered not cost-effective.

Figure 2 illustrates a way of presenting the full sectoral analysis using CHOICE results. The figure depicts the cost-effectiveness of multiple interventions in an epidemiological sub-region of Africa, called AfrD (see Table 1 for the countries in this sub-region). The figure includes a wide range of interventions, such as the provision of improved water and sanitation and preventive interventions to reduce cardiovascular risk factors. Intervention effectiveness in terms of DALYs averted is measured on the horizontal axis and annualized discounted costs of the interventions in international dollars are measured on the vertical axis. To enable the wide range of costs and effectiveness estimates for the individual interventions to be presented together, Figure 2 is drawn with the axes on a logarithmic scale. The lines drawn obliquely across the figure represent lines of equal cost-effectiveness. All points on the line at the south-east extreme have a cost-effectiveness ratio (CER) of I$1 per DALY averted. Because of the logarithmic scale, each subsequent line moving in a north-easterly direction represents a one order of magnitude increase in the CER, so all points on the next line have a CER of I$10, and the subsequent line represents a CER of I$100. The figure illustrates the variation in CERs across interventions within sub-region AfrD. Some interventions (for example, some preventive interventions aimed at reducing the incidence of HIV) avert one DALY at a cost of less than I$10. On the other hand, some preventive interventions to reduce cardiovascular risk factors cost almost I$100,000 per DALY averted. The figure allows the decision-makers to identify particularly bad buys (the brown shaded oval in Figure 2) and particularly good buys (the orange shaded oval in Figure 2) when choosing the mix of interventions they wish to ensure are provided in their setting.

Another potential use of the results from the WHO-CHOICE exercise is to assess the performance of health systems. In WHO's health systems performance framework, health system efficiency is assessed in terms of whether the system's resources achieve the maximum possible benefit in terms of outcomes that people value [15, 26]. Efficiency is the ratio of attainment (above the minimum possible in the absence of the resources) to the maximum possible attainment (also above the minimum). It reflects what proportion of the potential health system contribution to goal attainment is actually achieved for the observed level of resources. It could, in theory, be estimated for any of the health system goals or for all of them combined, but traditionally it has been limited to the assessment of the efficiency of translating expenditure into health outcomes using cost-effectiveness analysis. Figure 3 depicts the production frontier for a set of interventions to reduce the risks of cardiovascular disease in the countries of the Americas with very low rates of child and adult mortality, here called AmrA [27]. The vertical axis depicts the gain in the healthy life expectancy (HALE) of the population resulting from any given use of resources, while resource use or costs are shown on the horizontal axis. Available data on current coverage of the interventions and their costs and effectiveness allow current health attainment and costs to be estimated, represented as point *. The higher line shows the frontier estimated from information about the costs and effects of the most efficient mix of interventions at any given level or resource availability. Point * is below the frontier, suggesting that the health system is not achieving its full potential in terms of reducing the risks associated with high blood pressure and cholesterol [28]. The analysis could be used to evaluate how current resources used in preventing cardiovascular disease could be reallocated to achieve greater health benefits, as well as how any additional resources could be used most efficiently.

Factors to be considered in the contextualization of sub-regional cost-effectiveness data

In overcoming technical problems concerned with methodological consistency and generalizability, Generalized CEA has now generated data on avertable burden at a sub-regional level for a wide range of diseases and risk factors [19]. However, the existence of these CE data is no guarantee that findings and recommendations will actually change health policy or practice in countries. There remains a legitimate concern that global or regional CE results may have limited relevance for local settings and policy processes [29]. Indeed, it has been argued that a tension exists between Generalized CEA that is general enough to be interpretable across settings, and CEA that takes into account local context [30], and that local decision-makers need to contextualize sectoral CEA results to their own cultural, economic, political, environmental, behavioural, and infrastructural context [31].

In order to stimulate change where it may be necessary, there is a need to contextualize existing regional estimates of cost, effectiveness and cost-effectiveness to the setting in which the information will be used, since many factors may alter the actual cost-effectiveness of a given intervention across settings. These include: the availability, mix and quality of inputs, especially trained personnel, drugs, equipment and consumables; local prices, especially labour costs; implementation capacity; underlying organization structures and incentives; and the supporting institutional framework [32]. In addition, it may be necessary to address other concerns to ensure that the costs estimated on an ex-ante basis represent the true costs of undertaking an intervention in reality. For example, Lee and others [33–37]. (argue that cost estimates might not provide an accurate reflection of the true costs of implementing a health intervention in practice for a number of reasons: economic analyses can often be out of date by the time they are published [38]; the cost of pharmaceutical interventions may vary substantially depending on the type of contracts between payers, pharmacy benefits, management companies and manufacturers; or costs of care may be lowered by effective management (e.g. through negotiation, insurance companies may reduce prices). Likewise on the effectiveness side, there is a need for contextualisation. For example, effectiveness estimates used in CEA are often based on efficacy data taken from experimental and context-specific trials. When interventions are implemented in practice, effectiveness may well prove to be lower. According to Tugwell's iterative loop framework [39], the health care process is divided into different phases that are decisive in determining how effective an intervention will be in practice, including whether a patient has contact with the health care system or not, how the patient adheres to treatment recommendations, and with what quality the provider executes the intervention.

From regional to country-specific estimates

Figure 4 provides a schematic overview of the step-by-step approach by which WHO-CHOICE estimates derived at the regional level can be translated down to the context of individual countries. The following key steps are required:

The contribution of Generalized CEA to national-level priority-setting

Determination of the most cost-effective interventions for a set of diseases or risk factors, while highly informative in its own right, is not the end of the analytical process. Rather, it represents a key input into the broader task of priority-setting. For this task, the purpose is to go beyond efficiency concerns only and establish combinations of cost-effective interventions that best address stated goals of the health system, including improved responsiveness and reduced inequalities. Indeed, Generalised CEA has been specifically developed as a means by which decision makers may assess and potentially improve the overall performance (or efficiency) of their health systems, defined as how well the socially-desired mix of the five components of the three intrinsic goals is achieved compared to the available resources (Figure 1). Other allocative criteria against which cost-effectiveness arguments need to be considered include the relative severity and the extent of spillover effects among different diseases, the potential for reducing catastrophic household spending on health, and protection of human rights [12, 13, 18, 30, 31, 41]. Thus, priority-setting necessarily implies a degree of trading-off between different health system goals, such that the most equitable allocation of resources is highly unlikely to be the most efficient allocation. Ultimately, the end allocation of resources arising from a priority-setting exercise, using a combination of qualitative or quantitative methods, will accord to the particular sociocultural setting in which it is carried out and to the expressed preferences of its populace and/or its representatives in government. A sequential analysis of these competing criteria, however, indicates that for the allocation of public funds, priority should be given to cost-effective interventions that are public goods (have no market) and impose high spillover effects or catastrophic costs (particularly in relation to the poor) [41], which underscores the need for prior cost-effectiveness information as a key requirement for moving away from subjective health planning (based on historical trends or political preferences) towards a more explicit and rational basis for decision-making.

There are also a number of functions of a health system that shape and support the realisation of the above stated goals, including resource generation and financing mechanisms, the organisation of services as well as overall regulation or stewardship [15]. These functions inevitably influence the priority-setting process in health and hence contribute to variations in health system performance. Indeed, it has been argued that health strategies based on efficiency criteria alone may lead to sub-optimal solutions, owing to market failures in health such as asymmetry of information between providers and patients, as well as a number of adverse incentives inherent within health systems [42]. Accordingly, the results of an efficiency analysis such as a sectoral CEA are likely to be further tempered by a number of capacity constraints and organisational issues. As already noted above, the actual availability of human and physical resources can be expected to place important limits on the extent to which (cost-effective) scaling-up of an intervention's coverage in the population can be achieved. In addition, broader organisational reforms aimed at improving health system efficiency by separating the purchasing and provision functions can be expected to lead to some impact on the final price of health care inputs or the total quantity (and quality) of service outcomes. Finally, decisions relating to the appropriate mechanism for financing health, including the respective roles of the public and private sector, can be expected to have a significant influence on the end allocation of resources. For example, should the role of the public sector be to provide an essential package of cost-effective services, leaving the private sector to provide less cost-effective services [6], or should it be to provide health insurance where private insurance markets fail (such as unpredictable, chronic and highly costly diseases, for which only potentially less cost-effective interventions are available)? [42]. Even if both objectives are pursued – providing basic services to particularly vulnerable populations while catering to the majority's inability to pay for highly costly interventions [43] – a shift away from the most efficient allocation is still implied.