Do economic evaluation studies inform effective healthcare resource allocation in Iran? A critical review of the literature

To aid informed health sector decision-making, data from sufficient high quality economic evaluations must be available to policy makers. To date, no known study has analysed the quantity and quality of available Iranian economic evaluation studies. This study aimed to assess the quantity, quality and targeting of economic evaluation studies conducted in the Iranian context. The study systematically reviewed full economic evaluation studies (n = 30) published between 1999 and 2012 in international and local journals. The findings of the review indicate that although the literature on economic evaluation in Iran is growing, these evaluations were of poor quality and suffer from several major methodological flaws. Furthermore, the review reveals that economic evaluation studies have not addressed the major health problems in Iran. While the availability of evidence is no guarantee that it will be used to aid decision-making, the absence of evidence will certainly preclude its use. Considering the deficiencies in the data identified by this review, current economic evaluations cannot be a useful source of information for decision makers in Iran. To improve the quality and overall usefulness of economic evaluations we would recommend; 1) developing clear national guidelines for the conduct of economic evaluations, 2) highlighting priority areas where information from such studies would be most useful and 3) training researchers and policy makers in the calculation and use of economic evaluation data.


Introduction
Economic evaluations identify, measure, value and compare the cost and consequences of two or more alternative programs or interventions [1]. Economic evaluation is commonly used as a decision tool in health care systems where, due to resource constrains, policymakers have to choose between alternative activities with different implications for resources allocation [2].
Iran is a middle-income country with a population of 76 million. In 2009, Iran had a Gross National Income (GNI) per capita of US$ 10,250 and approximately 6% of its gross domestic production (GDP) per capita is spent on healthcare [3]. The constitution emphasizes the right of access to the highest level of health for all citizens and the Ministry of Health and Medical Education (MOHME) is responsible for fulfilling this goal through designing and implementing national level policies. There is at least one medical university in every province and at the provincial level, these universities play an important role in the provision of health services and medical education. The Dean of a medical university is the highest health authority at the province, reporting to the MOHME [4]. Health services are provided by public, quasi-public and philanthropic organizations, and a large network of private providers. The public sector (lead by MOHME) is the main provider of Primary Health Care across the country and provides a large part of secondary and tertiary health services [4]. The private sector mainly provides secondary and tertiary services in urban areas. Nearly 90% of the country's population is covered by health insurance, mainly public health insurance organizations [4,5]. In spite of the high insurance coverage rate, health services (secondary and tertiary) are not affordable for many due to high out-of-pocket expenditures (mainly form of co-payments) [5]. In 2011, out-of-pocket expenditure was 60% of total health expenditures [3].
Iran's health expenditure has been increasing rapidly over the last decade [5]. This trend is being driven by an ageing population, a rising prevalence of non-communicable and chronic diseases, an increase in the use of health technologies, a rise in domestic drug manufacturing and increasing prescription and consumption of medicines [6][7][8]. At the same time, the Iranian health system is being criticized for inefficient resource allocation, providing services without cost-effectiveness considerations, failing to regulate the private sector and over-utilization of new technologies [9][10][11][12]. Resource allocation decisions in Iran's healthcare system, have historically been based on implicit criteria such as pre-existing service availability, affordability of the insurance organisations and providers and political pressure [5,12]. Recently, interest in evaluating efficiency including estimating efficiency of health care providers [6][7][8] and economic evaluation of interventions and new technologies has been growing in Iran, like many other Asian countries [13]. The government and MOHME have developed various strategies to improve both the efficiency and equity of resource allocation, including; establishing the Ministry of Welfare and Social Security (MWSS) in 2005, incorporating all health insurance organisations under MWSS with the aim of separating the health care providers/MOHME from the financiers [5,11], and establishing the Technology Assessment Unit within MOHME in 2007 [14]. Despite these efforts, the extent to which evidence from economic analyses is used to inform resource allocation in national strategic planning and decision-making remains unclear [5,12,15].
Despite an increase in the number of economic evaluation studies worldwide, concern about the quality of these studies, among other factors, has been one of the major barriers which limit their application by policymakers [16][17][18]. In response to this policy concern, several guidelines have been developed to assess the quality of economic evaluation studies [1,19,20] and many systematic review studies has been conducted in different countries [21][22][23][24][25]. Arguably, these reviews focus disproportionately on high-income countries.
While a number of economic evaluations have been conducted in Iran, no known study has systematically reviewed the quality of Iranian economic evaluations. The purpose of this study was to provide a review of the state of economic evaluation within the context of Iran. Specifically, this review assessed whether Iranian economic evaluation studies had been performed according to current international standards and therefore whether their results are likely to prove useful to policy makers. Moreover, this study examined whether these studies were aimed at those conditions generating a higher burden of disease in Iran.

Literature search and study selection
A literature search was conducted independently by the first author in December 2011 and then verified by the second author and updated in May 2012. The search was performed using the following international databases: Medline/Pubmed, Embase, Web of Science, NHS Economic Evaluation Database (EED), Econlit and Google scholar. To identify articles published in national journals (both in Persian and English), the Scientific Information Database (SID) website was searched. In addition, the references of retrieved articles were manually searched for further papers. The search was continued until no new articles were found. The keywords used for the literature search were: "cost analysis", "cost*" "economic evaluation", "cost-effective", "cost-saving", "cost-effectiveness", "costutility", "cost-benefit", "cost-minimisation" and "Iran*" in the title or abstract of the articles. The full search strategy is available in Additional file 1. Studies were included in the review on the basis of the following inclusion criteria:

Data extraction and critical appraisal
Relevant papers were selected by screening the titles and abstracts (first step) and entire articles (second step) according to the inclusion criteria listed above. Screening of articles was conducted independently by HHB and AAK. Any disagreements about eligibility between the authors were solved through subsequent discussion with JSW.
Assessment of the quality of included studies was done using a questionnaire adapted from existing guidelines, checklists and other review articles of economic evaluations [1,19,20,24,26]. The questionnaire included both general information and economic evaluation features of the selected articles. General information included: year of publication, journal in which the study was published (national or international), affiliation of the first author (medical or non-medical), type of journal (medical, nonmedical), language (English or Persian) and geographical location covered by the study. The main economic evaluation features of the questionnaire included method of economic evaluation (CMA, CEA, CUA, CBA), study design, perspective (both stated and evaluated), type of sensitivity analysis performed, time horizon, type of outcome and its description, disease investigated, intervention type, description of intervention (competitors), type of data used (primary or secondary), types of costs included and if they were measured and valued properly, sample size, funding source, whether economic evaluation was the primary study goal, whether discounting were performed (if the costs and/or outcomes were from a study period of >1 year) and whether generalisability of results was discussed. Both first and second authors reviewed the selected articles independently and extracted the information into predesigned forms in Excel. Any disagreements were solved through subsequent discussion.
To investigate whether the published studies targeted high priority interventions, a study of the burden of disease and injuries in Iran was used to identify high burden health problems in the country [6]. The study, which used methods developed by the World Health Organization (WHO) [27] for national burden of disease (NBD) studies, measured disease burden in terms of disability-adjusted life years (DALYs) in the Iranian population in 2003. In any resource constrained environment, economic evaluation studies should ideally provide evidence on cost-effectiveness of interventions to address diseases with a high burden [2]. Therefore, analysing the disease burden and available evidence from economic evaluation studies would provide useful information for researchers and policy makers to establish gaps in knowledge and priorities for future study. This same approach has been previously used by a number of other researchers [2,28,29].

Results
A total of 258 articles were identified by the literature search, of which 150 articles were excluded at the initial stage because they violated basic inclusion criteria ( Figure 1). The full-texts of 108 articles were then reviewed and a further 78 articles were dropped because they failed one or more inclusion criteria. The remaining 30 articles  met the inclusion criteria of the study. Details of the reviewed studies are presented in Additional file 2: Table S1 and Additional file 3: Table S2.

General characteristics of included studies
As can be seen from the Figure 2, the numbers of full economic evaluation papers are scarce. Of the included articles, 17 (57%) were published in Persian. General characteristics of the reviewed studies are presented in Table 1. The majority of the papers (77%) were published in medical journals rather than specialized health economics or health care sciences journals, and in twenty-two papers (73%) the first authors had medical/clinical affiliations. Most studies (85%) covered sub-national geographical locations (province level), and only five studies had national coverage (Table 1).
Economic features and quality of included studies Table 2 reports economic and methodological features of the selected studies and Table 3 shows the extent to which these studies meet the recommendations for good reporting of economic evaluations. Of the 30 reviewed studies, 21 studies were CEA (70%). In terms of study design, most studies were observational (43%) following by randomized controlled trials (RCT) (27%). Only 6 studies (20%) employed a modelling approach, which among them, type of the model used was not specified in two studies [34,38]. While most studies did not specify their perspective (77%), we assessed the perspective of the studies, as shown in Table 2. The health care system/provider (77%) was the most frequent perspective. Two studies [30,58] stated their perspective as societal. The majority of the studies did not perform any sensitivity analysis (73%) and those that did so [30,32,[36][37][38][39]47,58] mainly conducted one-way sensitivity analysis (17%).
The analysis showed that 40% of the studies failed to report their time horizon clearly. Among those that did report the timeline, eight (27%) had time horizons of less than one year, five (17%) between 1 to 10 years and five over 10 years.
Among CUA studies (i.e. studies using comprehensive/ composite outcome measures such as quality-adjusted life years [QALYs] or DALYs as their primary outcome measure) three studies used QALYs [35,37,38] and two DALYs [39,52] as the primary effectiveness measure. Among the three studies using QALYs, one used utility scores obtained from other countries [38].
Among CEA studies (i.e. studies using intermediate measures or natural units such as deaths prevented or cases detected as the primary outcome measure), the majority of studies used disease specific outcome measures such as mortality rates, number of patients detected or number of complications, and none generic measures such as "life years gained/saved".
Only four CBA studies were performed in the context of Iran [41,42,48,56]. However, after evaluating these studies closely using the criteria recommended by Drummond, et al. [1] and Zarnke, et al. [18], these studies were not fully measured the benefits of the interventions under investigation. The benefits in these studies have been defined as cost savings (mainly savings in medical costs) without measuring and valuing the monetary values of health gained by the intervention.
The source of cost and outcome data was clearly specified in the majority of studies (80% for costs and 97% for outcomes). In 73% of cases, cost and outcomes data were gathered through primary data collection. In those studies collecting primary data, the median sample size was 120, ranging from 30 to 1,165,169. Three studies used a hypothetical cohort [34,39,58] and three studies did not clearly specify their sample size [30,38,52]. Among those studies using secondary data (8 studies), cost data were usually collected from national surveys/ reports or previous studies conducted in Iran. In four studies, outcomes data were obtained from studies conducted in settings other than Iran [30,32,34,38].
All studies included direct medical costs, however only four studies (13%) included direct non-medical costs [32,39,56,58]. Among the two studies that adopted a societal perspective [30,58], none estimated productivity loss and intangible costs. It is important to identify the relevant cost items for each intervention, to measure the resources used (in their physical units), and to value these resources (by their prices) properly. Only 9 studies (30%) identified all relevant costs related to each alternative intervention, considering their perspective. In most cases, only the costs of medicine and hospitalization were included and capital costs (such as building and equipment) and the cost of medical supplies were not often calculated. The data about hospitalisation costs were mainly extracted from patients' hospital records or obtained from the patients' hospital bills. Six studies [30,34,36,37,48,58] described clearly how they measured and valued their cost components. Out of the 12 studies in which the reported time horizon was more than one year, only two studies discounted both costs and outcomes at the same time [39,42]. A further two studies discounted either costs [30] or outcomes [48] only. The common discount rate utilised in the studies was 3%. Only 5 studies [32,35,36,38,39] calculated and reported incremental cost-effectiveness ratios (ICER). Three studies [41,42,48] reported benefit-cost ratios (BCR). Although discussing the generalizability of results to the national level or to other settings can be an important element of an economic evaluation, the majority of reviewed studies (87%) failed to do so. Only four studies discussed the issue of generalizability to other settings to some degree. Figure 3 compares the criteria for good reporting of economic evaluations. Based on the figure the English language studies, which were published in international journals, more often adhered to the recommendations for good reporting of economic evaluations compare to the Persian language studies. Figure 4 shows the distribution of full economic evaluation studies by disease category, compared to the burden of disease in the country. Based on the figure, the five major areas of health problems in Iran are injuries (with 28% of DALYs), mental health (with 16% of DALYs), circulatory system diseases (with 11% of DALYs), perinatal conditions (with 8% of DALYs) and musculoskeletal system diseases (with 6% of DALYs). There are no economic evaluation studies for the three disease categories with the highest burdens, namely, injuries, perinatal conditions and musculoskeletal system diseases. Moreover, only one study investigated mental disorders, which accounted for 16% of burden of disease in the country. The most common diseases category covered by the economic evaluation studies are "Endocrine, nutritional and metabolic diseases" (n = 5, 17%, in this group Congenital hypothyroidism was investigated by four studies) and "Pregnancy, childbirth and the puerperium" (n = 4, 13%). "Circulatory system diseases", "Cancers" and "Blood and blood-forming organs diseases" were other disease groups covered by the studies (in each group, n = 3 or 10%).

Discussion
High quality economic evaluation data on interventions targeting high burden of diseases are important to aid informed decision-making and resource allocation. This study enumerates the availability of economic evaluation studies and critically assesses the quality of those data in the Iranian context. The findings showed that the number of economic evaluation studies in Iran is limited. This is comparable with settings such as Bangladesh [60], Nigeria [26], Saudi Arabia [61], Zimbabwe [62], and lagging behind countries such as Thailand [29], South Korea [24], India [63] and South Africa [64]. The number of available studies remains very low compared with high-income countries where economic evaluation is a relatively well-established and formal part of the policy making process [21,65]. The findings from this review also showed that many published Iranian economic evaluation studies did not meet current international standards and were of sub-optimal quality.
The review showed that CEA was the most frequently published economic evaluation in Iran. The relative ease of obtaining effectiveness data in the form of intermediate outcomes or natural units, and the application of arguably more straightforward computational methods in these types of evaluations [60] are two features that make CEA a practical method of analysis in many settings. This approach may be particularly attractive, when collecting comprehensive outcome measures such as  DALYs and QALYs are time intensive and expensive to collect. CEA is useful in situations where outcome measures of interventions under investigation are similar. However, since the outcome measure may differ for different interventions, CEA can seldom be used to make comparisons across a broad set of interventions [1]. Taking into account this limitation, CUA or CBA arguably provide better tools for policy makers allocating resources across different health care programmes or even across different sectors [1]. Considering resource constraints, it would be expected that economic evaluations focus on interventions for diseases that have a significant impact on population health. However, the findings of the current review showed that the majority of economic evaluation studies concentrate on a small number of disease categories which do not include some of Iran's major health problems. For example,

Type of data used
Primary data 22 73 Secondary data 4 13  injuries and mental disorders, two disease categories which contribute more than 40% to burden of disease in the country [6] and are likely to increase in the future [6], are strongly underrepresented in the Iranian economic evaluation literature. In addition, the findings show that researchers have paid a great deal of attention to curative interventions, while relatively little attention has been given to preventive interventions and care delivery strategies. These findings are consistent with previous studies [2,28,29]. A mismatch of economic evaluations with disease burden does not necessarily indicate inefficient resource allocation. It is possible that studies do not focus on high burden illnesses because there are few effective interventions for some health conditions. It is also possible that there exist a number of well-established, cost-effective interventions for some disease categories and as such new work in the area is not required. Difficulties in obtaining data on the effectiveness of interventions in some areas, such as mental disorders, can be another barrier to the conduct of evaluations for high burden illnesses. The interest of researchers and funding agencies could also play a role as the majority of research in Iran is funded by universities or other governmental research centres. Multiple criteria including equity and social justice are frequently used to determining health priorities and the burden of disease is only one of those criteria [2,28,66]. Nonetheless, we believe that comparing the priorities reflected in the evidence regarding the burden of disease,  with those reflected in the economic evaluation literature in Iran can provide a useful starting point for discussing future priorities for economic evaluation research. The review indicated that the Iranian economic evaluation studies had low adherence to good practice criteria for the reporting of economic evaluations and suffer from several methodological flaws. The perspective of an economic evaluation study is an important issue as it affects the measurement of both costs and outcomes of interventions. Yet few of the reviewed studies (only 23%) specified their perspective. This suggests that many authors are unaware of the importance of the perspective adopted and its effect on the costs and outcomes. Among the studies which stated their perspective, many failed to include all appropriate costs associated with their chosen perspective. For example, two studies [30,58] stated their perspective as societal but they didn't measure indirect costs associated with their interventions and those studies that had taken a healthcare system or provider as their perspective, failed to include capital or equipment costs and some recurrent costs (e.g. overhead costs). In the context of Iran, obtaining data to estimate capital and overhead costs, particularly in hospital settings where most of the reviewed studies are based, might be difficult. Moreover, some part of these costs is usually incorporated in the inpatients and outpatients' costs or hospital bills, which many of the reviewed studies included in their analysis. Using hospital charges/bills (and also health care tariffs) in economic evaluation have been criticised since they may not reflect the actual costs [1,67]. In the context of Iran in particular, hospital charges do not reflect true hospital costs due to government subsidies for hospital services and medicines [68]. Moreover, the effects of adverse events associated with interventions on use of resources and outcomes were also rarely [37,53] included in the reviewed studies. Lack of transparency in reporting intervention costs, was another important shortcoming of the reviewed studies. In some studies [31,40,43,44,55] it becomes impossible to ascertain what authors had actually done. Many studies failed to describe clearly how they measured and valued costs and did not provide details on type and quantities of resources and their price, which limits the possibility of replicating the evaluation in other settings.

Mixed 4 13
Moreover, few studies calculated and reported ICER. Instead of an ICER, some studies reported the average cost-effectiveness ratio (i.e. total cost divided by total effect for the interventions being compared), which can flaw the conclusion of the evaluation and limit direct comparison between interventions. This is because an average ratio implies the comparison of each alternative with a hypothetical intervention with no costs and no effects [69].
A major weakness of the economic evaluation studies in the Iranian setting was the limited use of sensitivity analysis to explore the effect of uncertainty on findings. Only 27% of studies performed some sort of sensitivity analysis. Sensitivity analysis helps to assess reliability of the findings for the context of the study and can also facilitate consideration of the generalisability of findings to other settings [70]. Moreover, the review also showed that very few Iranian economic evaluation studies discounted costs and/or outcomes when the study period was more than one year. None of the studies provide justification for the discount rate used, even if that rate was zero, and none performed a sensitivity analysis on the rate used.
Low adherence to good practice for the conduct of economic evaluations and the other methodological shortcomings discussed above are not unique to the Iranian setting. Systematic reviews of economic evaluation studies in some other settings (both developed and developing countries) have reported similar shortcomings [21,23,24,26,29,[60][61][62][63][64][65]. For this reason, many countries have developed formal and informal guidelines to standardize and improve the quality of economic evaluation in health care [71].

Limitations of the review
Admittedly, this review may suffer from some limitations. This study included only published literature in peerreviewed journals and excluded grey literature such as government reports, pharmaceutical company reports, academic theses and conference proceedings. The inclusion of only published literature might have introduced publication bias, since studies with positive results are more likely to be published than studies with negative findings [72][73][74]. In addition, although the database used for searching the studies published in Persian consisted of the majority of journals published in Iran, some journals or studies may not have been included in this database. Furthermore, as in any review study, it is difficult to rule out selection bias or disagreement between the criteria of the reviewers. To minimise this bias, we used pre-defined inclusion criteria and discussion of disagreement between the investigators throughout of the review process.

Conclusion
The findings of this review indicate that the literature on economic evaluation in Iran is still at an early stage and these evaluations suffer from significant methodological flaws. Furthermore, the review reveals that economic evaluation studies have not focused on Iran's most significant health problems i.e. those contributing most to the country's burden of disease. Although interest in using inputs from economic evaluation and HTA studies has increased in Iran, evidence still points to scarce demand for and utilisation of these inputs by policy makers. This itself might undermine incentives to improve the quality of economic evaluation studies.