- Open Access
Field testing of a multicriteria decision analysis (MCDA) framework for coverage of a screening test for cervical cancer in South Africa
Cost Effectiveness and Resource Allocationvolume 10, Article number: 2 (2012)
Systematic and transparent approaches to priority setting are needed, particularly in low-resource settings, to produce decisions that are sound and acceptable to stakeholders. The EVIDEM framework brings together Health Technology Assessment (HTA) and multi-criteria decision analysis (MCDA) by proposing a comprehensive set of decision criteria together with standardized processes to support decisionmaking. The objective of the study was to field test the framework for decisionmaking on a screening test by a private health plan in South Africa.
Liquid-based cytology (LBC) for cervical cancer screening was selected by the health plan for this field test. An HTA report structured by decision criterion (14 criteria organized in the MCDA matrix and 4 contextual criteria) was produced based on a literature review and input from the health plan. During workshop sessions, committee members 1) weighted each MCDA decision criterion to express their individual perspectives, and 2) to appraise LBC, assigned scores to each MCDA criterion on the basis of the by-criterion HTA report.
Committee members then considered the potential impacts of four contextual criteria on the use of LBC in the context of their health plan. Feedback on the framework and process was collected through discussion and from a questionnaire.
For 9 of the MCDA matrix decision criteria, 89% or more of committee members thought they should always be considered in decisionmaking. Greatest weights were given to the criteria "Budget impact", "Cost-effectiveness" and "Completeness and consistency of reporting evidence". When appraising LBC for cervical cancer screening, the committee assigned the highest scores to "Relevance and validity of evidence" and "Disease severity". Combination of weights and scores yielded a mean MCDA value estimate of 46% (SD 7%) of the potential maximum value. Overall, the committee felt the framework brought greater clarity to the decisionmaking process and was easily adaptable to different types of health interventions.
The EVIDEM framework was easily adapted to evaluating a screening technology in South Africa, thereby broadening its applicability in healthcare decision making.
In any country, healthcare resource allocation decisions are complex and involve assessment of the available scientific evidence, clarification of priorities, value judgments and ethical considerations [1, 2]. In developing countries, which are generally low-resource settings, priority setting for healthcare becomes even more important. Not only are resources limited, but there are also other factors such as poor information, lack of policy, barriers to implementation, and political agendas, to name but a few . The result is that priority setting is inconsistent and unstructured . Transparent and explicit approaches to decisionmaking help produce decisions that are sound and acceptable to stakeholders [4–7].
In the healthcare sector of South Africa, current decisionmaking approaches are centered around evidence-based medicine, affordability and, where available, cost-effectiveness/costutility analysis (CEA/CUA) . Increasingly CEA/CUA is being used for priority setting at all levels: the patient; the healthcare service; and within populations. This is evident in the field of HIV/AIDS where cost-effectiveness analyses of antiretroviral (ARV) medicines has enhanced access to treatment and reduced drug prices [9, 10]. In many instances, a simple cost-minimization approach is all that is attempted. However, this approach has shortcomings, as there are a number of additional important dimensions, such as budget impact, equity, availability of alternatives, disease severity, etc. [11, 12], that are not incorporated. Where these are taken into consideration, they are often assessed in an ad hoc manner and there is a lack of transparency as to how they impact the final decision .
Thus, there is a need for a process that supports consideration of all dimensions impacting a decision in a systematic and explicit fashion, and increases transparency and access to the evidence upon which decisions are based. Multi-criteria decision analysis (MCDA) is a tool to support complex decisionmaking which allows a structured, objective consideration of factors that are both measurable and value-based in an open and transparent manner [1, 13, 14].
While MCDA has been used historically in sectors such as transport or agriculture, there is a growing interest in using and applying the principles of MCDA, and similar approaches based on multiple decision criteria, to resource allocation decisionmaking in health care [15–21]. The EVIDEM framework has been developed to bring together Health Technology Assessment (HTA) and MCDA by proposing a comprehensive set of decision criteria together with standardized processes/methods to develop HTA reports that are structured on these criteria . The aims of the framework are to facilitate concurrent consideration of multiple decision criteria, to stimulate reflection on priorities and values, and to promote transparency and communication within the decisionmaking committee as well as with outside stakeholders. The application of the EVIDEM framework is postulated to be wide ranging from decisionmaking by the healthcare provider, to coverage decisions by the funders or government policy setting.
A proof-of-concept study of the EVIDEM framework was performed in Canada involving a diverse panel of stakeholders appraising 10 medicines . The core framework was also further developed to include standardized contextual criteria and tested for clinical decisionmaking by a panel of pediatric endocrinologists and other stakeholders who applied it to appraise growth hormone for children with Turner syndrome in Canada . The framework was also tested as a support for drug formulary decisionmaking by a public healthcare payer in Canada .
The objective of the current study was to expand the scope of field testing both geographically and with respect to type of intervention: i.e., to field test the framework as a support for coverage decisionmaking on a cervical cancer screening test (liquid-based cytology, LBC) by a private health plan in South Africa.
The healthcare system in South Africa is dichotomous with a small (approximately 7 million lives) but resource-rich private healthcare sector and a growing (approximately 45 million lives) resource-scarce public sector. The private healthcare sector is largely funded by medical insurance companies (health plans), while the public sector is currently funded out-of-pocket or by the government. With respect to cervical cancer screening, the Department of Health guidelines stipulate that all women should be screened three times in their lifetime from the age of 30 years . In the private healthcare sector, cervical cancer screening is often recommended and carried out on an annual basis. Screening uptake is low in South Africa with poor accessibility to healthcare facilities, poorly trained staff, long turnaround times between laboratory and healthcare facility, as well as lack of education cited as some of the reasons . LBC for cervical cancer screening was selected by the health plan as a relevant case study due to its recent introduction as an alternative to conventional Pap smears.
The EVIDEM framework was field-tested in the private healthcare sector of South Africa with a clinical policy and decisionmaking committee of a major health plan. This panel of experts included doctors (specialists and general practitioners), pharmacists and nurses with at least 12 months experience in decision-making at a health policy level for the health plan. At the time of the study, the committee was already using evidence-based medicine, cost-effectiveness, affordability and some epidemiological principles in a multicriteria-based decisionmaking process.
The study design is presented in Figure 1. Based on a literature review and input from the clinical committee of the health plan, a structured HTA report on LBC for cervical cancer screening was produced and tailored to investigate each of the 14 MCDA decision criteria of the framework organized in the MCDA matrix . (Note that the framework contains 15 MCDA decision criteria; however, the criterion "Adherence to requirements of decisionmaking body" was not considered for appraisal in this field test.) Four contextual criteria were proposed by the healthcare funder: "Impact on future decisions", "Relationship with pathology providers", "Impact on screening intervals" and "Patient expectation"; these criteria were appraised qualitatively. The contents of the report were tailored to reflect the local context (i.e., South African private health plan). Each committee member was familiarised with the LBC technology through prior evaluation for clinical decisionmaking and/or review of the HTA report.
During workshop sessions, committee members assigned: 1) weights to each criterion of the MCDA matrix to express their perspectives; 2) scores for LBC for each criterion of the MCDA matrix based on data from the by-criterion HTA report; and 3) the qualitative impact of system-related criteria on the appraisal. The adoptability and utility of the EVIDEM approach were explored through a post-testing survey.
Health technology assessment report
Relevant data for each decision criterion of the framework was identified by searching PubMed, websites of HTA agencies (including the Canadian Agency for Drugs and Technologies in Health [CADTH], National Institute for Health and Clinical Excellence [NICE], US Preventative Services Task Force and Danish Centre for Evaluation and Health Technology Assessment) and the website of South Africa's Department of Health.
Data on the projected budget impact of LBC as well as local costing data was obtained from the health plan. The health plan also provided input for the four contextual criteria, which were appraised qualitatively. Collected data was extracted, analyzed and synthesized to produce a by-criterion HTA report.
The quality of the clinical evidence and the economic evaluation was assessed using previously developed instruments according to two quality criteria: "Completeness and consistency or reporting" and "Relevance and validity of evidence" . The HTA report was entered into an interactive web prototype (Tikiwiki v2.2) that was hyperlinked to the full-text sources from which data was extracted. The web prototype was designed to allow online appraisal of the proposed intervention and collect feedback from the committee.
Field testing by the committee
To explore their individual values, committee members were asked to assign weights to each MCDA decision criterion on a scale of 1 to 5, where 5 represents the greatest importance to their decisionmaking. Weights were to be assigned from individual perspectives in the context of the health plan, independent of the intervention to be appraised.
All committee members were asked to familiarise themselves with the study intervention (e.g. clinical trials, health economic studies, disease characteristics and treatments etc.) using the EVIDEM Synthesised evidence report prior to carrying out the evaluation.
To appraise the proposed intervention, committee members were then asked to score (on a 4-point scale from 0 to 3) LBC for cervical cancer screening with respect to each MCDA criterion of decision, based on synthesized data from the by-criterion HTA report. Committee members then considered what type of impact each of the four qualitative criteria might have on use of LBC in the context of the health plan.
Feedback on the framework and process was elicited during discussion periods at the first workshop and at a follow-up workshop, and from a questionnaire administered during the follow-up workshop. Committee members were surveyed about whether each of the framework's decision criteria should always, sometimes or never be considered in decisionmaking. They were also surveyed about how the EVIDEM approach compared to their current process regarding data provided, deliberative process and communication of decision (improved, same or worse).
Data collection and analyses
Weights, scores and impact obtained from committee members were entered into Excel software. Descriptive statistics were used and mean ± standard deviations (SD) were calculated for weights and scores. The MCDA value estimate for the use of LBC in cervical cancer screening was then obtained by applying an MCDA linear additive model . Thus, the value contribution of each decision criterion was calculated by multiplying its mean weight (normalized across the criteria to add up to 1 for each committee member) and score (standardized by division by the maximum score 3). The MCDA value estimate was then calculated as the sum of the value contributions of the 14 applicable criteria of the MCDA matrix.
Decision criteria and committee values
When surveyed on whether each of the criteria of the framework should always, sometimes or never be considered in the decisionmaking process, all committee members agreed that "Budget impact on health plan" should always be considered (Table 1). For eight further criteria ("Completeness and consistency of reporting evidence", "Relevance and validity of evidence", "Disease severity", "Clinical guidelines", "Improvement of efficacy/effectiveness", "Type of medical service", "Cost-effectiveness" and "Appropriate use"), eight out of nine (89%) committee members felt that these should always be considered. Among the 15 MCDA matrix criteria, "Improvement of PRO, convenience & adherence" and "Adherence to requirements of decisionmaking body" received the lowest acceptance from committee members, with only 44% and 33%, respectively, indicating that these should always be considered. "Political context" received least approval among all criteria included in the survey, with only one (11%) committee member expressing the opinion that it should always be considered. No committee member indicated that any criterion should never be considered.
Committee members were asked to weight each of the 14 MCDA decision criteria included in this field test according to their importance to the appraisal of an intervention, independently of the intervention that was to be appraised. (The criterion "Adherence to requirements of decisionmaking body" was not considered for appraisal in this field test.) The greatest weights (mean 4.7, on a scale of 1 to 5) were assigned to the criteria "Budget impact on health plan", "Cost-effectiveness of intervention" and "Completeness and consistency of reporting evidence", followed by "Improvement of efficacy/effectiveness" and "Relevance and validity of evidence" (mean 4.4) (Figure 2). The lowest weights were given to "Improvement of safety and tolerability" (mean 3.9) and "Public health interest" (mean 3.8). Weights for "Budget impact on health plan", "Cost-effectiveness of intervention" and "Completeness and consistency of reporting evidence" showed the least variation among committee members (SD 0.5). The largest divergence of weights was recorded for the criteria "Size of population affected" (SD 1.0) and "Public health interest" (SD 1.0).
Appraisal of liquid based cytology for cervical cancer screening
By-criterion Health Technology Assessment report
The HTA report for LBC was based on 14 references (Table 2, 'lite' version). (The full report is available online at http://www.evidem.org/tiki/?page=CCLBC-RecordMenu). Key items included in the assessment are reported here. In the context of cervical cancer screening, LBC refers to a technique in which the cervical sample is suspended into a preservative liquid, rather than, as in conventional cytology (CC, Pap test), being smeared onto a slide. Cervical cancer causes high morbidity and mortality [28–30]. With annual incidence rates of 9 per 100,000 among white and 40 per 100,000 among black women , cervical cancer is the second most common cause of cancer morbidity and mortality in South African women [32–34]. Current South African guidelines recommend screening with at least three cervical Pap smears (CC) per lifetime to detect precancerous lesions; they do not mention LBC . In addition to the occurrence of unsatisfactory samples, the chief limitations of CC are its relatively low specificity and sensitivity and its low uptake by the population [27, 35, 36]. There is considerable public health interest in improving screening methods and uptake to reduce the high burden of cervical cancer in South Africa [32, 33]. As a screening tool, LBC allows early detection which, with appropriate treatment, improves outcomes . A meta-analysis of RCTs found that LBC had 6.4% higher sensitivity but 4.0% lower specificity than CC, coupled with a slight reduction in the percentage of unsatisfactory samples . The most relevant outcomes, impact on morbidity and mortality due to cervical cancer, have not been measured. The incremental budget impact of LBC is estimated to be ZAR76.90 per patients screened and the incremental colposcopy cost per patient lifetime ZAR8 [36, 37]. The incremental cost per QALY gained was projected to be ZAR758,000 with a cost of ZAR764,000 per life-year gained . However, since these data are based on a Canadian economic evaluation model they may have limited relevance to the South African setting.
Quantitative considerations - scores
Using synthesized data from the HTA report in the MCDA matrix (Table 1), committee members assigned scores to appraise LBC for each criterion (Figure 3). The highest scores were assigned to the criteria "Relevance and validity of evidence" (mean 2.5, SD 0.3, on a scale of 0 to 3) and "Disease severity" (mean 2.2, SD 0.4). The lowest scores were given to "Cost-effectiveness of intervention" (mean 0.3, SD 0.7), "Improvement of safety & tolerability" (mean 1.0, SD 0.0) and "Completeness and consistency of reporting evidence" (mean 1.0, SD 0.0). The committee expressed unanimous agreement (SD 0.0) when scoring the criteria "Improvement of safety & tolerability" (mean 1.0), "Completeness and consistency of reporting evidence" (mean 1.0), and "Type of medical service" (mean 2.0). In contrast, 3-point differences across the full scoring scale (0 to 3) were seen for the criteria "Clinical guidelines" (mean 2.0, SD 1.2) and "Public health interest" (mean 2.0, SD 0.9).
MCDA value estimate
Combination of the committee's weights and scores yielded a mean MCDA value estimate for using LBC in cervical cancer screening of 46% (SD 7%) of the potential maximum value (Figure 4). Individual MCDA value estimates ranged from 36% to 61% among the nine committee members; 7 out of 9 estimates were in the range between 39% and 48%. The criterion "Relevance and validity of evidence" made the greatest contribution to the mean MCDA value estimate (13% of the total), followed by "Disease severity", "Clinical guidelines" and "Type of medical service" (11% each). The criterion "Cost-effectiveness" made the smallest contribution (2%) to the MCDA value estimate.
Qualitative considerations - impacts
Background information for the four contextual criteria proposed by the health plan was collected in the context of the health plan and provided to the committee to stimulate reflection and discussion (Table 3). Committee members considered that a decision to cover LBC may have a negative impact on future decisions (setting a precedent). When considering relationships with pathology providers, covering LBC was considered to have a positive impact. The possibility of extending screening intervals would also have a positive impact on the value of LBC. When considering patient expectations, committee members considered that patients would likely resist if co-payments were implemented for LBC.
Feedback from committee on overall approach
A feedback questionnaire was completed by the committee members following the evaluation. The elements of the questionnaire were pre-determined and part of the standardised set of tools available for the EVIDEM process. When surveyed on how the EVIDEM approach compared to existing process that committee members were familiar with, 50% of the committee members felt that it improved understanding of the intervention being appraised, access to quality assessment of the evidence on the intervention, and consideration of all key elements of the decision (Table 4). Fifty-six percent of committee members felt that the framework improved the transparency of the decision and the communication of the decision to stakeholders. No committee member rated the EVIDEM process as "worse". Concern was raised regarding the time and resources required to generate the HTA report, but on the whole, the committee felt it was a positive step forward in improving decisionmaking by bringing greater clarity to the decisionmaking process. The committee also thought that the approach could easily be applied to evaluation of other health interventions such as medicines and devices.
The EVIDEM process for LBC resulted in a consideration by the health plan to only fund for LBC up to the value of conventional pap smears. A negotiation process was started with the pathology laboratories to review their tariffs for this diagnostic. The fee for LBC was reduced to an amount which was considered appropriate for full funding.
The EVIDEM framework can be applied to assess the value of a screening intervention in the South African private health plan context, providing a practical tool for integrating HTA with an MCDA approach to support decisionmaking.
The use of the by-criterion HTA report allowed the clinical decisionmaking panel to assess all the criteria easily and from a single source. Where greater detail was required, hyperlinks embedded in the web-prototype provided access to the more detailed synthesized information as well as full source data. Available clinical evidence was fairly relevant and valid. Partial reporting of disaggregated data in the published economic evaluation limited its usefulness in the South African context although local costs were used to improve transferability. The fact that the committee gave the highest weighting to "Budget impact on health plan", "Cost-effectiveness of intervention" and "Completeness and consistency of reporting evidence" reflects its focus on issues of affordability and cost-effectiveness as well as quality of clinical evidence. This was also reflected by the low score (mean 0.3) the committee assigned to the cost-effectiveness of LBC (Figure 3), given an ICER of ZAR764,000 per QALY.
The MCDA estimate resulting from a combination of weights and scores, provided a comprehensive measure of the perceived value of LBC in the current context, i.e., relative to existing technology (conventional cytology), reflecting minor improvement at a significant cost, and also captured the importance of absolute elements of value such as disease severity, type of medical service and quality of evidence.
Non-quantifiable, contextual criteria of decision were also identified and considered. Given the lack of cost-effectiveness and high additional cost of LBC compared to conventional Pap smears, the budget impact would be substantial and therefore a co-payment would be considered. However, health plan members and many other South Africans are used to receiving Pap smears for free as part of health screening programmes and the implementation of co-payments would meet with strong resistance from both patients and healthcare providers. The use of the MCDA framework as a communication device to convey the structure and transparency of the decisionmaking process was thought to be an important tool to improve acceptability of the committee's decision.
Adaptation of the EVIDEM framework from medicines evaluation to a screening technology required minimal change to the structure of the framework. Since devices, procedures and technologies form a growing portion of the basket of healthcare interventions that require evaluation it was useful to assess the adaptability of the framework to these types of interventions.
Currently, there is no single HTA body (such as NICE or CADTH) in South Africa with the result that evaluations of new and existing health interventions are scattered and often duplicated. Skilled resources to carry out a proper HTA are scarce and a need is recognized for pooling these resources, allowing greater availability to all sectors in the country [38, 39]. Health technology assessments and reports are freely available from a number of sources such as NICE (UK), CADTH (Canada), AHRQ (USA), IQWIG (Germany), etc., as well as through subscription to knowledge bases such as the Cochrane Collaboration. However, none of these reports offers the opportunity to rationally and transparently measure the impact of the various criteria on the final clinical decision. This is left to clinical decisionmaking members who are intrinsically subjective based on their own priorities and knowledge. Using MCDA in healthcare decisionmaking adopts an existing, well established methodology based on sound, scientific principles .
On the whole, the committee members favoured the use of the EVIDEM process and were positive about its use. Some committee members rated their feedback of the EVIDEM process as "same" as existing process in the questionnaire, which may be explained by the fact that a more crude and simpler multicriteria approach was already being used by the committee.
This field test indicates that the EVIDEM framework may be useful for the evaluation of health technologies in the South African private healthcare sector context. However, further adaptation to a health plan's specific mission, scope and priorities is required to create a "custom-made" decisionmaking framework fully aligned with the local context. The framework's contextual tool ("extrinsic value tool") -- a result of a further development of this framework  -- facilitates this customization by providing generic criteria/themes on the basis of which contextual criteria can be identified, and if so desired, integrated into the MCDA model. Further development and validation of the framework's weighting and scoring methodologies is also currently underway.
The EVIDEM framework was easily adapted to evaluating a screening technology thereby broadening its applicability in healthcare evaluation. It was found to improve: understanding of intervention; access to quality assessment of evidence; consideration of key elements of decision; transparency of decision; and understandability of decision by stakeholders. Further field testing and instrument validation and development are ongoing to collaboratively advance MCDA approaches and contribute to more transparent and efficient healthcare decisionmaking.
Analysis of variance
Evidence and Value: Impact on DEcisionMaking
Health technology assessment
Intra-rater correlation coefficients
International Network of Agencies for Heath Technology Assessment
Multicriteria decision analysis
Patient reported outcomes
Randomized controlled trial
Baltussen R, Niessen L: Priority setting of health interventions: the need for multi-criteria decision analysis. Cost Eff Resour Alloc 2006, 4: 14. 10.1186/1478-7547-4-14
Tunis SR: Reflections on science, judgment, and value in evidence-based decision making: a conversation with David Eddy. Health Aff (Millwood) 2007, 26: w500-w515. 10.1377/hlthaff.26.4.w500
Youngkong S, Kapiriri L, Baltussen R: Setting priorities for health interventions in developing countries: a review of empirical studies. Trop Med Int Health 2009, 14: 930–939. 10.1111/j.1365-3156.2009.02311.x
Bryan S, Williams I, McIver S: Seeing the NICE side of cost-effectiveness analysis: a qualitative investigation of the use of CEA in NICE technology appraisals. Health Econ 2007, 16: 179–193. 10.1002/hec.1133
Dhalla I, Laupacis A: Moving from opacity to transparency in pharmaceutical policy. CMAJ 2008, 178: 428–431. 10.1503/cmaj.070799
Vuorenkoski L, Toiviainen H, Hemminki E: Decision-making in priority setting for medicines-a review of empirical studies. Health Policy 2008, 86: 1–9. 10.1016/j.healthpol.2007.09.007
Daniels N, Sabin J: Limits to health care: fair procedures, democratic deliberation, and the legitimacy problem for insurers. Philos Public Aff 1997, 26: 303–350. 10.1111/j.1088-4963.1997.tb00082.x
Doherty J: Cost-effectiveness analysis for priority setting in South Africa - what are the possibilities? S Afr Med J 2010, 100: 816–821.
Cleary SM, McIntyre D, Boulle AM: The cost-effectiveness of antiretroviral treatment in Khayelitsha, South Africa-a primary data analysis. Cost Eff Resour Alloc 2006, 4: 20. 10.1186/1478-7547-4-20
Cleary SM, McIntyre D, Boulle AM: Assessing efficiency and costs of scaling up HIV treatment. AIDS 2008,22(Suppl 1):S35-S42. 10.1097/01.aids.0000327621.24232.71
Camidge DR, Oliver JJ, Skinner C, Attwood B, Nussey F, Jodrell D, et al.: The impact of prognosis without treatment on doctors' and patients' resource allocation decisions and its relevance to new drug recommendation processes. Br J Clin Pharmacol 2008, 65: 224–229. 10.1111/j.1365-2125.2007.02996.x
Schlander M: The use of cost-effectiveness by the National Institute for Health and Clinical Excellence (NICE): no(t yet an) exemplar of a deliberative process. J Med Ethics 2008, 34: 534–539. 10.1136/jme.2007.021683
Baltussen R, Youngkong S, Paolucci F, Niessen L: Multi-criteria decision analysis to prioritize health interventions: Capitalizing on first experiences. Health Policy 2010, 86: 262–264.
Dolan JG: Multi-Criteria clinical decision support. A primer on the use of multiple-criteria decision-making methods to promote evidence-based, patient-centered healthcare. Patient 2010, 3: 229–248.
Browman GP, Manns B, Hagen N, Chambers CR, Simon A, Sinclair S: 6-STEPPPs: A modular tool to facilitate clinician participation in fair decisions for funding new cancer drugs. J Oncol Practice 2008, 4: 2–7. 10.1200/JOP.0812001
Martin DK, Pater JL, Singer PA: Priority-setting decisions for new cancer drugs: a qualitative case study. Lancet 2001, 358: 1676–1681. 10.1016/S0140-6736(01)06714-9
Mullen PM: Quantifying priorities in healthcare: transparency or illusion? Health Serv Manage Res 2004, 17: 47–58. 10.1258/095148404322772723
Wilson E, Sussex J, Macleod C, Fordham R: Prioritizing health technologies in a Primary Care Trust. J Health Serv Res Policy 2007, 12: 80–85. 10.1258/135581907780279495
Wilson EC, Peacock SJ, Ruta D: Priority setting in practice: what is the best way to compare costs and benefits? Health Econ 2009, 18: 467–478. 10.1002/hec.1380
Baltussen R, Ten Asbroek AH, Koolman X, Shrestha N, Bhattarai P, Niessen LW: Priority setting using multiple criteria: should a lung health programme be implemented in Nepal? Health Policy Plan 2007, 22: 178–185. 10.1093/heapol/czm010
Nobre FF, Trotta LT, Gomes LF: Multi-criteria decision making-an approach to setting priorities in health care. Stat Med 1999, 18: 3345–3354. 10.1002/(SICI)1097-0258(19991215)18:23<3345::AID-SIM321>3.0.CO;2-7
Goetghebeur M, Wagner M, Khoury H, Levitt R, Erickson LJ, Rindress D: Evidence and Value: Impact on DEcisionMaking - the EVIDEM framework and potential applications. BMC Health Services Research 2008, 8: 270. 10.1186/1472-6963-8-270
Goetghebeur MM, Wagner M, Khoury H, Levitt RJ, Erickson LJ, Rindress D: Bridging health technology assessment (HTA) and efficient health care decision making with multicriteria decision analysis (MCDA): Applying the EVIDEM framework to medicines appraisal (ahead of print). Med Decis Making 2011, in press.
Goetghebeur MM, Wagner M, Khoury H, Rindress D, Gregoire JP, Deal C: Combining multicriteria decision analysis, ethics and health technology assessment: applying the EVIDEM decisionmaking framework to growth hormone for Turner syndrome patients. Cost Eff Resour Alloc 2010, 8: 4. 10.1186/1478-7547-8-4
Tony M, Wagner M, Khoury H, Rindress D, Papastavros T, Oh P, et al.: Bridging health technology assessment (HTA) with multicriteria decision analyses (MCDA): field testing of the EVIDEM framework for coverage decisions by a public payer in Canada. BMC Health Serv Res 2011, 11: 329. 10.1186/1472-6963-11-329
Department of Health, South Africa: National guideline on: cervical cancer screening programme. [http://www.doh.gov.za/docs/factsheets/guidelines/cancer.pdf]
Cronje HS, Beyer E: Screening for cervical cancer in an African setting. Int J Gynaecol Obstet 2007, 98: 168–171. 10.1016/j.ijgo.2007.05.005
Alliance for Cervical Cancer Prevention: Planning and implementing cervical cancer prevention and control programs: a manual for managers. Chapter 10: Cancer treatment and palliative care. [http://screening.iarc.fr/planningmanual.php?lang=1&PHPSESSID=6c8ded0f31c8336a3ed5e9f6a4d59e63]
Adriano E, Jagoe JM, Harrison T, Riffenburgh RH, Johnstone PA: Survival of patients with untreated cervical carcinoma. Am J Clin Oncol 2003, 26: 369–373.
Waggoner SE: Cervical cancer. Lancet 2003, 361: 2217–2225. 10.1016/S0140-6736(03)13778-6
Department of Health: Statistical Notes. Focus of the Month: Cervical Cancer. [http://www.doh.gov.za/docs/facts/2006/countryhealth2006.pdf]
Franco EL, Duarte-Franco E, Ferenczy A: Cervical cancer: epidemiology, prevention and the role of human papillomavirus infection. CMAJ 2001, 164: 1017–1025.
Bailie RS, Selvey CE, Bourne D, Bradshaw D: Trends in cervical cancer mortality in South Africa. Int J Epidemiol 1996, 25: 488–493. 10.1093/ije/25.3.488
Department of Health: Research update 2003. [http://www.doh.gov.za/docs/reports/2004/nwhsr.pdf]
Canadian Agency for Drugs and Technologies in Health: Liquid-based techniques for cervical cancer screening: systematic review and cost-effectiveness analysis. Issue 103. [http://cadth.ca/index.php/en/publication/799]
Anderson N, Discovery Health: Internal Report prepared for Discovery Health: Summary of liquid-based cytology. 2008.
Discovery Health: Internal Report prepared for Discovery Health: Pap smear analysis updated 02. 2008.
Parrish A, Blockman M: Clinical excellence and the NICEties of value-based priority setting. S Afr Med J 2008, 98: 758–761.
Kenyon C, Ford N, Boulle A: When best practice is bad medicine: a new approach to rationing tertiary health services in South Africa. S Afr Med J 2008, 98: 350. 352, 354
Nutt DJ, King LA, Phillips LD: Drug harms in the UK: a multicriteria decision analysis. Lancet 2010, 376: 1558–1565. 10.1016/S0140-6736(10)61462-6
We wish to acknowledge the contributions of the members of the Clinical Policy Unit at Discovery Health who participated in this beta-testing. Internal sources of support for the study were provided by Discovery Health and BioMedCom Consultants inc.
The authors declare that they have no competing interests.
JKM, MMG, and MW designed the study and HK collected and analysed the data. JKM, MMG and MW drafted the manuscript. DR reviewed the manuscript and is one of the developers of the EVIDEM framework. All authors read and approved the final manuscript.