- Open Access
Decision uncertainty and value of further research: a case-study in fenestrated endovascular aneurysm repair for complex abdominal aortic aneurysms
Cost Effectiveness and Resource Allocation volume 16, Article number: 15 (2018)
Fenestrated endovascular aneurysm repair (fEVAR) is a new approach for complex abdominal aortic aneurysms, limited to a few specialist centers, with limited evidence base. We developed a cost-effectiveness decision model of fEVAR compared to open surgical repair (OSR) to investigate the likely direction of costs and benefits and inform further research projects on this technology.
A systematic review with meta-analysis and a four-state Markov model were used to estimate the cost-effectiveness of fEVAR versus OSR. We used a recent coverage with evidence development framework to characterize the main sources of uncertainty and inform decisions about the type of further research that would be most worthwhile and feasible.
Seven observational comparative studies were identified, of which four presented odds ratios adjusted for confounders. The odds ratios for operative mortality varied widely between studies. Assuming a central estimate of the odds ratio of 0.54 (95% CI 0.05–6.24), the decision model estimated that the incremental cost per quality adjusted life year (QALY) was £74,580/QALY with a probability of 9 and 16% of being cost-effective at standard cost-effectiveness thresholds of £20,000/QALY and £30,000/QALY, respectively. The Expected Value of Perfect Information over 10 years at a threshold of £20,000/QALY was £11.2 million. Operative mortality contributed to most of the uncertainty in the decision model.
In the case of “maturing technologies”, decision modelling indicates the likely direction of costs and benefits and guides the development of further research projects. In our analysis of fEVAR versus OSR, decision uncertainty, particularly around operative mortality, might be effectively resolved by a short-term RCT, or possibly a well-conducted comparative observational study. Decision makers may consider that a conditional coverage decision is warranted with assessments required to make this type of recommendation depending on local priorities and circumstances.
Health technology assessment (HTA) agencies are charged with providing guidance to practitioners and patients based on a systematic and comprehensive assessment of the clinical and cost-effectiveness profile of health technologies. Assessing medical devices (MDs), in particular, raises several challenges which require careful consideration . For a range of reasons, evidence on costs and effects relating to medical devices may be limited . This situation, usually but not exclusively associated to MD evaluation, leads to a dilemma that typically has proved challenging for HTA and policy makers. Making an early decision based on poor evidence carries a high risk of error that may be difficult to reverse, but delaying a decision until more evidence becomes available may leave current patients without effective treatment or health systems with inefficient technology.
Fenestrated endovascular repair fEVAR is an example of such dilemma. Fenestrated EVAR might be considered a maturing innovation in the typology of Sculpher et al. , that is a technology early in its life cycle, usually limited to a few specialist centers, with an evidence base comprising case series and small RCTs. Decision modelling in these situations is unlikely to provide a definitive statement about whether a technology is cost-effective, but rather indicate the likely direction of costs and benefits, the circumstances under which the new technology might represent good value for money, and to guide the development of further research projects . The aim of the study is to illustrate this approach through the clinical and cost-effectiveness evaluation of fEVAR compared to open surgical repair and to estimate the value of further research for the same decision problem.
Abdominal aortic aneurysms (AAAs) occur when the main artery in the body develops a balloon-like bulge of diameter exceeding 3 cm. The prevalence of AAAs in the UK has been estimated at between 1.2 and 7.6% in those over 50 years . The benefits of endovascular aneurysm repair (EVAR) compared to open surgical repair (OSR) of AAAs are well established by large multicenter randomized trials assessing all-cause or aneurysm related mortality and postoperative complications [5,6,7]. However, about 15% of AAAs are considered “complex” (cAAAs), i.e. aneurysms involving renal or visceral arteries or ‘juxta-renal AAAs’ . Surgical treatment options for cAAAs include OSR, usually requiring suprarenal aortic cross-clamping, or fEVAR [8,9,10]. In fEVAR the stent-graft fabric extends over the renal arteries, but perfusion to these arteries is preserved via accurately placed windows (fenestrations) within the stent-graft fabric. The fEVAR procedure is more challenging than standard EVAR as graft positioning requires both longitudinal and rotational alignment of the fenestrations with the target vessels .
Although a small number of systematic reviews have sought to compare efficacy and safety outcomes of fEVAR compared to OSR, their conclusions have been limited by lack of available robust evidence [12,13,14]. In particular, these reviews noted that there were no randomised controlled trials (RCTs), but there were several registries and case series, some of which were large and long-running.
This study evaluates the effectiveness, safety and cost-effectiveness of fEVAR versus OSR for elective repair of cAAA in patients suitable for either procedure. Another solution for more complicated aneurysms is branched EVAR (bEVAR). The term ‘branched’ refers to the need to bridge the gap, created by increased diameter of aorta, between the main body of aortic stent graft and target vessels and not to any actual branch from the graft itself . Based on the distinction between bEVAR and fEVAR in accordance with the Society of Vascular Surgery reporting standards on thoracic endovascular aortic repair , branched cAAAs endovascular procedures were excluded from this analysis.
Systematic review and meta-analysis
This updated systematic review was conducted and reported according to the Meta-analysis of Observational Studies in Epidemiology (MOOSE) reporting guidelines .
Data sources and search strategy
MEDLINE (via Ovid) was searched for previous systematic reviews or meta-analyses of fEVAR for the treatment of cAAAs. We also sought new primary studies published since the end date of the literature searches of the most recently undertaken systematic review (i.e. October 2013)  and up to 14th January 2015. A copy of the search strategy is available as an Additional file 1: Appendix S1. As this recent review found no RCTs, this updated study also includes observational evidence. Two reviewers (OC, RST) independently examined titles and abstracts of primary studies.
Relevant articles were obtained in full, and assessed against the inclusion/exclusion criteria described in Additional file 1: Table S1. Any disagreement was resolved through discussion.
Data extraction and risk of bias assessment
Details were extracted for each study including: year and country of publication, sample size, age and gender distribution, duration of follow up, anatomical location of aneurysms, and details of implant used. Outcomes of interest were: operative mortality, late mortality, complications, re-interventions, and resource use during the primary admission (i.e. length of stay, operative time, blood, and intensive care). Study quality was assessed using a modified list of criteria developed for case series  and the addition of three criteria for comparative studies. Data extraction and risk of bias assessment was undertaken by one author (OC) and verified by another (RST).
Where outcome data were consistently reported, results were pooled across studies using Der Simonian and Laird random-effects meta-analyses . For operative mortality, we considered that treatment effects should only be pooled where authors made statistical adjustment for relevant confounding factors. For resource use, we considered that unadjusted differences were adequate as these variables are relatively less influenced by factors such as body mass index, leukocyte count or other biomarkers. Moreover, no studies reported between-groups adjusted comparison of these measures. Recommendations from the Cochrane Handbook for systematic reviews of interventions were followed to deal with missing data . Heterogeneity was quantified using the I2 statistic and results displayed in a forest plot. All statistical analyses were performed using STATA® 13.1 StataCorp US.
This economic evaluation was undertaken and reported according to the Consolidated Health Economic Evaluation Reporting Standards (CHEERS) statement .
A decision analytic model was developed in Excel®2012 (Microsoft Corporation, US) to estimate the lifetime cost and QALY of each procedure. The structure of the model was adapted from a previously published economic evaluation of conventional EVAR for AAA repair, using essentially the same structure, with parameters estimated from the systematic review . The model consists of a lead-in period lasting 6 months, during which the cAAA repair is undertaken with fEVAR or OSR (Fig. 1). Perioperative survivors then enter a long-term three state Markov model, which extrapolates costs and outcomes over a lifetime horizon using 6-monthly cycles. During each cycle, patients may die of cAAA-related causes, such as rupture of the aneurysm, or die of other causes. Patients undergo surveillance with an outpatient visit and ultrasound every year after endovascular repair and every 5 years after open repair . Survivors have an ongoing risk of re-intervention, e.g. to correct complications of the device such as endoleaks which might be detected by surveillance. Patients can have only one re-intervention per cycle, but can have more than one re-intervention over their lifetime. The risk of re-intervention depends on initial procedure (OSR or fEVAR) and time from initial cAAA repair, but is assumed independent between cycles. Re-interventions reduce health-related quality of life (HRQOL) and incur a hospital cost. Mortality risks related to re-intervention are captured in the estimate of aneurysm-related deaths.
Perspective of the decision model and inputs required
Health outcomes were expressed in terms of quality-adjusted life years (QALYs). The perspective adopted was that of a publicly funded health system using an explicit cost-effectiveness threshold for decision making. The UK National Health Service (NHS)  is used here as an example, using thresholds in the range £20,000-£30,000 per QALY. The price year was 2016/17, and costs and benefits were discounted at an annual rate of 3.5% .
Table 1 summarises the main input variables required by the decision model and their distributions used to represent uncertainty. Parameter estimates for the model were obtained from the updated systematic literature review, personal communication with the device manufacturer, and routine UK National Health Service (NHS) sources. Where specific estimates for fEVAR were unavailable, the literature on conventional EVAR was considered. The mean age at which cAAAs repair is undertaken was 73 years . Mortality risks for non-AAA cause deaths were based on UK male population life tables .
Expected value of perfect information
Fenestrated EVAR is a relatively novel technology and the evidence relating to its costs and benefits is limited. In this regard, further research might be worthwhile if it reduces uncertainty and avoids the consequences of making a wrong decision about the use of the intervention. The expected value of perfect information (EVPI) estimates an upper bound for the value of a new study, if it could eliminate all uncertainty about which treatment was more cost-effective at the given cost-per-QALY threshold. Partial EVPI (EVPPI) is also calculated using web-based emulator software . EVPPI enables identification of those parameters that contribute particularly highly to decision uncertainty.
A decision timeframe of 10 years is used, to allow for obsolescence and product innovation by competing technologies. We assume that 1782 cAAA patients would be eligible for repair each year in the UK, based on German health insurance data  adjusted for UK population size. Net population benefit was calculated at cost-effectiveness thresholds used in the UK.
Decision uncertainty was estimated through probabilistic sensitivity analysis using 1000 Monte-Carlo simulations of the model .
Decision making and the value of further research
We used a recently developed framework to characterize the sources of uncertainty and to identify the most appropriate decision option for fEVAR given the available evidence [15, 29]. Four decision options were considered: accept fEVAR for immediate and general use in the national heath service in indicated patients (accept fEVAR), reject fEVAR (i.e. accept OSR), accept fEVAR provisionally but conditioned upon the collection of further evidence (accept with research, AWR), or reject fEVAR for general use until further evidence becomes available (only in research, OIR).
Systematic review and meta-analysis
The updated systematic review study selection process is illustrated in Fig. 2. A total of 22 studies involving 1641 fEVAR patients were included (see Additional file 1: Appendix S2 for list of included studies). Whilst we did not find any randomised trials comparing fEVAR and OSR, we did include seven comparative observational studies, reporting outcomes for both fEVAR and OSR patients [30,31,32,33,34,35,36].
A summary of the study and patient population characteristics and the interventions is reported in Additional file 1: Table S2. Study sample sizes were small, with a median of 78 fEVAR treated patients, a median patient age of 74 years and a median follow-up of 18.4 months. All studies showed high to moderate risk of bias as reported in Additional file 1: Table S3.
Relative risk of operative mortality
Seven observational studies compared perioperative [30, 36] or 30-day [31,32,33,34,35] mortality between fEVAR and OSR (Additional file 1: Table S4). The pooled odds ratio for mortality across all seven studies was 0.60 (95% CI 0.23–1.60, p = 0.014) (Additional file 1: Figure S1). However, there was a high degree of heterogeneity between studies (I2 statistic: 62.6%, p = 0.014).
Risk of bias for estimate of relative risk of operative mortality
In a context of poor available evidence, it is even more important to explore the heterogeneity and evaluate the risk of bias in order to reliably use the information for decision-making purposes. One source of heterogeneity was the method used by the study to account for possible confounding. Three studies did not make any adjustment for confounding variables [31,32,33]. Of the four that took account of confounding variables in some way, one used multivariate regression , one used propensity score matching , and two used a published risk score  to estimate what operative mortality hypothetically would have been in the fEVAR patients had they had undergone open repair [30, 36]. More on the risk of bias assessment performed for estimate of relative risk of operative mortality is reported in Additional file 1: Appendix S3.
Relative risk of operative mortality stratified by study methodology
To further explore the heterogeneity, we stratified the pooled estimate of effect size according to the methodology in the study employed to adjust for confounders. The Raux study  (propensity score matching) found an OR of 5.1 (95% CI 1.1–24). The Tsilimparis study  (multivariate regression) found an OR of 0.19 (95% CI 0.04–0.83). The two studies that used the V-POSSUM risk score 26,32 found a pooled OR of 0.37 (95% CI 0.20–0.69). The remaining studies that did not adjust for confounders found a pooled OR of 0.98 (95% CI 0.26–3.7).
Relative risk of operative mortality used as input to the decision model
The pooled estimate of the odds ratio over the four studies that performed some form of risk adjustment is 0.54 (95% CI 0.16–1.88) (Fig. 3). The decision model uses the OR estimate of 0.54 as the base-case (log(OR) = μ = − 0.61, SE(μ) = 0.644). However, this SE from random-effects meta-analysis does not take account of the full distribution of effect sizes across the heterogeneous studies . In order to make predictions about the likely effect size that might occur in a future new study or in the general practice use, we considered the estimate of τ2 from the four adjusted studies in the random-effects meta-analysis [30, 34,35,36] (τ2 = 1.15). This suggests a mean and confidence interval for a prediction for a new study of 0.54 (95% CI 0.05–6.24).
Baseline risk of operative mortality used as input to the decision model
Twenty-two studies reported operative mortality after fEVAR, with 43 deaths in 1670 patients (mean rate 2.6%) [30,31,32,33,34,35,36, 39,40,41,42,43,44,45,46,47,48,49,50,51,52,53]. However, there was considerable heterogeneity, with operative mortality varying from 0% to 9.5% (Additional file 1: Table S5). Analysis of UK national hospital episode statistics indicates the mean operative mortality after OSR in clinical practice, as opposed to selected case series in specialist centres, is 14%, with considerable heterogeneity . In the model, we use the mean operative mortality after fEVAR from the systematic review as the probability of AAA death during the first model cycle in the base-case and vary this from 0.5% to 9.5% in sensitivity analyses, assuming the baseline rate and the relative risk are independent.
Late aneurysm-related mortality
One study included in the review compared late aneurysm related mortality with about 1 year of follow-up between fEVAR and OSR  and found no late AAA-related deaths in either group.
Thirteen studies [31, 32, 36, 39,40,41,42, 44,45,46, 49, 51, 52] reported late AAA mortality after fEVAR. The overall rate of late AAA mortality was 6 deaths in 1558 patient-years (i.e. an average of 0.004 late AAA deaths per person-year) although 10 studies [36, 39,40,41,42,43,44,45,46,47] reported no late AAA-related deaths (Additional file 1: Table S5). The decision model assumes this 0.004 rate of late AAA mortality after fEVAR from > 6 months to 4 years (as found by this review), but that this rate would increase by 50% between 4 years and 8 years and doubles after 8 years (as found after conventional EVAR by the EVAR-1 trial ). Furthermore, the base-case model assumes relative risk of late AAA mortality would be greater after fEVAR compared with OSR (as found by the EVAR-1 trial ). Sensitivity analyses explore a scenario where the rate of late AAA mortality does not increase over time and is the same as OSR (as found by Chisci et al.  study at 1 year) over the patient’s lifetime.
Early and late re-interventions
Eleven studies reported a total of 81 early re-interventions (in-hospital or 30-day) in 1198 patients after fEVAR, an average probability of 0.07 [30, 31, 34, 39, 42, 43, 47, 50, 52]. Three studies compared the risk of early (in-hospital or 30-day) re-interventions for fEVAR compared to open repair [30, 31, 35]. All found more re-interventions after open repair with a pooled OR across the three trials of 0.42 (95% CI 0.26–0.68) (Additional file 1: Figure S4). These results were not adjusted for case-mix. Eleven studies reported a total of 132 late re-interventions in 1507 patient-years of follow-up after fEVAR, an average rate of 8.8 per 100 patient-years. No trial reported late re-interventions after open repair [32, 36, 41, 47, 48, 51, 52].
As the studies did not provide sufficient information on how the re-interventions rate might change over time, the decision model uses the rates of early re-interventions from the literature review, and the relative risk of late re-intervention (fEVAR vs OSR) from the EVAR-1 RCT  that compared EVAR versus open repair for conventional AAA, which was 6.29 (95% CI 3.09–12.78) from 6 months to 4 years, 1.60 (95% CI 0.81–3.15) from 4 to 8 years, and 1.51 (1.71–3.19) after 8 years .
Five studies provided comparative information on one or more healthcare resource use outcomes (Table 2, Table 3) [30,31,32,33, 35]. There was considerable heterogeneity across most of the resource use data. The decision model uses the average estimates of procedure resource use (i.e. length of stay, operating theatre, blood units and Intensive Care Unit stay) costed at UK prices in 2016/17 [55, 56]. The list price of the fEVAR device was £16,500 including extra stent parts and accessories (personal communication with Cook Medical, at 2015 prices). Sensitivity analyses were conducted by varying the price of the stent to reflect possible discounts. A reduction in cost might also be achieved if use of fEVAR was associated with lower length of stay or less use of other hospital resources, or if surveillance and re-interventions could be accurately targeted at patients with higher risk of complications. Other unit costs associated with the index procedure are given in Table 3. The cost of a AAA-related re-intervention was taken as £8670 (SE 831) .
Health-related quality of life
No study reported health related quality of life (HRQOL) after fEVAR or OSR. The EVAR-1 trial found that HRQOL was better after conventional EVAR than OSR for the first 6 months (mean difference in EQ-5D-3L 0.060, SE(0.017)) but there was no difference by 1 year . HRQOL is considerably diminished in the 6 months after a re-intervention and in the 6 months preceding death .
Peri-operative complications were poorly and heterogeneously reported, especially after open repair. The studies did not give a clear indication of the seriousness of the complication or its duration. Therefore complications were not explicitly costed or associated with QALYs in the model. However, some measure of their impact is included in the cost-effectiveness analysis, because serious complications and adverse events during the surgical procedure and hospital stay will be reflected in HRQOL, length of stay, operative theatre time, blood use, and use of ICU.
The mean difference in lifetime cost between fEVAR and OSR was £15,606 (95% CI 8390–22,512), and the lifetime difference in QALY was 0.209 (95% CI − 0.124–1.311), (Table 4, Additional file 1: Figure S2). The incremental cost-per-QALY is estimated to be £74,579/QALY with a probability of 0.090 of being cost-effective at a threshold of £20,000 per QALY, and a probability of 0.158 at £30,000 per QALY . Almost all of the additional cost is incurred in the price of the fEVAR device (Additional file 1: Figure S3).
External validation of model results
The predicted overall survival in the model after OSR at 5 years is 58% (Additional file 1: Figure S2), the same as reported from a study linking Hospital Episode Statistics to national mortality records in the UK .
The following univariate sensitivity analyses were conducted:
High relative risk of operative mortality (OR 5.1, 95% CI 1.10–24) ;
Low relative risk of operative mortality (OR 0.19, 95% CI 0.05–0.83) ;
High baseline risk of operative mortality after fEVAR of 9.5% ;
Low baseline risk of operative mortality after fEVAR of 1% ;
No difference in late (> 30 day) mortality between fEVAR and open repair ;
No difference in late re-interventions between fEVAR and open repair;
Price of fEVAR device of £7500.
The sensitivity analyses show the circumstances under which fEVAR plausibly might be cost-effective (Table 4). If the OR of operative mortality were that observed in the most favourable study, assuming that baseline risk remains unchanged, then fEVAR comes close to the £30,000/QALY threshold. fEVAR would also be close to this threshold if the cost of the device were £7500, rather than £16,500.
Expected value of perfect information
The EVPI over 10 years at a threshold of £20,000 per QALY was £11.2 million. EVPPI analysis showed that 92% of this value could be attributed to the two operative mortality parameters (baseline and odds ratio), and the remainder to the cost of the procedure.
The value of further research
Under the base-case model, fEVAR is unlikely to be cost-effective. This does not necessarily mean that it should be rejected, if further evidence might lead to different conclusions. Claxton et al. provide a checklist of the sequence of assessments that would be required for an HTA body considering whether further research should be mandated . We provide a structured discussion of the issues as they might apply to fEVAR given the results of this study. However, each decision maker would need to review and interpret the data as they apply to health services in their jurisdiction.
Is the technology expected to be cost-effective?
This would depend on the decision criteria used (e.g. the cost-per-QALY threshold) as well as the decision-maker’s interpretation of the appropriate inputs for the model. We assume it is likely not cost-effective on current evidence and from the perspective of the NHS in the UK .
Are there significant irrecoverable costs?
Implementing FEVAR would not require significant irrecoverable costs in NHS infrastructure, if most of the up-front investment in operating facilities and interventional radiography has already been made. However, on-going training for surgical teams would be required, and the procedure is of course irreversible for individual patients.
Does more research seem worthwhile?
There are no RCTs, there is considerable decision uncertainty from the model, and more research might plausibly reduce that uncertainty. Some assessment is therefore needed of the type of evidence and whether it can be conducted without approval.
Is the research possible with (without) approval?
Generally, most types of research are possible without approval, including RCTs. However, the patient population who might potentially benefit from fEVAR is small and hence population EVPI is only about £11 million at a threshold of £20,000/QALY in the UK. This means a large-scale, expensive, lengthy RCT would not be cost-effective. Nevertheless, in our EVPPI analysis, most of the uncertainty was around operative mortality, which might indicate that a RCT with short follow-up, or even a well-designed observational study would provide sufficient evidence for definitive approval or rejection. These factors would indicate OIR would be an appropriate decision.
Will other sources of uncertainty resolve over time?
A further source of uncertainty, which has not been reflected in our analysis, is the possibility of a learning curve for fEVAR. Fully understanding the learning curve might require the technology to be in widespread use. If this knowledge is important to the decision maker, it would argue in favour of AWR even though the technology is not thought to be cost-effective on current evidence. However, a recent analysis based on over 40,000 patients administrative data from year 2006 to 2013 in Germany using hierarchical regression models showed the learning curve for fEVAR on in-hospital mortality and length of stay is actually negligible .
In this study we investigated the clinical and cost-effectiveness evaluation of fEVAR compared to open surgical repair as an example of a technology with limited evidence base and diffusion across a few specialized sites where the decision modelling is intended to identify the circumstances under which the new technology might represent a worthwhile investment and to inform future evidence generation projects.
This updated systematic review and meta-analysis found no RCTs that compared fEVAR and OSR. A recent NIHR funded project has updated a similar review up to March 2017 confirming no RCTs is available to date and the most recent comparative FEVAR vs OSR study was Raux et al. 2014 as per our results . Seven observational studies compared these procedures, with varying methodological quality and substantial heterogeneity across all the outcomes.
Decision-makers responsible for setting prices of medical technologies or deciding re-imbursement policy need to make an assessment of whether the technology appears cost-effective on current evidence. In cases where the evidence base is highly uncertain and of poor methodological quality, decision makers may also need to assess what type of further research might be of value.
The usefulness of a decision model in these cases is not to make a definitive statement about cost-effectiveness, but rather to serve as a framework to organise the available data, identify the most important gaps in the evidence, and assess the plausible circumstances under which the technology might be cost-effective. In the fEVAR case, we were reliant on non-randomised comparative studies that are prone to bias and confounding. We assessed the risk of bias and focused our meta-analysis on four comparative studies that were judged to have performed adequate adjustment. These data were used in the model to estimate short-term outcomes. There was little data on late outcomes after fEVAR, particularly late AAA-deaths and complications. Long-term data from conventional endovascular devices clearly shows a greater rate of complications and re-interventions than after open repair, though it is difficult to know how generalizable these data are to fEVAR.
Using these data as base-case inputs, the model estimated that fEVAR is associated with greater survival and QALYs over a patient’s lifetime compared with OSR. However, the high acquisition cost of the device means that the cost of fEVAR considerably exceeds OSR. The EVAR-1 trial  showed that the frequency of re-interventions over the medium and long term is also likely to be greater under endovascular repair, though this is not a decisive driver of overall cost in the model. The overall treatment costs make fEVAR not cost-effective at conventional National Institute for Health and Care Excellence (NICE) thresholds, with an ICER exceeding £74,000/QALY. The cost-effectiveness threshold seeks to reflect the magnitude of health that other types of patients forgo when additional costs are incurred on a new technology rather than treatments from which they will benefit . NICE’s threshold range of £20,000 to £30,000 has no empirical link to such opportunity costs, and recent research estimates the threshold to be lower at around £13,000 per QALY . This would suggest that fEVAR is not cost-effective in the UK NHS based on current evidence. The model takes into account resource consumption estimates from the literature review of international studies, hence transferability of the model to other jurisdictions is possible, subject to availability of costs data from other countries. Criteria for decision-making rarely rely on a formal cost-effectiveness threshold outside UK , hence each decision maker would need to review and interpret these results in the context of the health systems in place in their jurisdiction. Sensitivity analyses showed that fEVAR would be close to being cost-effective at the upper range of NICE’s threshold if the OR of operative mortality were 0.19, as estimated by one of the studies . fEVAR would also be cost-effective at this threshold if the price of the stent was about £7500 instead of £16,500. High-volume centres may already be able to negotiate discounts with suppliers, although these are commercially confidential.
Expected value of further information analysis showed that the population EVPI is not especially great, because this is a specialist procedure with a limited target population. This would increase if one was willing to accept that research is a global public good that would benefit patients worldwide . Nevertheless, most of the decision uncertainty is around operative mortality. This might be effectively resolved by a short-term RCT, or possibly a well-conducted comparative observational study . Decision makers may consider that a conditional coverage decision is warranted with assessments required to make this type of recommendation depending on local priorities and circumstances.
The model we have used in this paper is very simple, as we have tried to remain faithful to the evidence. There may be several other considerations which would warrant further evidence collection. For example, the model assumes the OR of operative mortality is independent of the base-line risk. In practice there may be sub-group effects. The literature review does not discriminate between male and female patients. There is some evidence from conventional EVAR that outcomes for females are different, given the different anatomical morphology. Over 90% of incident AAA patients between 55 and 84 years age are male, and outcomes and costs in women are insufficiently described in the literature . There are some other important complications arising from cAAA repair, such as renal impairment or spinal cord injury that have not been taken into account in the model .
Nevertheless, the situation described in this study is typical of a medical-device procedure whose adoption and diffusion occur without strong clinical evidence supporting its marketing authorization, particularly in Europe. Based on current evidence, our model predicts that at conventional decision-making thresholds in the UK, fEVAR is not cost-effective and more research might plausibly reduce the decision uncertainty from the model. Staged approaches to evidence collection  or innovative and structured decision-making processes  linking the gathering of additional information with policies on approval for reimbursement around devices-related surgical procedures are recommended.
Tarricone R, Torbica A, Drummond M. Challenges in the assessment of medical devices: the MedtecHTA project. Health Econ. 2017;26:5–12.
Tarricone R, Torbica A, Drummond M. Key recommendations from the MedtecHTA Project. Health Econ. 2017;26:145–52.
Sculpher M, Drummond M, Buxton M. The iterative use of economic evaluation as part of the process of health technology assessment. J Health Serv Res Pol. 1997;2(1):26–30.
Paravastu SC, Jayarajasingam R, Cottam R, Palfreyman SJ, Michaels JA, Thomas SM. Endovascular repair of abdominal aortic aneurysm. Cochrane Database Syst Rev. 2014;1:004178.
Prinssen M, Verhoeven EL, Buth J, et al. A randomized trial comparing conventional and endovascular repair of abdominal aortic aneurysms. N Engl J Med. 2004;351:1607–18.
EVAR Trial Participants. Endovascular aneurysm repair and outcome in patients unfit for open repair of abdominal aortic aneurysm (EVAR trial 2): randomised controlled trial. Lancet. 2005;365:2187–92.
EVAR Trial Participants. Endovascular aneurysm repair versus open repair in patients with abdominal aortic aneurysm (EVAR trial 1): randomised controlled trial. Lancet. 2005;365:2179–86.
Jongkind V, Yeung KK, Akkersdijk GJ, et al. Juxtarenal aortic aneurysm repair. J Vasc Surg. 2010;52:760–7.
Cross J, Richards T. Current practice of aneurysm management in the UK. Ann R Coll Surg Engl. 2014;96:27–31.
Cross J, Raine R, Harris P, Richards T. Indications for fenestrated endovascular aneurysm repair. Br J Surg. 2012;99:217–24.
Greenberg RK, Lytle B. Endovascular repair of thoracoabdominal aneurysms. Circulation. 2008;117:2288–96.
Sun Z, Mwipatayi BP, Semmens JB, Lawrence-Brown MM. Short to midterm outcomes of fenestrated endovascular grafts in the treatment of abdominal aortic aneurysms: a systematic review. J Endovasc Ther. 2006;13:747–53.
Nordon IM, Hinchliffe RJ, Holt PJ, Loftus IM, Thompson MM. Modern treatment of juxtarenal abdominal aortic aneurysms with fenestrated endografting and open repair–a systematic review. Eur J Vasc Endovasc Surg. 2009;38:35–41.
Di X, Ye W, Liu CW, Jiang J, Han W, Liu B. Fenestrated endovascular repair for pararenal abdominal aortic aneurysms: a systematic review and meta-analysis. Ann Vasc Surg. 2013;27:1190–200.
Claxton K, Palmer S, Longworth L, et al. Informing a decision framework for when NICE should recommend the use of health technologies only in the context of an appropriately designed programme of evidence development. Health Technol Assess. 2012;16:1–323.
Rothery C, Claxton K, Palmer S, Epstein D, Tarricone R, Sculpher M. Characterising uncertainty in the assessment of medical devices and determining future research needs. Health Econ. 2017;26:109–23.
Stroup DF, Berlin JA, Morton SC, et al. Meta-analysis of observational studies in epidemiology: a proposal for reporting. Meta-analysis Of Observational Studies in Epidemiology (MOOSE) group. JAMA. 2000;283:2008–12.
Armstrong N, Burgers L, Deshpande S, et al. The use of fenestrated and branched endovascular aneurysm repair for juxtarenal and thoracoabdominal aneurysms: a systematic review and cost-effectiveness analysis. Health Technol Assess. 2014;18:1–66.
Moga C, Guo B, Schopflocher D, Harstall C. Development of a quality appraisal tool for case series studies using a modified Delphi technique. 2012. http://www.ihe.ca/publications/development-of-a-quality-appraisal-tool-for-case-series-studies-using-a-modified-delphi-technique. Accessed January 2015.
Higgins JPT, Green S, editors. Cochrane handbook for systematic reviews of interventions version 5.1.0 [updated March 2011]. The Cochrane Collaboration. 2011. www.handbook.cochrane.org.
Husereau D, Drummond M, Petrou S, et al. Consolidated Health Economic Evaluation Reporting Standards (CHEERS) statement. Value Health. 2013;16:e1–5.
Brown LC, Powell JT, Thompson SG, Epstein DM, Sculpher MJ, Greenhalgh RM. The UK EndoVascular Aneurysm Repair (EVAR) trials: randomised trials of EVAR versus standard therapy. Health Technol Assess. 2012;16:1–218.
Moll FL, Powell JT, Fraedrich G, et al. Management of abdominal aortic aneurysms clinical practice guidelines of the European society for vascular surgery. Eur J Vasc Endovasc Surg. 2011;41(Suppl 1):S1–58.
National Institute of Health and Care Excellence. Guide to the methods of technology appraisal 2013. 2013. https://www.nice.org.uk/article/pmg9/chapter/Foreword. Accessed Mar 15 2016.
Varabyova Y, Blankart CR, Schreyögg J. The Role of Learning in Health Technology Assessments: An Empirical Assessment of Endovascular Aneurysm Repairs in German Hospitals. Health Econ. 2017;26 Suppl 1:93–108.
Office for National Statistics. National life tables: United Kingdom. 2017. https://www.ons.gov.uk/peoplepopulationandcommunity/birthsdeathsandmarriages/lifeexpectancies/datasets/nationallifetablesunitedkingdomreferencetables. Accessed 5 July 2017.
Strong M, Oakley JE, Brennan A. Estimating multiparameter partial expected value of perfect information from a probabilistic sensitivity analysis sample: a nonparametric regression approach. Med Decis Making. 2014;34:311–26.
Claxton K, Sculpher M, McCabe C, et al. Probabilistic sensitivity analysis for NICE technology assessment: not an optional extra. Health Econ. 2005;14:339–47.
Rothery C, Claxton K, Palmer S, Epstein D, Tarricone R, Sculpher M. Characterising uncertainty in the assessment of medical devices and determining future research needs. In: University of York, 2016.
Canavati R, Millen A, Brennan J, et al. Comparison of fenestrated endovascular and open repair of abdominal aortic aneurysms not suitable for standard endovascular repair. J Vasc Surg. 2013;57:362–7.
Chisci E, Kristmundsson T, de Donato G, et al. The AAA with a challenging neck: outcome of open versus endovascular repair with standard and fenestrated stent-grafts. J Endovasc Ther. 2009;16:137–46.
Donas KP, Eisenack M, Panuccio G, Austermann M, Osada N, Torsello G. The role of open and endovascular treatment with fenestrated and chimney endografts for patients with juxtarenal aortic aneurysms. J Vasc Surg. 2012;56:285–90.
Moore Randy D, Macierewicz J, Cina C, Abuznadah W, Nutley M, Motamedi M. Fenestrated endovascular vs open repair for juxtarenal aortic aneurysmal disease. Vascular. 2006;14:S122–3.
Raux M, Patel VI, Cochennec F, et al. A propensity-matched comparison of outcomes for fenestrated endovascular aneurysm repair and open surgical repair of complex abdominal aortic aneurysms. J Vasc Surg. 2014;60:858–64.
Tsilimparis N, Perez S, Dayama A, Ricotta JJ 2nd. Endovascular repair with fenestrated-branched stent grafts improves 30-day outcomes for complex aortic aneurysms compared with open repair. Ann Vasc Surg. 2013;27:267–73.
British Society for Endovascular Therapy and the Global Collaborators on Advanced Stent-Graft Techniques for Aneurysm Repair (GLOBALSTAR) Registry. Early results of fenestrated endovascular repair of juxtarenal aortic aneurysms in the United Kingdom. Circulation. 2012;125:2707–15.
Mosquera D, Chiang N, Gibberd R. Evaluation of surgical performance using V-POSSUM risk-adjusted mortality rates. J Surg. 2008;78:535–9.
Higgins J, Thompson S, Spiegelhalter D. A re-evaluation of random-effects meta-analysis. J R Stat Soc Ser A Stat Soc. 2009;172:137–59.
Lee JT, Lee GK, Chandra V, Dalman RL. Comparison of fenestrated endografts and the snorkel/chimney technique. J Vasc Surg. 2014;60:849–56 (discussion 56-7).
O’Neill S, Greenberg RK, Haddad F, Resch T, Sereika J, Katz E. A prospective analysis of fenestrated endovascular grafting: intermediate-term outcomes. Euro J Vasc Endovasc Surg. 2006;32:115–23.
Tambyraja AL, Fishwick NG, Bown MJ, Nasim A, McCarthy MJ, Sayers RD. Fenestrated aortic endografts for juxtarenal aortic aneurysm: medium term outcomes. Euro J Vasc Endovasc Surg. 2011;42:54–8.
Verhoeven ELG, Vourliotakis G, Bos WTGJ, et al. Fenestrated stent grafting for short-necked and juxtarenal abdominal aortic aneurysm: an 8-year single-centre experience. Euro J Vascu Endovasc Surg. 2010;39:529–36.
Amiot S, Haulon S, Becquemin JP, et al. Fenestrated endovascular grafting: the French multicentre experience. Euro J Vasc Endovasc Surg. 2010;39:537–44.
Dijkstra ML, Tielliu IF, Meerwaldt R, et al. Dutch experience with the fenestrated Anaconda endograft for short-neck infrarenal and juxtarenal abdominal aortic aneurysm repair. J Vasc Surg. 2014;60:301–7.
Greenberg RK, Sternbergh WC 3rd, Makaroun M, et al. Intermediate results of a United States multicenter trial of fenestrated endograft repair for juxtarenal abdominal aortic aneurysms. J Vasc Surg. 2009;50(730–7):e1.
Quinones-Baldrich WJ, Holden A, Mertens R, et al. Prospective, multicenter experience with the Ventana Fenestrated System for juxtarenal and pararenal aortic aneurysm endovascular repair. J Vasc Surg. 2013;58:1–9.
Semmens JB, Lawrence-Brown MM, Hartley DE, Allen YB, Green R, Nadkarni S. Outcomes of fenestrated endografts in the treatment of abdominal aortic aneurysm in Western Australia (1997–2004). J Endovasc Ther. 2006;13:320–9.
Kristmundsson T, Sonesson B, Dias N, Tornqvist P, Malina M, Resch T. Outcomes of fenestrated endovascular repair of juxtarenal aortic aneurysm. J Vasc Surg. 2014;59:115–20.
Kristmundsson T, Sonesson B, Malina M, Malina M, Bjorses K, Bjorses K, Dias N, Dias N, Resch T. Fenestrated endovascular repair for juxtarenal aortic pathology. J Vasc Surg. 2009;49:568–75.
Vemuri C, Oderich GS, Lee JT, et al. Postapproval outcomes of juxtarenal aortic aneurysms treated with the Zenith fenestrated endovascular graft. J Vasc Surg. 2014;60:295–300.
Ziegler P, Perdikides TP, Avgerinos ED, Avgerinos ED, Umscheid T, Umscheid T, Stelter WJ. Fenestrated and branched grafts for para-anastomotic aortic aneurysm repair. J Endovasc Surg. 2007;14:513–9.
Banno H, Cochennec F, Marzelle J, Becquemin JP. Comparison of fenestrated endovascular aneurysm repair and chimney graft techniques for pararenal aortic aneurysm. J Vasc Surg. 2014;60:31–9.
Rolls AE, Jenkins M, Bicknell CD, et al. Experience with a novel custom-made fenestrated stent graft in the repair of juxtarenal and type IV thoracoabdominal aneurysms. J Vasc Surg. 2014;59:615–22.
Karthikesalingam A, Holt PJ, Patterson BO, et al. Elective open suprarenal aneurysm repair in England from 2000 to 2010 an observational study of hospital episode statistics. PLoS ONE. 2013;8:e64163.
Patel R, Powell JT, Sweeting MJ, Epstein DM, Barrett JK, Greenhalgh RM. The UK EndoVascular Aneurysm Repair (EVAR) randomised controlled trials: long-term follow-up and cost-effectiveness analysis. Health technol Assess. 2018;22:1–32.
Curtis L, Burns A. Unit costs of health and social care 2016. In: PSSRU, editor. Canterbury; 2017.
Epstein D. Health Related Quality of Life after Surgical Repair of Asymptomatic Abdominal Aortic Aneurysms. FEG Working Paper Series 02/16, Faculty of Economics and Business, University of Granada. 2016. https://ideas.repec.org/p/gra/fegper/02-16.html.
Claxton K, Palmer S, Longworth L, et al. A comprehensive algorithm for approval of health technologies with, without, or only in research: the key principles for informing coverage decisions EAES recommendations on methodology of innovation management in endoscopic surgery. Value Health. 2016;19:885–91.
Vallabhaneni S, Haycox A, Cook A, et al. A Risk-adjusted and anatomically stratified cohort comparison study of open surgery, endovascular techniques and medical management for juxtarenal aortic aneurysms: The UK COMplex AneurySm Study (UK-COMPASS). 2017. Accessed Feb 2018.
Claxton K, Martin S, Soares M, et al. Methods for the estimation of the National Institute for Health and Care Excellence cost-effectiveness threshold. Health Technol Assess. 2015;19:1.
Torbica A, Tarricone R, Drummond M. Does the approach to economic evaluation in health care depend on culture, values, and institutional context? https://doi.org/10.1007/s10198-017-0943-1. FAU - Torbica, Aleksandra.
Eckermann S, Willan AR. Globally optimal trial design for local decision making. Health Econ. 2009;18:203–16.
Howard DP, Banerjee A, Fairhead JF, Handa A, Silver LE, Rothwell PM. Age-specific incidence, risk factors and outcome of acute abdominal aortic aneurysms in a defined population. Br J Surg. 2015;102:907–15.
Sedrakyan A, Campbell B, Merino JG, Kuntz R, Hirst A, McCulloch P. IDEAL-D: a rational framework for evaluating and regulating the use of medical devices. BMJ. 2016;353:i2372.
OC lead the systematic review of the clinical evidence and meta-analyses. DE developed the decision analytic model. All authors read and approved the final manuscript.
The authors would like to thank the whole MedTecHTA consortium.
The authors declare that they have no competing interests.
Availability of data and materials
The datasets analysed during the current study are available from the corresponding author on reasonable request.
Consent for publication
Ethics approval and consent to participate
This paper is based on research funded by the European Union Seventh Framework Programme under grant agreement HEALTH-F3-2012-305694 (Project MedtecHTA). The funder had no role in the conduct of the research or in the decision to submit the paper for publication. The views and opinions expressed therein are those of the authors.
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Additional file 1. Appendix S1. Search strategy. Appendix S2. List of included studies. Appendix S3. More on the Risk of bias for estimate of relative risk of operative mortality. Table S1. Inclusion and exclusion criteria. Table S2. Characteristics of included studies. Table S3. Risk of bias assessment. Table S4. Thirty-day mortality or peri-operative mortality in comparative studies. Table S5. Operative mortality and late mortality after fenestrated EVAR. Figure S1. Odds ratios of operative mortality fEVAR vs OSR, all comparative studies. Figure S2. Predicted survival curves from the base-case model. Figure S3. Predicted costs (£) per patient over time (undiscounted). Figure S4. Early re-interventions (in-hospital or 30 day), unadjusted odds ratio.
About this article
Cite this article
Ciani, O., Epstein, D., Rothery, C. et al. Decision uncertainty and value of further research: a case-study in fenestrated endovascular aneurysm repair for complex abdominal aortic aneurysms. Cost Eff Resour Alloc 16, 15 (2018). https://doi.org/10.1186/s12962-018-0098-7
- Coverage with evidence development
- Medical devices
- Value of information