Health technology assessment (HTA) is a field of multidisciplinary research that aims to inform policy decisions and clinical practice around the use, introduction and reimbursement of health technologies. It uses specific methods to examine the health and social value, as well as cost implications of and ethical issues related to the use of a health technology in a systemic, transparent, unbiased and robust manner to inform decision-making [1, 2]. The overall goal of HTA research is to promote an ‘equitable, efficient, and high-quality health system’ [2]. HTA research can be conducted both by private (e.g. ICER in the United States) and by public actors, and the evidence HTA provides can inform decision makers about how best to ensure the health system is equitable, efficient, and of high quality.
While HTA research conducted by private actors can be informative to policy makers, several countries prefer to rely primarily on advice from publicly-funded national ‘HTA agencies’ that aim to serve the public interest. Two examples of such agencies are the HTA agencies of England (the National Institute for Health and Care Excellence, or NICE) and Thailand (the Health Intervention and Technology Assessment Program, or HITAP). Although the authority and responsibility given to HTA agencies varies from country to country, their prevalence across widely differing health systems is indicative of the power of HTA to add value across different contexts.
The launch of the UN’s Sustainable Development Goals (SDGs) in 2015 and its target to ‘achieve universal health coverage (UHC) […] for all’ [3] has drawn further attention towards the establishment of national HTA agencies, as they can offer a pathway to achieving and sustaining UHC even in severely resource-constrained environments. In particular, HTA’s rationale of directing resources towards health technologies that are ‘cost effective’—i.e. those that lead to large improvements in population health relative to the cost involved—allows nascent UHC initiatives to rapidly improve population health even under tight budgets.
However, with cost-effectiveness being such a crucial pillar of HTA, questions on the value added from investments into HTA agency capacity are inherently valid. After all, public resources invested in HTA agencies could have been diverted to frontline medical services offering much more ‘tangible’ health outcomes. Addressing the question of ‘value offered’ by HTA agencies is however not straightforward for a number of reasons. One reason for this is that existing HTA agencies around the world are highly heterogeneous, each operating in a different context, within different systems and under different budgets. As a result, the emerging impacts and externalities of HTA agencies may vary from context to context, highlighting the need to understand the question of how such agencies have impact.
To further position our study, we first discuss some key points from the literature to highlight the heterogeneity relating to the question “what is the impact of health technology assessment?”. The key points discussed put into context existing practice in the evaluation of impacts of HTA. We highlight a number of gaps in existing methodologies and approaches to the evaluation of HTA impact.
The heterogeneous nature of evaluating the impact of health technology assessment
There are a large number of studies relating to the question “What is the impact of health technology assessment?”. However, since this question is rather poorly defined, the literature is quite heterogeneous. To review in detail would require more space than we have here and, in any case, others have been here before us [4,5,6,7]. Specifically, a good starting point for accessing this literature is the study by Gerhardus et al. [6] which provides a helpful framework and clear summary of the literature up to about 2006. Another useful resource is Raftery et al. [4] which gives a detailed description of the various methodologies which have been deployed to measure the impact of health research with a view to evaluating the impact of HTA. Rather than summarising these papers immediately, we will highlight their main contributions to further position our study.
In what follows, we highlight some points for discussion around the heterogeneity of this body of literature. Specifically, we highlight four sources of heterogeneity: (1) variations in the purpose of the study; (2) differences in interpretation of “health technology assessment”; (3) differences in interpretation of “impact”; and (4) variability in scope and level of rigour of evaluation studies. We conclude by highlighting what we think we can and cannot learn from this literature to position our study.
Purpose of study
One source of heterogeneity in the literature is the purpose of the study. Some studies are relatively modest in aim, and are essentially descriptive, concerned with providing basic information about a sample of HTA reports and their findings [8,9,10,11,12,13]. Others have a stronger summative and analytical focus on the question of whether the investment in a health technology was worthwhile, in that they use either quantitative or qualitative data to explore the context in which the value is realised in more detail [14,15,16]. Yet other studies have a more formative purpose: how can the HTA system do a better job of delivering impact [17,18,19]? The nature of the evaluation assessment team also varies. Studies by external assessors, such as those commissioned to independent committees or pool of experts to evaluate the impact of HTA are often summative in nature, whereas studies by HTA or health system insiders are often descriptive or formative.
Definition of the term “health technology assessment”
Further heterogeneity stems from ambiguity in the use of the term “health technology assessment”. Some researchers frame the question as being one of the impact of “reports” (or “guidance” or “advice”) from an HTA agency [6, 8,9,10,11,12,13, 16, 18,19,20,21]. Others frame the question as being one of the impact of HTA “research” [4, 14, 18, 22,23,24,25,26]. These are not necessarily the same. “Reports” may be based on an overview of a relatively small body of evidence generated elsewhere, whereas “research” implies a piece of work which is of publishable standard. Research itself may not necessarily lead to a report (for example if its main conclusion is that the status quo should be maintained). The tendency to frame the impact of HTA as being about the impact of some sort of knowledge product (whether report or research) is helpful for tracing impacts (as the knowledge product provides a source to which impact can be tracked) but arguably means that the more diffuse benefits of a visible HTA presence (e.g. encouraging evidence-based practice; legitimising discussions about cost-effectiveness) are relatively neglected.
Definition of impact
Studies also differ in their interpretation of “impact” and conceptualisation of how impact occurs. For example, Gerhardus et al. [6] offer a six-stage model of impact which we paraphrase here:
-
1.
Awareness: the relevant stakeholder must know of the HTA report.
-
2.
Acceptance: the relevant stakeholder must see the HTA report as valid and a legitimate basis for action.
-
3.
Policy process: the policy process should explicitly utilise the HTA report.
-
4.
Policy decision: the policy decision should cite the HTA report.
-
5.
Practice: there should be “clear and measurable” changes in clinical practice in line with policy decision and thus the report.
-
6.
Outcome: health and economic outcomes should be realised on the basis of the changes in practice.
This six-stage model suggests that an ideal evaluation of the impact of HTA would provide evidence at all stages, and thus show that there was a clear chain from HTA study to health and economic outcomes. However, most studies omit some stages of this chain; some leave off the latter stages and some skip stages altogether. For example, studies which we designate as “model-based” studies [12, 16, 23] effectively skip the implementation chain almost entirely and provide estimates of health and financial benefits based on the HTA agency’s own cost-effectiveness studies, whether or not they have led to policy changes. This predominant focus on the endpoint of the chain characterises other types of evaluations, which appraise outcomes after changes in policy and practice occurred, rather than assuming that they will take place. This also applies to post-market field evaluations [20], where health technologies are assessed under real-world circumstances, or to studies that retrospectively analysed the longitudinal correlation between investments in health research and disease burden [24, 25]. Other studies integrate changes in policy and practice within the assessment, using varied approaches. For instance, primary or secondary data have been used to ascertain the extent to which preliminary HTA findings are actually implemented, thus adjusting pre-implementation model-based estimates to account for actual uptake and coverage [27, 28]. In other cases [8, 9, 11, 13, 19], analyses have assessed to what extent actual clinical practice and usage patterns adhered to guidance issued or appropriate use criteria. Others [29] primarily focussed instead on the time lag between HTA appraisal processes, policy decisions and, access for patients to approved medicines.
Scope and rigour of evaluations
Studies also differ in terms of their scope and level of rigour. There is a trade-off between scope and level of rigour: the most rigorous or in-depth analyses are often those which focus exclusively on a single or narrow set of HTA recommendations, and are published in clinical journals for a particular medical sub-speciality [30, 31]. By single or narrow set of HTA recommendations, we mean that studies often have, without clear explanation, focused on the impact of specific recommendations, rather than the impact of the HTA agency as a whole. At the other end of the scale, studies which focus on the impact of a broader set of HTA reports [11, 12, 28, 32, 33] are inevitably somewhat broad-brush.
From our perspective an effective study design would use a mix of methods, with a quantitative component using state-of-the-art-statistical techniques to detect changes in system behaviour [13, 19, 31], and a qualitative component which draws on knowledge from a wide range of system stakeholders [10, 16]. Such a study design would also have a plausible answer to the question of how changes in system behaviour are attributed to the HTA agency (for example by comparing between territories where the agency’s jurisdiction does and does not hold), and would fully account for all residual uncertainties and list all background assumptions [12, 13, 16, 26, 34].
In terms of study scope, it is also worth noting that the most of studies addressing the value of HTA are context-specific, in that they generally focus on a unique HTA agency or national system. Conversely, in a few cases, studies adopt a comparative perspective. These either contrast the performance of multiple HTA bodies operating within a country [29], or generate insights by applying a common evaluative framework across multiple national settings, though usually narrowing down the number of HTA recommendations considered [27, 30].
Summary of key points from the literature
We know, from the studies discussed in the previous section, that HTA studies have been conducted in several countries, and in many cases have influenced clinical practice and that there is also reasonably plausible evidence that—especially in medium-to large-sized countries—the benefits from implementing individual HTA recommendations can exceed the costs of performing the individual HTA by a substantial margin [15, 16, 20, 26,27,28, 35]. Moreover, the literature offers a rich resource of practical examples on how countries can evaluate their own HTA systems. However, we do not know everything. As highlighted above, from the standpoint of methodological perfection even the most rigorous studies have failings; there are significant gaps in terms of the coverage of time and space in the literature; and, as Raftery et al. [18] highlight, the available empirical literature is unlikely to be a random sampling of the entire human experience of HTA and most likely focusses on settings where HTA has been relatively successful.
Study aim
In this study, in the interests of moving forward, we will focus on addressing two main limitations in the knowledge base. Firstly, current studies are typically at the country level and there is limited ability to cross compare between countries due to variations in reporting. However cumulative knowledge building would be greatly advanced if there were at least a few minimally accepted indicators for evaluating the impact of an HTA agency which could also be used for international comparisons. Secondly, available frameworks for the evaluation of HTA reports (such as the Gerhardus et al. [6] framework above) tend to be somewhat “linear” (in the words of Raftery et al. [4]) and focussed on HTA as a ‘knowledge product’ leading inexorably to change in health service practice and thus health and economic outcomes. However, this is not really compatible with what has been observed in several contexts with an HTA agency operating at the centre of a system and interacting with various stakeholders, as presented in the introduction. As a result, in this study we aim to explore the pathways to, or mechanisms which lead to the impact of an HTA agency from an institutional perspective. This perspective allows us to explore rules of behaviour (both formal and informal) that influence the impact of HTA agencies [36]. The aim of doing so is to build an HTA agency impact evaluation framework that complements existing research in this space. Furthermore, such an approach allows us to take an in-depth analysis not only of an HTA agency’s structure but also of its broader institutional surroundings and how these contribute to its value added. We believe that this study will be useful not only for assessments of organisation-level HTA impact, but that it can also help guide the design and developments of HTA systems globally.
