Table 1 Framework for evaluating the impact of HTA systems

Mechanism 1: The effective conduct of HTA studies

Q1

Is the HTA agency doing/commissioning HTA studies that are fit for purpose?

Sub-questions:

 Are HTA studies quality-controlled, for example through submitting them for peer-reviewed publication in international journals, or by having them quality assessed and scored by experts (e.g. in-country academics; colleagues from sister agencyes in other countries)? Does the agency have published technical methods guidance, in order to facilitate quality control by providing a normative standard, and improve methods transparency? Are studies assessed against an international checklist, such as the CHEERS checklist? Do staff have appropriate formal qualifications, and are the duration and resourcing of studies in line with international benchmarks?

[15,16,17, 29, 33, 35]

Examples of effects:

 New method development; availability of good quality and relevant HTA evidence; transparent use of data in research; identification of research gaps; investment in and allocation of resources to HTA research

Mechanism 2: The effective use of HTA in agenda-setting and policy formulation processes

Q2

Are HTA studies used in agenda-setting/policy formulation?

Sub-questions:

 Do statute or administrative regulations clearly define who will use the outputs of the HTA agency and how (i.e. are there regulations in place for how HTA studies are to be implemented?), and is it possible to map the process by which HTA studies will impact clinical practice? The usage may be chained, in the sense that, for example, one body might use HTA studies to develop clinical guidelines, and then a further quality assurance body may monitor compliance with clinical guidelines. Is it possible to track study recommendations through the various steps on this process? Are some HTA studies cited outside of this process, for example in policy documents from the Ministry of Health or strategic plans of insurers?

[9,10,11, 17, 19, 35]

Examples of effects:

 HTA agency representation in policy decision-making; increased rigour in policy decision-making due to the use of HTA evidence; transparent policy decision-making due to use of HTA evidence; changes to the way policy decision-makers consider the healthcare system in decision-making

Mechanism 3: Effective engagement and external communications

Mechanism 4: Good institutional reputation and fit within healthcare system

Q3

Are HTA studies valued by stakeholders in the healthcare system?

Sub-questions:

 Are the various stakeholders in the healthcare system (for example, clinicians, including opinion leaders, manufacturers, managers of insurers and providers, patient groups, academics) aware of studies and do they perceive them to be credible? Where appropriate, do these stakeholders change their behaviour (for example clinical practice, regulatory actions, sales or purchasing) as a result of HTA studies? Is HTA perceived to have influenced conversations between different actors to focus more on evidence of effectiveness and cost-effectiveness? What are the formal mechanisms for engaging these stakeholders in the work of the HTA agency?

[8,9,10,11, 15, 17, 19, 35]

Examples of effects:

 Use of HTA principles by other stakeholders or by other institutions; acknowledgement of HTA agency recommendations in policy-making processes; increased transparency due to autonomy of HTA agency

Q4

Do HTA studies inform public debate?

Sub-questions:

 Are HTA studies picked up in non-specialist outlets, such as the public media? Do such studies stimulate public debate (e.g. editorials in mainstream newspapers, interviews or panel discussions in news analysis programmes, documentaries, public events)? Are the public aware of the role of the HTA agency and do they perceive it positively? What mechanisms does the HTA agency have for engaging members of the public in its deliberations?

[8, 33, 35]

Examples of effects:

 Increased public understanding of the challenges facing the healthcare system; increased public understanding of the value of HTA and its use in healthcare decision-making

Mechanism 5: Effective use of HTA as a tool for the negotiation of health technology prices

Q5

Do HTA studies support insurers or government in negotiation with manufacturers?

Sub-questions:

 Are HTA studies used in negotiations with manufacturers, and do the negotiators report price reductions, or other benefits such as risk-sharing agreements as a result? Do manufacturers litigate HTA agency recommendations (indicating that these recommendations are perceived to drive commercial consequences)?

[10, 12, 17, 33]

Examples of effects:

 Number of health technologies funded by public sector after successful price negotiation; healthcare payers can afford to pay for and provide access to health technologies; sustainability of manufacturer income; sustained innovation processes; increased transparency in price negotiation

Mechanism 6: Effective implementation of policy change

Q6

Do HTA studies result in changes in practice, and did such changes lead to measurable improvements in cost, health impacts, and wider social/economic impacts?

Sub-questions:

 Is it possible to track activity and prescribing rates (in a consistent way across the country and between subnational entities)? Do recommendations from HTA studies result in observable changes in practice? Is it possible to attribute the change in practice to the HTA agency recommendation (for example a similar change is not observed in other countries where no recommendation has been promulgated)? Is it possible to test key modelling assumptions about costs and health benefits of the technology once the recommendation has been implemented, and verify that the costs and health benefits are in line with what was modelled in the HTA study?

(10,11,12,13, 16, 19, 28, 41)

Examples of effects:

 Effective uptake of new health technology in practice through effective communication about policy change ensuring acceptability and receptiveness to policy change; improved provision of cost-effective care (including rejection or disinvestment of cost-ineffective health technologies); improved population health; wider social/economic impacts