Literature review and data extraction
To identify all relevant articles and available materials and to provide a rigorous scientific base for a comprehensive list of value propositions offered by VAMs, an SLR was conducted, supplemented by a targeted literature review (TLR). The SLR search syntax was designed to collect value propositions based on current examples with a scope to allow the framework to capture value delivered by potential future innovations as well. The search query was built up from a combination of four sets of keywords. The first two sets comprised of synonyms of value-added and medicine to identify compound words for VAMs that are composed according to this scheme. The third set was created to include those complex synonyms where the search terms could not be divided into phrases collected by sets #1 and #2, as described above. Therefore, set #3 collected terms analogous to the expression value-added medicine. As the aim of the research was to identify all potential value propositions, the fourth set contained synonym keywords on benefits and value. Since existing frameworks on the public domain were expected to report not only a list of potentially applicable value domains but also definitions or guides to measurement, set #4 also collected keywords on value frameworks and multi-criteria decision analysis (MCDA). The four sets were combined with Boolean operators (AND/OR) as follows: (((#1 AND #2) OR #3) AND #4) (Additional file 1: Table S2).
The search query was implemented in the Medline database through Pubmed on 21 June 2019. No restrictions on the publication date or article type were applied; however, only English-language articles were considered eligible during the screening process. Hits identified by the search syntax were further processed with the EndNote X6 software. Consideration of eligibility for the title and abstract screening, followed by full-text screening, was based on predefined explicit inclusion and exclusion criteria. Both screening steps were performed by two researchers independently and disagreements between reviewers regarding inclusion and exclusion were resolved by a principal researcher not involved in the primary screening processes.
The following exclusion criteria were used during the title and abstract screening: (1.1) no English abstract available, (1.2) English abstract of non-English full-text paper, (1.3) animal, in vitro study, in silico study or other preclinical studies; review article only mentioning repurposing but unlikely that even its full-text would contain any potential value propositions for VAMs, (1.4) abstract explicitly not related to the research topic (i.e., value assessment of VAMs), (1.5) value proposition from low-income countries without an established health care system.
All articles not excluded during the title and abstract screening phase were considered for full-text screening. The exclusion criteria for the full-text screening were the following: (2.1) no English abstract available, (2.2) non-English full text, (2.3) animal or in vitro, in silico or other preclinical studies, (2.4) full text not related to the research question, (2.5) value proposition from a low-income country without an established health care system, (2.6) full text with no value propositions reported, (2.7) duplicate publication, (2.8) full text not accessible. Publications not excluded during the full-text screening were eligible for data extraction. A snowball search was also performed on extracted articles to identify further relevant studies among their references.
Parallel to the SLR, a TLR was performed to identify additional documents from grey literature sources, including conference materials and reports recommended by member organizations of Medicines for Europe (MFE), an umbrella organization representing European pharmaceutical companies manufacturing generic, biosimilar and value-added medicines. Materials considered eligible for data extraction originated from (1) the SLR, (2) the snowball method and (3) grey literature materials identified by the TLR.
A predefined data extraction spreadsheet was developed in Microsoft Excel to standardize data collection among researchers. Data extracted by a researcher was double-checked by another one and disagreements were resolved by a principal researcher. The following data categories were extracted from the included studies:
-
1.
General study data including author, publication year, title, study type, study objectives, study conclusions (if relevant), country of origin;
-
2.
Active compound-related data including the name of active compound(s), the applied repurposing model (repositioning, reformulation or combination) and disease area;
-
3.
Value proposition-related data including the name, description, definition, information on the measurement of potential value propositions and example products (if mentioned).
A value proposition was considered to be a word, term or complex expression describing any health, monetary, social, etc. benefit(s) delivered by a VAM. Data extractors employed two distinct ways to capture all potential value attributes (even ones with seemingly minor relevance or without any existing reference medicine): a standard and an alternative method. During the standard method, reviewers extracted value propositions literally, with the precise wording, as it was reported by the publication. On the other hand, the alternative method was used to create value propositions by the data extractors inspired by the original text of the article. The method of extraction was required to be indicated next to all value propositions in the data extraction spreadsheet by the reviewers.
Framework development and validation
Transforming value propositions into value domains
The data extraction was followed by the development of potential value domains for an evaluation framework by grouping all the extracted value propositions. Therefore, the SLR and TLR data extraction spreadsheets were reviewed row by row, duplications and synonyms of terms were removed, and individual value propositions were grouped under draft umbrella terms for value domains. This process was also performed by two researchers.
Internal development and validation of draft domains
Subsequently, the initial list of value domains, along with the related value propositions, were reviewed by a group of experts (with experience on developing and adapting MCDAs or value frameworks and with a background in HTA, academics or patient advocacy) in multiple iterative rounds. The main goal of the development/internal validation was to minimize overlaps between the proposed domains and to provide an early feasibility assessment of using the draft domains in various decision-making contexts [22, 23]. Draft value domains were further merged under domain clusters, each cluster containing 2–3 domains. Following this internal validation, the Value Added Medicines sector group of Medicines for Europe had the opportunity to comment on the draft framework.
External validation of draft domains
To increase the validity of the draft evaluation framework, nine health policy experts and decision-makers with a thorough understanding of different decision-making contexts from selected Western European (United Kingdom, Germany), Southern European (Spain, Greece), Northern European (Denmark), and Central & Eastern European (Czech Republic, Slovakia, Slovenia, Lithuania) countries were invited to an external validation process. The selection criteria of experts were familiarity with the evaluation and reimbursement process of pharmaceuticals in different health systems. Efforts were made to achieve a balanced distribution of participants regarding their country of residence and stakeholder perspective. Therefore, experts represented countries with different economic status (i.e., high vs. low income), geographical location, HTA capacities and systems (i.e., primary reliance on relative effectiveness or cost-utility analyses), also different stakeholder groups (i.e., healthcare payers, HTA professionals, health policy experts) and sectors (public payer and governmental institute, private payer and academic). As a consequence of the COVID-19 pandemic, the originally planned face-to-face validation meeting was replaced by two consecutive half-day virtual meetings, where participants were encouraged to share their insights and provide recommendations for further improvement. During the first virtual meeting, researchers presented the background, methodology and development of the draft evaluation framework in detail and facilitated a short, moderated discussion for prompt questions. Between the two workshops, participants were asked to fill in a standardized feedback form on (1) the structure of the proposed evaluation framework, (2) the name, definition, full list of value propositions (as a word cloud) allocated to each individual value domain, (3) any issues regarding the evidence generation for VAMs and (4) applicability of the proposed value framework in different decision-making contexts.
After processing feedback from participants, researchers amended the draft evaluation framework for the second virtual meeting and prepared a set of illustrative test cases for further validation. This virtual meeting aimed to reflect on, discuss and create consensus regarding feedback on the draft framework through discussion and anonymous voting. Both virtual meetings took place in June 2020, with two weeks in-between. One representative of MFE participated in the meetings as a silent observer.