As shown in Fig. 1, our overall approach consists of five main steps in which we:
-
(1)
Operationalize WDHB strategic priorities into concrete health preference criteria;
-
(2)
Identify the dominant health conditions in the Waikato region;
-
(3)
Survey the general public in the Waikato region to estimate personal value weights for the health preference criteria;
-
(4)
Create an impact matrix of the dominant health conditions mapped to the health preference criteria, based on a systematic review of reports from the NZ MoH, the World Health Organization (WHO), the NZ Treasury, and other published studies from NZ, Australia, the United Kingdom, Canada, and the United States;
-
(5)
Compute a priority ranked list of health conditions in order of relative importance using a weighted, additive formula.
(1) Operationalize WDHB strategic priorities into concrete health preference criteria
In collaboration with WDHB executive staff, we identified five health preference criteria (Level I) and two sub-criteria (Level II) that encapsulated the values of WDHB’s strategic vision (Fig. 2) [16]. The health preference criteria are defined as follows:
-
1.
Scale of disease What are the morbidity and mortality impacts of the health condition?
-
2.
Household financial effects To what extent does the health condition cause personal financial difficulty by reducing earnings or diminishing personal savings?
-
3.
Cost-effectiveness What is the cost-effectiveness of prevention or treatment methods that are available for the health condition?
-
4.
Health disparities and inequities To what extent do vulnerable groups such as women, children, or certain ethnicities carry a disproportionate burden of the health condition? For the purposes of this application, we defined the burden of inequity based on gender and ethnicity sub-criteria (Level II). Rural communities in the Waikato region were not classified as a separate sub-criteria since extensive health data on this group were not available.
-
5.
Multimorbidity To what extent does the health condition contribute to a higher burden of multimorbidity?
(2) Identify the dominant health conditions in the Waikato region
As reported by the NZ MoH in 2014, over three-quarters of the deaths in NZ were attributable to cardiovascular disease, cancers, cerebrovascular disease, respiratory conditions, and mental/behavioral disorders [17]. Diabetes mellitus, maternal/neonatal complications, motor vehicle accidents, and HIV/AIDS were also identified as priority health conditions for the Waikato region based on the NZ MoH 2012 Mortality and Demographic Data Report and the WDHB Health Needs Assessment 2012 [18, 19]. Using this combination of national-level and district-level reporting on the leading causes of mortality and hospitalizations, we selected 25 priority health conditions for input into the priority setting algorithm. The final list was approved by WDHB executive staff to ensure that the dominant health conditions were not only limited to diseases of high mortality, morbidity, or hospitalization rates.
(3) Survey the general public in the Waikato region to estimate personal value weights for the health preference criteria
Since fair and accountable priority setting practices call for conditions of publicity, relevance, and customizability, we designed and administered a preference elicitation survey such that the relative weights of the health preference criteria defined in Step 1 would be based on personal values expressed by the general public [20].
Survey design
We estimated aggregate personal value weights for each of the five health preference criteria based on an anonymous, ordered-choice preference elicitation survey of the general public in the Waikato region. The survey consisted of three parts:
-
1.
Demographic questions: Respondents were asked to select their gender, age range, education, and ethnicity from a list of options validated by WDHB.
-
2.
Ordered choice questions: Value measurements are ideally estimated for all individual criteria and any corresponding sub-criteria, but to minimize the cognitive burden of our survey while maintaining completeness and a more flexible weighting system for the sub-criteria, we limited our survey to the five main criteria (Level I, Fig. 2) [21, 22]. The subsequent allocation of preference weights to the corresponding sub-criteria is detailed in Step 4. Respondents were presented with five hypothetical scenarios representing each of the five health preference criteria (Level I, Fig. 2), and were asked to select the option from an ordered-choice continuum that best reflected how important they personally believed each criterion to be [23]. For example, to estimate household financial effect preferences, respondents were presented with the following question: “A person who is the main income earner in their family is sick and cannot work. The family has to spend some of its weekly budget or savings to get proper treatment for the sick person. How important is it to you to address health conditions that cost families a lot of money due to lost income or increased medical expenses? Not at all important, A little important, Important, Very important, or Extremely important.”
-
3.
Free-text question: Respondents were asked to provide a free-text answer describing their personal top three health concerns. To assess the validity of our prioritization approach, this information was compared against the relative ranked output list computed by the algorithm.
The survey was written at a Flesch-Kincaid sixth grade reading level (Flesch Reading Ease score = 62.6). The full survey along with a description of the ethnicity and education classification scheme is provided in Additional file 1. The survey was pilot tested during April and May 2017 among 60 patients at Waikato Hospital, a major tertiary care hospital in Hamilton, NZ and was finalized based on resulting recommendations from WDHB staff and patients. The main survey was administered online and on paper to the general public in the Waikato region during June and July 2017.
Survey administration
As of June 2017, 6094 (~ 1.5%) Waikato region residents were enrolled in SmartHealth, a virtual health care service newly rolled out by WDHB. During June and July 2017, all SmartHealth enrollees received email invitations to take the anonymous online survey, built on the secure Qualtrics platform. To maximize the response rate, two different email subjects were A/B tested among a random half of SmartHealth patients; the subject line that generated a higher click rate was used to invite the remaining half. So that the health preference weights were not solely determined by the subset of patients who seek virtual health care, patients in the main outpatient clinic at the Waikato Hospital were invited to take a paper version of the survey from July 13–21, 2017. All completed paper surveys were entered into the secure Qualtrics survey database. Participants of both survey modes were presented with a consent page describing the voluntary and anonymous nature of the survey. Minors (< 18 years of age) were not allowed to participate. The survey met Institutional Review Board exemption criteria per the US Department of Health & Human Services and the Health and Disability Ethics Committee of NZ. Written permission was obtained from WDHB to administer the survey.
Health preference criteria weight calculations
In order to calculate the personal value weights for each of the five health preference criteria, the ordered-choice responses from each survey were ranked on a scale of 1–5 (e.g. Not at all important received a score of 1 and Extremely Important received a score of 5). These numerical values of the survey responses were stored in a preference matrix (A), where P = number of survey respondents and Q = number of health preference criteria (in this case, Q = 5):
$$A \in {\mathbb{R}}^{PXQ} .$$
We explored two different methods of computing aggregate percentage weights for the five health preference criteria:
-
1.
Normalized weights: Preference matrix A was normalized in three steps.
-
a.
Preference values were summed across the entire matrix:
$$s = \mathop \sum \limits_{i = 1}^{P} \left( {\mathop \sum \limits_{j = 1}^{Q} a_{ij} } \right).$$
-
b.
Each preference value was normalized, dividing it by the sum total:
$$A^{n} = \left( {{\raise0.7ex\hbox{$1$} \!\mathord{\left/ {\vphantom {1 s}}\right.\kern-0pt} \!\lower0.7ex\hbox{$s$}}} \right) A .$$
-
c.
Normalized preference values for each criterion were summed to create a vector of the percentage trade-off weights:
$$w\left( {normalized} \right) = \mathop \sum \limits_{i = 1}^{P} a_{i1 }^{n} , \mathop \sum \limits_{i = 1}^{P} a_{i2}^{n} , \ldots \mathop \sum \limits_{i = 1}^{P} a_{iQ}^{n} .$$
-
2.
Rank order centroid (ROC) weights [24]: For this method, we ranked the five criteria in order of importance based on the median preference value of each. The mean preference value was used to break any ties between criteria. A vector of the percentage trade-off weights was then computed using the formula below, setting the number of criteria (n) = 5 and the ranked order of each criterion (j) = 1 through 5.
$$w_{i} \left( {ROC} \right) = \frac{1}{n}\mathop \sum \limits_{j = 1}^{n} \frac{1}{j} \quad i = 1, \ldots , n .$$
To ensure that the computed percentage weights added up to 100%, we used the Hare–Niemeyer procedure for rounding the weights in both methods [25].
Free-text analysis
To calculate the frequencies of the most common health concerns among our survey respondents, we processed free-text responses according to standard text-analysis methods such as excluding common stopwords (e.g. “a”, “the”, “and”, etc.), removing punctuation, and reducing words to their root stem (e.g. “hypertension” or “hypertensive” was reduced to “hypertens”) [26]. From here, we created a term-document matrix, which consisted of one row for every processed free-text word and one column per respondent [27]. Row sums were calculated to yield the overall frequency count for each health concern. If the same word was mentioned more than once by a single respondent, it did not receive multiple counts. To assess the validity of our prioritization approach, health concerns with the highest frequency counts from the term-document matrix were compared against the algorithm output list of health conditions, ranked in order of relative importance.
(4) Create an impact matrix of the dominant health conditions mapped to the health preference criteria
To quantify the health, economic, and social impacts of each health condition, we created an impact matrix in which scores were assigned to a given health condition for each of the five health preference criteria: scale of disease, household financial effect, cost-effectiveness, health equity, and multimorbidity [28, 29]. Impact scores from 1 to 5 were derived from a systematic review of reports from the NZ MoH, the WHO, the NZ Treasury, and other published studies from NZ, Australia, the United Kingdom, Canada, and the United States. The complete impact matrix of all 25 health conditions mapped to the five health preference criteria is provided in Additional file 2.
1. Scale of disease
We used DALYs for all ages from the WHO Global Health Estimates 2015 summary tables to estimate the morbidity and mortality impact of each of the 25 health conditions identified in Step 2 [30]. The scale of disease impact score for each health condition is defined by its proportional contribution to the total DALYs in NZ (Eq. 1):
-
≥ 5.0% of total NZ DALYs was ranked at level 5.
-
≥ 3.0% and < 5.0% of total NZ DALYs was ranked at level 4.
-
≥ 2.0% and < 3.0% of total NZ DALYs was ranked at level 3.
-
≥ 1.0% and < 2.0% of total NZ DALYs was ranked at level 2.
-
< 1.0% of total NZ DALYs was ranked at level 1
$$Scale \;impact\; score = DALY \;Level.$$
(1)
2. Household financial effect
Since over 80% of total health expenditure in NZ is from government sources, the personal household financial effect (HFE) was defined in terms of the extent to which household income is diminished by a particular health condition [31]. In December 2015, the NZ Treasury released a working paper that quantified the percentage impact of eight priority health conditions on employment, income support, and personal monthly income [32]. We used these findings to calculate the decrease in personal annual income for a median income earner in NZ with a particular health condition. For health conditions that were not included in the NZ Treasury report or other NZ studies, we used data on decreased income by health condition from Australia, Canada, and the United Kingdom. These high-income Commonwealth countries share comparable health expenditure patterns with NZ, where the total health spending in 2016 was estimated between 9.2 and 10.3% of the total GDP [33]. Quintiles for diminished personal annual income for the median income earner in NZ were calculated using these data gathered for all 25 health conditions. The HFE impact score for each health condition was then calculated based on the quintile ranking of diminished annual income on a scale of 1–5 (Eq. 2):
-
> 5091 New Zealand Dollars (NZD) was ranked at level 5.
-
> 2789 NZD and ≤ 5091 NZD was ranked at level 4.
-
> 1735 NZD and ≤ 2789 NZD was ranked at level 3.
-
> 750 NZD and ≤ 1735 NZD was ranked at level 2.
-
≤ 750 NZD was ranked at level 1
$$HFE \;impact \;score = Diminished \;Annual\; Income \;Level.$$
(2)
3. Cost-effectiveness
Technically and economically established interventions in NZ were identified for each of the 25 health conditions. The cost-effectiveness (CE) of the primary interventions per health condition was assessed using incremental cost-effectiveness ratios (ICER; NZD per quality-adjusted life year in comparison with no treatment) from the Burden of Disease Epidemiology, Equity and Cost-Effectiveness Programme (BODE3), a rich epidemiological database combined with simulated economic models created by the University of Otago in Wellington, NZ [34]. For interventions that were not available in BODE3, we used ICER or cost/quality-adjusted life year (QALY) data from the NZ Pharmaceutical Management Agency (PHARMAC), the WHO, the National Institute for Health and Care Excellence (NICE) in the United Kingdom, the Assessing Cost Effectiveness (ACE) Prevention Study in Australia, and other published studies. Historical average conversion rates from 2017 were used to convert ICERs from foreign currencies into NZD [35]. The CE impact score for each intervention pertaining to a corresponding health condition was then calculated by ranking the ICER or cost/QALY data on a scale of 1–5, where more cost-effective interventions received higher scores (Eq. 3). Depending on the availability of data, either the cost-effectiveness of prevention or treatment interventions was used for a given health condition.
-
≤ 5000 NZD was ranked at level 5.
-
> 5000 NZD and ≤ 10,000 NZD was ranked at level 4.
-
> 10,000 NZD and ≤ 20,000 NZD was ranked at level 3.
-
> 20,000 NZD and ≤ 25,000 NZD was ranked at level 2.
-
> 25,000 NZD was ranked at level 1
$$CE \;impact\; score = ICER \;or\;\frac{Cost}{QALY}\; Level.$$
(3)
4. Health disparities and inequities
We quantified health equity by estimating the burden of each disease across gender and ethnicity, as measured by mortality or prevalence rate ratios. While women in NZ have better health outcomes than men on average, common mental disorders such as anxiety and depression, maternal complications, and breast/reproductive cancers continue to impact women at significant rates [36]. Similarly, while NZ has exhibited rapid health improvements at the macro level (as measured by declines in age-standardized total DALY rates from 1990 to 2015), serious inequities persist by ethnicity and socioeconomic status (SES) [37]. For example, the mortality rate for Māori in 2012 was almost double the non-Māori rate (649.3 vs. 362.0 deaths per 100,000 respectively) [18]. Data published by the NZ MoH also demonstrated that across all SES levels, Māori were more disadvantaged than non-Māori in outcomes related to education, personal income, employment rates, and living conditions [38]. Since ethnicity and SES are so closely linked in NZ, we focused on the burden of disease for Māori compared to non-Māori for the purposes of this study. Burden of disease ratios (i.e. mortality or prevalence rate ratios) by gender and ethnicity were then ranked from 1 to 5:
-
> 2.00 was ranked at level 5.
-
> 1.51 and ≤ 2.00 was ranked at level 4.
-
> 1.26 and ≤ 1.51 was ranked at level 3.
-
> 1.11 and ≤ 1.26 was ranked at level 2.
-
≤ 1.11 was ranked at level 1.
Reflecting the poorer health outcomes of Māori across the board, the health equity impact score, defined by the Level II sub-criteria in Fig. 2, was calculated by weighting the burden of disease for ethnicity two times higher than the burden of disease for gender (Eq. 4).
$$Equity\; impact \;score \; = \; {\raise0.7ex\hbox{$2$} \!\mathord{\left/ {\vphantom {2 3}}\right.\kern-0pt} \!\lower0.7ex\hbox{$3$}}\; Ethnicity \;Level + {\raise0.7ex\hbox{$1$} \!\mathord{\left/ {\vphantom {1 3}}\right.\kern-0pt} \!\lower0.7ex\hbox{$3$}} \;Gender\; Level.$$
(4)
5. Multimorbidity
The coexistence of two or more chronic diseases in a single patient (i.e. multimorbidity) affects a substantial proportion of the general population and is estimated to impact most individuals above the age of 65 [39]. In response to the rising tide of complex chronic diseases and multimorbidity, especially among the elderly, the NZ MoH established Care Plus, a funding initiative to improve complex chronic care management [40]. In addition, the Multimorbidity Project at the University of Otago created a multimorbidity (M3) index to predict 1-year mortality, mutually adjusted for 61 chronic conditions [41, 42]. To quantify the average multimorbidity burden associated with each health condition, we summed 1-year mortality log hazard ratios (HR) derived from the M3 index, weighted by multimorbidity prevalence estimates from published literature (see Additional file 2 for details). For a few health conditions that were not present in the M3 index, we used 1-year mortality HRs from other published studies. The multimorbidity impact score was then calculated by ranking the multimorbidity HRs on a scale from 1 to 5 (Eq. 5):
-
> 2.00 was ranked at level 5.
-
> 1.51 and ≤ 2.00 was ranked at level 4.
-
> 1.26 and ≤ 1.51 was ranked at level 3.
-
> 1.11 and ≤ 1.26 was ranked at level 2.
-
≤ 1.11 was ranked at level 1
$$Multimorbidity \;impact\; score = Multimorbidity \;Mortality \;HR \;Level .$$
(5)
(5) Compute a priority ranked list of health conditions in order of relative importance using a weighted, additive formula
Using the aggregate health preference values derived from Step 3 to weight the impact matrix from Step 4, a composite algorithm score \((A_{score} )\) was computed for each health condition (Eq. 6). The 25 health conditions were then ranked in order of descending Ascore such that greater prioritization values were ascribed to health conditions of higher importance based on multiple criteria, weighted by public opinion.
$$A_{score} = {{\left( {w_{s} Scale + w_{h} HFE + w_{c} CE + w_{e} Equity + w_{m} Multimorbidity} \right)} \mathord{\left/ {\vphantom {{\left( {w_{s} Scale + w_{h} HFE + w_{c} CE + w_{e} Equity + w_{m} Multimorbidity} \right)} {100}}} \right. \kern-0pt} {100}}.$$
(6)
All analyses were performed using R version 3.2.1 (The R Foundation for Statistical Computing).
