Priority setting in health: development and application of a multi-criteria algorithm for the population of New Zealand’s Waikato region

Priority setting strategies in the health sector must account for the fact that resources are limited and that tradeoffs are required to decide which health conditions deserve the most attention and which interventions should be used to address them [1, 2]. Even in developed countries, competing investment decisions are influenced by electoral implications, monetary pressures, and divergent ethical perspectives that make significant resource demands on health care systems. The situation is no different in New Zealand (NZ), where the publicly funded health system accounts for over one-fifth of government spending, decentralized across 20 District Health Boards [3, 4]. The NZ Treasury has recently called for allocating health resources more explicitly and sustainably within District Health Boards to deal with the increasing burden of chronic diseases on the aging population and persistent disparities in health outcomes for Māori and Pacific peoples [4].

In an earlier effort to promote efficient and equitable allocation of resources, the National Health Committee advised the NZ Ministry of Health (MoH) from 1993 to 2016 in deciding which technologies and services should be included in the publicly funded health package [5]. This committee approached priority setting in a systematic manner, leveraging criteria of technical and allocative efficiency (as measured by cost-effectiveness) and health equity [6–8]. Instead of depending solely on the judgment of health care professionals, their approach took into consideration the preferences of the general public. However, the prioritization and implementation of the National Health Committee recommendations at the District Health Board level was often slow and cumbersome, with limited customizability for local needs and preferences [9, 10]. In addition, these efforts around health prioritization were focused on health technology assessments, which recognized the importance of evaluating various health technologies/interventions, but did not explicitly prioritize the health conditions themselves to allow for more strategic policymaking [10–12]. In this paper, we present the development and application of an explicit, rational, and comprehensive framework for prioritizing health conditions in alignment with local strategic priorities and public preferences within the Waikato region of NZ.

The Waikato region on the upper North Island of NZ is home to over 400,000 people (almost 10% of the total NZ population). The age profile in this region is similar to the national distribution—approximately 34% of the population is above the age of 50. However, the Waikato region has a larger proportion of Māori than the national average (22.9% vs. 15.8%) and a larger proportion of people in the highest economic deprivation group (25% vs. 20%) [13]. Over 10% of the residents of the Waikato region live in rural areas with low influence from urban employment and without easy access to tertiary care [14]. The Waikato District Health Board (WDHB) is responsible for the health care of all Waikato residents and provides funding for all health services including primary care, pharmaceuticals, community health services, hospital services, research and development, and public health education [15]. In July 2016, the WDHB released its strategy document, “Healthy People Excellent Care” that describes the current public health challenges posed by increasing morbidity due to chronic long-term health conditions, inequities for underserved populations such as Māori who face significantly higher morbidity and mortality rates compared to non-Māori, and increasing demands on the health system from an aging population [16].

To assist WDHB in achieving explicit, transparent, and rational health priority setting in this context, we developed and applied an algorithm to rank the region’s dominant health conditions in order of relative importance, based on a customized set of health criteria. These criteria included measures of disease burden such as disability-adjusted life years (DALYs), economic measures such as cost-effectiveness, and social objectives related to health equity and poverty reduction. Subjective preference weights for each criterion were estimated based on the preferences of the general public in the Waikato region. The goal of this paper is to show how this customizable multi-criteria algorithm can explicitly further a vision of policymaking bodies for rational and transparent priority setting driven by local strategies and preferences.

The 25 dominant health conditions selected for input into the algorithm accounted for approximately 70% of the 31,168 total deaths across NZ in 2014 and almost 60% of the 1,059,254 total DALYs in 2015. Of these dominant health conditions, 24 were non-communicable diseases or injuries, with HIV/AIDS being the only communicable disease.

Ordered-choice preference profiles

Over 70% of respondents ranked Scale and HFE criteria as Extremely important, compared to 61.2% for Multimorbidity, 48.3% for Equity, and 30.7% for CE (Fig. 3). Māori respondents had higher average preferences for all criteria compared to non-Māori respondents, especially for Equity and CE (Fig. 4). The average Equity preference was 9.2% higher for Māori (4.6 vs. 4.2, two-sample t-test: t = 7.5324, df = 403.33, p-value < 0.01) and the average CE preference was 6.2% higher (3.8 vs. 3.6, two-sample t-test: t = 2.5534, df = 333.85, p-value = 0.01). Since both these differences for Māori respondents were more than 5%, we computed separate aggregate percentage weights by ethnicity.

Respondents 50 years of age and older also tended to have higher preference profiles compared to younger respondents (Fig. 5). The average Multimorbidity preference was 4.2% higher for older respondents (4.6 vs. 4.4, two-sample t-test: t = − 4.1571, df = 604.17, p-value < 0.01) and the average Scale preference was 2.3% higher (4.7 vs. 4.6, two-sample t-test: t = − 2.8375, df = 607.81, p-value < 0.01). However, since both these differences for older respondents were less than 5%, we did not compute separate percentage weights by age.

Comparison of the normalized ranked list with the free-text health conditions

The top 100 health conditions of concern listed by the survey respondents are shown in Fig. 6. Except for “arthritis”, “obesity”, and “pain”, the first ten diseases of concern listed by the survey respondents were included in the list of WDHB priority health conditions. In close alignment with our algorithm, heart disease, cancers, and diabetes, were the top three broad health concerns of the survey respondents. Respondents did not appreciably differentiate between types of cancers, so we could not reliably compare the algorithm rankings and free-text rankings by type of cancer. On the other hand, mental and cognitive disorders such as suicide and dementia were ranked within the top ten health conditions by our algorithm, above asthma and hypertension which were explicitly identified as higher priority health conditions according to the free-text responses (Fig. 7). Premature birth and chronic obstructive pulmonary disease (COPD) also received higher rankings from our algorithm, but were not listed with appreciable frequency in the free-text responses.

The development and application of our multi-criteria algorithm closely follows the best practices recommended by the International Society For Pharmacoeconomics and Outcomes Research (ISPOR) Task Force on multiple criteria decision analysis [28]. This strategic priority setting algorithm allowed us to estimate what health conditions should be prioritized and why [12]. To address these priority health conditions, a follow-up tactical approach, such as health technology assessment, can be used to inform which interventions should be appropriately funded [43]. In order to generate our ranked list of strategically prioritized health conditions, we elicited the underlying health preference weights and pressing health concerns of the general public while building on detailed, preexisting datasets pertaining to the morbidity and mortality rates, financial impacts, and burdens of inequity of various health conditions in NZ. According to the Corruption Perceptions Index 2016, NZ ranks second out of 176 countries in the world [44]. This exceptionally high level of data transparency made it possible for us to demonstrate a direct, modeled approach to determining which health conditions are most important to address, while quantifying a link between five, near-equal underlying priorities that drive the health concerns of the general public in the Waikato region.

In this discussion, we provide three recommendations for applying our algorithm to strategic health priority setting. First, to achieve accountable, relevant, and transparent health care priority setting, the public perspective can be used to weigh public health evidence across multiple criteria [45]. We measured the preferences of the general public in the Waikato region to estimate health preference criteria weights and to validate the algorithm output against the free-text responses. Our findings suggest that there is broad correspondence and alignment between the algorithm output and the explicit health concerns of the public. Certain health conditions are of concern to the public not just because of the morbidity and mortality they cause, but also because of household financial effects, multimorbidity risks, and burdens of inequity. However, a majority of the free-text responses were in broader, more general bins than our algorithm output list, so fine-grained comparability with the algorithm ranking was not possible. Future iterations of this approach might incorporate more clinical specificity in asking respondents to specify the types of cancers or chronic diseases that concern them the most. Misalignments between the public free-text health concerns and the algorithm output list can help policymakers identify specific health conditions that are rare, underdiagnosed, or lack public awareness. For example, our algorithm ranked suicide as the fourth most important health outcome of concern, but since young Māori males—who were underrepresented in our survey—are at a particularly high risk of suicide, our findings suggest that awareness of suicide and its risk factors in NZ might be lacking [46]. Similarly, COPD was ranked higher than asthma and hypertensive disorders even though the latter two conditions were listed as higher priority according to our survey respondents. This misalignment supports the fact that while asthma in NZ receives considerable attention and publicity, COPD is often underdiagnosed and less well understood [47]. In consultation with WDHB, our analysis primarily focused on higher burden diseases in the Waikato region. To ensure that orphan health conditions also receive appropriate attention, future iterations of this study might be expanded to include rarer, high impact diseases.

Our second finding is that estimating health preferences of the general public is only as strong as the preference elicitation and subsequent weighting methodologies that are used. We recommend that the ROC weighting method be used if only the ordinal ranking between criteria is available or if the raw multi-criteria preferences are substantially different from each other. If raw multi-criteria preferences are near-equal (as expressed by our survey population), normalized weights more consistently capture this information compared to ROC weights, which are forced into a much wider range across criteria. In our survey population, Māori respondents ranked concerns of health equity higher than non-Māori respondents. However, since Māori respondents also tended to rank the other health preference criteria higher than non-Māori respondents did, the resulting normalized weights for the five health preference criteria were not appreciably different between both groups, with health equity carrying a near-equal percentage weighting along with scale of disease, household financial effects, and multimorbidity. As a result, lung cancer, which has a mortality rate three times higher among Māori compared to non-Māori and has arguably not received enough public health emphasis due to factors of stigma and blame, was ranked third in the algorithm output list, close to breast cancer and bowel cancer—diseases that have traditionally received more attention in NZ [48, 49]. It is unclear from our study whether the comparable percentage weightings of all five health preference criteria underscored the relatively equal importance they represented for our survey respondents or whether the survey questions were sensitive enough to measure the respondents’ true preferences. How a survey question is framed can skew public perception, especially if there is no opportunity for broader clarification and discourse with survey respondents regarding accurate and precise definitions of the criteria under question. For example, while it might not be surprising that cost-effectiveness received the lowest ranking of the five health preference criteria, public opinion would likely be more favorable to incorporating considerations of cost-effectiveness into priority setting if the burden of cost-ineffective treatments was presented as the direct responsibility of private taxpayers vs. the indirect responsibility of public hospitals.

Therefore, our third recommendation is that our algorithm output along with the survey of underlying preferences and explicit health concerns of the general public are meant to inform, not replace, representative and deliberative priority setting processes. Our survey response rate was approximately 18%, in line with the expectation that internet-based surveys without incentives or multimodal follow-up typically yield response rates less than 25% [50, 51]. The survey population mirrored the ethnic distribution in the broader Waikato region, but respondents were more likely to be older females with higher education levels compared to the general Waikato population. Our survey and algorithm results highlight the importance of soliciting diverse preferences using multiple modalities that are not limited by literacy, especially when underserved populations are among the primary stakeholders. Fair and representative policymaking will ideally be based on concepts of deliberative democracy, in which representative groups of stakeholders interpret the algorithm output in the light of multiple competing special interests, through transparent, in-person, and iterative discourse. These deliberative processes will ideally consider how the preferences of relevant groups in the general public vary from each other, what levels of diversity might be involved in the deliberative discourse, and how these decisions will influence policy outcomes [52].

This study can be improved by expanding the range of non-redundant health preference criteria to more accurately capture the broader impact of ill-health. For example, foregone income alone due to disease burden is likely an underestimate of household financial effects. We considered lost income due to absenteeism, but did not explicitly include estimates of presenteeism or loss of productivity from working while ill [53]. Similarly, over 7% of Waikato region residents reported looking after a household member who was ill or had a disability, but estimates for caretaking costs were not included in this study [54]. While we considered the 1-year mortality impact of average multimorbidity estimates, we did not account for the overlap between the scale of disease criterion and multimorbidity, nor did we consider the downstream social impacts of certain health conditions (e.g. stigma associated with mental health conditions, HIV/AIDS, etc.). Finally, survey respondents in our study were not fully representative of the broader Waikato population. Young men with lower education levels were especially underrepresented in our study. Pacific peoples, like the Māori, face significant health inequities but were not weighted separately in our dataset because of the small number of Pacific respondents. Future applications of this priority setting approach might include a survey with stratified sampling and multimodal follow-up strategies to improve response rates and to minimize sampling bias. Study design can also include contextualized community-based participatory research methodologies such as deliberative polling, in which the full range of criteria scales and algorithm inputs/outputs might be iterated among representative stakeholders [55–57].

As mentioned earlier, the main strengths of this study come from leveraging public health preferences to weight a detailed dataset relevant to the NZ context. The preference elicitation methodology we used is relatively easy to implement and analyze. Operationalizing strategic priorities and estimating criteria weights from the general public in this manner can be replicated by other public health institutions with the goal of understanding which health conditions deserve the most attention and why.

We developed and applied a strategic priority setting approach in the Waikato region of NZ, in which preexisting disease indicators were combined in an algorithm with public preferences to quantify the relative importance of dominant health conditions. Policymakers can use this two-pronged approach not only to determine the immediate health needs of their constituency, but even to predict the importance of new health conditions as they emerge. Ideally in the context of democratic deliberation, future applications of this model can be extended to developed and developing countries alike, where scarce health care resources must be more efficiently and equitably allocated.