Does technique matter; a pilot study exploring weighting techniques for a multi-criteria decision support framework

Table 1 The effect of weighting technique on rank order of criteria and weight differences between criteria

Characteristics	Five point rating scale (n = 60)		Five point rating (hierarchical) (n = 15)		Point allocation (hierarchical) (n = 15)		Ranking (n = 15)		Pairwise comparison (hierarchical) (n = 15)		Best worst scaling (n = 15)
Criteria rank order (mean group weight)
1st	D2	0.100	D1	0.087	D1	0.092	D1	0.095	D2	0.118	D2	0.091
2nd	D1	0.089	Q1	0.086	I3	0.089	C2	0.088	D1	0.111	I1	0.090
3rd	I3	0.085	T1	0.083	D2	0.084	I3	0.086	I3	0.110	D1	0.089
4th	I1	0.080	D2	0.082	C2	0.082	Q1	0.083	C2	0.083	I3	0.083
5th	Q2	0.076	Q2	0.077	Q1	0.078	D2	0.080	T1	0.082	T1	0.078
6th	T1	0.075	C2	0.076	I1	0.077	I1	0.078	I1	0.071	E1	0.077
7th	C2	0.072	T2	0.070	E3	0.070	E3	0.077	Q2	0.060	E2	0.074
8th	Q1	0.069	I3	0.070	E2	0.068	Q2	0.069	I2	0.056	I2	0.073
9th	E2	0.066	I1	0.067	Q2	0.065	E2	0.065	Q1	0.056	E3	0.069
10th	I2	0.065	C1	0.063	E1	0.062	E1	0.063	C1	0.054	C2	0.064
11th	E1	0.061	E2	0.062	I2	0.060	C1	0.056	E1	0.054	Q1	0.057
12th	E3	0.057	E1	0.062	T1	0.059	I2	0.055	T2	0.052	T2	0.056
13th	T2	0.057	E3	0.061	T2	0.058	T1	0.053	E2	0.049	Q2	0.054
14th	C1	0.052	I2	0.053	C1	0.056	T2	0.053	E3	0.044	C1	0.046
Weight difference
1st-3rd^*	0.027 (0.04)		0.009 (0.01)		0.030 (0.03)		0.034 (0.03)		0.075 (0.10)		0.017 (0.02)
Most-least^**	0.091 (0.07)		0.064 (0.02)		0.100 (0.05)		0.110 (0.05)		0.156 (0.12)		0.076 (0.02)
Equal weights^***	0.022 (0.01)		0.017 (0.01)		0.027 (0.01)		0.027 (0.01)		0.034 (0.02)		0.018 (0.01)
Rank difference
Group ranks	#		2.8 (2.8)		2.0 (1.8)		2.5 (2.1)		1.6 (1.4)		2.3 (2.1)
Individual ranks	#		7.5 (1.7)		7.6 (2.6)		7.8 (3.5)		8.3 (1.9)		6.2 (2.5)
Correlation
Mean across-respondents	##		0.83		0.83		0.71		0.78		0.97
Mean within-respondents	##		0.53		0.50		0.43		0.45		0.66

^*Within respondent difference between the weight of the first and third ranked criteria (group mean and standard deviation). ^**Within respondent difference between the weight of the first and last ranked criteria (group mean and standard deviation). ^***Within respondent difference between weights elicited with the technique and equal weights for all criteria (mean and standard deviation). # rank order based on five point weighting scale (nH) criteria ordering is the reference ranking. ## weights of five point rating scale (nH) is the reference weight. Criteria abbreviations: D1: Disease severity, D2: Size of population, I1: Improvement of efficacy/effectiveness, I2: Improvement of safety/tolerability, I3: Improvement of patient-reported outcomes, Q1: Completeness and consistency of reporting evidence, Q2: Relevance and validity of evidence, C1: Clinical guidelines, C2: Comparative interventions limitations, T1: Public health interest, T2: Type of medical service, E1: Budget impact on health plan, E2: Cost-effectiveness of intervention, E3: Impact on other spending.

ISSN: 1478-7547