AA

Assessing the Outcome Measures – Methods

Assessment Protocol:

Who was involved?
  • Team of reviewers who assessed each measure consisted of clinicians and scientists who have long established expertise in a wide variety of relevant research areas. For details, please see the Editors and the Acknowledgements page.

What did they do?

  • Data was extracted from papers reporting findings about the psychometric properties and several “pragmatic” factors (acceptability, feasibility, etc.) for each of the various measures. Extraction was done with heavy reliance on work of Fitzpatrick and colleagues (1998) for methods and standards. For an example of the data extraction forms used for the project see Appendix 1.
  • Data was evaluated – our evaluation criteria as well as the standard for quantifying the rating where possible are presented in Table 3.
  • Summaries were generated for each measure.

Table 3 Criteria for Rating Properties of Outcome Measures

 

Standard

Reliability

Validity

Responsiveness

High (+++)

IC

ICC and Kappa Coefficient

Construct or Convergent and Concurrent ROC analysis  – AUC Evidence of change in expected direction using methods such as standardized effect sizes:
Small Moderate Large

>0.80

>0.75

 

>0.60

 

>0.90

 

 

<0.5

 

 

> 0.5 –0.8

 

>0.8

 

Moderate

(++)

0.71 – 0.79

>0.4 –0.74

 

 

>0.31-0.59

 

 

 

> 0.70-0.89

 

 

 

Evidence of moderate/less change than expected; conflicting evidence

 

 

Low

(+)

<0.70

<0.4

 

 

< 0.31

 

 

 

< 0.70

 

1. Weak evidence based solely on p-values (statistical significance)
2. Floor or ceiling effects:

Excellent

Adequate

Poor

 

No floor or ceiling effects

 

 

 

Floor and ceiling effects is <20% of patients who attain either the minimum (floor) or maximum (ceiling) score.

 

>20%

 

 

Not Applicable

(-)

N/A

N/A

N/A

N/A

N/A

N/A

N/A

Countless numbers of books and manuscripts have been written classifying and discussing psychometric principles and standards for the selection or validation of clinical measures. It is not our intent to replicate this process, but instead refer the reader to a couple of key dispositions such as Streiner and Norman’s Health Measurement Scales (2003) and Portney and Watkin’s chapters (4, 5 and 6) on reliability and validity (2000). For an excellent overview that provides insightful tips for selecting measures directly related to rehabilitation, read Finch and colleagues Physical Rehabilitation Outcome Measures (1999).