Methodological Recommendations for Content Validation of Community Resilience Indicators

Abstract

Community resilience indicators increasingly have been used in disaster resilience planning and decision making. However, the methods used to validate that such indicators measure what they claim to measure continue to be lacking. Part of the issue is that validation is a multidimensional concept, and there is limited guidance in the literature on which dimensions are most relevant to community resilience indicators. Consequently, most studies focus on tests that require little original data collection or simple statistical methods, such as whether indicators covary with disaster impacts, or on reliability—a related but different concept akin to measuring consistency. Guidance from fields that heavily utilize indicators, such as medicine and education, suggests that although these tests are necessary, they are not sufficient to properly validate indicators. Specifically, statistical validation methods should be paired with content validation methods, which establish that the selected indicators have appropriate theoretical and empirical underpinnings. Furthermore, content validity should draw from both the scientific literature and rigorous expert judgment protocols, because each type of evidence generation provides a check on the other. In a comprehensive review of the community resilience literature, this study shows that out of 43 relevant community resilience indicator studies, one-third included no expert judgment at all, and only 2 studies followed a rigorous expert judgment protocol. Thus, a vast majority of community resilience indicator frameworks rely heavily on evidence generated from the literature without the balance of structured expert judgment. Given the relative lack of adoption of existing methods, this study presents a review of content validation methods from other relevant fields and provides recommendations based on the specific needs of developing community resilience indicators.