Assessing Quality of User Submitted Need Statements From Large Scale Needfinding: Effects of Expertise and Group Size

Abstract

Collecting data on user needs often results in a surfeit of candidate need statements. Additional analysis is necessary to prioritize a small subset for further consideration. Previous analytic methods have been used for small quantities (often fewer than 75 statements). This study presents a simplified quality metric and online interface appropriate to initially screen and prioritize lists exceeding 500 statements for a single topic or product area. Over 20,000 ratings for 1697 need statements across three common product areas were collected in 6 days. A series of hypotheses were tested: (1) Increasing the quantity of participants submitting needs increases the number of highquality needs as judged by users; (2) increasing the quantity of needs contributed per person increases the number of highquality needs as judged by users; and (3) increasing levels of selfrated user expertise will not significantly increase the number of highquality needs per person. The results provided important quantitative evidence of fundamental relationships between the quantity and quality of need statements. Higher quantities of total needs submitted correlated to higher quantities of highquality need statements both due to increasing group size and due to increasing counts per person using novel contentrich methods to help users articulate needs. Based on a multivariate analysis, a user's topicspecific expertise (selfrated) and experience level (selfrated hours per week) were not significantly associated with increasing quantities of highquality needs.