A Dual-Stage Framework for Automated Review Labeling: Integrating Keyword Detection and Large Language Models for Subfeature Analysis

Abstract

Abstract. Incorporating user needs into design strategies is a promising approach for successful product design. To achieve this, numerous studies extract design implications from user-generated data through supervised and unsupervised learning techniques. While supervised learning methods generally deliver superior performance, they require extensive data labeling, which is time-consuming and labor-intensive. This study presents a domain-specific framework for automating the labeling of product review data, aimed at supporting fine-grained analysis of customer feedback—particularly at the subfeature level. The proposed framework consists of two pseudo-labeling mechanisms, keyword detection and large language model (LLM) application. The first stage extracts keywords for the target topic and then labels datasets by checking if the data contains these keywords. The second stage employs an LLM and labels the remainder of the first stage based on their context. This article presents two applications of LLMs tailored to the characteristics of the target data. (i) Prompting LLM: This approach appends a task-specific template to the input text (reviews) and predicts the masked token representing the label. (ii) Fine-tuned LLM: Leveraging domain knowledge, this method involves fine-tuning the LLM to classify the input data (reviews) with improved accuracy and contextual relevance. The framework is evaluated through real-world case studies in two product categories: smartphones and blood pressure monitors. Results show that the proposed method achieves F1 scores ranging from 83% to 97%, outperforming a baseline model, which yields F1 scores between 53% and 89%.