Representation Learning of Problem-Solving Behavior in Virtual Reality Manufacturing Environment Using Long Short-Term Memory Networks

Abstract

Abstract. The number of manufacturing jobs in the US has been consistently increasing, driven by a rapidly evolving industrial landscape and the implementation of a new strategic plan. At the same time, concerns have emerged about the problem-solving abilities of engineering students, who represent the future workforce. This highlights the need for systematic evaluation and deeper insight into how these students approach problem-solving. In this article, we introduce a virtual reality (VR)-based manufacturing environment combined with a data-driven analytical workflow to evaluate engineering students’ problem-solving performance. Within the VR system, students complete assembly tasks to build car toys that meet specific design criteria. During the process, we capture real-time eye-tracking data, reflecting the spatial and temporal dynamics of their visual attention and assembly actions. We extract latent features from this data via a long short-term memory-based supervised representation learning for problem-solving performance evaluation. Our approach outperforms the traditional performance metrics-based evaluation by capturing the nonlinear dynamics of the in situ problem-solving process. Experimental results, including benchmarking against alternative architectures and ablation analyses, show that the learned feature representations yield the clearest distinctions in categorizing students' problem-solving performance among the evaluated methods when integrating full behavioral input. The proposed evaluation framework holds broader potential for improving problem-solving assessments across various manufacturing systems and workforce training programs.