Augmented Reality for Maintenance Tasks with ChatGPT for Automated Text-to-Action

Abstract

Advancements in sensor technology, artificial intelligence (AI), and augmented reality (AR) have unlocked opportunities across various domains. AR and large language models like GPT have witnessed substantial progress and increasingly are being employed in diverse fields. One such promising application is in operations and maintenance (O&M). O&M tasks often involve complex procedures and sequences that can be challenging to memorize and execute correctly, particularly for novices or in high-stress situations. By combining the advantages of superimposing virtual objects onto the physical world and generating human-like text using GPT, we can revolutionize O&M operations. This study introduces a system that combines AR, optical character recognition (OCR), and the GPT language model to optimize user performance while offering trustworthy interactions and alleviating workload in O&M tasks. This system provides an interactive virtual environment controlled by the Unity game engine, facilitating a seamless interaction between virtual and physical realities. A case study (N=30) was conducted to illustrate the findings and answer the research questions. The Multidimensional Measurement of Trust (MDMT) was applied to understand the complexity of trust engagement with such a human-like system. The results indicate that users can complete similarly challenging tasks in less time using our proposed AR and AI system. Moreover, the collected data also suggest a reduction in cognitive load when executing the same operations using the AR and AI system. A divergence of trust was observed concerning capability and ethical dimensions.