Large Language Model and Knowledge Graph-Based Intelligent Control Framework for Smart Manufacturing

Abstract

Abstract. The rapid development of large language models (LLMs) and robotics is fundamentally reshaping the manufacturing sector. However, there are challenges for applying LLMs in robotic control, particularly in understanding the spatial and logical relationships between different objects. To address the problem, this article proposes a novel industrial robot control scheme called the LLM knowledge graph powered intelligent agent framework (LKPI). The goal is to leverage the external knowledge and logical relationships provided by knowledge graph (KG) to enhance robotic perception and decision-making in complex production environments. In experimental validation, LKPI demonstrated superior logical reasoning and environmental adaptability in a dynamic and noise-prone pecan processing case study. Even in the presence of noise, LKPI utilized environmental background information and relationship information stored in the KG to accomplish transportation and navigation. By integrating real-time recognition and depth measurement from an RGB-D camera, the robot continuously adjusts its path in response to changes in the position of moving targets. By leveraging Internet of Things technologies to continuously update the knowledge graph in real time, LLMs can achieve real-time environmental awareness through the use of retrieval-augmented generation. This research demonstrates the reasoning control capability, dynamic perception capability, and real-time adaptation capability of the LKPI framework. Experimental results show that the LKPI framework achieved a 96.67% success rate and successfully realized dynamic tracking of target objects. These results demonstrate LKPI’s capability for dynamic information acquisition and robotic control in complex industrial environments, thereby improving the overall task completion rate.