Task and Motion Planning With Large Language Models Enhanced by Spatially Temporally Aware Tools for Smart Manufacturing

Abstract

Abstract. In smart factories, robots play a crucial role in executing critical operations such as material loading and unloading. Task and motion planning (TAMP) approaches enable robots to perform these tasks in an efficient and coordinated manner. Compared with traditional TAMP approaches, large language models (LLMs) demonstrate superior comprehension and reasoning capabilities, making them particularly suitable for solving multitype and interleaved manipulation tasks in smart manufacturing environments. However, the performance of LLMs in TAMP heavily relies on prompt design, for which no unified standard currently exists in the robotics domain. Moreover, LLMs are inherently weak in spatial and temporal reasoning in a manufacturing context, which limits their applicability to practical, domain-specific problem solving. To address these limitations, we propose a tool-augmented LLM approach for TAMP in a smart factory. The core idea is that we first design a unified, domain-specific symbolic prompt engineering scheme tailored for robotic applications. Then, we develop a suite of spatially temporally aware tools to enhance the LLM's reasoning capabilities. Building on these components, the tool-augmented LLM can perform iterative task solving, enabling effective scheduling of machining operations. Both simulation and physical experiments are conducted, and results show that the proposed approach significantly improves the LLM's success rate and generalizability. This work contributes to the development of advanced tool-augmented LLMs for robot task and motion planning in the context of smart manufacturing.