A Mobile Robot Framework for Learning to Detect New Objects With Large Language Models

Abstract

Abstract. Truly autonomous mobile robots must detect both known and unknown objects. This article proposes a fast, real-time open set object detector (OSOD) that enables mobile robots to identify both known and unknown objects. By leveraging a small YOLO model with pseudolabels provided by a multimodal large language model (LLM), we develop an effective open set object detector for edge devices. We replace the traditional human-in-the-loop for interpreting novel objects with a multimodal LLM, which automatically provides semantic information (name and properties) from images, automating the information learning process. Once a mobile robot acquires the semantics of an unknown object from an LLM, a vision-language model classifies repeated instances of the object, reducing the number of slow LLM queries. Meanwhile, an object's properties provided by an LLM allow a mobile robot to operate more effectively around the new object. To facilitate incremental learning, images and labels of novel objects are stored as they are encountered. Once a sufficient number of instances of a novel object are compiled, the original object detector is retrained to include the new object in the set of known objects. Demonstrated with real mobile robots in an academic office building, the incremental learning method showcases how mobile robots can learn to detect novel objects without human intervention. Source code is available at the URL provided in Note 2.