| description abstract | Abstract. Engineering rulebooks and technical standards contain multimodal information like dense text, tables, and illustrations that are challenging for retrieval augmented generation (RAG) systems. Building upon the DesignQA framework (Doris, A. C., Grandi, D., Tomich, R., Alam, M. F., Ataei, M., Cheong, H., and Ahmed, F., 2025, “Designqa: A Multimodal Benchmark for Evaluating Large Language Models’ Understanding of Engineering Documentation,” J. Comput. Inf. Sci. Eng., 25(2), p. 021009. 10.1115/1.4067333), which relied on full-text ingestion and text-based retrieval, this work establishes a multimodal ColPali-enhanced retrieval and reasoning framework (MCERF), a system that couples a multimodal retriever with large language model reasoning for accurate and efficient question answering from engineering documents. The system employs ColPali, which retrieves both textual and visual information, and multiple retrieval and reasoning strategies: (i) hybrid lookup mode for explicit rule mentions, (ii) vision to text fusion for figure- and table-guided queries, (iii) high-reasoning LLM mode for complex multi modal questions, and (iv) SelfConsistency decision to stabilize resp onses. The modular framework design provides a reusable template for future multimodal systems regardless of the underlying model architecture. Furthermore, this work establishes and compares two routing approaches: a single-case routing approach and an agent-based system, both of which dynamically allocate queries to optimal pipelines. Evaluation on the DesignQA benchmark illustrates that this system improves average accuracy across all tasks with a relative gain of +32.6% from baseline RAG best results, which is a significant improvement in multimodal and reasoning-intensive tasks without complete rulebook ingestion. This shows how vision-language retrieval, modular reasoning, and adaptive routing enable scalable document comprehension in engineering use cases. MCERF is publicly available online. | |
