| dc.description.abstract | Abstract. Environmental simulations are critical in early-stage design, offering insights into factors like solar radiation, airflow, and climate responsiveness. However, interpreting these simulation outputs is time consuming and requires expert reasoning, which can slow down design workflows. Vision-language models (VLMs) offer potential for automated interpretation support, yet their reliability on architectural simulation outputs remains unexplored, particularly across different computational scales. This study introduces VizArQA, a benchmark for evaluating VLM performance on architectural environmental simulation interpretation. VizArQA comprises 121 questions across five standard simulation types (radiation roses, wind roses, illuminance diagrams, solar insolation, and computational fluid dynamics plots), with both binary and multiple-choice formats that assess visual comprehension and design-relevant interpretation capabilities. We evaluate six models representing both open-source and closed-source approaches across varying computational scales: GPT-5, GPT-5-Mini, and GPT-5-Nano, alongside Qwen3-235B, Qwen3-8B, and Qwen3-2B. Results show performance variations across model sizes and architectures, with larger models achieving substantially higher accuracy: Qwen3-235B achieved 84.30%, followed by GPT-5 and Qwen3-8B at 80.99%, while smaller models achieved 57.02–74.38%. Performance varied across simulation types, with wind rose analysis showing high accuracy (73.91–100%) and radiation rose analysis proving challenging (40.00–76.00%). These findings establish baseline performance metrics for VLM-assisted environmental analysis and highlight the potential of AI-assisted workflows, potentially enabling a path toward integrating VLMs into design processes to enhance efficiency, insight generation, and accessibility for nonexperts. | |
