Radar–Vision Fusion Method for Traffic Event Detection with Roadside Perception

Abstract

Detecting traffic flow and events on highways is conventionally dominated by vision-centric approaches and rule-based methods. These systems focus on identifying both micro (e.g., wrong-way driving, abnormal parking, emergency lane usage) and macro (e.g., congestion, stop-and-go waves) traffic events. The development of technologies of 4D millimeter-wave radar has significantly expanded the application of roadside sensors. The utilization of 4D radar data presents several challenges, including sparsity of detection points, presence of noise, and limited feature extraction capacity of existing roadside perception systems. The end-to-end deep learning method can greatly facilitate traffic event detection by integrating traffic participant detection and event detection processes. Moreover, it enables hardware optimization of edge computing devices and on-device data transmission. Hence, this paper proposes an end-to-end radar–vision fusion method for traffic event detection with roadside perception. First, we construct a highway event data set with radar–vision fusion in a 3D environment through joint simulation using Simulation of Urban Mobility (SUMO) software and Carla. Then, we develop an encoder-decoder network for feature extraction. Finally, we present an end-to-end approach for traffic event detection. We define the average event detection precision (AEDP) as a criterion for traffic event detection. Our method maintains robustness in detecting microscale traffic events and demonstrates a 20% improvement in congestion detection using the rule-based method and a 32% improvement in the detection of stop-and-go waves.