Feasibility Study to Identify Brain Activity and Eye-Tracking Features for Assessing Hazard Recognition Using Consumer-Grade Wearables in an Immersive Virtual Environment

Abstract

Hazard recognition is vital to achieving effective safety management. Unmanaged or unrecognized hazards on construction sites can lead to unexpected accidents. Recent research has identified cognitive failures among workers as being a principal factor associated with poor hazard recognition levels. Therefore, understanding cognitive correlates of when individuals recognize hazards versus when they fail to recognize hazards will be useful in combating poor hazard recognition. Such efforts are now possible with recent advances in electroencephalograph (EEG) and eye-tracking technologies. This paper presents a feasibility study that combines EEG and eye tracking in an immersive virtual environment (IVE) to predict when safety hazards will be successfully recognized during hazard recognition efforts using machine learning techniques. Workers wear a virtual reality (VR) head-mounted device (HMD) that is equipped with an eye-tracking sensor. Together with an EEG sensor, brain activities and eye movements are recorded as the workers navigate a simulated virtual construction site and recognize safety hazards. Through an experiment and a feature extraction and selection process, 13 best features out of 306 features from EEG and eye tracking were selected to train a machine learning model. The results show that EEG and eye tracking together can be leveraged to predict when individuals will recognize safety hazards. The developed IVE can be potentially used to first identify hazard types that are correlated with higher arousal and valence. Also, the developed IVE can be potentially used to evaluate the correlation among arousal, valence, and hazard recognition.