Weighing Votes in Human–Machine Collaboration for Hazard Recognition: Inferring a Hazard-Based Perceptual Threshold and Decision Confidence from Electroencephalogram Wavelets

Abstract

Human–machine collaboration is a promising approach to improve on-site hazard inspection because it can complement the inherent limitations of human cognitive functions. Nevertheless, research on the effective integration of opinions from humans and machines to form optimal group decision-making is lacking. Prior work suggests that a confidence-weighted voting strategy is superior, but self-reported decision confidence is often unreliable. Thus, this study proposes an innovative methodological framework to predict workers’ hazard response choices and decision confidence from brain activities captured by a wearable electroencephalogram (EEG) device. First, we developed a Bayesian inference–based algorithm to ascertain the decision threshold above which a hazard is reported characterized by the power of human brain activity. Furthermore, we describe hazard recognition as a process of probabilistic inference involving a decision uncertainty evaluation. Benchmarking against an optimal Bayesian observer, the optimal criteria to differentiate between low-, medium-, and high-confidence levels were obtained based on numerical simulations. The proposed method was tested empirically with a predesigned experiment in which 77 construction workers participated in a hazard recognition task while their EEG data were simultaneously collected. Cross-validating with behavioral indexes of the signal detection theory, the results confirmed the possibility of EEG measurement to observe workers’ internal representations when discriminating hazards. Parietal α-band EEG power was chosen as a proxy for confidence-level evaluation prior to responses. Theoretically, this framework characterizes workers’ mental model when recognizing hazards. Practically, it enables the prediction of workers’ hazard responses and decision uncertainty, supporting the design of future hazard confirmation mechanisms in the collaborative human–machine systems research field.