Looking Beyond Self-Reported Cognitive Load: Comparing Pupil Diameter Against Self-Reported Cognitive Load in Design Tasks

Abstract

As designers experience greater mental demands from the increased complexity of new design tools and methods, it is important to understand designers' cognitive load when performing design tasks. Several researchers have identified task- and designer-related factors that affect cognitive load, such as time or expected outcome. However, most of these design studies used self-report measures of cognitive load, which have been observed to be inaccurate and, to some extent, incomplete. In contrast, physiological measures (e.g., eye tracking) provide an objective assessment of mental workload. However, little research in engineering design has compared self-reported measures of cognitive load against physiological measures and our aim in this paper is to provide motivation and a starting point for such work. Specifically, we present a rich dataset comprising pupil diameter collected with ten student designers performing an hour-long loosely controlled design task using various design representations (e.g., computer-aided design and sketching). We also collected self-reported cognitive load using the NASA-TLX after the design task was completed. A preliminary analysis revealed that self-reported physical demand correlated with the minimum latent pupil diameter, whereas performance satisfaction correlated with the maximum latent pupil diameter. Furthermore, design representations vary in the range of cognitive load experienced by designers when utilizing them. These findings highlight the importance of statistical moments in the interpretation of physiological indicators such as pupil diameter. These findings also call for the use of a multi-modal approach for measuring cognitive load. Moreover, the accompanying dataset enables future research toward such studies.