Proto-TAI++: Exploring Perceptually Consistent Creation of Planar Shape Assemblies Through Multimodal Tangible Interactions

Abstract

We explore tangible 3D interactions that allow for geometric and perceptual correspondence between a midair modality and the 3D elements it controls. To demonstrate our approach, we use a concrete application scenario through Proto-TAI++, a multimodal system using pen-based drawing of planar shapes and their subsequent midair assembly via a hand-held planar proxy. The planarity of the proxy is a key element that physically embodies virtual planar shapes during 3D manipulation. We also investigate advanced operations such as patterning and blending using asymmetric bimanual interactions that augment midair motion with multitouch gestures. We describe a three-stage user evaluation with our system wherein our goal is to (a) study the effectiveness of the planar proxy as a tangible 3D modality, (b) contrast key features of our approach with a GUI-based planar shape assembly system, and (c) evaluate user experience and performance in creative tasks using Proto-TAI++.