Transparent Tangible UI for Volumetric Data Viewing & Simulation using Projection Mapping + Pressure Imaging Sensing

Sheng Kai Tang, Yusuke Sekikawa, Ken Perlin, Kent Larson, Hiroshi Ishii
Tangible Media Group, MIT Media Lab


inSide is a novel TUI for physical interaction with volumetric CAD data. Much 3D volumetric CAD data involves simulation. For example, surgeons use 3D models of organs to form surgical operation plans, mechanical engineers use 3D CAD or architecture models with stress simulation to design structures. Many current approaches to graphical user interfaces (GUI) are inefficient for such purposes, particularly when interacting with the inside of 3D structures. Many ideas have been proposed for combining TUI with wearable AR techniques to provide more intuitive interactions. But most such systems require carefully calibration and are difficult apply in many practical situations. Our approach uses pre-distorted projection mapping, based on head position, object location and 3D CAD data to form perspectively corrected textures mapped directly onto a 3D object model, to create the illusion that one is viewing the interior of the physical object. Users can observe internal structure from arbitrarily view positions. Furthermore we demonstrate a novel interactive simulation method that incorporates a pressure imaging sensor pad to allow a user to use their own hand to directly input simulated parameters, such as pressure or heat, onto a tangible object model.