未分類

inSide

inSide

Transparent Tangible UI for Volumetric Data Viewing & Simulation using Projection Mapping + Pressure Imaging Sensing

Sheng Kai Tang, Yusuke Sekikawa, Ken Perlin, Kent Larson, Hiroshi Ishii
Tangible Media Group, MIT Media Lab

Introduction

inSide is a novel TUI for physical interaction with volumetric CAD data. Much 3D volumetric CAD data involves simulation. For example, surgeons use 3D models of organs to form surgical operation plans, mechanical engineers use 3D CAD or architecture models with stress simulation to design structures. Many current approaches to graphical user interfaces (GUI) are inefficient for such purposes, particularly when interacting with the inside of 3D structures. Many ideas have been proposed for combining TUI with wearable AR techniques to provide more intuitive interactions. But most such systems require carefully calibration and are difficult apply in many practical situations. Our approach uses pre-distorted projection mapping, based on head position, object location and 3D CAD data to form perspectively corrected textures mapped directly onto a 3D object model, to create the illusion that one is viewing the interior of the physical object. Users can observe internal structure from arbitrarily view positions. Furthermore we demonstrate a novel interactive simulation method that incorporates a pressure imaging sensor pad to allow a user to use their own hand to directly input simulated parameters, such as pressure or heat, onto a tangible object model.

 

Tangible Cityscape

Tangible Cityscape

Adopting 2.5D Shape Display and Digital Shadow to Assist Collaborative Urban Massing

Sheng Kai Tang, Yusuke Sekikawa, Daniel Leithinger, Sean Follmer, Hiroshi Ishii
Tangible Media Group, MIT Media Lab

Introduction

Urban massing is an intermediary stage which realizes urban policies by determining appropriate sizes, functions and relations of buildings. It also helps planners to balance the dependencies and conflicts among various complicated urban systems. Better collaboration of domain experts during the urban massing process helps to avoid conflicts among diverse urban systems. However, current design tools respectively support different parts of the process, due to a huge gap between physical (atoms) and digital (bits) information representations. It forces planners to frequently switch between manual and computational tools andresults in ineffective and inefficient in process.

In this research, we propose Tangible CityScape consisting of a 2.5D actuated shape display, a digital shadow display and a gestalt view display to assist the urban massing process. With Tangible CityScape, planners can convert abstract information into tangible forms and cross-refer tangible information of multiple city systems. They can also manipulate both digital and tangible representations and leave annotations with both collocated and remote partners. We are going to realize the proposed goals by a five-step process and conduct a qualitative user study by think aloud protocol analysis to assure a seamless urban massing process.

Pinball

Pinball

Merging the Physical and Digital Properties to Achieve Inter-Material Interaction

Sheng Kai Tang, Yusuke Sekikawa, Daniel Leithinger, Sean Follmer, Hiroshi Ishii
Tangible Media Group, MIT Media Lab

Introduction

In this project, we create a system that can record and reconstruct the movement of a physical ball and represent it on a 2.5D shape display. Through this, we are going to discover the potential of inter-material interaction by merging the physical and digital properties of objects.

Multi-flash Camera

or HW3 HCI by Sheng Kai Tang (Tony)

0. Target

1. Try One

 

2. Try Two

3. Try Three

Lightfields

for HW2 Lightfields by Sheng Kai Tang (Tony)

0. Shift and Add in Processing

The idea of lightfields taught in the class is to create a 1D or 2D array of pinhole cameras to capture images . With slightly shifting of images based on camera positions, adding up values of pixels, and averaging these pixels values, we can create a general sense of lightfields. The code below is written in Processing according to the concept of 1D array:


int imageWidth;// image width
int imageHeight;// image height
PImage show;//final image
PImage[] images;//source images
String[] fileNames;//source filenames
int imageAmount;//number of source images
color c;//pixel color
float r, g, b;//r,g,b of a pixel color
int l = 0;//keycode counter


void setup(){
imageWidth = 420;//set image width
imageHeight = 240;//set image height
imageAmount = 15;//set image amount
size(imageWidth, imageHeight);//set canvas size
images = new PImage[imageAmount];//instantiate source images
fileNames = new String[imageAmount];//instantiate image filenames


//load image filenames
for(int i = 0; i < fileNames.length; i++){
fileNames[i] = "IMG_24" + (i + 10) + ".JPG";
}
//load source images
for(int i = 0; i < images.length; i++ ){
images[i] = loadImage(fileNames[i]);
}
//instantiate the final image
show = createImage(width, height, RGB);
}


void draw(){
//shift and add
show.loadPixels();
for(int i = 0; i < width; i++){
for(int j = 0; j < height; j++){
for(int k = 0; k < imageAmount; k++){
r = r + red(images[k].get(i + l*k, j));
g = g + green(images[k].get(i + l*k, j));
b = b + blue(images[k].get(i + l*k, j));
}
c = color(r/imageAmount, g/imageAmount, b/imageAmount);
show.set(i, j, c);
r = 0;
g = 0;
b = 0;
}
}
show.updatePixels();
image(show, 0, 0);
}


void keyPressed() {
if (key == CODED) {
if (keyCode == UP) {
l++;
}
else if (keyCode == DOWN) {
l--;
}
}
}

1. Code Test 

With the code developed above, we conduct test by importing 16 photos provided.

1. Focus at the window

2. Focus at the dots board

3. Focus at the Almo

4. Focus at the Pumpkin

 

2. Refocus and See through

 

3. Interactive Application

 Shift and Add (Mac version, Windows 32, Windows 64), original photos and source codes are included.

 

 

 

 

 

 

 

 

進度

2/19  課程介紹 –> 成為一個創造者(Maker)

2/28  228放假

3/06  創意的方法論 –>從形式創意、功能創意到使用者經驗創意

3/13  掌握傳統媒材1 –> 回收物創意

3/20  掌握傳統媒材2 –> 泡綿談造型創意

3/27  掌握傳統媒材3 –> 瓦愣紙板創造新的結構

4/03  春假|展場規劃與設計

4/10  展覽準備

4/17  期中考

4/24  第一階段展覽

5/01

5/08

5/15

5/22

5/29

6/05

6/12

6/19

6/26

進度

2/19  實體互動簡介:從設計科技到設計運算

2/28  228放假

3/06  基礎電學:歐姆定律、串聯與並聯之限流電阻計算

3/13  物件導向程式語言(一):variable、function

3/20  物件導向程式語言(二):if….elsefor

3/27  數位輸入與輸出裝置及其電路:數位開關、下拉與上升電阻、LED控制

4/03  春假

4/10  設計檢討(一):設計概念、零件表、電路圖|類比輸入與輸出裝置及其電路:類比開關、switch…case

4/17  設計檢討(二):硬體、程式與外觀檢討|狀態機進階

4/24  設計檢討(三):硬體、程式與外觀檢討|序列埠溝通

5/01  設計檢討(四):原型檢討|訊號辨識

5/08  期中設計案評圖與展示

5/15

5/22

5/29

6/05

6/12

6/19

6/26

Bio

I am currently a Staff Interaction Designer and Engineer in the Customer eXperience Lab of Samsung Research America. I trained my out-of-the-box thinking in MIT, CMU and Harvard. I was hired as an independent contributor by ASUS and as a consultant by ITRI and III in Taiwan. My works and participated projects were reported on ABC, published at CHI, HCI International, TEI, and CSCW, and demonstrated in CES, CeBIT, and Computex.

I dedicate in creating unique and novel interactions and experiences for Graphical UI, Tangible UI, AR and VR. I can ideate, design, prototype, conduct user research and lead directions. I do sketching, hand crafting, 3D modeling, coding, and electronics to make prototypes inspiring people and iterating ideas.

In short, I am a maker making innovation.

Adaptive Typing

Adaptive Typing

Adaptive Typing

A Study Looking for User’s Mental Model to Achieve Intuitive Typing

Sheng Kai Tang
User Experience Design, ASUS

Introduction

A laptop with dual screens is a trend for the coming future. It means typing on an LCD screen displaying dynamic content instead of a fixed physical keyboard is an emergent design possibility. However, the current approach only showing a static keyboard graphically on the screen wastes the potential of a sensible and displayable touch screen. In this research, we are going to design an “Adaptive Typing” mechanism that actively detects user’s hand palm resting on the screen and automatically predicts key locations. This mechanism also provides minimum visual clues to assist novice users. In order to realize this idea, we design an experimental desk to collect and observe users’ mental models and hand ergonomics while typing. As shown in the picture, the upper camera collects data about keys and fingers; the lower camera collects data of hand palms through a transparent part with which a conventional keyboard and a touch pad embedded in. We believe that by overlapping image data of upper and lower cameras, the hidden relationships between hand palm and typing can be further revealed.


Tangible Slider

Tangible Slider

A Capacitive Touch Slider Enabling Intuitive Sidebar Manipulation and Control

Sheng Kai Tang
User Experience Design, ASUS

Introduction

Sidebar is a kind of widely used component for most window-based applications. A sidebar is defined as a container where application shortcuts are located. Currently, a sidebar is called out from hidden by pointing the cursor at a hot-area or a hot-edge. After calling out a sidebar, users can further point at a shortcut icon and click to trigger. In this project, such a conventional interaction model is replaced by a “Tangible Touch Slider” to increase the usability. The Tangible Touch Slider is a horizontal sensible area with lighting feedbacks on the C part of a laptop computer. By touching on this sensible area, the related sidebar on the screen will be called out; by sliding on it, user can switch among shortcut icons; by releasing from the sensible area, the selected shortcut will be triggered. After a preliminary test based on GOMS, this new interaction model can save up to 1.5 seconds especially when users are doing typing oriented works.


Sneak Peeking Bars

Sneak Peeking Bars

Detecting Eye Position to Control the Presence of Side Bars

Sheng Kai Tang
User Experience Design, ASUS

Introduction

The control sidebar is essential for window-based application. Most of them have auto-hide mechanism to spare more window space for users. Users can bring out a sidebar at will by moving the mouse cursor to the edge where the bar is hidden. In this project, we propose a new way of bringing out sidebars; we called it “sneak peeking sidebars”. First of all, we put all sneak peeking sidebars outside the frame of the window, which means users can’t see them while looking at the window straightly. However, when users move their head a bit to peek at edges of a window, and they will easily see sidebars hidden outside the window. The sneak peeking mechanism is actually based on the idea of “dynamic perspective” widely adopted in the interactive computer graphic to create real 3D scene. The dynamic perspective technique is to actively detect the eye position of the user based on which the perspective scene is generated dynamically.


Different predispositions of design and scientific thinking

Different predispositions of design and scientific thinking

Sheng Kai Tang
Computational Design Program, CMU

Introduction

The important role of design in HCI is providing design thinking to support scientific thinking especially for generating novel ideas in the concept forming stage. However, why couldn’t scientific thinking generate novel ideas by itself and need the assistance of design thinking in HCI research is an interesting issue. In this paper, we conduct two experiments for this problem. We assume representation is a reliable evidence for human thinking. By studying representations, we can understand process of human thinking. We propose a classification table as our coding schemes for analyzing representations. This classification is based on observation of Leonardo da Vinci’s manuscripts and correlation between the former and Buchana’s ideas. Finally, we assure the need of design thinking when dealing with a problem in terms of generating novel ideas. We also prove that those characteristics of design thinking that scientific thinking lacks, such as reframing problems and inventing oriented problem solving, can benefit searching for novelty We finally propose the idea that the type of a problem is not rigid but a predisposition resulted from design or scientific thinking.

Documentation

Tang, S.K.: 2008, Different predisposition of design and scientific thinking, Final Report for “Design Perspective in HCI” offered by Prof. Jodi Forlizzi.


TeleEGGs

TeleEGGs

TeleEGGs

A Development of Multi Modal Communication Devices

Sheng Kai Tang
COmputational DEsign Lab, CMU

Introduction

In our daily life, there are many subtle actions which can deliver our emotions in very unique ways especially those between acquaintances. From a glance, one can perceive not only the other’s existence but demands and feelings. Via a smile and a blink, one can deliver good intentions and concerns to the other. Even though there is no verbal or physical communication further needed, the communications still happen. These are what we defined as indirect emotional communications. However, when two people are not in the same space, these kinds of communications seem not to happen. TeleEGGs is a pair of devices we developed to solve this problem. In the other word, this device can assist long distance indirect emotional communication.

Publication

Tang, S.K.: 2006, TeleEGGs: A Development of Multi Modal Communivation Devices, Final Document for “Physical Computing” offered by Prof. Pamela Jennings.