Current Projects:

Taehyun Rhee / 3D Modeling and Animation

Real-Time Skinning on the GPU        [more info here]
WPSD (Weighted Pose Space Deformation) is an example based skinning method for articulated body animation. The per-vertex computation required in WPSD can be parallelized in a SIMD (Single Instruction Multiple Data) manner and implemented on a GPU. While such vertex-parallel computation is often done on the GPU vertex processors, further parallelism can potentially be obtained by using the fragment processors. We develop a parallel deformation method using the GPU fragment processors. Joint weights for each vertex are automatically calculated from sample poses, thereby reducing manual effort and enhancing the quality of WPSD as well as SSD (Skeletal Subspace Deformation). We show sufficient speed-up of SSD, PSD (Pose Space Deformation) and WPSD to make them suitable for real-time applications.

Human Hand Modeling from Surface Anatomy        [more info here]
The human hand is an important interface with complex shape and movement. In virtual reality and gaming applications the use of an individualized rather than generic hand representation can increase the sense of immersion and in some cases may lead to more effortless and accurate interaction with the virtual world. We present a method for constructing a person-specific model from a single canonically posed palm image of the hand without human guidance. Tensor voting is employed to extract the principal creases on the palmar surface. Joint locations are estimated using extracted features and analysis of surface anatomy. The skin geometry of a generic 3D hand model is deformed using radial basis functions guided by correspondences to the extracted surface anatomy and hand contours. The result is a 3D model of an individual's hand, with similar joint locations, contours, and skin texture.

Jinhui Hu / Augmented Virtual Environments

Texture Painting from Video        [more info here]
With the fast development of technologies in modeling and remote sensing, it becomes increasingly more feasible to model a large-scale environment. However, the acquisition of static textures for such a large-scale environment is still a challenge task, usually demanding lots of tedious and time-consuming manual work, let alone dynamical textures. In this research, we present a system called "texture painting from video", which not only can acquire textures automatically from videos in a convenient way, but also can update the texture information in real time to capture the most up-to-date image information of the environment. Using our technology, we can model, capture and render a large-scale adaptive realistic environment.

Zhenyao Mo / Non-photorealistic Rendering, Human Computer Interaction

Caricature and Portraiture: a Model-based Method        [more info here]
The human face offers unique challenges and opportunities for NPR techniques. While the face is a subject of particular importance to human observers, it also has a canonical structure, and face features even have an approximate importance ordering (e.g., with the eyes and mouth being more important than the nose and ears). Important facial structure is clearly reflected in portraits, even those drawn by amateur artists. However, general purpose NPR algorithms may not respect this structure and ordering, because they have no way of knowing that a shadow edge is less important than edges delineating the mouth or eyes. A model-based approach is developed to produce stylized portraiture. Further, by learning a face space from 500 human face examples, we can automatically locate distinctive facial features and transform a face photograph into a caricature by exaggerating those features.

Gesture-driven User Interface Design        [more info here]
While gestures have been recognized as natural and powerful human computer interaction mechanisms for many years, it is still a challenging problem to build a gesture-driven user interface. The difficulties lie not only in recognizing gestures (the vision problem), but also in describing and implementing interfaces to meet specific requirements of various applications (the design problem). The purpose of this project is to develop a framework which will facilitate to build gesture-driven interfaces efficiently and painlessly.

Ismail Oner Sebe / Augmented Virtual Environments

Real-Time Tracking from Video         [more info here]
Robust and accurate tracking of multiple objects is a key challenge in video surveillance. Tracking algorithms generally suffer from either one or more of the following problems, excluding detection errors. First, objects can be incorrectly interpreted as one of the other objects in the scene. Second, interactions between objects, such as occlusions, may cause tracking errors. Third, globally-optimum tracking is hard to achieve since the combinatorial assignment problem is NP-Complete. We present a modified Multiple-Hypothesis Tracking algorithm, MHT, for globally optimum tracking of moving objects. The system defines five states for tracked objects: appear, disappear, track, split, and merge, and these states cover all the interactions of object pairs. After the detection of objects in the current frame, a resemblance matrix is computed for every object pair. We convert the two-dimensional resemblance matrix into a three-dimensional state-likelihood structure and use a MHT technique to solve the state-assignment problem in 3D. This prevents incorrect assignments due to local minima in the assignment process. Moreover, the method models occlusion cases with the split and merge states. Finally, this method approximates a globally optimum state assignment in polynomial time complexity.

Previous Projects:

Zhigang Deng / 3D Modeling and Animation, Human-Computer Interaction

Data-Driven Facial Animation Synthesis        [more info here]
The goal of this data-driven facial animation synthesis project is to synthesize realistic 3D talking faces, including expressive speech and facial gestures. Also, our system should provide intuitive high-level controls for users. The approach we used is to adopt a machine learning strategy that has successfully been applied in many other areas and "learn" models for facial motion capture data.

Audio-Visual Speech Processing        [more info here]
The interaction between human beings and computers will be more natural if computers are able to perceive and respond to human non-verbal communication such as emotions. Although several approaches have been proposed to recognize human emotions based on facial expressions or speech, relatively limited work has been done to fuse these two, and other, modalities to improve the accuracy and robustness of the emotion recognition system. By working with speech processing researchers, we are investigating joint audio-visual speech information to build novel multi-modal human computer interaction models.

J.P. Lewis / 3D Modeling and Animation, Human Computer Interaction

Blend Shape Animation        [more info here]
Blendshapes (linear shape interpolation models) are perhaps the most commonly employed technique in facial animation practice. A major problem in creating blendshape animation is that of blendshape interference: the adjustment of a single blendshape ``slider'' may degrade the effects obtained with previous slider movements, since the blendshapes have overlapping, non-orthogonal effects. In this project, we try to solve this problem in order to make blend-shape animation more controllable and effective.

Visual IDs        [more info here]
Although the desktop metaphor was intended to engage our considerable visual and spatial skills, casual computer users often have no idea where they put their data. We argue that the problem is that data lack scenery (distinctive appearnace), and propose VisualIDs, a form of automatically generated scenery applied to individual data items such as files. Our experiments show significantly improved speeds in a simulated file finding task after only a few minutes of use, and with no training.

Reyes Enciso / 3D Modeling and Visualization
3D Modeling and Visualization

3D modeling of Faces: The goal is to create photo-realistic 3D meshes of human faces including geometry and texture maps. Our method is based on a pair of calibrated stereo cameras and volume morphing techniques. We take different pictures of the person, and create 3D meshes. A generic 3D model containing animation parameters is then fitted to take the shape of the new person, using volume morphing. Some point correspondences are introduced manually.
3D Visualization: This project aims to develop generic and data independent 3D visualization techniques including 2D image warping for 3D model visualization multi-texturing of 3D models and volume and surface rendering of all kinds of data including medical and ground radar data (Preliminary Results).

[more info here]

Douglas Fidaleo / Facial Expression and Analysis
Facial Analysis for Emotion Recognition and Avatar Control

This project deals with the creation, control, and animation of 3D representations of human faces. The primary application of this research is 3D teleconferencing. Current state-of-the-art 2D teleconferencing applications all suffer from the inability to maintain eye contact between the conference participants. Due to constraints in network bandwidth, the participants are forced to look at small images of people's faces on a flat screen. This problem is amplified when there are multiple (more than 2) participants. . . . We are developing technology that will enable the use of 3D representations of people's faces that allow eye contact preservation and assembling of the avatars into seamless virtual environments.

[more info here]

Tae-Yong Kim / Hair Modeling and Rendering
Modeling, Rendering, and Animating Human Hair

Hair is an indispensible ingredient in realizing virtual human charater. In IMSC, we have developed a set of techniques for human hair modeling, rendering, and animation. For human hair modeling, we developed two techniques focusing on different aspects of hair modeling. The thin shell volume (TSV) method is suited for flat straight hairstyle where hair-hair interaction such as static charge and combing is important. Based on the TSV concept, we extend the idea of separating abstract geometry and detail. The multi-resolution cluser (MRC) model generalizes to wider class of hairstyles, such as curly hairs, short spiky hairs as well as long hairs.

[more info here]

Bolan Jiang / Tracking for Augmented Reality
Incremental Extendible Tracking for Augmented Reality

One of the key requirements for an augmented reality system is a tracking system that determines the user's viewpoint accurately. Many vision-based tracking systems exhibit limited tracking range due to their dependence on pre-calibrated features. We develop an incremental extendible tracking method that integrates pre-calibrated landmarks and unknown natural features for AR tracking. Extendible and robust tracking is achieved by dynamically calibrating prior unknown 3D features. the camera pose is estimated by incrementally integrating partial information provided by each measurement.

[more info here]

Thomas Pintaric / Virtual Reality
Immersive Panoramic Video

The acquisition and presentation of high-resolution panoramic video presents a number of technical difficulties. We demonstrate a system that acquires high-resolution (>3k x 480) panoramic images. These images are recorded at 30Hz frame rates and played back for later viewing. During playback users wears a head-mounted display (HMD) and a head-tracking device that allows them to turn their heads freely to observe the desired portions of the panoramic scene. User impressions initially indicate that the experience produces a strong sense of immersion and this new form of media offers new options for creating immersive simulations.

[more info here]

Jun-Yong Noh / Facial Animation and Cloning
Expression Cloning

Instead of creating new facial animations from scratch for each new model created, we take advantage of existing animation data in the form of vertex motion vectors. Our method allows animations created by any tool or methods to be easily retargeted to new models. We call this process expression cloning (EC), and it provides a new alternative for creating facial animations for character models. EC makes it meaningful to compile a high-quality facial animation library since this data can be reused for new models. EC transfers vertex motion vectors from a source face model to a target model having different geometric proportions and mesh structure (vertex number and connectivity). With the aid of an automated heuristic correspondence search, EC typically requires a user to select fewer than ten points in the model. Cloned expression animations preserve the relative motions, dynamics, and character of the original facial animations.

[more info here]

Jong Weon Lee / Vision-based tracking
Motion-Based Pose Tracking Method

Most vision-based pose (position and orientation) tracking methods require a priori knowledge about the environment and correspondences between 3D environment features and 2D images. This environmental information is difficult to acquire accurately for large working volumes and it may not available at all. As a result, most vision-based pose tracking methods are designed for small working spaces. We are currently developing a vision-based method that requires no priori knowledge of the 3D environments. The pose of a camera is tracked through two 5 degree-of-freedom (DOF) motion estimations, which requires only 2D-to-2D correspondences. Therefore, the presented method can easily be applied to varied working space sizes.

[more info here]

Clint Chua / Object deformation
Adaptive Free Form Modeling and Animation

Static objects populate most virtual environments. While this may be sufficient for some applications, the addition of deformable objects to a scene will not only increase realism but also broaden the application's utility. The limiting factors for including deformable objects in an environment are object specific deformation software, lack of standard deformation primitives and complex mesh representations of deformable objects. The goal of our research is to propose a standard geometric deformation primitive and simplify complex deformable object representation.

[more info here]