Nils Krahnstoever
Dept. of Computer Science and Engineering
The Pennsylvania State University
220 Pond Lab
University Park, PA 16802

I'm a Ph.D. student at the Department of Computer Science and Engineering at the Pennsylvania State University. My research deals with computer vision based understanding of human motion, especially in the context of multimodal human computer interfaces. My current research focuses on robust visual tracking and problems related to track initialization. More specifically, I'm developing a methodology for learning articulated (especially human) models from video without any prior knowledge about size, location or topology for subsequent model-based tracking.

Other areas of interest are gesture recognition, multimodal fusion of gestures with spoken commands, large scale computer vision system design, gait recognition and -analysis, motion segmentation, visual surveillance and medical image processing.

Real-Time Framework for Natural Multimodal Interaction with Large Screen Displays
Development of a large scale software framework that provides the basis to several multimodal interactive systems developed in our lab and at Advanced Interface Technologies, Inc. The system combines a large number of computer vision components (face detection, user localization, hand detection, head and hand tracking, gesture recognition) and multimedia components (speech, recognition, speech synthesis, 3D rendering engine) into a flexible, multithreaded real-time framework. The entire system runs smoothly at 30 fps on a dual Pentium III, 500 MHz.

Details: Here!
Related papers: ,.
Videos: AVI, Codec: Cinepack (4.9 MB) | MPEG (5.7 MB) | Quicktime, Codec: Cinepack (4.9 MB).

Multimodal Human Computer Interaction for Crisis Management Systems
One example is XISM, a multimodal crisis management system that incorporates unconstrained hand gestures and speech in a real-time interactive interface. This paper provides insights into the design aspects of the XISM system. In particular, it addresses the issues of extraction and fusion of gesture and speech modalities to allow more natural interactive behavior. This project was the foundation for later research systems developed in our lab (e.g., DAVE_G) and commercial systems developed at Advanced Interface Technologies, Inc.

Details: Here!
Related papers: ,,.
Videos: AVI, Codec: Divx (15.2MB) | MPEG-2 (15.8MB).
See also: iMap, Advanced Interfaces Inc., eOz.tv.

Automatic Acquisition and Initialization of Articulated Models from Video
Model based approaches to human motion tracking and analysis are actively explored by many researchers. However, model acquisition, initialization and adaptation are still relatively under-investigated problems. In this project we develop methods for inferring kinematic structure from visual data. We use Bayesian motion segmentation to extract and initialize kinematic models from visual data from the ground up. Image sequences are decomposed into layers (rigid links) that undergo parametric (affine) motion. The relative motion of the links is used to obtain joint information. The resulting components are assembled into articulated models which are then used for visual tracking without any manual initialization or adaptation.

Related papers: ,.
Videos: Here!

Model Based Tracking
Development of a robust model-based tracking framework that utilizes highly specific appearance models acquired online from video data. Our approach is able to track many different type of objects in challenging (e.g., cluttered, non-stationary) environments.

Related papers: ,,,.

Layer Based Approaches for Motion Segmentation
Investigation of layer based approaches for motion segmentation. There are a set of elegant methods for handling motion estimation and segmentation in a unified Bayesian framework. We are working on extending existing methods to handle occlusion and obtain estimates based on multiple frames.

Related papers: ,.

Tracking and Analysis of Human Motion
Development of a framework for detecting, tracking and analyzing non-rigid motion based on learned motion patterns. The framework features an appearance based approach to represent the spatial information and Hidden Markov Models (HMM) to encode the temporal dynamics of the time varying visual patterns. The low level spatial feature extraction is fused with the temporal analysis, providing a unified spatio-temporal approach to common detection, tracking and classification problems. The method allows us to perform a set of important tasks such as activity recognition, gait-analysis and keyframe extraction.

Related papers: .

Curvature Guided Surface Triangulation of 3D Image Data
Development of a new approach for obtaining surface triangulations of 3D image data. The algorithm is fast, size adaptive and curvature guided. The method is robust and flexible and can triangulate large data sets as well as a single voxels correctly. The work was conducted with Cristian Lorenz, PHILIPS Research, Medical Image Processing Group.

Related papers: .

Statistical Shape Modeling
Development of point based three-dimensional statistical shape models. Given a set of medical objects, a statistical shape model can be obtained by Principal Component Analysis. This technique requires that a set of complex shaped objects is represented as a set of vectors that uniquely determine the shapes of the objects, and at the same time are suitable for a statistical analysis. The correspondence between the vector components and the respective shape features has to be identical in order for all shape parameter vectors to be considered. We developed a novel approach to the correspondence problem for arbitrary three-dimensional objects, which involves developing a template shape and fitting this template to all objects to be analyzed. The method is successfully applied to obtain a statistical shape model for the lumbar vertebrae. The work was conducted during a stay at PHILIPS Research with Cristian Lorenz.

Related papers: , , .

Beside my academic work I have been working on commercial vision based human computer interfaces at Advanced Interface Technologies, Inc.

I have visited and collaborated with the PHILIPS Research Laboratory in Hamburg, Germany where I worked with C. Lorenz on generating three dimensional statistical shape models of anatomical objects and on curvature guided surface triangulations of 3D image data.

I have received my Masters in Physics from the Christian-Albrechts Universität Kiel, Germany. Back then I was doing numerical simulations of nonlinear dynamical systems in the Plasma Dynamics Group with A. Piel, T. Klinger (now at the MPI in Greifswald) and F. Greiner. My thesis was on "Controlling Chaos in the Pierce Diode" [pdf (in German)].

1. R. Sharma, M. Yeasin, N. Krahnstoever, I. Rauschert, G. Cai, I. Brewer, A. MacEachren, K. Sengupta , "Speech-Gesture Driven Multimodal Interfaces for Crisis Management," Proceedings of IEEE special issue on Multimodal Human-Computer Interface, accepted for publication.
2. N. Krahnstoever, C. Lorenz, "Computing Curvature Adaptive Surface Triangulations of Three Dimensional Image Data," The Visual Computer, International Journal of Computer Graphics, accepted for publication.
3. B. Paulhamus, N. Krahnstoever, R. Sharma, "A Vision-based Approach to Creating Dynamic Transparent Textures for Augmented Reality," submitted for publication.
4. N. Krahnstoever, M. Yeasin, and R. Sharma, "Automatic Acquisition and Initialization of Articulated Models," Machine Vision and Applications, to appear, 2003. [details,pdf,ps].
5. E. Ozyildiz, N. Krahnstoever, R. Sharma, "Adaptive Texture and Color Segmentation for Tracking Moving Objects," Pattern Recognition, Vol. 35, Nr. 10, pp. 2013-2029, 2002. [details,pdf,ps]
6. C. Lorenz, N. Krahnstoever, "Generation of Point-Based 3D Statistical Shape Models for Anatomical Objects," Computer Vision and Image Understanding, Nr. 77, pp. 175-181, 2000. [pdf,ps]
7. A. Piel, F. Greiner, T. Klinger, N. Krahnstoever, T. Mausbach, "Chaos and Chaos Control in Plasmas," Physica Scripta, T84, pp. 128-131, 2000.
8. N. Krahnstoever, T. Klinger, F. Greiner, A. Piel, "Controlling Chaos in the Pierce Diode," Phys. Lett. A, 239(1,2), pp. 103-109, 1998.

1. N. Krahnstoever, R. Sharma, "Appearance Management and Cue Fusion for 3D Model-Based Tracking," IEEE Conference on Computer Vision and Pattern Recognition, Madison, Wisconsin, June 16-22, 2003, accepted for publication. [details,pdf,related techreport]
   
2. N. Krahnstoever, R. Sharma, "Robust Probabilistic Estimation of Uncertain Appearance for Model Based Tracking," IEEE Workshop on Motion and Video Computing, The Orlando World Center Marriott Orlando, FL USA, December 5-6, 2002. [details,pdf,ps]
3. N. Krahnstoever, E. Schapira, S. Kettebekov, R. Sharma, "Multimodal Human Computer Interaction for Crisis Management Systems," IEEE Workshop on Applications of Computer Vision, The Orlando World Center Marriott Orlando, FL USA, December 3-4, 2002. [details,pdf,ps]
4. N. Krahnstoever, S. Kettebekov, M. Yeasin, R. Sharma, "A Real-Time Framework for Natural Multimodal Interaction with Large Screen Displays," Fourth International Conference on Multimodal Interfaces (ICMI'2002), Pittsburgh, PA USA, October 14-16, 2002. [pdf,ps]
5. S. Kettebekov, M. Yeasin, N. Krahnstoever, R. Sharma, "Prosody based Co-Analysis of Deictic Gestures and Speech in Weather Narration Broadcast," Workshop on Multimodal Resources and Multimodal Systems Evaluation, 3rd Int. Conf. on Language Resources and Evaluation (LREC 2002), Palacio de Congreso de Canarias Las Palmas, Spain, June 2002. [pdf].
6. N. Krahnstoever, M. Yeasin, R. Sharma, "Automatic Acquisition and Initialization of Kinematic Models," IEEE Conference on Computer Vision and Pattern Recognition, Technical Sketches, Kauai Marriott, Hawaii, USA, Dec, 2001. [details,pdf,ps]
7. N. Krahnstoever, M. Yeasin, R. Sharma, "Towards a Unified Framework for Tracking and Analysis of Human Motion," Int. Conference on Computer Vision, Workshop on Detection and Recognition of Events in Video, Vancouver, Canada, July 8, 2001. [details,pdf,ps]
8. E. Ozyildiz, N. Krahnstoever, R. Sharma, "Fusion of Texture and Color for Robust Visual Tracking," ARL Federate Laboratory 4th Annual Symposium, College Park, MD, March, 2000. [related publication]
9. S. Kettebekov, N. Krahnstoever, M. Leas, E. Polat, H. Raju, E. Schapira, R. Sharma, "i2Map: Crisis Management using a Multimodal Interface," ARL Federate Laboratory 4th Annual Symposium, College Park, MD, March, 2000. [related publication]
10. C. Lorenz, N. Krahnstoever, "3D statistical shape models for medical image segmentation," Proceedings of the Second International Conference on 3-D Digital Imaging and Modeling (3DIM) '99, Ottawa, Canada, October 4-8, 1999. [pdf,ps]
11. N. Krahnstoever, C. Lorenz, "Development of a point based shape representation of arbitrary three-dimensional medical objects suitable for statistical shape modeling," Medical Imaging 1999: Image Processing, Conf., San Diego, February 1999. [pdf,ps]
12. F. Greiner, T. Klinger, H. Kolinsky, N. Krahnstoever, T. Mausbach, A. Piel, "The Pierce-Diode as a Model for Selfoscillations and Controlling Chaos in Thermionic Discharges," 1998 International Congress on Plasma Physics and 25th EPS Conference on Controlled Fusion and Plasma Physics, Praha, June 29 - July 3, H023PR, p. 2240, 1998. [pdf]
13. N. Krahnstoever, T. Klinger, F. Greiner, A. Piel, "Systematische Chaoskontrolle in der Pierce-Diode," DPG Tagung Bayreuth, pp. 326, 1998.

1. R. Sharma, N. Krahnstoever, E. Schapira, "Method and System for Detecting Conscious Hand Movement Patterns and Computer-Generated Visual Feedback for Facilitating Human-Computer Interaction," Advanced Interfaces, Inc., Patent Pending, U.S. Patent Application Number 10/403,234, April 2, 2002.
2. R. Sharma, E. Schapira, N. Krahnstoever, N. Jung, "Method and Apparatus for Providing Virtual Touch Interaction in the Drive-Thru," Advanced Interfaces, Inc., Patent Pending, U.S. Provisional Patent No. 60/415,690, October 3, 2002.
3. R. Sharma, N. Krahnstoever, E. Schapira, "Method and Apparatus for Robustly Tracking Objects," Advanced Interfaces, Inc., Patent Pending, U.S. Provisional Patent No. No. 60/426,574, November 15, 2002.

1. N. Krahnstoever, S. Kettebekov, M. Yeasin, R. Sharma, "A Real-Time Framework for Natural Multimodal Interactionwith Large Screen Displays," Technical Report CSE-02-010, Department of Computer Science and Engineering, The Pennsylvania State University, PA 16802, USA, May 2002. [pdf,ps]
2. N. Krahnstoever, R. Sharma, "Appearance Management and Cue Fusion for 3D Model-Based Tracking," Technical Report CSE-03-009, Department of Computer Science and Engineering, The Pennsylvania State University, PA 16802, USA, April 2003. [pdf,hires version]

1. N. Krahnstoever, "Chaoskontrolle in der Pierce-Diode," Diplomarbeit (Masters Thesis), Institut für Experimentelle und Angewandte Physik, Christian-Albrechts-Universität Kiel, Germany, April 1998. [pdf (in German)]