A Real-Time Framework for Natural Multimodal Interaction with Large Screen Displays
   
Jump to: Complete Reference, Abstract, Download, BibTeX Entry, Links and Notes, Images, Additional Material
 
Complete Reference
  N. Krahnstoever, S. Kettebekov, M. Yeasin, R. Sharma, "A Real-Time Framework for Natural Multimodal Interaction with Large Screen Displays", Fourth International Conference on Multimodal Interfaces (ICMI'2002), Pittsburgh, PA USA, October 14-16, 2002.
 
Abstract
  This paper presents a framework for designing a natural multimodal human computer interaction (HCI) system. The core of the proposed framework is a principled method for combining information derived from audio and visual cues. To achieve natural interaction, both audio and visual modalities are fused along with feedback through a large screen display. Careful design along with due considerations of possible aspects of a systems interaction cycle and integration has resulted in a successful system. The performance of the proposed framework has been validated through the development of several prototype systems as well as commercial applications for the retail and entertainment industry. To assess the impact of these multimodal systems (MMS), informal studies have been conducted. It was found that the system performed according to its specifications in 95% of the cases and that users showed adhoc proficiency, indicating natural acceptance of such systems.
 
Download
  PDF Color (923 KB) | Postscript (11.1 MB)
 
BibTeX Entry
 
@inproceedings{krahnstoever02multimodal,
author={N. Krahnstoever and S. Kettebekov and M. Yeasin and R. Sharma},
booktitle={Proc. of Fourth Intl. Conference on Multimodal Interfaces (ICMI 2002), Pittsburgh, PA, USA},
month={October},
title={A Real-Time Framework for Natural Multimodal Interaction with Large Screen Displays},
year={2002}
}
 
Links and Notes
  ICMI 2002 Conference
 
Images
   
 
Additional Material
 


VIDEOS:

User detection, hand detection and head and hand tracking:

AVI, Codec: Cinepack (4.9 MB) | MPEG (5.7 MB) | Quicktime, Codec: Cinepack (4.9 MB)