Model-based, multimodal interaction in document browsing

Eslambolchilar, P. and Murray-Smith, R. (2006) Model-based, multimodal interaction in document browsing. Lecture Notes in Computer Science, 4299, pp. 1-12. (doi: 10.1007/11965152_1)

Abstract

In this paper we introduce a dynamic system approach to the design of multimodal interactive systems. We use an example where we support human behavior in browsing a document, by adapting the dynamics of navigation and the visual feedback (using a focus-in-context (F+C) method) to support the current inferred task. We also demonstrate non-speech audio feedback, based on a language model. We argue that to design interaction we need models of key aspects of the process, here for example, we need models for the dynamic system, language model and sonification. We show how the user’s intention is coupled to the visualization technique via the dynamic model, and how the focus-in-context method couples details in context to audio samples via the language identification system. We present probabilistic audio feedback as an example of a multimodal approach to sensing different languages in a multilingual text. This general approach is well suited to mobile and wearable applications, and shared displays.

• Item Type: Articles
• Status: Published
• Refereed: Yes
• Glasgow Author(s) Enlighten ID: Murray-Smith, Professor Roderick
• Authors: Eslambolchilar, P., and Murray-Smith, R.
• Subjects: Q Science > QA Mathematics > QA75 Electronic computers. Computer science
• College/School: College of Science and Engineering > School of Computing Science
• Journal Name: Lecture Notes in Computer Science
• ISSN: 0302-9743
• ISSN (Online): 1611-3349
• Published Online: 23 January 2007