Multimodal HCI is at the intersection of several disciplines that include psychology, computer vision, user interface design ,and artificial intelligence just to name a few. Computers have become integrated into everyday objects and tasks (often termed ubiquitous and pervasive computing) and so it has become important to facilitate a natural interaction between users and computers that simulate the way humans interact with each other face-to-face. Multimodal HCI aims to discern the ways in which humans interact with each other in order to translate these processes to the human and computer interaction.
Multimodal HCI encompasses a wide range of modalities. A modality is a mode of communication according to human sense or type of computer input device. Typically modalities are divided into two categories:
1.Human senses: sight, touch, hearing, smell, taste, and cognitive/emotional abilities
2.Computer input devices: cameras, haptic sensors, microphones, olfactory (all meant to emulate the human senses)
Together these modalities can be used to construct various types of user interface designs in order to increase usability. In this article I will briefly overview the functions of each modality and then discuss the benefits of multimodal user interface design.
Multimodalities and their application in user interface design
The above mentioned categories exist to understand how, particularly related to user interface design, humans and computers utilize the same functions and can thus complement each other for a more positive user experience.
The human senses category helps designers understand how humans take in and process information. Sight has to do with what a user sees on an interface words, images etc. Hearing has to do with the audio aspect of a user interface, whether it be music, commands, or conversation between two people. While smell and, especially, taste are less important in relation to traditional user interface design, they are still important for designers to consider since they can affect the other senses, and the synergy of all the senses ultimately affects the users cognitive and emotional state. From a design perspective, it is critical to understand the cognitive and emotional state that results from a user employing a user interface. This can help to detect and track the users task execution, motivations, and level of satisfaction with a design.
Computer input devices mimic the human senses in order to help people relate to computer interfaces (and conversely to help computers relate to people). Cameras function like an eye and mimic the visual sense, and haptic sensors are a tactile feedback technology (think of the mouse) mimicing the sense of touch. Microphones and speakers enable a user to hear and make sounds. Olfactory devices can stir up a users sense of smell; a process called machine olfaction employed by SMBO instruments (commonly referred to as electronic noses) uses a complex array of chemical sensors and pattern recognition algorithms to allow computers to mimic the sense of smell.
Why is multimodal HCI important to user interface design?