Fusion of multi-sensory saliency maps for automated perception and control

David J. Huber; Deepak Khosla; Paul A. Dow

doi:10.1117/12.820504

11 May 2009 Fusion of multi-sensory saliency maps for automated perception and control

David J. Huber, Deepak Khosla, Paul A. Dow

Proceedings Volume 7336, Signal Processing, Sensor Fusion, and Target Recognition XVIII; 73360Z (2009) https://doi.org/10.1117/12.820504
Event: SPIE Defense, Security, and Sensing, 2009, Orlando, Florida, United States

Abstract

In many real-world situations and applications that involve humans or machines (e.g., situation awareness, scene understanding, driver distraction, workload reduction, assembly, robotics, etc.) multiple sensory modalities (e.g., vision, auditory, touch, etc.) are used. The incoming sensory information can overwhelm processing capabilities of both humans and machines. An approach for estimating what is most important in our sensory environment (bottom-up or goal-driven) and using that as a basis for workload reduction or taking an action could be of great benefit in applications involving humans, machines or human-machine interactions. In this paper, we describe a novel approach for determining high saliency stimuli in multi-modal sensory environments, e.g., vision, sound, touch, etc. In such environments, the high saliency stimuli could be a visual object, a sound source, a touch event, etc. The high saliency stimuli are important and should be attended to from perception, cognition or/and action perspective. The system accomplishes this by the fusion of saliency maps from multiple sensory modalities (e.g., visual and auditory) into a single, fused multimodal saliency map that is represented in a common, higher-level coordinate system. This paper describes the computational model and method for generating multi-modal or fused saliency map. The fused saliency map can be used to determine primary and secondary foci of attention as well as for active control of a hardware/device. Such a computational model of fused saliency map would be immensely useful for a machine-based or robot-based application in a multi-sensory environment. We describe the approach, system and present preliminary results on a real-robotic platform.

Citation Download Citation

David J. Huber, Deepak Khosla, and Paul A. Dow "Fusion of multi-sensory saliency maps for automated perception and control", Proc. SPIE 7336, Signal Processing, Sensor Fusion, and Target Recognition XVIII, 73360Z (11 May 2009); https://doi.org/10.1117/12.820504

ACCESS THE FULL ARTICLE

INSTITUTIONAL
Select your institution to access the SPIE Digital Library.

SELECT YOUR INSTITUTION

PERSONAL
Sign in with your SPIE account to access your personal subscriptions or to use specific features such as save to my library, sign up for alerts, save searches, etc.

PERSONAL SIGN IN

No SPIE Account? Create one

PURCHASE THIS CONTENT

SUBSCRIBE TO DIGITAL LIBRARY

50 downloads per 1-year subscription

Members: $195

Non-members: $335 ADD TO CART

25 downloads per 1 - year subscription

Members: $145

Non-members: $250 ADD TO CART

PURCHASE SINGLE ARTICLE

Includes PDF, HTML & Video, when available