An automatic multimodal data registration strategy for 2D/3D information fusion

Jonathan Schierl; Vijayan Asari; Nina Singer; Theus Aspiras; Andrew Stokes; Brett Keaffaber; Andre Van Rynbach; Kevin Decker; David Rabb

doi:10.1117/12.2618564

27 May 2022 An automatic multimodal data registration strategy for 2D/3D information fusion

Jonathan Schierl, Vijayan Asari, Nina Singer, Theus Aspiras, Andrew Stokes, Brett Keaffaber, Andre Van Rynbach, Kevin Decker, David Rabb

Author Affiliations +

Proceedings Volume 12100, Multimodal Image Exploitation and Learning 2022; 1210004 (2022) https://doi.org/10.1117/12.2618564
Event: SPIE Defense + Commercial Sensing, 2022, Orlando, Florida, United States

Abstract

With the emergence of advanced 2D and 3D sensors such as high-resolution visible cameras and less expensive lidar sensors, there is a need for a fusion of information extracted from senor modalities for accurate object detection, recognition, and tracking. To train a system with data captured by multiple sensors the regions of interest in the data must be accurately aligned. A necessary step in this process is a fine, pixel-level registration between multiple modalities. We propose a robust multimodal data registration strategy for automatically registering the visible and lidar data captured by sensors embedded in aerial vehicles. The coarse registration of the data is performed by utilizing the metadata, such as timestamps, GPS, and IMU information, provided by the data acquisition systems. The challenge is these modalities contain very different sets of information and are not able to be aligned using classical methods. Our proposed fine registration mechanism employs deep-learning methodologies for feature extraction of data in each modality. For our experiments, we use a 3D geopositioned aerial lidar dataset along with the visible data (coarsely registered) and extracted SIFT-like features from both of the data streams. These SIFT features are generated by appropriately trained deep-learning algorithms.

Conference Presentation

Citation Download Citation

Jonathan Schierl, Vijayan Asari, Nina Singer, Theus Aspiras, Andrew Stokes, Brett Keaffaber, Andre Van Rynbach, Kevin Decker, and David Rabb "An automatic multimodal data registration strategy for 2D/3D information fusion", Proc. SPIE 12100, Multimodal Image Exploitation and Learning 2022, 1210004 (27 May 2022); https://doi.org/10.1117/12.2618564

ACCESS THE FULL ARTICLE

INSTITUTIONAL
Select your institution to access the SPIE Digital Library.

SELECT YOUR INSTITUTION

PERSONAL
Sign in with your SPIE account to access your personal subscriptions or to use specific features such as save to my library, sign up for alerts, save searches, etc.

PERSONAL SIGN IN

No SPIE Account? Create one

PURCHASE THIS CONTENT

SUBSCRIBE TO DIGITAL LIBRARY

50 downloads per 1-year subscription

Members: $195

Non-members: $335 ADD TO CART

25 downloads per 1 - year subscription

Members: $145

Non-members: $250 ADD TO CART

PURCHASE SINGLE ARTICLE

Includes PDF, HTML & Video, when available