Cross-modality attention-based multimodal fusion for non-small cell lung cancer (NSCLC) patient survival prediction

Ruining Deng; Nazim Shaikh; Gareth Shannon; Yao Nie

doi:10.1117/12.3006036

3 April 2024 Cross-modality attention-based multimodal fusion for non-small cell lung cancer (NSCLC) patient survival prediction

Ruining Deng, Nazim Shaikh, Gareth Shannon, Yao Nie

Author Affiliations +

Proceedings Volume 12933, Medical Imaging 2024: Digital and Computational Pathology; 1293309 (2024) https://doi.org/10.1117/12.3006036
Event: SPIE Medical Imaging, 2024, San Diego, California, United States

Abstract

Cancer prognosis and survival outcome predictions are crucial for therapeutic response estimation and for stratifying patients into various treatment groups. Medical domains concerned with cancer prognosis are abundant with multiple modalities, including pathological image data and non-image data such as genomic information. To date, multimodal learning has shown potential to enhance clinical prediction model performance by extracting and aggregating information from different modalities of the same subject. This approach could outperform single modality learning, thus improving computer-aided diagnosis and prognosis in numerous medical applications. In this work, we propose a cross-modality attention-based multimodal fusion pipeline designed to integrate modality-specific knowledge for patient survival prediction in non-small cell lung cancer (NSCLC). Instead of merely concatenating or summing up the features from different modalities, our method gauges the importance of each modality for feature fusion with cross-modality relationship when infusing the multimodal features. Compared with single modality, which achieved c-index of 0.5772 and 0.5885 using solely tissue image data or RNA-seq data, respectively, the proposed fusion approach achieved c-index 0.6587 in our experiment, showcasing the capability of assimilating modality-specific knowledge from varied modalities.

Conference Presentation

(2024) Published by SPIE. Downloading of the abstract is permitted for personal use only.

Citation Download Citation

Ruining Deng, Nazim Shaikh, Gareth Shannon, and Yao Nie "Cross-modality attention-based multimodal fusion for non-small cell lung cancer (NSCLC) patient survival prediction", Proc. SPIE 12933, Medical Imaging 2024: Digital and Computational Pathology, 1293309 (3 April 2024); https://doi.org/10.1117/12.3006036

ACCESS THE FULL ARTICLE

INSTITUTIONAL
Select your institution to access the SPIE Digital Library.

SELECT YOUR INSTITUTION

PERSONAL
Sign in with your SPIE account to access your personal subscriptions or to use specific features such as save to my library, sign up for alerts, save searches, etc.

PERSONAL SIGN IN

No SPIE Account? Create one

PURCHASE THIS CONTENT

SUBSCRIBE TO DIGITAL LIBRARY

50 downloads per 1-year subscription

Members: $195

Non-members: $335 ADD TO CART

25 downloads per 1 - year subscription

Members: $145

Non-members: $250 ADD TO CART

PURCHASE SINGLE ARTICLE

Includes PDF, HTML & Video, when available