Disentangled multimodal brain MR image translation via transformer-based modality infuser

Jihoon Cho; Xiaofeng Liu; Fangxu Xing; Jinsong Ouyang; Georges El Fakhri; Jinah Park; Jonghye Woo

doi:10.1117/12.3006502

2 April 2024 Disentangled multimodal brain MR image translation via transformer-based modality infuser

Jihoon Cho, Xiaofeng Liu, Fangxu Xing, Jinsong Ouyang, Georges El Fakhri, Jinah Park, Jonghye Woo

Proceedings Volume 12926, Medical Imaging 2024: Image Processing; 129262H (2024) https://doi.org/10.1117/12.3006502
Event: SPIE Medical Imaging, 2024, San Diego, California, United States

Conference Poster

Abstract

Multimodal Magnetic Resonance (MR) Imaging plays a crucial role in disease diagnosis due to its ability to provide complementary information by analyzing a relationship between multimodal images on the same subject. Acquiring all MR modalities, however, can be expensive, and, during a scanning session, certain MR images may be missed depending on the study protocol. The typical solution would be to synthesize the missing modalities from the acquired images such as using generative adversarial networks (GANs). Yet, GANs constructed with convolutional neural networks (CNNs) are likely to suffer from a lack of global relationships and mechanisms to condition the desired modality. To address this, in this work, we propose a transformer-based modality infuser designed to synthesize multimodal brain MR images. In our method, we extract modality-agnostic features from the encoder and then transform them into modality-specific features using the modality infuser. Furthermore, the modality infuser captures long-range relationships among all brain structures, leading to the generation of more realistic images. We carried out experiments on the BraTS 2018 dataset, translating between four MR modalities, and our experimental results demonstrate the superiority of our proposed method in terms of synthesis quality. In addition, we conducted experiments on a brain tumor segmentation task and different conditioning methods.

(2024) Published by SPIE. Downloading of the abstract is permitted for personal use only.

Citation Download Citation

Jihoon Cho, Xiaofeng Liu, Fangxu Xing, Jinsong Ouyang, Georges El Fakhri, Jinah Park, and Jonghye Woo "Disentangled multimodal brain MR image translation via transformer-based modality infuser", Proc. SPIE 12926, Medical Imaging 2024: Image Processing, 129262H (2 April 2024); https://doi.org/10.1117/12.3006502

ACCESS THE FULL ARTICLE

INSTITUTIONAL
Select your institution to access the SPIE Digital Library.

SELECT YOUR INSTITUTION

PERSONAL
Sign in with your SPIE account to access your personal subscriptions or to use specific features such as save to my library, sign up for alerts, save searches, etc.

PERSONAL SIGN IN

No SPIE Account? Create one

PURCHASE THIS CONTENT

SUBSCRIBE TO DIGITAL LIBRARY

50 downloads per 1-year subscription

Members: $195

Non-members: $335 ADD TO CART

25 downloads per 1 - year subscription

Members: $145

Non-members: $250 ADD TO CART

PURCHASE SINGLE ARTICLE

Includes PDF, HTML & Video, when available