Multiple imaging modalities are commonly jointly used for investigating biological phenomena or diagnosis purposes. In this study, we propose a deep-learning-based cross-modality imaging technique that utilizes one imaging modality to computationally predict another. A novel neural network architecture, featuring recurrent multi-stage refinement controlled by gated activation, was developed for this purpose. To demonstrate the effectiveness of the proposed method, we conducted experiments on predicting organelle fluorescence images from stimulated Raman scattering (SRS) imaging. The results of the experiments indicate that our method outperforms the current state-of-the-art techniques across multiple datasets, in terms of both accuracy and efficiency. The neural network architecture was able to produce high-quality predictions with clear boundaries and high prediction accuracy through the multi-stage refinement process. The proposed method presents a versatile framework that addresses the limitations of current deep-learning-enabled cross-modality image prediction techniques and has potential applications in the field of medical and biological imaging.
KEYWORDS: Databases, Video, Robots, Data modeling, Detection and tracking algorithms, Facial recognition systems, Neural networks, Cameras, Video processing, Digital cameras
Empowering retail service robots with empathy is one of the current research hotspots in the field of artificial intelligence. Identifying consumer emotions, understanding the changes in shopping interests, and developing appropriate sales strategies is a challenging task for retail service robots. We investigate the feasibility of using computer vision methods for empowering robots with empathy by examining the correlation between consumer emotion and levels of shopping interest. To this end, we construct the first video database of consumer sentiment changes in a business context and propose a deep learning method that uses multimodal information to infer consumers’ shopping intentions, and conduct preliminary experimental validation on this database. The experimental results show that the proposed method is 7% and 10% more accurate than manual assessment (n = 40) in identifying consumer emotions and predicting consumer shopping interest levels, respectively. Thus, the proposed method is valid and effective. We anticipate that the results of this study will have considerable implications for human–computer interaction research in service robots.
Cell classification is a fundamental task in biological research and medical practices. In this study, we proposed a singlecell classification pipeline through machine learning and hyperspectral stimulated Raman scattering imaging. The pipeline proposed is validated by using hyperspectral SRS images of two types of pancreatic cancer cells before and after the treatment of drugs that affects cellular cholesterol level. The result demonstrates that the proposed machine learning pipeline is capable of classifying cells with different metabolite dynamics, which provides possibilities for wide applications in cell analysis.
Optical phase microscopy is widely adopted for quantitative imaging of optical density in transparent cells and tissues that lack absorption contrast. Fundamentally, the phase information of the sample is contained in the wavefront of the probe beam, often detected by interferometry-based techniques. Here, a novel approach has been developed based on the phase-sensitive second harmonic signals that are generated after the sample. A deep learning algorithm is developed for efficient recovery of the original phase images. Inheriting the advantages of the second harmonic imaging, our second harmonic phase imaging is a label-free technique with a demonstrated phase sensitivity of 1/100 wavelength and high robustness against noises, facilitating applications in biological imaging and remote sensing.
Infrared (IR) spectroscopy depicts molecular structure and dynamics based on vibrational absorption of chemical bonds. Spatially resolved IR spectroscopy, i.e. IR imaging, further enabled label-free in situ chemical imaging for dynamics in complex systems. However, IR imaging suffers from low spatial resolution at a few micrometers due to diffraction limit, thus having difficulty in applications such as sub-cellular imaging. Recently, by visible light probing of the photothermal effect of vibrational absorption, mid-infrared photothermal imaging (MIP) overcomes the limitations of conventional IR microscopy and has achieved sub-micron resolution. In this work, we built an optimized MIP system to boost the spatial resolution and sensitivity, and demonstrated MIP imaging of nanometer-sized polymeric microspheres and living cells with a high spatial resolution of 200 nm.
Optical phase microscopy is widely adopted for quantitative imaging of optical density in transparent cells and tissues, yet lacks the chemical selectivity. To address this challenge, a bond-selective transient phase imaging (BTSP) technique was developed, in which a transient change in phase induced by infrared excitation of molecular vibrations was detected by a diffraction phase microscope. BTSP achieved chemically selective phase imaging of live cells. We further demonstrated an IR-pump visible-probe phase microscopy based on second harmonic generation after the sample, enabled by deep learning. The phase-sensitive information is encoded into the second harmonic signal, which is decoded using a deep learning algorithm. It presents a label-free technique featured by high phase sensitivity and high robustness against noises, which has promising applications in biological and medical imaging and remote sensing.
Access to the requested content is limited to institutions that have purchased or subscribe to SPIE eBooks.
You are receiving this notice because your organization may not have SPIE eBooks access.*
*Shibboleth/Open Athens users─please
sign in
to access your institution's subscriptions.
To obtain this item, you may purchase the complete book in print or electronic format on
SPIE.org.
INSTITUTIONAL Select your institution to access the SPIE Digital Library.
PERSONAL Sign in with your SPIE account to access your personal subscriptions or to use specific features such as save to my library, sign up for alerts, save searches, etc.