Guest Editors Ivana Išgum, Bennett A. Landman, and Tomaž Vrtovec introduce the JMI Special Section on Advances in High-Dimensional Medical Image Processing.

Purpose: Segmentation of lung nodules in chest CT images is essential for image-driven lung cancer diagnosis and follow-up treatment planning. Manual segmentation of lung nodules is subjective because the approach depends on the knowledge and experience of the specialist. We proposed a multiscale fully convolutional three-dimensional UNet (MF-3D UNet) model for automatic segmentation of lung nodules in CT images.

Approach: The proposed model employs two strategies, fusion of multiscale features with Maxout aggregation and trainable downsampling, to improve the performance of nodule segmentation in 3D CT images. The fusion of multiscale (fine and coarse) features with the Maxout function allows the model to retain the most important features while suppressing the low-contribution features. The trainable downsampling process is used instead of fixed pooling-based downsampling.

Results: The performance of the proposed MF-3D UNet model is examined by evaluating the model with CT scans obtained from the Lung Image Database Consortium and Image Database Resource Initiative (LIDC-IDRI) dataset. A quantitative and visual comparative analysis of the proposed work with various customized UNet models is also presented. The comparative analysis shows that the proposed model yields reliable segmentation results compared with other methods. The experimental result of 3D MF-UNet shows encouraging results in the segmentation of different types of nodules, including juxta-pleural, solitary pulmonary, and non-solid nodules, with an average Dice similarity coefficient of 0.83  ±  0.05, and it outperforms other CNN-based segmentation models.

Conclusions: The proposed model accurately segments the nodules using multiscale feature aggregation and trainable downsampling approaches. Also, 3D operations enable precise segmentation of complex nodules using inter-slice connections.

Purpose: The technique of fusing or integrating medical images collected from single and various modalities is known as medical image fusion. This is done to improve the quality of the images and combine information from several medical images. The whole procedure aids medical practitioners in gaining correct information from single images. Image fusion is one of the fastest-growing research topics in the medical imaging field. Sparse modeling is a popular signal representation technique used for image fusion with different dictionary learning approaches. We propose a medical image fusion with sparse representation (SR) and block total least-square (BLOTLESS) update dictionary learning.

Approach: The domain of dictionary learning is the most significant research domain related to SR. An efficient dictionary increases the effectiveness of sparse modeling. Due to SR being an ongoing interesting research area, the medical image fusion process is done with a modified image fusion framework with recently developed BLOTLESS update dictionary learning.

Results: The experimental results are compared for the image fusion process using other state-of-the-art dictionary learning algorithms, such as simultaneous codeword optimization, method of optimal directions, and K-singular value decomposition. The effectiveness of the algorithm is evaluated based on image fusion quantitative parameters. Results show that the BLOTLESS update dictionary algorithm is a promising modification for the sparse-based image fusion with its applicability in the fusion of images related to different diseases.

Conclusions: The experiments and results show that the dictionary learning algorithm plays an important role in the sparse-based image fusion general framework. The fusion results also show that the proposed improved image fusion framework for medical images is promising compared with frameworks with other dictionary learning algorithms. As an application, it is also used as a tool for the fusion of different modularities of images related to brain tumor and glioma.

Purpose: Covariance between gray-matter measurements can reflect structural or functional brain networks though it has also been shown to be influenced by confounding factors (e.g., age, head size, and scanner), which could lead to lower mapping precision (increased size of associated clusters) and create distal false positives associations in mass-univariate vertexwise analyses.

Approach: We evaluated this concern by performing state-of-the-art mass-univariate analyses (general linear model, GLM) on traits simulated from real vertex-wise gray matter data (including cortical and subcortical thickness and surface area). We contrasted the results with those from linear mixed models (LMMs), which have been shown to overcome similar issues in omics association studies.

Results: We showed that when performed on a large sample (N  =  8662, UK Biobank), GLMs yielded greatly inflated false positive rate (cluster false discovery rate >0.6). We showed that LMMs resulted in more parsimonious results: smaller clusters and reduced false positive rate but at a cost of increased computation. Next, we performed mass-univariate association analyses on five real UKB traits (age, sex, BMI, fluid intelligence, and smoking status) and LMM yielded fewer and more localized associations. We identified 19 significant clusters displaying small associations with age, sex, and BMI, which suggest a complex architecture of at least dozens of associated areas with those phenotypes.

Conclusions: The published literature could contain a large proportion of redundant (possibly confounded) associations that are largely prevented using LMMs. The parsimony of LMMs results from controlling for the joint effect of all vertices, which prevents local and distal redundant associations from reaching significance.

Purpose: Muscle, bone, and fat segmentation from thigh images is essential for quantifying body composition. Voxelwise image segmentation enables quantification of tissue properties including area, intensity, and texture. Deep learning approaches have had substantial success in medical image segmentation, but they typically require a significant amount of data. Due to the high cost of manual annotation, training deep learning models with limited human label data is desirable, but it is a challenging problem.

Approach: Inspired by transfer learning, we proposed a two-stage deep learning pipeline to address the thigh and lower leg segmentation issue. We studied three datasets, 3022 thigh slices and 8939 lower leg slices from the BLSA dataset and 121 thigh slices from the GESTALT study. First, we generated pseudo labels for thigh based on approximate handcrafted approaches using CT intensity and anatomical morphology. Then, those pseudo labels were fed into deep neural networks to train models from scratch. Finally, the first stage model was loaded as the initialization and fine-tuned with a more limited set of expert human labels of the thigh.

Results: We evaluated the performance of this framework on 73 thigh CT images and obtained an average Dice similarity coefficient (DSC) of 0.927 across muscle, internal bone, cortical bone, subcutaneous fat, and intermuscular fat. To test the generalizability of the proposed framework, we applied the model on lower leg images and obtained an average DSC of 0.823.

Conclusions: Approximated handcrafted pseudo labels can build a good initialization for deep neural networks, which can help to reduce the need for, and make full use of, human expert labeled data.

Purpose: Coronary artery calcium (CAC) score, i.e., the amount of CAC quantified in CT, is a strong and independent predictor of coronary heart disease (CHD) events. However, CAC scoring suffers from limited interscan reproducibility, which is mainly due to the clinical definition requiring application of a fixed intensity level threshold for segmentation of calcifications. This limitation is especially pronounced in non-electrocardiogram-synchronized computed tomography (CT) where lesions are more impacted by cardiac motion and partial volume effects. Therefore, we propose a CAC quantification method that does not require a threshold for segmentation of CAC.

Approach: Our method utilizes a generative adversarial network (GAN) where a CT with CAC is decomposed into an image without CAC and an image showing only CAC. The method, using a cycle-consistent GAN, was trained using 626 low-dose chest CTs and 514 radiotherapy treatment planning (RTP) CTs. Interscan reproducibility was compared to clinical calcium scoring in RTP CTs of 1662 patients, each having two scans.

Results: A lower relative interscan difference in CAC mass was achieved by the proposed method: 47% compared to 89% manual clinical calcium scoring. The intraclass correlation coefficient of Agatston scores was 0.96 for the proposed method compared to 0.91 for automatic clinical calcium scoring.

Conclusions: The increased interscan reproducibility achieved by our method may lead to increased reliability of CHD risk categorization and improved accuracy of CHD event prediction.

Purpose: Ensembles of convolutional neural networks (CNNs) often outperform a single CNN in medical image segmentation tasks, but inference is computationally more expensive and makes ensembles unattractive for some applications. We compared the performance of differently constructed ensembles with the performance of CNNs derived from these ensembles using knowledge distillation, a technique for reducing the footprint of large models such as ensembles.

Approach: We investigated two different types of ensembles, namely, diverse ensembles of networks with three different architectures and two different loss-functions, and uniform ensembles of networks with the same architecture but initialized with different random seeds. For each ensemble, additionally, a single student network was trained to mimic the class probabilities predicted by the teacher model, the ensemble. We evaluated the performance of each network, the ensembles, and the corresponding distilled networks across three different publicly available datasets. These included chest computed tomography scans with four annotated organs of interest, brain magnetic resonance imaging (MRI) with six annotated brain structures, and cardiac cine-MRI with three annotated heart structures.

Results: Both uniform and diverse ensembles obtained better results than any of the individual networks in the ensemble. Furthermore, applying knowledge distillation resulted in a single network that was smaller and faster without compromising performance compared with the ensemble it learned from. The distilled networks significantly outperformed the same network trained with reference segmentation instead of knowledge distillation.

Conclusion: Knowledge distillation can compress segmentation ensembles of uniform or diverse composition into a single CNN while maintaining the performance of the ensemble.

Purpose: The quantitative detection, segmentation, and characterization of glomeruli from high-resolution whole slide imaging (WSI) play essential roles in the computer-assisted diagnosis and scientific research in digital renal pathology. Historically, such comprehensive quantification requires extensive programming skills to be able to handle heterogeneous and customized computational tools. To bridge the gap of performing glomerular quantification for non-technical users, we develop the Glo-In-One toolkit to achieve holistic glomerular detection, segmentation, and characterization via a single line of command. Additionally, we release a large-scale collection of 30,000 unlabeled glomerular images to further facilitate the algorithmic development of self-supervised deep learning.

Approach: The inputs of the Glo-In-One toolkit are WSIs, while the outputs are (1) WSI-level multi-class circle glomerular detection results (which can be directly manipulated with ImageScope), (2) glomerular image patches with segmentation masks, and (3) different lesion types. In the current version, the fine-grained global glomerulosclerosis (GGS) characterization is provided, including assessed-solidified-GSS (associated with hypertension-related injury), disappearing-GSS (a further end result of the SGGS becoming contiguous with fibrotic interstitium), and obsolescent-GSS (nonspecific GGS increasing with aging) glomeruli. To leverage the performance of the Glo-In-One toolkit, we introduce self-supervised deep learning to glomerular quantification via large-scale web image mining.

Results: The GGS fine-grained classification model achieved a decent performance compared with baseline supervised methods while only using 10% of the annotated data. The glomerular detection achieved an average precision of 0.627 with circle representations, while the glomerular segmentation achieved a 0.955 patch-wise Dice dimilarity coefficient.

Conclusion: We develop and release an open-source Glo-In-One toolkit, a software with holistic glomerular detection, segmentation, and lesion characterization. This toolkit is user-friendly to non-technical users via a single line of command. The toolbox and the 30,000 web mined glomerular images have been made publicly available at https://github.com/hrlblab/Glo-In-One.

Purpose: Quantitative lung measures derived from computed tomography (CT) have been demonstrated to improve prognostication in coronavirus disease 2019 (COVID-19) patients but are not part of clinical routine because the required manual segmentation of lung lesions is prohibitively time consuming. We aim to automatically segment ground-glass opacities and high opacities (comprising consolidation and pleural effusion).

Approach: We propose a new fully automated deep-learning framework for fast multi-class segmentation of lung lesions in COVID-19 pneumonia from both contrast and non-contrast CT images using convolutional long short-term memory (ConvLSTM) networks. Utilizing the expert annotations, model training was performed using five-fold cross-validation to segment COVID-19 lesions. The performance of the method was evaluated on CT datasets from 197 patients with a positive reverse transcription polymerase chain reaction test result for SARS-CoV-2, 68 unseen test cases, and 695 independent controls.

Results: Strong agreement between expert manual and automatic segmentation was obtained for lung lesions with a Dice score of 0.89 ± 0.07; excellent correlations of 0.93 and 0.98 for ground-glass opacity (GGO) and high opacity volumes, respectively, were obtained. In the external testing set of 68 patients, we observed a Dice score of 0.89 ± 0.06 as well as excellent correlations of 0.99 and 0.98 for GGO and high opacity volumes, respectively. Computations for a CT scan comprising 120 slices were performed under 3 s on a computer equipped with an NVIDIA TITAN RTX GPU. Diagnostically, the automated quantification of the lung burden % discriminate COVID-19 patients from controls with an area under the receiver operating curve of 0.96 (0.95–0.98).

Conclusions: Our method allows for the rapid fully automated quantitative measurement of the pneumonia burden from CT, which can be used to rapidly assess the severity of COVID-19 pneumonia on chest CT.

Purpose: To evaluate the usefulness of computed tomography (CT) texture descriptors integrated with machine-learning (ML) models in the identification of clear cell renal cell carcinoma (ccRCC) and for the first time papillary renal cell carcinoma (pRCC) tumor nuclear grades [World Health Organization (WHO)/International Society of Urologic Pathologists (ISUP) 1, 2, 3, and 4].

Approach: A total of 143 ccRCC and 21 pRCC patients were analyzed in this study. Texture features were extracted from late arterial phase CT images. A complete separation of training/validation and testing subsets from the beginning to the end of the pipeline was adopted. Feature dimension was reduced by collinearity analysis and Gini impurity-based feature selection. The synthetic minority over-sampling technique was employed for imbalanced datasets. The ML classifiers were logistic regression, SVM, RF, multi-layer perceptron, and K-NN. The differentiation between low grades/ high grades, grade 1/grade 2, grade 3/grade 4, and between all grades was assessed for ccRCC and pRCC datasets. The classification performance was assessed and compared by certain metrics.

Results: Textures-based classifiers were able to efficiently identify ccRCC and pRCC grades. An accuracy and area under the characteristic operating curve (AUC) up to 91%/0.9, 91%/0.9, 90%/0.9, and 88%/1 were reached when discriminating ccRCC low grades/ high grades, grade 1/grade 2, grade 3/grade 4, and all grades, respectively. An accuracy and AUC up to 96%/1, 81%/0.8, 86%/0.9, and 88%/0.9 were found when differentiating pRCC low grades/ high grades, grade 1/grade 2, grade 3/grade 4, and all grades, respectively.

Conclusion: CT texture-based ML models can be used to assist radiologist in predicting the WHO/ISUP grade of ccRCC and pRCC pre-operatively.

Purpose: For a patient at risk of having lung cancer, accurate disease staging is vital as it dictates disease prognosis and treatment. Accurate staging requires a comprehensive sampling of lymph nodes within the chest via bronchoscopy. Unfortunately, physicians are generally unable to plan and perform sufficiently comprehensive procedures to ensure accurate disease staging. We propose a method for planning comprehensive lymph node staging procedures.

Approach: Drawing on a patient’s chest CT scan, the method derives a multi-destination tour for efficient navigation to a set of lymph nodes. We formulate the planning task as a traveling salesman problem. To solve the problem, we apply the concept of ant colony optimization (ACO) to derive an efficient airway tour connecting the target nodes. The method has three main steps: (1) CT preprocessing, to define important chest anatomy; (2) graph and staging zone construction, to set up the necessary data structures and clinical constraints; and (3) tour computation, to derive the staging plan. The plan conforms to the world standard International Association for the Study of Lung Cancer (IASLC) lymph node map and recommended clinical staging guidelines.

Results: Tests with a patient database indicate that the method derives optimal or near-optimal tours in under a few seconds, regardless of the number of target lymph nodes (mean tour length = 1.4% longer than the optimum). A brute force optimal search, on the other hand, generally cannot reach a solution in under 10 min. for patients exhibiting >16 nodes, and other methods provide poor solutions. We also demonstrate the method’s utility in an image-guided bronchoscopy system.

Conclusions: The method provides an efficient computational approach for planning a comprehensive lymph node staging bronchoscopy. In addition, the method shows promise for driving an image-guided bronchoscopy system or robotics-assisted bronchoscopy system tailored to lymph node staging.

Purpose: Radiologists exhibit wide inter-reader variability in diagnostic performance. This work aimed to compare different feature sets to predict if a radiologist could detect a specific liver metastasis in contrast-enhanced computed tomography (CT) images and to evaluate possible improvements in individualizing models to specific radiologists.

Approach: Abdominal CT images from 102 patients, including 124 liver metastases in 51 patients were reconstructed at five different kernels/doses using projection domain noise insertion to yield 510 image sets. Ten abdominal radiologists marked suspected metastases in all image sets. Potentially salient features predicting metastasis detection were identified in three ways: (i) logistic regression based on human annotations (semantic), (ii) random forests based on radiologic features (radiomic), and (iii) inductive derivation using convolutional neural networks (CNN). For all three approaches, generalized models were trained using metastases that were detected by at least two radiologists. Conversely, individualized models were trained using each radiologist’s markings to predict reader-specific metastases detection.

Results: In fivefold cross-validation, both individualized and generalized CNN models achieved higher area under the receiver operating characteristic curves (AUCs) than semantic and radiomic models in predicting reader-specific metastases detection ability (p < 0.001). The individualized CNN with an AUC of mean (SD) 0.85(0.04) outperformed the generalized one [AUC = 0.78 ( 0.06 ) , p = 0.004]. The individualized semantic [AUC = 0.70 ( 0.05 ) ] and radiomic models [AUC = 0.68 ( 0.06 ) ] outperformed the respective generalized versions [semantic AUC = 0.66 ( 0.03 ) , p = 0.009; radiomic AUC = 0.64 ( 0.06 ) , p = 0.03].

Conclusions: Individualized models slightly outperformed generalized models for all three feature sets. Inductive CNNs were better at predicting metastases detection than semantic or radiomic features. Generalized models have implementation advantages when individualized data are unavailable.