2.1.

In-House MxIF Data

Samples were studied in deidentified form from Vanderbilt University Medical Center under Institutional Review Board approval (IRB #191738 and #191777). The samples were labeled at the slide level by a pathologist as normal, quiescent, mild, moderate, or severe, with respect to Crohn’s disease activity. The samples were formalin-fixed and paraffin-embedded. We used 28 whole slides imaged at $0.32\mu \mathrm{m}$ per pixel (14 from the ascending colon and 14 from the terminal ileum) from 20 patients. Not every sample from a Crohn’s disease patient depicts the disease.

Of the 28 slides, six were control tissue, six were normal tissue, six were quiescent Crohn’s disease, six were mild Crohn’s disease, two were moderate Crohn’s disease, and two were severe Crohn’s disease. The distribution of slides per patient was as follows: 14 patients had one slide, five patients had two slides, and one patient had four slides. Slides were split at the patient level during training, validation, and testing, which we discuss in further detail in Sec. 2.7. We studied 17 out of 27 stain channels to annotate cell types on MxIF: NaKATPase, PanCK, Muc2, CgA, Vimentin, DAPI, SMA, Sox9, OLFM4, Lysozyme, CD45, CD20, CD68, CD11B, CD3d, CD8, and CD4 (Table 1). Although we used 17 stains for annotation, we annotated 14 classes because each of the 17 stains did not always directly map to a unique nucleus/cell type. These stains came from a subset of a previously described protocol.²⁰

2.2.

Public Real H&E Data

The public real H&E data came from the CoNIC Challenge 2022 and is from normal colon tissue as well as colon tissue depicting cancer, dysplasia, and inflammation.⁶ There were 4981 image patches of size $256\times 256\text{pixels}$, which came from images with a resolution of $\sim 0.5\mu \mathrm{m}$ per pixel. Nuclei in this dataset had labels for six cell types: epithelial, lymphocyte, plasma, eosinophil, neutrophil, and connective. The data came from multiple sites in the United States, England, and China.³

Moreover, the data were aggregated from five source datasets: CRAG,²⁸ GlaS,²⁹ DigestPath, PanNuke,³⁰ and CoNSeP.³¹

2.3.

Label Generation Leveraging MxIF

The multi-stain nature of MxIF allows cell subclassification by design, which we leveraged to automatically generate our 14 class labels. Generating classification labels for nuclei and cells required identifying whether a stain bound or did not bind to cells at the whole slide image level. To determine stain binding required picking a threshold for positive and negative nuclei and cells, for each stain. In this work, a senior digital pathology researcher manually selected and applied stain-wise thresholds. These thresholds were determined separately by stain channel for each MxIF whole slide image. Thresholds were determined on images with autofluorescence removed but without further preprocessing such as normalization.

To classify the nuclei, we needed to know their locations. The MxIF did not have cell centroids labeled, so we opted for an automatic approach. To determine the location of each nucleus/cell, we obtained an instance segmentation by performing inference with the pretrained DeepCell Mesmer model.³² We passed the Mesmer model a single grayscale channel image as input. This single channel input was the sum of the MxIF DAPI and Muc2 channels. Merging the channels through addition was reasonable because DAPI identifies nuclei, and Muc2 identifies the goblets associated with goblet cells. Each nucleus or cell was then categorized as positive or negative for each stain type by computing its mean stain intensity and applying the manual thresholds. We assigned each instance a single class label based on a series of biological rules.²⁵ The key stains needed to identify the cell/nucleus types are highlighted in Table 1. After label generation, we had 14 classes, 13 of which were nucleus classes: enteroendocrine, enterocyte, epithelial progenitor, fibroblast, stromal (undetermined), monocyte, macrophage, helper T, cytotoxic T, CD3d+ CD4− CD8− T, B, myeloid (other), and leukocyte (other), and one of which was a cell class: goblet. Each cell/nucleus in the dataset had one single label. Although leukocyte and myeloid are parent classes for some of the other cell types, a cell with the label leukocyte or myeloid was unable to be identified with any more detail given the ground truth staining information for that nucleus. As such, we refer to leukocyte as leukocyte (other) and myeloid as myeloid (other).

We differentiate goblet cells as not being nuclei because they were identified via MUC2 positive regions, which are the goblets from goblet cells, rather than the nuclei of the goblet cells. For progenitors, as 96% were epithelial-positive in our dataset, we refer to progenitors and epithelial progenitors interchangeably throughout this paper. In addition, the nucleus class stromal (undetermined) refers to cells in the stroma that were not classified further into any specific subclasses.

2.4.

Virtual H&E Via Style Transfer

As our goal was to segment and classify from H&E data, we used a pretrained network to style transfer the MxIF (from which the cell classification labels were derived) to virtual H&E. The virtual H&E was inferred from all of the available 27 MxIF stains for the 28 MxIF whole slide images. We used a pretrained CycleGAN-based model that performed style transfer. This model was trained on the same 28 whole slide images of in-house MxIF and 28 close-cut tissue slices of in-house real H&E. Close-cut refers to adjacent tissue slices cut from the same block of tissue. The architecture and training strategy [named “Proposed-(8)”] were described in detail in a previous work from our team.²⁰ In our paper, the style transfer from MxIF to virtual H&E was performed using the pretrained “Proposed-(8)” model, exactly as described by Bao et al.²⁰

2.5.

Segmentation Model and Training Approach

The instance segmentation model we used on virtual H&E was a Hover-Net.³¹ We trained the Hover-Nets on virtual H&E input. The matching instance segmentation labels were created from another prediction model, not from a human rater or a human-in-the-loop labeling approach. The labels were created using predictions from the pretrained DeepCell Mesmer model on MxIF. The MxIF data were downsampled via cubic interpolation to $0.5\mu \mathrm{m}$ per pixel so that the Hover-Nets would work at the same resolution as our target real H&E data.^3,6 The training strategy that we selected was the default from the Hover-Net public GitHub repository: https://github.com/vqdang/hover_net.

2.6.

Classification Model and Training Approach

We selected ResNet-18²⁷ as our classification model. The model was initialized from the PyTorch 1.12.1 default ResNet-18²⁷ pretrained on ImageNet.³³ To remove negative effects from class imbalance and batch size differences between training and testing, we replaced each batch normalization³⁴ layer with instance normalization.³⁵ The instance normalization layers were implemented with PyTorch using default parameters.

The ResNet-18 was trained for classification on image patches of virtual H&E. The patches were resampled via cubic interpolation to a standard H&E resolution of $0.5\mu \mathrm{m}$ per pixel. Our patch extraction strategy involved selecting patches of size $41\times 41\text{pixels}$ centered on a nucleus or cell. All patches were individually intensity normalized between 0 and 1. A single class label was assigned to each patch, corresponding to the center nucleus or cell (Fig. 1). To address heavy class imbalance during training, at each batch, we ensured that each example had an equal likelihood of coming from any of the 14 classes. The model was trained for 20,000 steps using a batch size of 256, the Adam optimizer,³⁶ a learning rate of 0.001, cross-entropy loss, and the one-cycle learning rate scheduler.³⁷

All of the classification code was implemented in Python 3.8 using PyTorch 1.12.1 and Torchvision 0.13.1. In addition, all classification training and inference were performed using an Nvidia RTX A6000.

2.7.

Cross-validation and External Validation

We performed five-fold cross-validation separately on both the instance segmentation and classification models on the virtual H&E. To avoid data contamination, the training, validation, and testing data were split at the patient level (Fig. 3). To maintain consistency across folds, we specified that in each fold, the training data contained 12 patients, validation contained four patients, and testing contained four patients. To reduce bias, we always included data from both the ascending colon and terminal ileum, healthy and diseased, in each training, validation, and testing set. The folds and splits used for training, validation, and testing were the same for the instance segmentation model and the classification model.

Fig. 3

To balance the quality of the models trained on each fold of data, we split training, validation, and testing data at the patient level (each column is an image). In addition, each training, validation, and testing fold contained tissue without Crohn’s disease activity (normal) and tissue with Crohn’s disease activity (diseased) from the ascending colon and terminal ileum.

As mentioned in Sec. 2.3, on our virtual H&E we created labels for goblet cells, which we identified by finding the Muc2+ goblets associated with the cells, rather than by finding their nuclei. Goblets look very different from cell nuclei on H&E (Fig. 1). On the real H&E data that we selected for external validation, there were no labels on the goblets associated with goblet cells because these data came from the CoNIC Challenge, which only dealt with nuclei. Hence, we needed to train two instance segmentation models (Hover-Nets): the first to segment nuclei and goblets so that we could evaluate our approach on our virtual H&E, and the second to only segment nuclei so that we could evaluate the real H&E from the CoNIC Challenge.

We selected weights for evaluation based on the step with the lowest validation loss for each fold. The evaluation was performed on the segmentation and classification models both separately and combined.

Our entire cross-validation approach was performed on virtual H&E. To test how well our virtual-trained models generalized to unseen real H&E data, we used the publicly available CoNIC dataset.⁶ This dataset contained 4981 image patches from five source datasets. To reduce the effects of the domain shift between the stain color of our virtual H&E and the real H&E data, we trained separate CycleGANs to make the real H&E staining more similar to our virtual H&E.

The CycleGANs to correct H&E stain differences were trained using a sample of 20,000 patches of virtual H&E as well as the real H&E patches from the CoNIC dataset. Due to systematic staining differences from the different sites within the CoNIC dataset, a separate CycleGAN was trained for each site within the CoNIC dataset, which resulted in five CycleGANs (one for each site). The GANs were trained for 100 epochs using a batch size of 2 and a learning rate of 0.0002. All patches used in training the stain-correcting CycleGAN were of size $256\times 256\text{pixels}$ and at a resolution of $0.5\mu \mathrm{m}$ per pixel. The weight selection was determined based on the qualitative evaluation of the ability of each CycleGAN to realistically change the stain color in one direction, from the specific CoNIC site stain color to the virtual H&E stain color. The style-transferred patches of real H&E were then used for external validation of the segmentation and classification approach.

3.1.

Results for Nucleus/Cell Instance Segmentation

The instance segmentation was evaluated with precision, recall, $F1$ score, and intersection over union (IoU) of the true positives (Fig. 4). We defined true positives as nuclei/cells with an IoU greater than 0.25 between the prediction and the label. In the true positives, each label was only matched to a single prediction, and each prediction was only matched to a single label. If there was no matching label for a prediction, then the nucleus/cell was a false positive. If a label had no matching prediction, then the nucleus/cell was a false negative. Defining true positive, false positive, and false negative using a lenient IoU threshold of 0.25 was necessary to account for the differences in the label sets between our virtual H&E and real H&E. More specifically, the instance segmentation labels on our virtual H&E (inferred by the pretrained DeepCell Mesmer model) were not as closely cropped to the boundary of each nucleus/cell as the labels on the real H&E from the CoNIC dataset.

Fig. 4

Instance segmentation was better on recall than precision due to a lower number of false negatives than false positives. A drop in performance across metrics on real H&E highlights domain shift. The performance gap for intersection over union (IoU) of true positives implies that the size and shapes of predicted nuclei/cells more closely matched the virtual H&E than the real H&E. The five data points shown for each metric are from the five folds of cross-validation—each datapoint is the mean of the metrics computed at the nucleus level for that fold.

On the virtual H&E and real H&E, recall was better than precision (Fig. 4). In addition, on virtual H&E, when goblet cells were included, precision increased compared with when goblet cells were excluded. On the real H&E, there was performance variability due to the site.

In Fig. 5, we further investigated the instance segmentation on the real H&E through visualization. The model sometimes confused darker-colored style transferred structures with nuclei. In addition, the labels on the real H&E did not always contain every nucleus. Both factors contributed to a higher number of false positives.

Fig. 5

To counteract domain shift in H&E staining, instance segmentation was performed on a style-transferred version of the real H&E. In the column titled: “TP, FP, FN, TN”, nuclei were matched via a lenient IoU threshold of 0.25, as described in Sec. 3.1. True positive (green) and false negative (blue) nuclei were plotted based on the segmentations provided in the ground truth labels. Because of the lenient IoU threshold, false positive (red) nuclei (which were predictions rather than labels) could have some pixel overlap with ground truth labels. In areas where pixel overlap occurred between false positives (predictions plotted) and false negatives or true positives (labels plotted), we mixed the individual RGB colors, resulting in these additional plotted categories: “TP & FP” in yellow, and “FN & FP” in magenta. Lower precision on instance segmentation occurred when (1) the labels were not exhaustive and (2) style transfer made tissue darker and presented more like nuclei. Note that the colors for nuclei are not expected to match between the prediction and label columns—these colors were used to visually separate instances but do not have any classification meaning.

3.2.

Results for Virtual H&E Nucleus/Cell Classification

On virtual H&E, the classification accuracy of the ResNet showed some learning behavior for a subset of classes (Fig. 6). These classes were helper T, macrophage, enterocyte, epithelial progenitor, enteroendocrine, and fibroblast nuclei, as well as goblet cells. We show that this classification was stable when the classification was performed using the ground truth centroids, as well as when the centroids were predicted with the Hover-Nets.

Fig. 6

Although not all classes were learned from virtual H&E (shown for fivefold cross-validation), learning behavior can be seen for helper T, macrophage, enterocyte, progenitor, enteroendocrine, and fibroblast nuclei, as well as goblet cells. The classification model’s ability to learn MxIF label information (derived from 17 stain channels) on virtual H&E (three RGB channels) implies that there is a signal present in our virtual H&E to learn some of our fine-grained classification subtypes. From left to right, we show classification accuracy is stable when using label centroids and predicted centroids.

Looking in more detail at classification performance on virtual H&E, we computed the positive predictive value (PPV), negative predictive value (NPV), and prevalence (Fig. 7). When prevalence is low, we expect PPV to be low and NPV to be high. Likewise, when prevalence is high, we expect PPV to be high and NPV to be low. A cutoff for reliable classification could be selected based on PPV (Fig. 7 at 0.3). The cutoff of PPV at 0.3 was selected based on visual inspection of Fig. 7. The classes above this threshold are helper T, enterocyte, epithelial progenitor, fibroblast, and stromal (undetermined) nuclei, as well as goblet cells. When considering prevalence, we note that PPV is high for helper T and epithelial progenitor nuclei, and NPV is high for goblet cells and enterocyte nuclei. These results are relatively stable when using the ground truth centroids or predicted centroids. Qualitative results of nucleus/cell classification are reasonable and detailed in Fig. 8.

Fig. 7

Virtual H&E metrics of PPV, NPV, and prevalence are shown highlighting which classes can be reasonably identified. These metrics give more insight into the usefulness of model predictions than the accuracy shown in a confusion matrix. Helper T, enterocyte, epithelial progenitor, fibroblast, and stromal (undetermined) nuclei, as well as goblet cells, show reasonable learning. In addition, performance is relatively stable when comparing classification using ground truth centroids, predicted centroids (all nuclei/cells), or predicted centroids (all except goblet).

Fig. 8

Classification is visually consistent at multiple zoomed scales for enterocyte and progenitor nuclei, as well as goblet cells. Stromal (undetermined) nuclei are often incorrectly classified as a variety of immune cells. Nuclei in an area with immune activity are often correctly identified as helper T cells, although the predictions do include a non-trivial number of false positives for helper T. Crypts, visible here as oval structures at zoomed scales mostly outlined by collections of progenitor (orange) nuclei, are expected to be comprised of epithelial cells. However, progenitors around the crypts in the label set are not surprising because 96% of the progenitors in this dataset are positive for stains that identify epithelium.

3.3.

Results for Real H&E Nucleus Classification

We evaluated our virtual-trained models on real H&E, for five of the six classes that showed learning behavior on PPV and NPV (helper T, enterocyte, epithelial progenitor, fibroblast, and stromal undetermined). The sixth class that was not transferred was goblet cells. In our virtual H&E, we identified goblet cells by the goblet, not the nucleus. Because there were no appropriate corresponding labels in the real H&E data (only nuclei were labeled), we excluded goblet cells from this evaluation. We matched our selected virtual H&E classes to the closest matching and available parent classes on the real H&E. The matching scheme between the virtual H&E nucleus classes to real H&E nucleus classes was as follows: helper T to lymphocyte, enterocyte to epithelial, epithelial progenitor to epithelial, fibroblast to connective, and stromal (undetermined) to connective. Because we matched to parent classes, we could not compute PPV and NPV directly and so instead computed bounds on PPV and NPV.

Some of the information for some of the classes learned on virtual H&E can transfer to an external testing set of real H&E (Fig. 9). Looking in more detail, we compared performance on different datasets/sites of real H&E by using prevalence-normalized PPV (PPV divided by prevalence). We found that prevalence-normalized PPV showed instability across datasets/sites, despite the use of style transfer to address differences in staining (Fig. 10).

Fig. 9

Nucleus classification learned on virtual H&E transfers reasonably well to the real H&E external testing data, despite it being diverse and coming from multiple source datasets. Boxplots for positive predictive value and negative predictive value are shown for both virtual H&E and all source datasets from the real H&E on a subset of cell types. The bar plots denote class prevalence for virtual H&E and the closest matching class prevalence on the real H&E. A thin red connecting bar is used on real H&E to denote that the metrics available are upper bounds on PPV and upper and lower bounds on NPV. Because of the lack of real H&E labels at the granularity of our cells of interest, our NPV and PPV are bounded metrics based on the available parent class labels—this means that we can only compute an upper bound for PPV, which is optimistic, and an upper and lower bound for NPV, which allows for an estimate of NPV. Goblet cell performance is not shown for real H&E because the goblets were not labeled in the real H&E testing data, and so, there was no appropriate available matching label.

Fig. 10

To assess the stability of our virtual-trained cell identification and classification pipeline on real H&E, we evaluated our approach across the source datasets within the real H&E dataset (Consep, Crag, DigestPath, Glas, Pannuke). Because the real H&E does not have labels to the granularity of our shown subclasses (helper T, enterocyte, epithelial progenitor, fibroblast, stromal undetermined), the PPV that we computed is an upper bound based on the matching available parent class labels. To compare the upper bound PPV across datasets for each cell type, we normalized by class prevalence (upper bound PPV divided by prevalence), which varies across source datasets. For each cell type shown, we computed the Friedman test to assess for differences across datasets. Values of $p<0.05$ are denoted with * and $p<0.001$ with **. Statistical significance for helper T, enterocyte, epithelial progenitor, fibroblast, and stromal (undetermined) nuclei backs up what is visually evidenced by the boxplots—performance on the PPV upper bound varies across datasets.