2.1

Self-supervised learning of feature extractor

The motivation of self-supervised learning of each feature extractor is to speed up the whole process of classifier training. For a specific scale of image patches, a huge number of patches can be generated (e.g., with overlapped regular sampling from each large-size WSI slide), and thus training an CNN classifier (both the feature extractor and the classifier head) using the huge number of image patches would be very time-consuming. While a pre-trained and fixed feature extractor based on natural image dataset (e.g., ImageNet) can be used such that only the classifier head needs to be trained quickly for classification of pathological image patches, the ability of the pre-trained feature extractor based on natural images may not be powerful enough for extraction and representation of pathological image features. To make full use of the huge number of pathological image patches and meanwhile speed up the training process, we propose using a relatively smaller number of image patches to train the feature extractor in a self-supervised manner, and then using the huge number of patches to train only the classifier head (often a two- or three-layer MLP), with the pre-trained feature extractor fixed.

In this study, the contrastive learning strategy was applied to self-train each feature extractor, and a novel way of forming positive and negative pairs of image patches was proposed for contrastive learning. Specifically, positive pairs are generated not only based on two image augmentations of each image patch (Figure 1 left, from original patch x_i to its augmented versions ), but also novelly based on augmentations of two neighboring image patches (Figure 1 left, from two original neighboring patches x_i and x_i to their augmented versions ). Augmented neighboring image patches are considered as positive pairs because adjacent image patches are from the same tissue which is composed of cell groups with similar morphology and same function and therefore often have similar morphological features. On the other hand, for negative pair generation, the MoCo method¹¹ was adopted, by which a large number of negative pairs can be generated based on memory buffer storing feature representations of image patches from both current mini-batch and hundreds of previous mini-batches. In general, the corresponding two patches for each negative pair are often from two different WSI slides or from different locations of one WSI slide. Based on the generated positive and negative pairs with a specific patch scale, the corresponding MoCo model can be well trained. Consequently, the trained encoder from each well-trained MoCo was kept as the feature extractor for the corresponding patch classifier (Figure 1 right).

Figure 1.

Overview of the proposed method. Left: Construction of positive pairs () and (). Right: the training process of a EBV classifier, with the feature extractor trained by self-supervised learning (top) and then the classifier head (a MLP) trained with labelled image patches of a unique magnification (bottom).

2.2

Multi-scale ensemble classifier

Once a feature extractor was self-trained with image patches of a unique patch scale, the feature extractor can be used to extract a feature vector representation for each labelled image patch with the same scale, where the labelled image patches are from the annotated EBV regions from each WSI slide. Then, built on the feature extractor, a MLP as the classifier head can be trained for prediction of each patch into EBV or non-EBV category. Compared to training a CNN classifier (i.e., the CNN feature extractor and the final FC output layer), training a two- or three-layer MLP is much more efficient because of much smaller number of model parameters and smaller-size patch representation (i.e., feature vectors rather than image patches as input).

Once each MLP is trained based on the corresponding feature extractor for image patches of a unique scale, then the feature extractor and the MLP are combined as a CNN classifier. With three different image magnifications, three such CNN classifiers can be ensembled for prediction of any new image patch (Figure 2). It is worth noting that during inference, at each location (often representing a local region) in a new WSI, three image patches with different magnifications are cropped and fed into corresponding classifiers, resulting in three output probability predictions. The average of the predictions is used as the final prediction probability for the location in the WSI slide.

Figure 2.

Multi-scale ensemble classifier, consisting of three individual classifiers, each of which is associated with one magnification.

3.1

Experimental setup

Three pathological image datasets of gastric cancer were used for evaluation of the proposed method, including one public dataset TCGA and two prividate datasets SYSUCC-Internal and SYSUCC-MultiCenter (Table 1 for more details). While SYSUCC-Internal was used to self-train the feature extractors, SYSUCC-MultiCenter and TCGA were used to train the classifier heads and evaluate the performance of the classifiers respectively. SYSUCC-MultiCenter and TCGA were respectively split into five folds at slide level and the five-fold cross-validation strategy was adopted for evaluation, each time with three folds for MLP training, one fold for validation, and the remaining fold for testing. The average area under the ROC curve (AUC) over five testing folds were reported on SYSUCC-MultiCenter and TCGA respectively. Note that only image patches from tissue regions in SYSUCC-Internal were regularly extracted for feature extractor training, and patches from annotated tumor regions in SYSUCC-MultiCenter and TCGA for classifier head training and classifier evaluations.

Table 1.

Statistics of datasets.

The three feature extractors have the same model backbone, i.e., the ResNet-50 convolutional layers. Since the MoCo v2¹² method was used train the feature extractor, a two-layer MLP projection head as in MoCo v2 was attached to the feature extractor to reduce the dimension of feature vector from 2048 to 128. Each feature extractor was trained by the MoCo method with suggested hyper-parameters. SGD optimizer with batch size 128 was used to train each feature extractor for 100 epochs. Each image patch was pre-processed by the Vahadane’s method¹³ for color normalization and randomly cropped to 224×224 pixels, and general data augmentations was performed, including random rotation, horizontal and vertical flip, brightness changes, contrast and saturation changes.

The three classifier heads also shared one backbone, i.e., a three-layer MLP with layer output dimensions being 2048, 2048 and 2 respectively. Each MLP was trained by the SGD optimizer (batch size 256, momentum 0.9) over 100 epochs with the initial learning rate of 0.001. In the first 20 iterations, the learning rate was adjusted by linear warmup, and then dynamically adjusted by cosine annealing in the remaining epochs. In terms of data processing, each patch from TCGA and SYSUCC-MultiCenter was center cropped into the size of 224×224 pixels and fed to the corresponding well-trained feature extractor to obtain the feature vector as the input of classifier head.

3.2

Performance evaluation

Both patch-level and WSI-level AUCs were reported with SYSUCC-MultiCenter and TCGA respectively. For slide-level AUCs, the classification probabilities of all patches in each WSI were averaged as the classification probability of the corresponding WSI slide. The original MoCo and MoCo v2 were used as two self-supervised learning baselines for comparison. In addition, simultaneously fine-tuning of each pre-trained feature extractor (initially based on ImageNet dataset) and training classifier head was also used as a baseline (‘Finetune’), where batch size was set 32 considering the large memory consumption. From Table 2, it can be observed that on TCGA, our method achieved the best patch-level and slide-level performance at 2.5× magnification. On SYSUCC-MultiCenter, our method achieved the best patch-level performance and comparable slide-level performance at 10× magnification. This clearly demonstrates that although the proposed self-supervised learning can help improve classification performance, patch scale affects model performance and its effect may vary over different datasets. Compared to those single-scale classifiers which were affected by patch scales, the proposed multi-scale ensemble classifier (Table 2, last row) was shown to be more stable, achieving either best patch-level and slide-level performance on SYSUCC-MultiCenter or comparable performance on TCGA.

Table 2.

Performance comparison between our method and baselines.