1.

Introduction

Receiver operating characteristic (ROC) analysis is a widely used and accepted method for performance assessment of two-class, i.e., binary, classification tasks. It investigates true positive fraction (i.e., sensitivity or recall) and false positive (FP) fraction (1-specificity) pairs over the range of the decision variable.¹ The area under the ROC curve (AUC) is commonly used as a summary metric of performance for binary classification tasks including those addressed by many artificial intelligence models in computer aided diagnosis (CADx) in medical imaging.² The AUC summarizes in a single value (with an uncertainty such as a 95% confidence interval, CI) the classification performance in the context of the entire range of thresholds for the decision variable. It has been reported, however, that ROC analysis alone may not be sufficient to thoroughly assess performance.³ In addition, in the case of imbalanced datasets, i.e., datasets with very low or very high disease prevalence, methods such as precision–recall analysis⁴ may be preferred. At the same time, a disadvantage of precision–recall analysis is that the precision (also known as the positive predictive value, PPV) and the area under the precision–recall curve depend on the disease prevalence in the dataset. In addition, some authors have noted that while PPV and the negative predictive value (NPV) can be investigated in a framework analogous to ROC analysis, these curves and prevalence-based metrics are not as easily interpreted.⁵

Moreover, when considering clinical use of CADx methods, the selection of an operating point (or operating points) is important.⁶ The selected threshold value for the decision variable trades off type I (FP) and type II (false negative, FN) errors. Reducing the number of FP and FN decisions is of clinical importance in medical imaging and in other fields.^7–9 In ROC analysis, several methods have been developed for the identification of a single “optimal” decision threshold for this purpose.^10,11 Most are constructed in the context of a “preferred” sensitivity or specificity (such as 95% for either), a combination of these,¹² or their extension into cost basis or utility.^2,13,14 The “preferred” target values for sensitivity or specificity depend on disease prevalence, the clinical task, and the tolerance for (or cost of) type I and type II errors. However, the methods to optimize the decision threshold in ROC-like analysis using prevalence are not yet well developed.^5,15–18

In this work, we therefore propose a framework of performance metric curves (PMCs) to simultaneously view sensitivity, specificity, PPV, and NPV as a function of the decision variable threshold. We investigated six different commonly used choices of operating points in the context of disease prevalence and variability in training and test sets. While our framework is applicable to performance evaluation of two-class classification tasks in general, we illustrate its use by applying it to the evaluation of classification performance for the distinction between malignant and benign breast lesions imaged with dynamic contrast-enhanced magnetic resonance imaging (DCE-MRI), using both a clinical dataset and simulation studies.

2.

Methods

2.4.

Statistical Analysis

The overall performance in the task of classifying (actual or simulated) lesions as malignant or benign was evaluated using ROC analysis and the nonparametric Wilcoxon method to estimate the AUC for each test fold. The median and 95% CI for AUC were derived empirically from all test folds. Note that no $.632+$ bootstrap correction of AUC^28,29 was needed in this work since the training and test folds were each sampled from completely separate training and test sets.

To demonstrate the utility of our proposed PMC framework, the performance metrics of sensitivity, specificity, PPV, and NPV were obtained at thresholds over the range of the decision variable at intervals of 0.01. (Note that intervals smaller than 0.01 did not provide any numerical advantage or change figure interpretations.) Six example desired operating points on the ROC curve were investigated (Table 3), and all thresholds for the decision variable corresponding to the desired operating points were determined using each training fold and then applied to the corresponding test fold. That is, this training set-determined decision threshold was applied to the cases in the corresponding test fold. The example operating points were not selected to be exhaustive but rather a selection of operating points used in the literature. In addition, the “best” choice of operating point(s) from a clinical perspective depends on many factors including the clinical task, population, and risk/benefits, and we did not investigate the superiority of any of these operating points in this classification task.

Table 3

Example operating points investigated in the study. A complete description of the operating points can be found in the references as noted when applicable.

Operating point
(A) Maximum sensitivity at minimum FP rate30
(B) A “preferred” target sensitivity of 0.95
(C) A “preferred” target specificity of 0.95
(D) An “optimal” sensitivity and specificity that $\text{maximized sensitivity}+\text{specificity}-1$ (based upon the Youden index)12,31
(E) An “optimal” sensitivity and specificity point that minimized the distance of the ROC curve from perfect sensitivity and specificity (i.e., the Euclidean distance)12,32
(F) The maximum of mutual information $I$, the amount of information expected to be gained by a diagnostic test 5,18,33

To construct the PMCs: (1) the values for the performance metrics were aggregated for the training and test folds, and median values and 95% CIs were empirically calculated and (2) the median values and 95% CI of the threshold of the decision variable corresponding to each operating point type were determined (Table 4). By combining these performance metrics in a single figure, PMCs allow for simultaneous assessment of sensitivity, specificity, PPV, and NPV over the range of a decision variable, and of the impact of decision threshold value selection for both the training set and the test set.

Table 4

Characteristics of performance metric curves.

Characteristic	For the training set	For the test set
Performance metrics	Median sensitivity, specificity, PPV, and NPV	Median sensitivity, specificity, PPV, and NPV
Classifier threshold	Median classifier threshold for the desired operating point(s)	n/a
Error estimates	95% CI of the decision threshold(s) corresponding to the desired operating point(s)	95% CI of all performance metrics across the range of the decision variable

CI, confidence interval.

In addition, to gain insight in the actual number of type I and type II errors, the median and 95% CI of the number of FP decisions and FN decisions were determined for each operating point of interest. A workflow figure describing the preparation of the datasets and the analysis is shown in Fig. 1.

Fig. 1

Workflow depiction of the preparation of the datasets and the construction of the PMCs. (a) Separation of the unique cases into training and test sets by year of acquisition after image acquisition, lesion segmentation, and extraction of radiomic features, (b) the clinical dataset study, and (c) the simulated datasets study. Dotted lines are used to indicate dataset and PMCs preparation using the training set, contrasted with using the test set. LDA, linear discriminant analysis; PPV, positive predictive value; NPV, negative predictive value; AUC, area under the receiver operating characteristic curve.

3.

Results

3.1.

Clinical Dataset Study

3.1.1.

ROC results from the clinical dataset study

The results in terms of ROC for the clinical test set ($N=435$ cases, cancer prevalence 74%) demonstrate the difference of operating points for a single prevalence in terms of the uncertainty of sensitivity and specificity for each operating point, as measured by both the median and the 95% CI of these (Fig. 2).

Fig. 2

Clinical dataset study ROC curves (solid line) with 95% CI (shading) and example operating points (see Table 3) with 95% CIs for the clinical dataset test folds (sampled from 435 clinical test cases) in the task of distinguishing between benign and malignant lesions. The area under the ROC curve (AUC), median and 95% CI, across the bootstrap folds is given as well. (Note that the shading (95% CI) along the ROC curve is very narrow, reflected in the narrow CI of the AUC).

3.1.2.

Performance metric curves from the clinical dataset study

The results in terms of PMC for the clinical test set ($N=435$ cases, cancer prevalence 74%) demonstrate the difference of operating points for a single prevalence in terms of the decision threshold for each operating point, as measured by both the median and the 95% CI of these (Fig. 3).

Fig. 3

Clinical dataset study (435 test cases) PMC showing sensitivity, specificity, PPV, and NPV as a function of decision threshold for the training set (median, dashed lines) and the test set cases (median, solid lines, and 95% CI, shading). Vertical lines indicate the median classifier threshold from the training set needed to attain the example operating points (Table 3), with the 95% CI of the threshold shown with error bars above the lines. The decision threshold is across the range of the classifier output, which is the posterior probability of malignancy.

3.1.3.

Performance metric curves interpretation on the clinical dataset study

The characteristics of PMCs (Table 4) can be interpreted for the clinical dataset (Fig. 3) as follows:

Performance metrics (training set and test set)

The median sensitivity, specificity, PPV, and NPV of the test set were found to be mostly similar to those of the training set across the decision variable. Exceptions were NPV over the range of $\sim 0$ to 0.3 and specificity over the range of $\sim 0.1$ to 0.6 for the decision threshold. There may be some degree of overfitting in these ranges, and operating points that utilize these metrics in these decision threshold ranges may be negatively impacted.

Classifier threshold (training set)

The median decision threshold associated with the Youden index and Euclidean distance was similar for this clinical dataset ($\sim 0.6$).

Error estimates (training set)

The classifier threshold for the Youden index varied more than for the Euclidean distance. Using the Euclidean index may be preferable over the Youden index due to the reduced error from training set variability if an operating point of their type is desired. There was substantial variability in the classifier threshold associated with the mutual information operating point.

Error estimates (test set)

The 95% CI of NPV, specificity, and sensitivity in the test set was substantial over the range of 0 to 0.3, 0.1 to 0.6, and 0.6 to 0.8, respectively. On the other hand, it was particularly small for sensitivity and PPV in the range of the target 95% sensitivity operating point.

3.1.4.

Extension of performance metric curves into clinical impact from the clinical dataset study

As noted above, PMCs can be extended into clinical impact by identifying type I and type II errors, and their uncertainty, in terms of operating point selection. For this dataset, the trade-offs in sensitivity, specificity, PPV, and NPV across the decision threshold range can be viewed in terms of impact on resulting FP (type I errors) and FN (type II errors) decisions (Fig. 4).

Fig. 4

Clinical dataset study (435 test cases) median number of (left) FP decisions (type I errors) and (right) FN decisions (type II errors) across the range of the decision threshold for each (lines) and 95% CI thereof (shading) for the test cases. The decision threshold is across the range of the classifier output, which is the posterior probability of malignancy.

This can specifically be seen for the different example operating points, (A)–(F) (Table 3), which demonstrate the impact on the median number of FP and FN decisions as well as their corresponding uncertainties (Fig. 5).

Fig. 5

Clinical dataset study (435 test cases) example operating points and error bars (95% CI) of the number of FP decisions (type I errors) and FN decisions (FN, type II errors).

For example, Fig. 5 shows the trade-off between using the Youden index and the Euclidean distance as the operating point in the clinical dataset for type I errors (median and 95% CI: 48 [28, 77] and 54 [39, 61], respectively). The median number of FP decisions was similar between the two operating points, but the precision was shown to be higher when the Euclidean distance was used. In addition, Fig. 5 demonstrates that when the target 95% specificity operating point was used for this dataset, while there is a very low number of type I errors (7 [5, 9]), the uncertainty in the already high number of type II errors was substantial (140 [120, 169]). In contrast, the 95% CI of the sensitivity was low in the decision threshold region for the target 95% sensitivity classifier, which resulted in high precision in type I and type II errors for this operating point (48 [41, 55] and 18 [14, 22], respectively).

3.2.

Simulated Datasets Study

3.2.1.

ROC results from the simulated datasets study

As expected, the ROC curves and AUC values in the task of distinguishing between malignant and benign cases in the simulated test set showed little difference by prevalence (Fig. 6). For example, AUC $(\text{median},[95\%\mathrm{CI}])=0.914$ [0.905, 0.921] for a $\text{prevalence}=0.1$, while $\mathrm{AUC}=0.912$ [0.902, 0.919] for a $\text{prevalence}=0.9$. Concurrently, the median (1 − specificity, sensitivity) of most example operating points demonstrated different levels of uncertainty among them (as indicated by the width of their 95% CIs), a result of the variability of the training set. The most substantial uncertainty across the simulated prevalences was observed in sensitivity for the mutual information operating point at a $\mathrm{prevalence}=0.9$ (95% CI: 0.428).

Fig. 6

Simulated datasets study ROC curves (solid lines) with 95% CI (shading) plus example operating points (see Table 3) with 95% CIs for the simulation test folds (sampled from 2000 simulated test cases) in the task of distinguishing between benign and malignant lesions. The area under the ROC curve (AUC), median and 95% CI, across the bootstrap folds is given as well. (Note that the shading (95% CI) along the ROC curve is very narrow, reflected in the narrow CI of the AUC). For $\text{prevalence}=0.1$ and 0.25, the operating point for the Youden index is obscured by marker for the operating point for the Euclidean distance. For $\text{prevalence}=0.5$, the operating points for the Youden index and the Euclidean distance are obscured by the marker for the operating point for mutual information.

3.2.2.

Performance metric curves from the simulated datasets study

The associated proposed PMCs (Fig. 7) show that while sensitivity and specificity as a function of decision threshold were similar across prevalence, PPV and NPV depended on prevalence. These observations were as expected but their simultaneous presentation gives insight into the impact of decision variable threshold selection on type I and type II errors by prevalence (unnecessary biopsies and missed cancers, respectively, in our example). Moreover, the additional presentation of the decision variable threshold obtained from the training folds (median and 95% CI) for the selected operating points, (A)–(F), provides valuable insight into how reliable the CADx/AI output is for a given operating point, and how it may change with prevalence for some operating points (e.g., 95% specificity and mutual information) more than others. For example, the median decision variable threshold for 95% specificity was 0.79, 0.82, 0.85, 0.91, and 0.95 as prevalence increased, while for mutual information it was 0.61, 0.59, 0.54, 0.28, 0.25 as prevalence increased. In contrast, the median decision variable threshold for 95% sensitivity was fairly stable at 0.28, 0.3, 0.3, 0.32, and 0.28 as prevalence increased.

Fig. 7

Simulated dataset study performance metric threshold (PMC) curves of sensitivity, specificity, PPV, and NPV as a function of the decision threshold (obtained from the training folds) for the simulated training folds (dashed lines, median) and the simulated test folds (solid lines, median, with shading, 95% CI). The decision threshold is across the range of the classifier output, which is the posterior probability of malignancy. Vertical lines indicate the median classifier threshold from the simulated training folds needed to attain the example operating points (Table 3), with the 95% CI of the threshold shown with error bars above the lines.

3.2.3.

Performance metric curves interpretation on the simulated datasets study

The characteristics of PMCs (Table 4) can be interpreted for the simulated dataset (Fig. 7) as follows:

Performance metrics (training set and test set)

For most prevalences, the median sensitivity, specificity, PPV, and NPV of the test set were similar to that of the training set, with the exception of NPV over the range of 0 to 0.5 for the decision threshold. This effect increased with prevalence. There may be some degree of overfitting in this range.

Classifier threshold (training set)

The median decision threshold remained similar across prevalence for the operating points of maximum sensitivity, 95% sensitivity, the Youden index, and Euclidean distance. The median decision threshold associated with the mutual information operating point changed substantially with prevalence, while that of the target 95% specificity operating point did so to a lesser extent.

Error estimates (training set)

The classifier threshold for the Euclidean distance varied more than for the Youden index, with increasing 95% CI as prevalence increases. Using the Youden index may be preferable over the Euclidean distance due to the reduced error from training set variability if an operating point of their type is desired. There was substantial variability in the classifier threshold associated with the mutual information operating point.

Error estimates (test set)

The 95% CI of NPV and sensitivity in the test set was substantial over the range of 0 to 0.5 and 0.6 to 0.8, respectively. It improved for sensitivity as prevalence decreases.

3.2.4.

Extension of performance metric curves into clinical impact from the simulated datasets study

As noted above, PMCs can be extended into clinical impact by identifying type I and type II errors and their uncertainty, in terms of operating point selection. The number of FP decisions (type I errors) and FN decisions (type II errors) was different by prevalence across the decision thresholds, as was the uncertainty (95% CI) thereof (Fig. 8). The uncertainty was largest for very low disease prevalence (FP decisions) or very high disease prevalence (FN decisions), as expected.

Fig. 8

Simulated datasets study median number of FP decisions (type I errors, left) and FN decisions (type II errors, right) across the range of the decision threshold (posterior probability of malignancy) for each prevalence investigated (lines) and 95% CI thereof (shading) for the 2000 simulated test cases.

For a given example operating point, the absolute numbers of FP and FN decisions differed with prevalence in terms of both median values and 95% CI (Fig. 9). For example, the number of FP and FN decisions decreased and increased with prevalence, respectively, for most operating points, especially maximum sensitivity and target 95% specificity.

Fig. 9

Simulated datasets study example operating points and error bars (95% CI) of the number of FP decisions (type I errors) and FN decisions (type II errors) at each prevalence investigated.

Notably, the 95% CI of type I errors for the maximum sensitivity operating point ranged from 572 FP decisions (out of 2000 test cases) at a $\mathrm{prevalence}=0.1$ to 64 FP decisions at a $\mathrm{prevalence}=0.9$, while the 95% CI of type II errors for the target 95% specificity operating point ranged from 20 FN decisions at a $\mathrm{prevalence}=0.1$ to 760 FN decisions at a $\mathrm{prevalence}=0.9$. However, these trends were not observed for the mutual information operating point.

4.

Discussion

There is great interest in the use of graphical tools to support understanding of classification performance and association with decision thresholds. Some of these include likelihood ratios,^34,35 $F$-measure curves to convey information related to precision–recall,³⁶ a variety of methods to display information related to cost^37–39 and utility¹⁴ (including indifference curves^40–42), and combinations of these, such as that of prevalence and cost of misclassification into “prevalence value accuracy” (PVA) curves.⁴³ Most of these rely on display of the performance in terms of pairs of metrics. Some, like the PVA curve, which plots minimum cost of misclassification based upon the number of FPs and FNs as a function of values of prevalence and unit cost ratio, are three-dimensional. The authors who proposed PVA curves noted that the three-dimensional nature of the figures could inhibit their usability. The PMC graphical presentation, however, maintains two-dimensional data visualization while still aiding in identification of the number of type I and type II errors at example operating points and the associated uncertainty in those errors in the test set. The two-dimensional nature of PMCs has an advantage of close association with metrics given in traditional ROC curves, reducing difficulty in transferring the conceptual framework to a new visualization.

The proposed PMC method utilizes bootstrapping in both the training set and the test set so in the graphical presentation the curves of sensitivity, specificity, PPV, and NPV are displayed as their median and 95% CI from the test set and are easily viewed with respect to the median and 95% CI of the thresholds for operating points from the training set, similar to our presentation of sensitivity and specificity in this format in previous work.^44,45 Interpreting performance metrics and example operating points in the context of variability enhances the interpretation of them and supports their translation to a clinical decision-making process.

The decision threshold associated with the Youden index was consistently associated with the intersection of sensitivity and specificity. Greiner et al. proposed that the intersection of sensitivity and specificity as a function of decision threshold, called a “two-group ROC” curve,⁴⁶ can be used to graphically identify a useful decision threshold. Our proposed method expands upon this concept by incorporating additional information from prevalence, the PPV and NPV. Similar to these “two-group ROC” curves, if desired, PMCs can additionally provide a reference point where sensitivity is equal to PPV and where specificity is equal to NPV. Simple algebra demonstrates that at this crossing point of sensitivity and PPV, the number of FN decisions is equal to the number of FP decisions: $\mathrm{FN}=\mathrm{FP}$. The same relationship is true for the decision threshold where specificity is equal to the NPV, i.e., $\mathrm{FN}=\mathrm{FP}$. This decision threshold is simple and intuitive: for decision thresholds less than the crossing point, the number of type I errors will begin to be greater than the number of type II errors. On the other hand, for decision thresholds greater than the crossing point, the number of type II errors will begin to be greater than the number of type II errors. While this decision threshold does not take into account different weights or costs for type I and type II errors, it could potentially serve as an additional intuitive, simple reference point to observe on PMCs.

The use of mutual information to determine a desired operating point is a method that can also incorporate prevalence information via PPV and NPV since it intrinsically depends upon prevalence, in contrast to the other operating points used in this study, which are independent of prevalence by definition. Hughes et al.¹⁸ recently investigated this operating point outside of the context of CADx but additionally note that the concept was introduced by Metz et al. as the term “information capacity.”³³ Their work suggests the utility of predictive ROC curves (PROC), which display PPV as a function of 1-NPV, as a graphical complement to traditional ROC curves. An index similar to the Youden index, called the PSEP index (which stands for “simple index of separation”¹⁶) by Altman and Royston, which measures an optimal position on the PROC curve has been previously proposed by separate investigators,^16,17 also outside of CADx. In the study presented here, implementing the maximum of mutual information as a criterion for decision threshold determination was associated with substantial uncertainty, as indicated by the 95% CI of the decision thresholds and the resulting numbers of cases that were classified as FN or FP. This may be due to the increased number of terms which are used for the measure. We were initially interested in combining the Youden index and PSEP as a possible, beneficial optimal decision threshold. However, preliminary results suggested that for our dataset, the decision thresholds associated with this were binormally distributed, at times corresponding to the threshold from the Youden index and at others to the threshold from the PSEP index, which reduced the usefulness of the approach. This result was not entirely unexpected because the optimal thresholds for the Youden index and for the PSEP index will most likely be different.¹⁸ Future studies will investigate other potential methods for combining information from the Youden index and the PSEP index.

There were some limitations to our study. For example, we demonstrated PMCs on both clinical and simulated data from a classification task related to the use of radiomic features extracted from MR images of breast lesions. While the utility of PMCs is expected to easily translate to other applications, the application of this method to radiomic features extracted from other modalities, such as full-field digital mammography, or to transfer learning from deep learning networks is a topic of future work. Second, we did not make quantitative comparisons of the impacts of training set variability, operating point selection, and prevalence for this classification task and this clinical dataset. Doing so will likewise be of interest in future work as we apply PMCs to other applications and classification tasks. Third, in our study, the disease prevalence in the training and test sets was the same. Prevalence scaling may be necessary in studies with training and test sets of different prevalence.⁴⁷ Fourth, the trends of the performance metrics in the training and test sets were very similar, due to the homogeneous nature of the clinical dataset. This similarity in trends will not be seen in more heterogeneous datasets, but this emphasizes the benefits of using PMCs in assessing classification. Finally, PMCs are information-dense and this could potentially reduce their utility somewhat. It could be useful to create a graphical user interface which displays example operating points one at a time and overlays their median and CI on the PMC figure itself.

5.

Conclusion

We have presented a graphical method and interpretation framework to extend performance evaluation in ROC analysis beyond AUC. PMC analysis

Thus, PMC figures and related analysis may support more comprehensive consideration of machine learning model design, enhancing traditional two-class classification performance evaluation methods.