Discrimination and confidence error in detector-reported scores

Timothy D. Ross; Mark E Minardi

doi:10.1117/12.542161

2 September 2004 Discrimination and confidence error in detector-reported scores

Timothy D. Ross, Mark E Minardi

Proceedings Volume 5427, Algorithms for Synthetic Aperture Radar Imagery XI; (2004) https://doi.org/10.1117/12.542161
Event: Defense and Security, 2004, Orlando, Florida, United States

Abstract

A decision system may compute a score that reflects its confidence in one of its decisions. This paper considers methods for evaluating such scores. There is a class of measures-of-performance (MOPs) for each of the score's two roles: discrimination (how well it separates targets and clutter) and confidence (how well it predicts its own accuracy). Area-under-the-ROC and probability of error are considered as discrimination MOPs. Error in the posterior (EP) and normalized cross entropy (NCE) are considered as confidence MOPs. MOPs for the scores are assessed using Monte Carlo simulations where known score distributions are sampled, allowing comparison of true and estimated MOPs. Classical data-direct ROC estimates are found to be equivalent to those based on explicit distribution estimation using probability mass functions (pmfs). An alternative distribution estimation based on histograms is recommended for empirical ROCs, being accurate and avoiding the unnatural stair-step character of data-direct ROCs. Confidence MOPs are more difficult to estimate than discrimination MOPs and NCE estimates are especially poor. EP may be meaningfully estimated through histogram density estimates and it is recommended as a replacement for the AdaptSAPS binned EP confidence MOP.

Citation Download Citation

Timothy D. Ross and Mark E Minardi "Discrimination and confidence error in detector-reported scores", Proc. SPIE 5427, Algorithms for Synthetic Aperture Radar Imagery XI, (2 September 2004); https://doi.org/10.1117/12.542161

ACCESS THE FULL ARTICLE

INSTITUTIONAL
Select your institution to access the SPIE Digital Library.

SELECT YOUR INSTITUTION

PERSONAL
Sign in with your SPIE account to access your personal subscriptions or to use specific features such as save to my library, sign up for alerts, save searches, etc.

PERSONAL SIGN IN

No SPIE Account? Create one

PURCHASE THIS CONTENT

SUBSCRIBE TO DIGITAL LIBRARY

50 downloads per 1-year subscription

Members: $195

Non-members: $335 ADD TO CART

25 downloads per 1 - year subscription

Members: $145

Non-members: $250 ADD TO CART

PURCHASE SINGLE ARTICLE

Includes PDF, HTML & Video, when available