Shadow detection plays an important role in remote sensing applications. Shadow should be detected with damage assessment algorithms, and it should be removed from the ground surface with semantic labeling applications. The procedure of a typical shadow detection method includes defining a shadow index and thresholding it, either automatic or manually. An automatic shadow detection method is proposed to facilitate the process of automatic applications. Specifications of shadow and nonshadow areas are analyzed to construct a new spectral–spatial shadow index. Spectral elements of the index can handle the dark shadow extraction. The spatial element of the index provides a high separability between light shadow and dark nonshadow areas such as water bodies. Index definition follows a segmentation algorithm to provide a segment-based analysis. Thresholding is a nonseparable part of shadow detection methods as it divides the region into shadow and nonshadow areas. To avoid high false positive or negative results, the proposed thresholding method is dependent on the enhanced bimodality test. Bimodal distribution can easily be thresholded by typical techniques such as the Otsu thresholding, whereas a special clustering-based thresholding is proposed in the unimodal distribution. The evaluations show a great improvement in shadow detection of very high-resolution RGB images.