A local density-based anomaly detection (LDAD) method is proposed. LDAD is a nonparameter model-based method, which utilizes the pixel’s local density in hyperspectral images as a criterion to determine the pixel’s anomalousness. In this method, the local density is calculated as a function of the spectral distance between pixels. Distinct from the statistical-based method, there are no assumptions made on the background distributions. Due to the pairwise distance calculation between pixels, LDAD’s computational complexity is quadratic to the total number of pixels. To improve the efficiency, an optimization strategy by pruning is implemented to reduce the unnecessary computational costs. Experiments on real hyperspectral image suggest that the proposed anomaly detector can achieve better detection performance than its counterparts, while keeping the computational cost relatively low by applying the optimization.