Edge extraction from high spatial resolution (HSR) remotely sensed images is one of the essential tasks for image segmentation and object identification. We present an optimal Gabor-based edge detection method which mainly focuses on selecting optimal parameters, including central frequency and spectrum scale, for Gabor filter. The central frequency is automatically optimized by phase randomization and the human visual system-based structure similarity index. Next, the optimal spectrum scale is determined based on two-dimensional power spectrum density. The edge detection method is comprehensively discussed in the analysis of parameter sensitivity, overall performance, and comparative tests with several widely used methods. Qualitative and quantitative experimental studies, performed on six test images with various spatial resolution, show that the proposed method provides a promising solution to edge detection from HSR remotely sensed images.