Image matching has been a central issue in the field of computer vision and image processing for decades. It is normally based on the maximum similarity between two given images. We propose a similarity measure criterion based on structural similarity (SSIM) for image matching. We use the similarity measure criterion to compute the similarity of the feature points of two images to obtain the matching points. The results show that the correct matching rate of the proposed similarity measure criterion is higher than that of the normalized correlation coefficient. As for the mismatches in the initial set of matches, we use the proposed algorithm to compute the mean structural similarity (MSSIM) index of their neighborhood window image and eliminate those whose values of the MSSIM index are below the threshold. Then we use the geometric distribution of corresponding points in the image space to eliminate the mismatches that are difficult to eliminate using the SSIM algorithm. The experimental result shows that the proposed algorithm is superior to the random sample consensus algorithm in terms of mismatch detection and computational time.