This paper presents a point cloud optimization method of low-altitude remote sensing image based on least square matching (LSM). The proposed method is designed to be especially effective for addressing the conundrum of stereo matching on the discontinuity of architectural structures. To overcome the error matching and blur on building discontinuities in three-dimensional (3-D) reconstruction, a pair of mutually perpendicular patches is set up for every point of object discontinuities instead of a single patch. Then an error equation is built to compute the optimal point according to the LSM method, space geometry relationship, and collinear equation constraint. Compared with the traditional patch-based LSM method, the proposed method can achieve higher accuracy 3-D point cloud data and sharpen the edge. This is because a geometric mean patch in patch-based LSM is the local tangent plane of an object’s surface. Using a pair of mutually perpendicular patches instead of a single patch evades the problem that the local tangent plane on the discontinuity of a building did not exist and highlights the edges of buildings. Comparison studies and experimental results prove the high accuracy of the proposed algorithm in low-altitude remote sensing image point cloud optimization.