Remotely sensed data, especially unmanned aerial vehicle images, provide more details about intensive ground objects. An algorithm with a solid capability to effectively handle this massive information is highly desired. The state-of-the-art algorithms proposed for building detection mainly focus only on buildings in use, ignoring those under construction. For buildings under construction, various types of soil are the main obstructions that impede building identification. Unmanned aerial vehicle images are used as experimental data for discriminating constructions (both in use and under construction) from other ground objects. A mask for potential constructions is created before the exact detection. A random forest classifier, together with a high dimensional textural feature, is used to remove soils that share similar texture characteristics with constructions. Experimental results suggest that our method can be widely used to detect construction (both in use and under construction) and has the ability to effectively handle heavy amounts of information from large-scale images with very high spatial resolution. It provides a method for soil exclusion from remotely sensed images with very high resolution.