We present a method to upsample low-resolution light detection and ranging (LiDAR) data using coregistered panoramic images collected in urban environments. The context is a mobile mapping vehicle equipped with a LiDAR scanner and spherical camera system, where the goal is to recover a three-dimensional model of the built environment. What makes this problem challenging is that as LiDAR samples become sparse, interpolation becomes difficult as connectivity between points can become ambiguous. The idea behind this paper is to use structural context and information in the associated panoramic images to determine the location of surfaces in the scene, and by inference, the surface membership of the LiDAR samples. With the latter determined, surfaces can then be reconstructed via interpolation using a function appropriate to the task at hand. We show that this method is effective in recovering details that are not apparent in the initial dataset. In the event that images are either uninformative or not available, we introduce a fallback method for surface segmentation that infers depth discontinuities from the properties of the surface triangulation. Experimental results are presented showing the performance of both methods on data acquired from an urban setting.