2.1.

Thermal Camera Intrinsic Calibration

In order to use a camera for photogrammetric purposes, its calibration¹⁵ should be carried out to determine the interior orientation parameters and to compensate for the effects of lens geometric distortion. In the applications considered in this study, neglecting the correction of distortions may lead to a significant worsening of the final quality of the textured model.

In the case of IRT sensors, the pinhole camera model can be assumed and calibration applied by using standard photogrammetric methods. However, IRT cameras are not designed for metric purposes and their calibration may not be an easy task for a series of reasons (see also Ref. 14):

All these aspects should be taken into account for planning the calibration project in order to fix a proper set of “best practice rules” to be generally adopted.

The camera AVIO TVS700, further used in this work for IRT surveys, was calibrated by using the Brown’s model,¹⁶ which is based on eight parameters (principal distance, principal point coordinates, three coefficients for radial distortion compensation, and two parameters for decentering distortion). These parameters can be estimated by using a proper calibration target set (Fig. 2), which must be imaged from different positions. An important aspect concerning the target set is that its points have to be clearly visible in the IR images. The solution here adopted is based on a set of 40 iron nails fixed in a wooden structure. When exposed to sunlight, nails warm up faster than wooden background and become clearly visible in IR images.¹⁷ Each of these nails was also measured with a first order theodolite Leica TS30. The 3-D coordinates of all nails and the corresponding image coordinates, manually measured on all images are included in a bundle adjustment, whose solution includes also the Brown’s model calibration parameters. The average precision of this procedure can be evaluated through the estimated statistical quality parameters computed along with the bundle adjustment (see Sec. 2.3 for more details on bundle adjustment technique, which is also adopted in the image registration stage). In the case under consideration, the computed set of parameters allowed us to transform points from the images to the real word with an average residual error of $\pm 2\mathrm{mm}$.

Fig. 2

The wooden panel with the iron nails used for IR camera calibration as it is depicted in an RGB image (a) and in IR image (b), respectively. The distortion effects are quite large in the IR image.

Some other aspects of the calibration procedure need a special remark. First, in order to correctly estimate the calibration parameters, the points used for the bundle adjustment computation should be evenly distributed on the image, covering also those regions close to the image borders where distortions are usually larger. In addition, the inclusion in the calibration block of at least two rolled images (i.e., in portrait position, with the others in landscape orientation) can help to reduce correlations between the estimated parameters. Indeed, high correlation between a couple of parameters means that they have not been estimated in a reliable way. Consequently, one of these parameters should be excluded from the final set. In particular, in the presented case, the radial distortion parameters $k_2$ and $k_3$ showed a correlation $>90\%$. For this reason, in the final estimation, the parameter $k_3$ was excluded from the estimation and only seven parameters from the Brown’s model were used. As the autofocusing systems of the camera could not be turned off, the images were acquired more or less from the same distance from the polygon. However, the principal distance estimation in the case of real surveys still remains a problematic aspect. Finally, Fig. 3 highlights the difficulty of acquiring strongly convergent images due to the limited FoV of IR cameras compared with RGB ones.

Fig. 3

Visualization of camera stations for the calibration of an infrared thermography (IRT) camera (a) and for a standard RGB camera (b).

In this application, the camera calibration parameters were estimated after including in the bundle-adjustment some ground control point (GCP) obtained from theodolite surveying. However, the estimation can be carried out also in a “free-net adjustment” without any GCP.¹⁸

2.2.

Reconstruction of the Building Model

A 3-D model of building is necessary for mapping thermal IR images. Such a model can be derived either starting from existing drawings or from surveying. In the case of large constructions or historical building where the façades can feature complex architectural elements, photogrammetric and TLS techniques may provide high-resolution models without requiring the direct contact with the object. Both techniques can also be successfully applied in an integrated way to exploit their complementary key properties. In Refs. 19 and 20, this topic is described in more detail. The most complete product that can be obtained from photogrammetry/laser scanning is a vector 3-D model. Unfortunately, this would require a largely time-consuming manual work, especially in the case of complex historical buildings where façades usually have an abundance of decorations. An alternative approach can be based on the use of point clouds which are directly achievable by means of TLS or alternatively can be derived from image matching techniques applied to photogrammetric images.²¹ A point cloud is a set of 3-D points describing the surface of an object. Points are not usually related by any topological relationship but, if their density is sufficiently high to sample the surface in a nonambiguous way, they give a useful model for mapping IR images. Usually, problems rise up in correspondence of edges, corners, and other features, where the spatial resolution of the point cloud is not enough to give a complete description of geometry. This is the motivation of why photogrammetry can increase the quality of point cloud because linear features like edges can be extracted in an easier way from images. Point clouds can be directly mapped by assigning to each point the radiometric value found in the corresponding positions in the images. On the other hand, this method is adequate only for rapid and general visualization and not to map details. Moreover, it suffers from low resolution of the point cloud. To overcome this drawback, raw point clouds derived from field surveying are further processed and a triangular mesh or triangular irregular network (TIN) describing the surface of the object is achieved. In the polygonal model, the raw points are connected with triangles which do not intersect or overlap.¹¹ The triangularization process can be carried out in an automatic way, but usually it requires a final editing work. In Fig. 4, an example of a part of laser scanning point cloud at different stages of the modeling process is reproduced (raw point cloud, TIN surface obtained from automatic processing, TIN after editing).

Fig. 4

Three steps in the laser scanning process: (a) row point cloud, (b) automatically generated mesh with wrong triangles, (c) edited model.

2.3.

Registration of Images and 3-D Model

In the last phase, thermal images have to be mapped and mosaicked on the TIN model of the building. Given a calibrated camera as described in Sec. 2.1, this process is made up of two stages. The first one is the reconstruction of the spatial position of each image with respect to the 3-D model. This task is called image registration and will be described in this subsection. In a second step, the content of each image is projected over the 3-D model (texturing), as described in Sec. 2.4. Typically, the same geometric transformation adopted for registration is also adopted for texturing. In theory, registration and texturing of IR images do not differ from the same procedures for the case of RGB images. In practice, it is a much more complicated task to accomplish due to the smaller FoV, the smaller format, and the different radiometric content.

In many applications that carried are out in standard thermal surveys of buildings, registration is carried out by using homographic transformations, one per each image to map.²² Estimation of homography requires the identification of at least four corresponding control points (CPs) on both image and surface. Not only does this method have strict limitations regarding the surface’s geometry that must be almost flat, but it can also be difficult to apply in the case where image depicts a homogeneous area where no CPs can be found. This situation quite frequently occurs in the case of IR images due to the small format and the homogeneous temperature over the region depicted in the image (Fig. 5).

Fig. 5

The basic principle of a single image texture mapping based on space resection: the method works only if the object and the thermal image allow the operator to manually measure a set of corresponding points.

A more comprehensive approach to image registration is based on the collinearity equations,¹³ which are normally used in photogrammetry to describe the spatial central perspective transformation approximating the process of image formation. Such equations relate a 3-D point in the object space $P_i$ $(X_i,Y_i,Z_i)$ to its corresponding projected point $p_{ij}$ $(x_{ij},y_{ij})$ on the image (indexes $i$ and $j$ refer to point and image, respectively). Collinearity equations are parameterized through the so called exterior orientation (EO) parameters, which include three orientation angles used to build up a spatial rotation matrix $R_j$, and 3-D coordinates of the perspective center $(X_{0j},Y_{0j},Z_{0j})$ of image $j$:

In Eqs. (1) and (2), image coordinates $x_{ij}$ and $y_{ij}$ have been previously corrected for distortion (see Sec. 2.1). Indeed, all cameras have to be calibrated before computing image registration although it would also be possible to do both registration and intrinsic calibration together; this last option, however, would require some strict conditions on the layout of the photogrammetric block, which usually differs from the one adopted during the surveying stage.²³ Calibration also provides the principal distance $c_j$ and the coordinates of principal point $(x_{0j},y_{0j})$, which are used in the collinearity equations [Eqs. (1) and (2)]. The registration of each image (space resection) can be directly performed by using these equations, whereas some GCPs should be independently measured (e.g., by using a theodolite or directly from the point cloud derived from TLS). Coordinates of GCPs have to be measured on the images as well. By using at least three GCPs, it is possible to write down a system of three pairs of Eqs. (1) and (2) for a total number of six equations. This system, after linearization, allows one to solve for the six EO parameters related to the image under consideration. In practice, the number of GCPs should be redundant to make it possible to work out the solution by least squares;²⁴ this solution also helps to strengthen the numerical stability of the estimation. Another problem comes out from the linearization of Eqs. (1) and (2), which requires the knowledge of an approximate solution in advance for computing the Taylor expansion; this problem, although crucial in the whole process, is not adescribed here, but reader is referred to Ref. 13.

An alternative approach which is usually adopted for space resection of thermal images is the projective model provided by direct linear transformation (DLT).²⁵ In this case, the relationship between two-dimensional (2-D) and 3-D coordinates is modeled by means of 11 algebraic coefficients, which implicitly incorporate both calibration and EO parameters. By this method, each image is processed individually by measuring at least six GCPs; no approximations for the unknown parameter are necessary because DLT equations can be linearized in a straightforward manner. Sometimes, DLT is used to provide approximations for the successive application of the collinearity equations.

Even if this space resection is the most popular technique for registration of IR images and it is implemented in several commercial packages, it suffers from at least three main drawbacks: (1) it is time consuming because each image has to be processed independently from others; (2) in the case of IR images with poor texture, the identification of the minimum number of GCPs may be difficult and the final orientation accuracy may be questionable; and (3) mosaicking of IR images on the 3-D model is prone to show discontinuities in overlapping areas.

To overcome these drawbacks, usually in photogrammetry, the bundle adjustment approach is applied. Basically, several images are registered together into the same reference system through the solution of a linearized system of Eqs. (1) and (2). Also some approaches based on DLT exist, but are less popular. The unknowns of the system are the six EO parameters per each image, whereas the intrinsic calibration parameters are usually determined in an independent way. Some GCPs are used to setup the reference system where the registration is needed. In this case, GCPs have to be shared with the laser scanning point cloud. However, GCPs should not necessarily be measured on all images, mitigating the problem (2) in space resection methods. They have to be positioned to fix a general 3-D similarity transformation (seven parameters) that setup the reference system. This call for using at least three GCPs is not aligned. As a best practice, GCPs’ number should be larger and they should be spread out on the overall surveyed area. This has the advantage of either raising the numerical redundancy of solving system of equations or improving the stability of the solution. On the other hand, if the number of GCPs is reduced, the system would show a rank deficiency that needs to be filled by using the so called tie-points (TPs). TPs are corresponding points measured on at least two images, giving rise to at least four additional Eqs. (1) and (2). TPs do not need to be measured in the object space because their object coordinates are also computed during the adjustment. It is clear that every TP in two images introduces four equations and three new unknowns (the 3-D coordinate in the object space), resulting in a positive contribution to the global redundancy of the system. This contribution is much higher if a TP is seen in more than two images. Consequently, a good configuration for the bundle adjustment requires multiple TPs. This solution not only increases the global redundancy, but also improves the local redundancy of single observations, with a greater chance to detect errors. The block geometry should be properly selected as it would require large overlaps to find many multiple TPs, whose measurements can be carried out in a manual or automatic way.²⁶ The second option is possible only if the images have an adequate texture. In addition, bundle adjustment also has the advantage of exploiting common GCPs between images, reducing the total number of points to be measured on the images. However, in many cases, the configuration of IR camera poses is quite different from the one of photogrammetric blocks made up of RGB images. This is strictly related to the characteristic of close-range IR sensors: the narrower FoV increases the number of images to be processed and reduces the baselines (i.e., the distance in space between the camera stations) between consecutive images, causing a worsening in the orientation accuracy. Moreover, in most cases the baselines are so short compared to the camera/object distance, that the numerical stability of the solving numerical system is dropped.

For the sake of completeness, some specific acquisition systems designed to speed up the thermal image orientation can be also found in the literature. A “bi-camera” system, coupling a traditional RGB camera with an IR sensor, can be also used. In this case, only RGB images are processed in the bundle adjustment deriving their EO parameters. Then, from the knowledge of the relative orientation between RGB and IR cameras, which could be determined in a calibration phase, thermal images are oriented.¹⁴ However, the bi-camera system can be quite bulky and not always applicable. In the case of mobile mapping systems integrating IR cameras, the image orientation can be performed in a two-step way. Thermal images are first relatively oriented using Neister’s “five point algorithm.”²⁷ Then, the obtained sequence is matched to the recorded GPS camera path and then to the building façade.²⁸

The solution presented here is based on a global photogrammetric bundle adjustment combining both IR and RGB images, which tries to overcome some drawbacks connected to the standard space resection and to the bundle adjustment of IR images only. The procedure starts with the acquisition of an adequate set of RGB images with a calibrated camera, meaning that the image block should satisfy the standard requirements of a close-range survey in terms of image overlap, baseline between consecutive images, and image resolution.²⁹ For instance, in the case of a planar-like façade, a simple strip of images with an overlap of about 80% can be a good choice as it allows one to find TPs on three to four images. In building surveys, some factors like occlusions caused by surroundings, buildings and/or trees, logistics limitation, and the like might influence the block design. Consequently, a trade-off between concurring requirements has to be defined. Then, RGB images are oriented within a standard photogrammetric bundle adjustment, which is based on a set of TPs measured on the images and some GCPs that are used to register the project in the reference system of the laser scans. An important consideration deserves to be mentioned: TPs individuated in this first step will then be adopted for registration of IR images. For this reason, their spatial distribution should be carefully planned. In particular, TPs should be preferably measured in correspondence of elements that are clearly visible in both RGB and IR images (e.g., windows’ and doors’ corners). This strategy may result in increasing the processing time of the bundle adjustment of the RGB images, if compared to a standard photogrammetric project where the integration of IR images is not needed. For example, in our application up to more than 100 TPs were individuated on a single RGB image. On the other hand, the larger number of TPs will help the registration of the IR data.

Once RGB images have been registered by means of bundle adjustment, IR images can be added to the block by measuring those TPs which have been previously determined in the RGB images. A final combined bundle adjustment including all data is actually carried out to obtain the EO parameters of all images simultaneously. To account for the different precision of TPs measured in RGB and IR images in the computation of least squares solution, different kinds of observations are properly weighted. Basically, weights of image coordinates in the RGB images are derived from the statistics of bundle adjustment including the RGB images only. Weights of IR images are assigned on the basis of the pixel size of the adopted thermal camera. This combined adjustment has several advantages:

On the other hand, TPs have to be measured manually both in RGB and IR images and, in addition, they have to be measured in abundance in RGB images in order to cope with the limited FoV of IR camera.

An application example of this orientation technique is the “Nave” building of Politecnico di Milano (Fig. 6).

Fig. 6

The Nave building of Politecnico di Milano University.

First, 17 RGB images acquired with a Nikon D80 with a 20 mm Sigma lens were registered within a bundle adjustment. In this project, 16 retro-reflective targets measured with a theodolite Leica TS30 and also employed for the registration of laser scans were used as GCPs during bundle adjustment. As can be seen in Fig. 7(b), the irregular distribution of both RGB (red cameras) and IR images (blue cameras) depends on the presence of several occlusions due to obstacles like other buildings and trees. Then, 65 IR images were included in the bundle adjustment by using more than 600 TPs previously measured in the RGB images. Thermal images were acquired with two different IRT cameras (AVIO TVS700 and NEC TH9260) and were included in a unique adjustment with different sets of intrinsic calibration parameters. The high number of TPs used is connected with the narrow FoV of both thermal cameras that limits the number of elements clearly measurable in each image (Fig. 7). Statistics of the combined bundle adjustment show a final RMS of about 1.2 pixels. This result can be considered as acceptable due to the low geometric resolution of IR images. In fact, the ground sample distance of the thermal images was about 1 cm, while the one of RGB images was 1 mm, meaning 1 order of magnitude of difference. An important remark concerns the identification of TPs on RGB and IR images. As previously mentioned, TPs identified in RGB images are abundant [Fig. 7(a)], resulting in a higher increment of the processing time of RGB images. However, TP localization is not a problem. On the other hand when IR images are used, the presence of repeated elements, such as windows or doors [Fig. 7(c)], may determine some ambiguities in the identification of TPs. For this reason, in the surveying phase some notes should be taken. This measurement is quite laborious and can last several hours and, in the case of large projects, even days.

Fig. 7

In the upper row (a) an RGB image with the measured tie points (TPs) (red crosses); and a 3-D view showing camera poses of both RGB (red) and thermal (blue) datasets (b). In the lower row (c), some IR images used for texturing the 3-D model with measured TPs; red lines are a graphical representation of the residuals after registration through the bundle adjustment (a magnification factor 50 is applied).

2.4.

Texture Mapping of the IR Images

Once IR and RGB images are registered in the same reference system of laser scanning point cloud, they can be mapped and mosaicked on the model of the façade. The process aimed at applying artificial or reality-based textures to a 3-D model is known as texture mapping. In the former case, a predefined pattern is applied to each elementary surface which forms the digital surface model of the object. In the latter case, an image captured on the real object has to be mapped on the corresponding portion of the object model. Several works in the fields of photogrammetry, computer vision, and computer graphics focused on how to obtain the automatic photorealistic texture mapping from RGB images.^30–33 In many cases, the general workflow is similar. As a preliminary stage, images and surface model (e.g., a 3-D TIN), which have been recorded independently, are registered in a common reference system. Then, the texture mapping process can be performed. As each portion of the façade is usually visible in more than one image, the second step is the implementation of an automatic criterion to select the image that is more suitable for mapping that area. This choice should take into account the image quality and the presence of occlusions. For instance, images used in the mapping phase are selected to maximize a score function or to optimize some geometric parameters such as viewing direction, level of detail, distance from the object, etc. Finally, the pointwise correspondences between pixels in the image and points on the elementary surface to be textured are established. In the application presented in this paper, the elementary surface is each triangle of the TIN generated from the laser scanner point cloud, and Eqs. (1) and (2) are used for mapping. Eventually, the obtained textures should be corrected for radiometric differences due to different lighting conditions.

Some commercial software packages (e.g., 3-D Studio Max®, Geomagic® Studio, PhotoModeler® Scanner, ShapeTexture®, etc.) accomplish texture mapping. However, in many cases, the process is manual, meaning that the measurement of corresponding points ($\text{images}\leftrightarrow \text{model}$) is carried out manually for each image. Also in the case that the image orientations can be uploaded in the software allowing a more automated processing, a significant editing phase is needed to fix problems related to occlusions, variations in lighting, and camera settings. On the other hand, when a small number of IR images or a simplified mesh model is used, these problems are less relevant. But when the number of IR images to adopt is large (also more than 50 to 100 images) and the façade model presents a high degree of complexity, a more efficient approach should be applied. The algorithm adopted here for texture mapping has been designed in order to minimize self occlusions and texture assignment problems.³⁴

The procedure that identifies occluded areas is called visibility analysis. Figure 8 helps to understand the basic idea of this method: if triangle $T_i$ is not visible from the viewpoint $I_k$, there will be at least another triangle $T_j$ which occludes it. In the image space, the occlusion is reflected by an intersection between the two back-projected triangles. In order to understand which is the occluded triangle and which is the occluding one, the reciprocal distance between the vertices of the triangles and the image projection center is calculated. The farther triangle is occluded, whereas the closer is the occluding one.

Fig. 8

The visibility analysis principle: (a) the two triangles do not occlude each other and no triangle intersection exists in the image space; (b) the green triangle occludes the magenta one (blue region intersection in the image space).

From a computational point of view, this phase is very demanding because the analysis should be repeated for all the triangles composing the whole mesh (computation complexity $O^2$). This might be a problem in the case of models composed of millions of triangles. To reduce the number of triangles to be checked, three different strategies have been implemented: (1) view frustum culling, (2) back-facing culling, and (3) triangle distance culling. The view frustum culling is based on labeling as occluded all triangles outside the camera view frustum [Fig. 9(a)] since they are not directly visible from viewpoint $I_k$. The camera view frustum is the region of space where objects might have been imaged from a given camera station. In this case, it corresponds to the FoV of the adopted camera. The nearest and farthest planes limiting the camera view frustum are, respectively, the plane of the camera sensor and the plane at infinity. The classification of any triangle outside this region as “occluded” limits the number of visibility analyses to accomplish when dealing with triangles inside the view frustum. A further reduction of the number of triangles to check can be obtained using the back-facing culling [Fig. 9(b)]. If the mesh is oriented, the normal vector for each triangle is defined. All triangles whose normal forms an angle $>90\mathrm{deg}$ with the camera viewing direction are geometrically not visible from image viewpoint $I_j$ because they are occluded by the triangles facing toward the camera. In the case where the angle is smaller, it is labeled as potentially visible [blue face in Fig. 9(b)]. Depending on the dataset structure, the number of testing triangles can be significantly reduced.

Fig. 9

View frustum (a) and back-facing culling (b) principles.

The triangle distance culling strategy (3) is based on the observation that in the visibility analysis, a lot of time is spent to analyze those triangles whose back projection in the image fall quite far from the reference triangle. On the other hand, triangles that back projected in the image are close to the reference one will have a higher probability of intersection. For this reason, we limit the intersection test only to a certain number of nearest neighbor triangles with respect to the reference one. Their number depends on the quality of the geometric model and on the image scale.

The second important aspect that has been implemented in the procedure for texture mapping concerns the choice of the best image to be used for texturing in overlapping areas. This choice is performed evaluating two texture quality parameters: the resolution of the image in object space and the camera viewing direction. The image whose quality parameters are better ranked is used as the source for texturing. Finally, the texture coordinates for the triangle are calculated by back projecting the triangle coordinates in the object space onto the selected image by means of Eqs. (1) and (2).

The described algorithm works independently for each triangle. Therefore, depending on the illumination conditions during acquisition, sharp color differences in the textured model may appear. This is expected especially when two different images are used to map adjacent triangles of the model. Nonhomogeneous texture can be reduced by using a color/brightness correction. The developed color/brightness correction is not performed in the traditional RGB space but in the $L*a*b*$ color space.³⁵ Indeed, unlike the RGB color model, $L*a*b*$ color space is designed to approximate human vision. In particular, its $L$ component closely matches human perception of lightness, whereas $a$ and $b$ channels define the color plane. This distinction can thus be used to make accurate color balance corrections by modifying both $a$ and $b$ components and to adjust the lightness contrast using the $L$ component. The color and brightness corrections are estimated by exploiting common small areas detected in multiple images.

The procedure for texture mapping IR images has been applied to the Nave building of Politecnico di Milano (see Fig. 6). The textured 3-D model was then rotated by using a viewing direction normal to the façade plane and both IR and RGB orthoimages (Fig. 10) were generated. Such products are useful for visual and analytical inspections. As can be seen, some holes, which are filled with RGB images, are present in the IR data orthoimage due to some obstacles which prevented the acquisition of IR images.

Fig. 10

RGB (a) and IR (b) orthoimages of the Nave building of Politecnico di Milano and the 3-D model textured by IR thermal images (c).

2.5.

Comparison with Other Methods

In this section, a comparative analysis between the method described in this section and other techniques which are normally adopted in the current practice (i.e., homography and space resection) is carried out. As previously discussed, in both homographic transformations and space resection methods, images are processed individually, requiring the identification of a high number of CPs between images and the building model. In contrast, in the method based on the combined bundle adjustment of RGB and IR images, several TPs are measured in the RGB images instead of the laser scans. In the case of homographic or space resection methods, CPs can be measured in two different ways: (1) from a theodolite survey, a process that may become highly time consuming due to the high number of points to survey or (2) by identifying some features in the laser scanning point cloud. In this second case, the quality of the point cloud and, in particular, its resolution play an important role in the accuracy of the registration.

The comparative analysis is carried out considering a portion of the “Nave” building covered by 10 IR images. For this test site, IR images are registered by using all three discussed methods (bundle adjustment of RGB and IR images, homography, and space resection) and a final orthophoto is generated by adopting the same geometric model for the building. The comparison among the three different approaches is performed considering two different parameters:

In the case of homography and space resection methods, CPs used for image registration were directly derived from the laser scanning point cloud. Moreover, because these two approaches are based on the individual processing of each image, the average RMSE were evaluated on the whole set of 10 IR images analyzed. On the other hand, the simultaneous bundle adjustment of all IR images gives out a single statistic for the whole block orientation. As can be seen from Table 1, RMSE for bundle adjustment is in the order of 1 pixel as previously anticipated, while those corresponding to homography and space resection methods are significantly larger. This is mainly due to the fact that the identification of corresponding points between IR images and point cloud is limited by lower resolution of laser scanning data. On the other hand, the bundle adjustment is based on the measurements of TPs in RGB images instead of laser scans, resulting in an improved precision of point measurement and finally in a higher registration accuracy.

Table 1

RMS comparison for homography, space resection and combined bundle adjustment methods.

The second comparison is directly performed on the derived orthophotos. In this case, some check points which were clearly visible on both the RGB and thermal orthophotos were selected, and the error between IR and RGB data was evaluated.

As can be seen from Table 2 and Fig. 11, discrepancies for homography and space resection are much larger than those obtained from the combined bundle adjustment method.

Table 2

RMS comparison for homography, space resection and combined bundle adjustment methods.

Fig. 11

IR thermal orthophoto derived from (a) homography, (b) space resection and (c) combined bundle adjustment of RGB and IR images.

2.6.

Application of the Whole Procedure

The examples reported along the previous section demonstrated the results obtained by different steps. In this section, a complete case study is presented to show the entire process. The building is still located in the main campus of Politecnico di Milano University and it is a wing of the Padiglione Sud, which mainly hosts classrooms and offices (Fig. 12). In this case, the building was constructed in the 1920s and presented typical decorations and architectural features of that time. The aim of IRT investigation was the detection of thermal anomalies due to stucco delamination, thermal loss of the structure, and damages of the finishing.

Fig. 12

Photo (at top) and 3-D laser scanning model (at bottom) of Padiglione Sud building at Politecnico di Milano.

The laser scanning survey was based on several stations from which scans were acquired with a Riegl LMS-Z420i instrument to cover the whole façade. It should be noted that this long-range instrument undoubtedly was not the best solution for architectural surveys. Considering the short ranges involved (a few tens of meters), a more precise, faster and less noisy phase-shift scanner should be preferred for this application. On the other hand, from one side that was the only instrument available when needed. Second, the aim of survey was not to derive an accurate 3-D model of the building but only to construct a TIN surface for mapping IR images. In addition, a photogrammetric survey to collect RGB images was carried out with a camera Nikon D80. Also in this case, the scope of this survey was not the geometric modeling of the building³⁶ but only to provide image for texturing the 3-D model and to be used for helping the registration of IR images. A set of retroreflective targets positioned on the façade were adopted to establish a local reference system for georeferencing the laser scans and also to be used as GCPs in the combined bundle adjustment of RGB/IR images.

IR thermal images were registered by using the method presented in Sec. 2.3. Other solutions like the use of a “bi-camera system” as reported in Ref. 14 were tried. However, occlusions due to a railing close to the façade and many trees made this solution not feasible due to the bulky coupling equipment. On the other hand, the combined bundle adjustment method allowed the acquisition of IR images simply by placing the camera on a photographic tripod. Also a method based only on homography estimation resulted in improper images due to the presence of stucco and concrete decorations which altered the flat model needed for this registration model. The orientation was carried out by using the commercial photogrammetric software PhotoModeler®. This software allows for identification of GCPs and TPs in the images and does not require any approximation of the EO parameters for the bundle adjustment. First, 30 RGB images were oriented by using 20 retro reflective targets as GCPs and by identifying 150 TPs all over the façade. Then, about 90 IR images were oriented in a combined bundle adjustment with RGB images, offering a final average precision for image coordinates of $\sim 1\text{pixel}$ (Fig. 13).

Fig. 13

IRT and RGB camera poses for the Padiglione Sud building.

The processing time for this phase was about four business days. In parallel, the laser scans were registered together and the TIN model was generated and edited. Finally, the 3-D digital textured model was obtained by using the procedure presented in Sec. 2.4 and then the final orthoimages (Fig. 14) with RGB and IR data derived by simply projecting the images onto the model of the façade. Also in this case, some gaps in the thermal map are clearly visible, mainly due to the presence of obstacles (vegetation, fence, …). The obtained model and the derived orthoimages were used to identify defects and delamination in the stucco using the procedure presented in Ref. 37.

Fig. 14

RGB and IR orthoimages derived from the mosaic of 30 RGB and 90 IR images, respectively.