Letters

Wavelet-domain satellite image fusion based on a generalized fusion equation

[+] Author Affiliations
Bora Jin, Nam Ik Cho

Seoul National University, Department of Electrical and Computer Engineering, INMC, 1 Gwanakro, Seoul 151-744, Republic of Korea

Gibak Kim

Soongsil University, School of Electrical Engineering, 369 Sangdoro, Seoul 156-743, Republic of Korea

J. Appl. Remote Sens. 8(1), 080599 (May 21, 2014). doi:10.1117/1.JRS.8.080599
History: Received February 9, 2014; Revised April 25, 2014; Accepted April 28, 2014
Text Size: A A A

Open Access Open Access

Abstract.  A generalized wavelet domain image fusion method which imposes weights on each of the wavelet coefficients for improving the conventional wavelet domain approach is presented. The weights are controlled in the least-squares sense for enhancing the details while suppressing excessive high frequency components. In experiments with IKONOS and QuickBird satellite data, we demonstrated that the proposed method shows a comparable or better performance than conventional methods in terms of various objective quality metrics.

Figures in this Article

For decades, various image fusion techniques have been developed to obtain a high-resolution multispectral (HRM) satellite image from a set of sensor data: a high-resolution panchromatic (HRP) image containing only intensity information and several low-resolution multispectral (LRM) images with color information.1 Among them, the intensity-hue-saturation (IHS) domain method and principal component analysis (PCA) method replace the intensity component of LRM images with that of an HRP image and a PCA-applied HRP image, respectively.1,2 These methods are relatively simple and easy to implement but known to cause distortion in color information.

Recently, wavelet decomposition was adopted for the satellite image fusion, which is reported to have less color distortion.35 In the conventional wavelet-based methods, the “à trous” algorithm is known to be suitable for obtaining wavelet planes of satellite images,3,6 where the wavelet plane is computed from the difference between the given image and its filtered image with the B3 cubic spline function. The wavelet-based methods are generally classified into the substitution method [substitute wavelet (SW)] and the additive method [additive wavelet (AW)] according to the merging strategy of wavelet planes.3 In the substitution method, the wavelet planes of LRM images are substituted by the wavelet planes of a panchromatic image, whereas those are added by the wavelet plane of a panchromatic image in the additive method. Since the SW discards high frequency components of the LRM images, it usually fails to attain sufficient high frequency details and sometimes loses the information in the LRM images. In contrast, the AW considers the high frequency components of all the LRM images and the HRP image, and thus possibly introduces excessive high frequency details in the synthesized image. Accordingly, there have been efforts to overcome these drawbacks using some improved wavelet-domain methods utilizing weighted merging.4,5 Otazu et al.4 proposed a method in which the wavelet plane of the HRP image is added to each LRM image in proportion to its color intensity value [AW-luminance proportional (AWLP)], whereas Kim et al.5 proposed to add the difference between the wavelet planes of HRP image and each LRM image with or without considering the relative radiometric signature of the LRM images [improved AW (IAW) and IAW proportional (IAWP)].

To develop a more optimized way of adding the wavelet planes, we propose a generalized fusion equation in the form of a weighted composition of wavelet planes, where the weights are determined by the least-squares method. The proposed fusion equation includes wavelet planes of HRP, LRM, and degraded HRP images to control the high frequency injection. In the experiments with IKONOS and QuickBird satellite images, the proposed method is compared with the conventional methods in terms of various objective quality metrics. The results show that the proposed method does not introduce noticeable color distortion and enhances the details better than the conventional methods.

We propose a wavelet-based image fusion method which extracts wavelet planes by the “à trous” algorithm3 and combines them using a generalized fusion equation. The weights included in the fusion equation are determined by the least-squares method.

Generalized Fusion Equation

The generalized fusion equation includes n wavelet planes and is defined as Display Formula

HRMi=LRMi+αij=1nωHRP,j+βij=1nωLRMi,j+γij=1nωLRP,j,(1)
where HRMi is the i’th HRM image that we want to obtain, LRMi is the i’th LRM image, and αi, βi, γi are the weights for the i’th multispectral image fusion. In this fusion equation, the low-resolution panchromatic (LRP) image is a spatially degraded image of HRP through the decimation (by four) and interpolation (by four), and ωI is the wavelet plane of the image I, e.g., ωHRP is the wavelet plane of the HRP image. Each LRM image is interpolated (by four) and then added to the weighted sum of the HRP wavelet planes. The weighted LRM wavelet planes and LRP wavelet planes are added or subtracted to inject high frequency components without introducing excessive high frequency components. The “generalized” equation means that our fusion Eq. (1) includes the conventional wavelet-based fusion methods by setting the weights as shown in Table 1.

Table Grahic Jump Location
Table 1Various wavelet-based fusion algorithms derived from Eq. (1).

As shown in Fig. 1, the LRM images are enlarged to the size of the HRP, and then histogram matching between the HRP and the intensities of the LRM is performed. For the wavelet decomposition of these images, we adopt the “à trous” algorithm as presented in Refs. 36. The “à trous” algorithm is known to be more suitable for image fusion than the Mallat algorithm in terms of artifacts and structure distortion, because the “à trous” algorithm is an undecimated and dyadic algorithm which preserves the structure continuity, whereas the Mallat algorithm is a decimated algorithm which causes the loss of linear continuity.3,6

Graphic Jump LocationF1 :

The block diagram for the proposed image fusion procedure.

Computation of Weights by Least-Squares Method

The fusion equation (1) can be rewritten with matrices as Display Formula

HRMi=[LRMij=1nωHRP,jj=1nωLRMi,jj=1nωLRP,j][1αiβiγi]T,(2)
which is in the form of Yi=Aiwi. Based on the matrix representation, finding weights (αi, βi, γi) is an over-determined problem and the weight vector wi can be computed by using the pseudoinverse as Display Formula
wi=(AiTAi)1AiTYi.(3)

In the actual implementation, since the HRM image Ai is not available, we perform the estimation in the lower resolution. This approach is motivated by an example-based super-resolution, where the relation between an original image patch and the corresponding high-resolution one is extracted using the relation between the original image patch and its degraded (i.e., blurred and downsampled) one.7 This process is explained in Fig. 1, which shows that the LRP and LRM images are degraded by decimation and interpolation to generate LLRP and LLRM images (lower resolution images of LRP and LRM), respectively, and Ai is built in this lower resolution. In the computation of weights by the least-squares method in the lower resolution, however, the relation between the HRM images and LRM images is possibly different from that between the LRM images and the LLRM images, which might degrade the quality of the fused multispectral images. To compensate for the discrepancy, we adopt a scaling factor which is multiplied by the weights. That is, the weights αi, βi, and γi are computed in the lower resolution and then multiplied by the scaling factor.

Twenty-eight IKONOS and 10 QuickBird images were used to evaluate the performance of the proposed fusion method. The spatial resolutions for the HRP and LRM images are 1 and 4 m, respectively. The HRP size is 512×512 and the LRM size is 128×128. It is difficult to measure the quality of the synthesized HRM images objectively since the original HRM images for the reference do not exist. For the quantitative comparison of the fused image, we used the original LRM images with a 4-m resolution as the reference HRM images. Then, the reference HRM images are compared to the HRM images which are obtained by fusing the degraded HRP and LRM images from the original satellite images to 4 and 16-m resolution, respectively.

Figures 2 and 3 show the HRM images synthesized by the proposed method and the existing algorithms, such as IHS,2 substitute wavelet intensity (SWI),8 and IAWP.5 The reference HRM and input LRM images are also included. In these figures, the input LRM images are upsampled by four and interpolated for the purpose of comparison. In Fig. 2, color distortion is noticeable for the IHS method, whereas it is not noticeable for the wavelet-based methods (SWI, IAWP), including the proposed method. It can also be seen that the details are well restored in the synthesized HRM image by the proposed algorithm when compared with the others (Fig. 3). For the objective evaluation, we compute various visual quality metrics,1 such as correlation coefficient (CC), root mean squared error (RMSE), mean structural similarity (MSSIM), universal image quality index (UIQI), quality nonrequiring reference (QNR), spectral angle mapper (SAM), ERGAS, and peak signal-to-noise ratio (PSNR). For the metrics CC, UIQI, MSSIM, QNR, and PSNR, a larger value means a better performance, whereas a smaller value implies better performance for RMSE, ERGAS, and SAM. The overall visual quality assessments are summarized in Tables 2 and 3. The CPU times for the MATLAB® implementation on a personal computer (Intel Core i5 CPU 750 @2.67 GHz) are also measured for the assessment of the computational complexity. Approximately 80% of the time for the proposed algorithm is occupied by the extraction of the wavelet planes for the HRP, LRM, and LRP images. IHS, Gram-Schmidt adaptive (GSA), GIHSA, AdapIHS, AdapCS, and MMSE are not based on the wavelet decomposition, whereas SWI, AWLP, IAWP, NAW, and the proposed algorithms are based on the wavelet decomposition. In Tables 2 and 3, the best two results for each assessment are highlighted in bold. The assessments show that the proposed algorithm is included in the best two for all the objective evaluation metrics except for the UIQI and QNR tests in the QuickBird images, where the proposed algorithm takes third place for both cases. The performance of the proposed method is comparable to the MMSE method9 for the IKONOS data and the proposed method is slightly better than the MMSE except for UIQI in the QuickBird test. The full-size synthesized images for all the algorithms used in the comparison and MATLAB p-codes for the proposed algorithm are available at http://ispl.snu.ac.kr/~idealgod/image_fusion.

Graphic Jump LocationF2 :

Results of fusing IKONOS1 images [cropped images (64×64) from fused images (512×512)]. (a) Reference HRM, (b) input LRM, (c) IHS, (d) SWI, (e) IAWP, (f) proposed method.

Graphic Jump LocationF3 :

Results of fusing IKONOS2 images [cropped images (64×64) from fused images (512×512)]. (a) Reference HRM, (b) input LRM, (c) IHS, (d) SWI, (e) IAWP, (f) proposed method.

Table Grahic Jump Location
Table 2The spectral quality assessment of image fusion methods (averaged values for 28 IKONOS images). The best two results for each assessment are highlighted in bold.
Table Grahic Jump Location
Table 3The spectral quality assessment of image fusion methods (averaged values for 10 QuickBird images). The best two results for each assessment are highlighted in bold.

The scaling factor multiplied by the weights is set to 0.65, which is experimentally chosen and applied to all the tested images. Figure 4 shows the performance variation with respect to the scaling factor for IKONOS images. As the scaling factor increases, more high frequency components are injected, whereas a small scaling factor induces less sharpened results. We choose 0.65 as the optimal scaling factor since a scaling factor of 0.65 yields the lowest accumulated ranking values (=13) for all the evaluation metrics (cf. 15 for 0.75, 22 for 0.55).

Graphic Jump LocationF4 :

Performance variation of the proposed method with the scaling factor.

In this article, we have proposed a wavelet domain image fusion method, which can be considered a generalization of the existing methods. The fusion equation consists of the weighted composition of wavelet planes, which includes the terms for enhancing the details and avoiding excessive high frequency components as well. The weights are computed by the least-squares method in the low resolution, and the scaling factor is exploited to compensate the discrepancy incurred in the lower resolution computation of the weights. Experimental results show that the proposed method reduces color distortion and provides high-resolution details.

This work was supported by Basic Science Research Program through the National Research Foundation of Korea (NRF) funded by the Ministry of Science, ICT And Future Planning (2009-0083495).

Amro  I. et al., “A survey of classical methods and new trends in pansharpening of multispectral images,” EURASIP J. Adv. Signal Process.. 79,  (2011), CrossRef.
Tu  T.-M. et al., “A fast intensity-hue-saturation fusion technique with spectral adjustment for IKONOS imagery,” IEEE Geosci. Remote Sens. Lett.. 1, (4 ), 309 –312 (2004), CrossRef. 1545-598X 
Nunez  J. et al., “Multiresolution-based image fusion with additive wavelet decomposition,” IEEE Trans. Geosci. Remote Sens.. 37, (3 ), 1204 –1211 (1999), CrossRef. 0196-2892 
Otazu  X. et al., “Introduction of sensor spectral response into image fusion methods. Application to wavelet-based methods,” IEEE Trans. Geosci. Remote Sens.. 43, (10 ), 2376 –2385 (2005), CrossRef. 0196-2892 
Kim  Y. et al., “Improved additive-wavelet image fusion,” IEEE Geosci. Remote Sens. Lett.. 8, (2 ), 263 –267 (2011), CrossRef. 1545-598X 
Gonzlez-Audcana  M. et al., “Comparison between Mallat’s and the trous discrete wavelet transform based algorithms for the fusion of multispectral and panchromatic images,” Int. J. Remote Sens.. 26, (3 ), 595 –614 (2005), CrossRef. 0143-1161 
Glasner  D., Bagon  S., Irani  M., “Super-resolution from a single image,” in  2009 IEEE 12th Int. Conf. on Computer Vision , pp. 349 –356,  IEEE ,  Kyoto, Japan  (2009).
Gonzalez-Audicana  M. et al., “Fusion of multispectral and panchromatic images using improved IHS and PCA mergers based on wavelet decomposition,” IEEE Trans. Geosci. Remote Sens.. 42, (6 ), 1291 –1299 (2004), CrossRef. 0196-2892 
Garzelli  A., Nencini  F., Capobianco  L., “Optimal MMSE pan sharpening of very high resolution multispectral images,” IEEE Trans. Geosci. Remote Sens.. 46, (1 ), 228 –236 (2008), CrossRef. 0196-2892 
Aiazzi  B., Baronti  S., Selva  M., “Improving component substitution pan-sharpening through multivariate regression of MS+pan data,” IEEE Trans. Geosci. Remote Sens.. 45, (10 ), 3230 –3239 (2007), CrossRef. 0196-2892 
Rahmani  S. et al., “An adaptive IHS pan-sharpening method,” IEEE Geosci. Remote Sens. Lett.. 7, (4 ), 746 –750 (2010), CrossRef. 1545-598X 
Choi  J., Yu  K., Kim  Y., “A new adaptive component-substitution-based satellite image fusion by using partial replacement,” IEEE Trans. Geosci. Remote Sens.. 49, (1 ), 295 –309 (2011), CrossRef. 0196-2892 
Sathya Bama  B. et al., “New additive wavelet image fusion algorithm for satellite images,” Pattern Recognit. Mach. Intell., Lect. Notes Comput. Sci.. 8251, , 313 –318 (2013), CrossRef. 0302-9743 

© The Authors. Published by SPIE under a Creative Commons Attribution 3.0 Unported License. Distribution or reproduction of this work in whole or in part requires full attribution of the original publication, including its DOI.

Citation

Bora Jin ; Gibak Kim and Nam Ik Cho
"Wavelet-domain satellite image fusion based on a generalized fusion equation", J. Appl. Remote Sens. 8(1), 080599 (May 21, 2014). ; http://dx.doi.org/10.1117/1.JRS.8.080599


Figures

Graphic Jump LocationF1 :

The block diagram for the proposed image fusion procedure.

Graphic Jump LocationF2 :

Results of fusing IKONOS1 images [cropped images (64×64) from fused images (512×512)]. (a) Reference HRM, (b) input LRM, (c) IHS, (d) SWI, (e) IAWP, (f) proposed method.

Graphic Jump LocationF3 :

Results of fusing IKONOS2 images [cropped images (64×64) from fused images (512×512)]. (a) Reference HRM, (b) input LRM, (c) IHS, (d) SWI, (e) IAWP, (f) proposed method.

Graphic Jump LocationF4 :

Performance variation of the proposed method with the scaling factor.

Tables

Table Grahic Jump Location
Table 1Various wavelet-based fusion algorithms derived from Eq. (1).
Table Grahic Jump Location
Table 2The spectral quality assessment of image fusion methods (averaged values for 28 IKONOS images). The best two results for each assessment are highlighted in bold.
Table Grahic Jump Location
Table 3The spectral quality assessment of image fusion methods (averaged values for 10 QuickBird images). The best two results for each assessment are highlighted in bold.

References

Amro  I. et al., “A survey of classical methods and new trends in pansharpening of multispectral images,” EURASIP J. Adv. Signal Process.. 79,  (2011), CrossRef.
Tu  T.-M. et al., “A fast intensity-hue-saturation fusion technique with spectral adjustment for IKONOS imagery,” IEEE Geosci. Remote Sens. Lett.. 1, (4 ), 309 –312 (2004), CrossRef. 1545-598X 
Nunez  J. et al., “Multiresolution-based image fusion with additive wavelet decomposition,” IEEE Trans. Geosci. Remote Sens.. 37, (3 ), 1204 –1211 (1999), CrossRef. 0196-2892 
Otazu  X. et al., “Introduction of sensor spectral response into image fusion methods. Application to wavelet-based methods,” IEEE Trans. Geosci. Remote Sens.. 43, (10 ), 2376 –2385 (2005), CrossRef. 0196-2892 
Kim  Y. et al., “Improved additive-wavelet image fusion,” IEEE Geosci. Remote Sens. Lett.. 8, (2 ), 263 –267 (2011), CrossRef. 1545-598X 
Gonzlez-Audcana  M. et al., “Comparison between Mallat’s and the trous discrete wavelet transform based algorithms for the fusion of multispectral and panchromatic images,” Int. J. Remote Sens.. 26, (3 ), 595 –614 (2005), CrossRef. 0143-1161 
Glasner  D., Bagon  S., Irani  M., “Super-resolution from a single image,” in  2009 IEEE 12th Int. Conf. on Computer Vision , pp. 349 –356,  IEEE ,  Kyoto, Japan  (2009).
Gonzalez-Audicana  M. et al., “Fusion of multispectral and panchromatic images using improved IHS and PCA mergers based on wavelet decomposition,” IEEE Trans. Geosci. Remote Sens.. 42, (6 ), 1291 –1299 (2004), CrossRef. 0196-2892 
Garzelli  A., Nencini  F., Capobianco  L., “Optimal MMSE pan sharpening of very high resolution multispectral images,” IEEE Trans. Geosci. Remote Sens.. 46, (1 ), 228 –236 (2008), CrossRef. 0196-2892 
Aiazzi  B., Baronti  S., Selva  M., “Improving component substitution pan-sharpening through multivariate regression of MS+pan data,” IEEE Trans. Geosci. Remote Sens.. 45, (10 ), 3230 –3239 (2007), CrossRef. 0196-2892 
Rahmani  S. et al., “An adaptive IHS pan-sharpening method,” IEEE Geosci. Remote Sens. Lett.. 7, (4 ), 746 –750 (2010), CrossRef. 1545-598X 
Choi  J., Yu  K., Kim  Y., “A new adaptive component-substitution-based satellite image fusion by using partial replacement,” IEEE Trans. Geosci. Remote Sens.. 49, (1 ), 295 –309 (2011), CrossRef. 0196-2892 
Sathya Bama  B. et al., “New additive wavelet image fusion algorithm for satellite images,” Pattern Recognit. Mach. Intell., Lect. Notes Comput. Sci.. 8251, , 313 –318 (2013), CrossRef. 0302-9743 

Some tools below are only available to our subscribers or users with an online account.

Related Content

Customize your page view by dragging & repositioning the boxes below.

Related Book Chapters

Topic Collections

Advertisement
  • Don't have an account?
  • Subscribe to the SPIE Digital Library
  • Create a FREE account to sign up for Digital Library content alerts and gain access to institutional subscriptions remotely.
Access This Article
Sign in or Create a personal account to Buy this article ($20 for members, $25 for non-members).
Access This Proceeding
Sign in or Create a personal account to Buy this article ($15 for members, $18 for non-members).
Access This Chapter

Access to SPIE eBooks is limited to subscribing institutions and is not available as part of a personal subscription. Print or electronic versions of individual SPIE books may be purchased via SPIE.org.