臺灣博碩士論文加值系統

衛星全色態影像銳化(Image Pan-sharpening)作為一種提升多光譜影像空間解析度的技術，是衛星影像處理中的基礎且重要的研究議題。該技術通過融合高空間解析度的全色態(Panchromatic, PAN)影像與低空間解析度的多光譜（Multispectral, MS）影像，生成一幅高空間解析度且多光譜的影像。儘管目前已存在多種先進的全色態影像銳化技術，但仍面臨一些難以解決的問題。首先，現今的全色態影像銳化技術主要依賴深度學習進行影像融合。深度學習模型的複雜性使得不同的全色態影像銳化技術難以輕易地在不同應用場景中移植。不只模型的泛化性能低下，也增加研究成本。其次，原始MS影像和輸出影像之間存在一定程度的光譜強度差異。這種差異在光譜特徵分析中顯而易見。生成影像與原始影像之間的差異過大，如植被區域失去原有特徵。融合影像的可靠性不足。
本文旨在解決光譜強度差異與結構維持問題。提出一個新的訓練方法以擴展現有模型的性能。該方法透過三個額外的項修改損失函數。第一個項對生成影像和地真資料中的每個波段進行光譜強度的全局比較。旨在透過最小化各波段強度的平均值差異，矯正影像光譜強度偏差。第二個項計算兩種影像每個波段光譜強度的標準差，以控制影像的值域表現。目的是防止因平均值計算導致的影像波段強度分佈失真；第三個項將影像分解為多個局部區域，計算兩種影像中每個局部區域的相關性。為了維持影像的結構一致性引入影像金字塔概念，使模型學習影像在不同尺度下的細節紋理等高頻特徵。本文透過修改損失函數，在不大幅修改既有全色態影像銳化模型的前提下，使訓練方法具有通用性，彌補深度學習模型僵化問題。
在Worldview-3資料集上的實驗結果表明，該訓練方法在定性和定量方面均取得了優異的成果。

Satellite image pan-sharpening, a technique used to enhance the spatial resolution of multispectral images, is a fundamental and significant research topic in satellite image processing. This technique generates a high spatial resolution multispectral image by fusing high spatial resolution panchromatic (PAN) images with low spatial resolution multispectral (MS) images. Despite various advanced pan-sharpening technologies, some challenges remain unsolved. First of all, current pan-sharpening techniques rely on deep learning technology for image fusion. The complexity of deep learning models makes it difficult to transfer pan-sharpening techniques easily to various application scenarios. This not only results in poor generalization performance but also increases research costs. Second, there is spectral intensity drift between the original MS image and the output image. This drift is evident in the analysis of spectral features. The resulting image may differ significantly from the original image, such as the loss of original features in vegetated areas. Therefore, the reliability of fused images is deficient.
This work aims to solve the problem of spectral intensity drift and structural preservation. It proposes a new training method to extend the performance of existing models. This method modifies the loss function through three additional terms. The first term globally compares the spectral intensity of each band between the generated images and the ground truth data. It aims to correct spectral intensity drift by minimizing the average intensity difference of each band. The second term calculates the standard deviation of the spectral intensity for each band of both images. This is done to control the range performance of the intensity and prevent the distortion caused by averaging. The third term segments the images into multiple local regions and calculates the correlation of each local region in both images. To maintain structural consistency, the image pyramid concept is introduced, allowing the model to learn high-frequency features, such as detailed textures, at different scales. This work generalizes the training method without significantly modifying the existing pan-sharpening model by modifying the loss function. This can address the rigidity problem of deep learning models.
Experimental results on the Worldview-3 dataset indicate that this training method achieves outstanding qualitative and quantitative outcomes.

摘要 i
Abstract ii
Acknowledgements iii
Content iv
List of Figures v
List of Table vi
Chapter 1 Introduction 1
1.1 Introduction 1
1.2 Satellite image 2
1.3 Degradation 3
1.4 Motivation 5
1.5 Contribution 6
Chapter 2 Related work 7
2.1 Traditional method 7
2.2 Deep learning method 8
Chapter 3 Methodology 13
3.1 LPPN Architecture 13
3.2 Loss function of LPPN 15
3.3 Proposed Loss Terms 15
3.3.1 Global Intensity De-drift 15
3.3.2 Standard Deviation Preservation 19
3.3.3 Local Detail Compensation 20
3.4 Proposed Loss function 21
Chapter 4 Experimental Results and Discussion 22
4.1 Experimental Datasets 22
4.2 Model Training details 22
4.3 Metrics 23
4.4 Model training 24
4.5 Model Performance and Comparisons 26
4.6 Ablation experiments 27
4.6.1 Quantitative Analysis 28
4.6.2 Histogram analysis 30
4.6 Statistical Analysis 32
Chapter 5 Conclusions and Future Works 34
Reference 36
Appendix 39

Aiazzi, B., A;parone, L., Baronti, S., Garzelli, A., & Selva, M. (2006). MTF-tailored Multiscale Fusion of High-resolution MS and Pan Imagery. Photogrammetric Engineering & Remote Sensing, 5, 591-596.

Burt, P., & Adelson, E. (1983). The Laplacian Pyramid as a Compact Image Code. IEEE Transactions on Communications, 31(4), 532-540.

Fasbender, D., Radoux, J., & Bogaert, P. (2008). Bayesian Data Fusion for Adaptable Image Pansharpening. IEEE Transactions on Geoscience and Remote Sensing, 46(6), 1847-1857.

He, K., Zhang, X., Ren, S., & Sun, J. (2016) Deep Residual Learning for Image Recognition. arXiv:1512.03385.

Huang, W., Xiao, L., Wei, Z., Liu, H., & Tang, S. (2015). A New Pan-Sharpening Method With Deep Neural Networks. IEEE Geoscience and Remote Sensing Letters, 12(5), 1037-1041.

Jacobsen, J.-H., Smeulders, A., & Oyallon, E. (2018). i-RevNet: Deep Invertible Networks. arXiv:1802.07088.

Jin, C., Deng, L. J., Huang, T. Z., & Vivone, G. (2022). Laplacian pyramid networks: A new approach for multispectral pansharpening. Information Fusion, 78, 158-170.

Jin, X., Zhang, P., Jiang, Q., Miao, S., Yao, S., & Zhou, W. (2022). F-UNet++: Remote Sensing Image Fusion Based on Multipurpose Adaptive Shuffle Attention and Composite Multi-Input Reconstruction Network. IEEE Transactions on Instrumentation and Measurement, 72.

Kingma, D. P., & Ba, J. (2014) Adam: A Method for Stochastic Optimization. arXiv:1412.6980.

Masi, G., Cozzolino, D., Verdoliva, L., & Scarpa, G. (2016). Pansharpening by Convolutional Neural Networks. Remote Sensing, 8(7), 594.

Mulla, D. J. (2013). Twenty five years of remote sensing in precision agriculture: Key advances and remaining knowledge gaps. Biosystems Engineering, 114(4), 358-371.

Olkkonen, H., & Pesola, P. (1996). Gaussian Pyramid Wavelet Transform for Multiresolution Analysis of Images. Graphical Models and Image Processing, 58(4), 394-398.

Otazu, X., Gonzalez-Audicana, M., Fors, O., & Nunez, J. (2005). Introduction of sensor spectral response into image fusion methods. Application to wavelet-based methods. IEEE Transactions on Geoscience and Remote Sensing, 43(10), 2376-2385.

Palsson, F., Sveinsson, J. R., Ulfarsson, M. O., & Benediktsson, J. A. (2014). Model-Based Fusion of Multi- and Hyperspectral Images Using PCA and Wavelets. IEEE Transactions on Geoscience and Remote Sensing, 53(5), 2652-2663.

Ranchin, T., & Wald, L. (2000). Fusion of high spatial and spectral resolution images: the ARSIS concept and its implementation. Photogrammetric engineering and remote sensing, 66 (1), 49-61.

Restaino, R., Mura, M. D., Vivone, G., & Chanussot, J. (2016). Context-Adaptive Pansharpening Based on Image Segmentation. IEEE Transactions on Geoscience and Remote Sensing, 55(2), 753-766.

Shettigara, V. K. (1992). A generalized component substitution technique for spatial enhancement of multispectral images using a higher resolution data set. Photogrammetric engineering and remote sensing, 58(5), 561-567.

Tao, H., Li, J., Hua, Z., & Zhang, F. (2023). DUDB: Deep Unfolding-Based Dual-Branch Feature Fusion Network for Pan-Sharpening Remote Sensing Images. IEEE Transactions on Geoscience and Remote Sensing, 62, 1-17.

Tu, T.-M., Su, S.-C., Shyu, H.-C.,& Huang, P. S. (2001). A new look at IHS-like image fusion methods. Information Fusion, 2(3), 177-189.

Wald, L., Ranchin, T., & Mangolini, M. (1997). Fusion of satellite images of different spatial resolutions: Assessing the quality of resulting images. Photogrammetric engineering and remote sensing, 63(6), 691-699.

Wang, Z., Bovik, A. C., Sheikh, H. R., & Simoncelli, E. P. (2004). Image quality assessment: from error visibility to structural similarity. IEEE Transactions on Image Processing, 13(4), 600-612.


West, H., Quinn, N., & Horswell, M. (2019). Remote sensing for drought monitoring & impact assessment: Progress, past challenges and future opportunities. Remote Sensing of Environment, 232.

Ye, F., Guo, Y., & Zhuang, P. (2019). Pan-sharpening via a gradient-based deep network prior. Signal Processing: Image Communication, 74, 322-331.

Yang, J., Fu, X., Hu, Y., Huang, Y., Ding, X., & Paisley, J. (2017). PanNet: A Deep Network Architecture for Pan-Sharpening. 2017 IEEE International Conference on Computer Vision (ICCV), 1753-1761.

Yuhas, J. R. H., Goetz, A. F., & Boardman, J. M. (1992). Discrimination among semi-arid landscape endmembers using the Spectral Angle Mapper (SAM) algorithm. JPL Airborne Geoscience Workshop, 1, 147-149.

Zhou, M., Yan, K., Huang, J., Yang, Z., Fu, X., & Zhao, F. (2022). Mutual Information-driven Pan-sharpening. 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 1788-1798.