臺灣博碩士論文加值系統

影像超解析度成像技術旨在從低解析度影像中恢復丟失的結構和紋理資訊。此技術雖已被廣泛研究多年，仍舊面臨兩項主要挑戰：1. 大多數研究僅專注於網路架構設計，藉由增加神經網路的參數及完善神經元的連接以達到更好的效果；然而，這些方法忽視了模型中大量的冗餘特徵，導致模型過於巨大，難以部署於實際應用中。2. 大多數研究分別地針對失真優化或針對感知優化，卻少有研究專注於針對感知與失真之間的權衡優化，因此難以生成既精確且逼真的高解析度影像。在本研究中，我們提出自增強之遞迴式架構，以解決上述兩個挑戰。首先，我們藉由置信度之預測，將影像拆解成不同的困難度區域。接著，藉由提出的自增強損失函數，讓每個子網路自適應地專注於不同的困難度區域，藉此減少網路之間大量的冗餘特徵。最後，我們提出融合網路，將不同子網路的輸出融合，並藉由置信圖與融合損失函數，達到更好的感知失真權衡。實驗結果顯示此架構可以有效地增強基於感知及基於失真的超解析度成像的效果，並且能夠達到更好的感知失真權衡。更進一步地，實驗結果也顯示了此架構可泛用至其他研究所提出的網路架構。

Single-image super-resolution restores the lost structures and textures from low-resolved images, which has achieved extensive attention from the research community. However, it still faces two major challenges: 1. Most research only focuses on network architecture design, and the performance improvement in such a design comes from an increase of the parameter number and the elaboration of neural connection; however, these methods don’t consider the large number of redundant features in the model, resulting in a huge model that is difficult to deploy in practical applications. 2. Most research focus on optimization for distortion or perception separately, but few studies focus on optimization for the tradeoff between them, that can help the model to generate high-resolved images which are both accurate and photo-realistic. In this research, we proposed a recursively boosting framework to solve the above two challenges. First, we measure the confidence to decompose the image into different difficulty areas. Then, with the proposed boosting loss function, each sub-network can adaptively pay attention on different difficulty areas, thereby reducing the redundant features between networks. Finally, we propose a fusion network, which fuses the outputs of different sub-networks, and achieves a better perception-distortion tradeoff through the confidence map and the fusion loss function. Experimental results show that this framework can effectively boost the performance of both perceptual-based and distortion-based image super-resolution, and can also achieve a better perception-distortion tradeoff. Furthermore, the experimental results show that this framework can be generalized to various network architectures proposed by other researches.

誌謝辭
摘要
Abstract
目錄
圖目錄
表目錄
第一章 緒論
1.1 研究背景與動機
1.2 研究目標與困難
1.3 論文架構
第二章 文獻探討與技術背景
2.1 影像超解析度成像(Image Super-resolution)
2.2 感知失真權衡(Perception-Distortion Tradeoff)
2.3 置信度(Confidence)於影像恢復演算法之應用
第三章 自適應與自增強之遞迴式影像超解析度成像
3.1 整體架構
3.2 置信度估計(Confidence estimation)
3.2.1 置信網路(Confidence Network)架構
3.2.2 最大似然估計(maximum likelihood)與置信度之關係
3.3 自增強重建(Boosting reconstruction)
3.3.1 自增強網路(Boosting Network)架構
3.3.2 失真自增強損失函數(Distortion boosting loss)
3.3.3 感知自增強損失函數(Perception boosting loss)
3.3.4 其他損失函數
3.4 置信度指導之影像融合(Confidence guided image fusion)
3.4.1 權重影像融合(Weighted image fusion)
3.4.2 可學習之影像融合(Learnable image fusion)
3.4.2.1 融合網路(Fusion network)架構
3.4.2.2 融合損失函數(Fusion loss)
第四章 系統實現
4.1 系統平台
4.2 網路架構
第五章 實驗結果
5.1 資料集
5.2 評估指標
5.2.1 客觀影像品質評估指標
5.2.2 感知影像品質評估指標
5.3 實驗比較
5.3.1 置信圖可視化
5.3.2 失真自增強結果
5.3.3 感知自增強結果
第六章 討論
6.1 結論
6.2 未來展望
參考文獻

[1]Y. Zhang, Y. Tian, Y. Kong, B. Zhong, and Y. Fu, “Residual Dense Network for Image Super-Resolution,” in IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018.
[2]C. Dong, C. C. Loy, K. He, and X. Tang, “Learning a Deep Convolutional Network for Image Super-Resolution,” in European Conference on Computer Vision (ECCV), 2014.
[3]J. Kim, J. K. Lee and K. M. Lee, “Accurate Image Super-Resolution Using Very Deep Convolutional Networks,” in IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
[4]W. Shi, J. Caballero, F. Husza ́r, J. Totz, A. P. Aitken, R. Bishop, D. Rueckert, and Z. Wang, “Real-Time Single Image and Video Super-Resolution Using an Efficient Sub-Pixel Convolutional Neural Network,” in IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
[5]Y. Zhang, K. Li, K. Li, L. Wang, B. Zhong, and Y. Fu, “Image Super-Resolution Using Very Deep Residual Channel Attention Networks,” in European Conference on Computer Vision (ECCV), 2018.
[6]T. Dai, J. Cai, Y. Zhang, S. T. Xia, and L. Zhang, “Second-order Attention Network for Single Image Super-Resolution,” in IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
[7]M. Haris, G. Shakhnarovich, and Norimichi Ukita, “Deep Back-Projection Networks for Super-Resolution,” in IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018.
[8]Y. Guo, J. Chen, J. Wang, Q. Chen, J. Cao, Z. Deng, Y. Xu, and M. Tan, “Closed-loop Matters: Dual Regression Networks for Single Image Super-Resolution,” in IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
[9]Z. Li, J. Yang, Z. Liu, X. Yang, G. Jeon, and W. Wu, “Feedback Network for Image Super-Resolution,” in IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
[10]Y. Qiu, R. Wang, and D. Tao, “Embedded Block Residual Network: A Recursive Restoration Model for Single-Image Super-Resolution,” in IEEE International Conference on Computer Vision (ICCV), 2019.
[11]C. Ledig, L. Theis, F. Husza ́r, J. Caballero, A. Cunningham, A. Acosta, A. Aitken, A. Tejani, J. Totz, Z. Wang, and W. Shi, “Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Network,” in IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017.
[12]M. S. M. Sajjadi, B. Scholkopf, and M. Hirsch, “EnhanceNet: Single Image Super-Resolution Through Automated Texture Synthesis,” in IEEE International Conference on Computer Vision (ICCV), 2017.
[13]Y. Blau and T. Michaeli, ”The Perception-Distortion Tradeoff,” in IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018.
[14]X. Deng, R. Yang, M. Xu, and P. L. Dragotti, “Wavelet Domain Style Transfer for an Effective Perception-distortion Tradeoff in Single Image Super-Resolution,” in IEEE International Conference on Computer Vision (ICCV), 2019.
[15]R. Yasarla and V. M. Patel, “Uncertainty Guided Multi-Scale Residual Learning-using a Cycle Spinning CNN for Single Image De-Raining,” in IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
[16]A. Eldesokey, and F. S. Khan, “Confidence Propagation through CNNs for Guided Sparse Depth Regression,” IEEE transactions on pattern analysis and machine intelligence (PAMI), 2019.
[17]I. Daubechies, “The wavelet transform, time-frequency localization and signal analysis,” IEEE Transactions on Information Theory, 36(5):961–1005, 1990.
[18]N. Ahmed, T. Natarajan, and K.R. Rao, “Discrete Cosine Transform,” IEEE Transactions on Computers, 1974.
[19]Y. Freund and R. E. Schapire, “A Decision-Theoretic Generalization of On-Line Learning and an Application to Boosting,” Journal of Computer and System Sciences, 1997.
[20]J. Johnson, A. Alahi, and F. F. Li, “Perceptual Losses for Real-Time Style Transfer and Super-Resolution,” in European Conference on Computer Vision (ECCV), 2016.
[21]R. Zhang, P. Isola, A. A. Efros, E. Shechtman, and O. Wang, “The Unreasonable Effectiveness of Deep Features as a Perceptual Metric,” in IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018.
[22]J. M. S. Prewitt, “Object enhancement and extraction,” Journal of Picture Processing and Psychopictorics, 1970.
[23]R. Timofte, E. Agustsson, L. V. Gool, M. H. Yang, and L. Zhang, “Ntire 2017 challenge on single image super-resolution: Methods and results,” in IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPR workshops), 2017.
[24]A. Mittal, R. Soundararajan and A. C. Bovik, “Making a ‘Completely Blind’ Image Quality Analyzer,” IEEE Journal on Signal Processing Letters, 2013.
[25]X. Wang, K. Yu, C. Dong, C. C. Loy, “Recovering Realistic Texture in Image Super-resolution by Deep Spatial Feature Transform,” in IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018.
[26]Y. Zhang, K. Li, K. Li, B. Zhong, and Y. Fu, “Residual Non-local Attention Networks for Image Restoration,” in International Conference on Learning Representations (ICLR), 2019.
[27]Y. Tai, J. Yang, and X. Liu, ”Image Super-Resolution via Deep Recursive Residual Network,” in IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017.
[28]J. Kim, J. K. Lee, and K. M. Lee, “Deeply-Recursive Convolutional Network for Image Super-Resolution,” in IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
[29]M. Irani and S. Peleg, “Improving resolution by image registration,” CVGIP: Graphical models and image processing, 53(3):231–239, 1991.
[30]J. Yang, J. Wright, T. S. Huang, and Y. Ma, “Image super-resolution via sparse representation,” IEEE Transactions on Image Processing, 2010.
[31]L. A. Gatys, A. S. Ecker, and M. Bethge, “Image style transfer using convolutional neural networks,” in IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
[32]Y. Zeng, Z. Lin, J. Yang, J. Zhang, E. Shechtman, and H. Lu, “High-Resolution Image Inpainting with Iterative Confidence Feedback and Guided Upsampling,” in European Conference on Computer Vision (ECCV), 2020.
[33]D. Liu, B. Wen, Y. Fan, C. C. Loy, and T. S. Huang, “Non-Local Recurrent Network for Image Restoration” in Advances in Neural Information Processing Systems (NeurIPS), 2018.
[34]M. Bevilacqua, A. Roumy, C. Guillemot, and M. L. Alberi-Morel, “Low-complexity single-image super-resolution based on nonnegative neighbor embedding,” in Proceedings of British Machine Vision Conference (BMVC), 2012.
[35]R. Zeyde, M. Elad, and M. Protter, “On single image scale-up using sparse-representations,” in Proceedings of International conference on curves and surfaces, 2010.
[36]P. Arbelaez, M. Maire, C. Fowlkes, and J. Malik, “Contour detection and hierarchical image segmentation,” IEEE Transactions on Pattern Analysis and Machine Intelligence (PAMI), 2011.
[37]J. B. Huang, A. Singh, and N. Ahuja, “Single image super-resolution from transformed self-exemplars,” IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2015.
[38]Y. Matsui, K. Ito, Y. Aramaki, A. Fujimoto, T. Ogawa, T. Yamasaki, and K. Aizawa, “Sketch-based manga retrieval using manga109 dataset,” Multimedia Tools and Applications, 2017.
[39]M. Irani and S. Peleg, “Motion analysis for image enhancement: Resolution, occlusion, and transparency,” Journal of Visual Communication Image Representation, 1993.
[40]Z. Wang, A. C. Bovik, H. R Sheikh, E. P. Simoncelli, et al, “Image quality assessment: from error visibility to structural similarity,” IEEE Transactions on Image Processing (TIP), 2004.