臺灣博碩士論文加值系統

隨著科技演進，人們對多媒體的體驗愈來愈注重。隨著3D顯示技術的進步，從最初需要搭配專屬眼鏡的雙視角立體播放模式演變至今的多視角裸眼3D顯示。為了兼顧2D廣播系統之相容性，框兼容傳輸為目前3D廣播的主流。它能夠將一個視角的影像以及其對應的深度圖包裝在一幀中，在接收端需要經由基於深度圖像渲染(DIBR)的演算法以推論出多個視角的影像。其中影像置中深度包裝(CTDP)屬於一種框兼容的包裝格式。在本論文中我們結合了前人所提出基於CTDP解包裝格式的增進方法，並與DIBR演算法結合成一套3D播放器。為了實現即時播放，我們利用計算統一設備(CUDA)加速程式庫配合C語言在圖形顯示器(GPU)的平台上為各個演算法提出平行化實現的方法。此外，為了保持解包裝後深度圖的質量，我們還提出基於深度學習的引導式深度上採樣網路。實驗結果證明我們提出的3D播放器能夠針對9-view的3D電影進行即時播放，提出的上採樣網路也能夠使產生的視圖達到比傳統演算法更佳的質量。

As technology evolves, people pay more attention to the multimedia experience. With the advancement of 3D display technology, the original 3D stereo-view broadcasting mode requiring exclusive 3D glasses has evolved to the multiple-view naked eye 3D display. To take into consideration the compatibility of 2D broadcasting systems, frame-compatible is still the major 3D broadcasting format. It packs one view and its corresponding depth map into a frame which could be generate multiple views by the depth image-based rendering (DIBR) algorithm at the receiver if we can use the centralize texture depth packing (CTDP) frame-compatible format. In this thesis, we combine the improved CTDP depacking methods proposed by previous researchers with DIBR to form a 3D player. In order to achieve real-time display, we propose the parallelized implementation of each algorithm on the GPU platform using CUDA accelerated library and C language. In addition, to maintain the quality of the depacked depth maps, we propose a deep learning-based guided depth upsampling network. The experimental results demonstrate that the proposed 3D displayer can generate 9-view 3D movies in real-time, and the proposed upsampling network can achieve a better quality of the generated views than traditional algorithms.

摘要 I
Abstract II
誌謝 III
Contents IV
List of Tables VII
List of Figures VIII
Chapter 1 Introduction 1
1.1 Research Background 1
1.2 Motivations 3
1.3 Thesis Organization 5
Chapter 2 Related Work 6
2.1 Centralized Texture Depth Packing Format (CTDP) 6
2.1.1 Packing Details of CTDP formats 7
2.1.2 Assigned Color Depth Packing Method 9
2.1.3 Improved Depth Upsampling 11
2.2 Depth Image Based Rendering (DIBR) 13
2.3 Single Image Super-Resolution (SISR) Network 16
2.4 Residual Dense Network (RDN) 18
2.5 Multi Scale Residual Block (MSRB) 19
2.6 Cross-Level Gating Decoder 20
2.7 Compute Unified Device Architecture (CUDA) 21
2.7.1 Common Notations and Configurations in CUDA Kernels 23
Chapter 3 The Proposed CTDP Depacking System with Guided Depth Upsampling Network and DIBR System 25
3.1 Overview of the Proposed System 26
3.2 Realization of Centralize Texture Depth De-packing 27
3.2.1 Implementation of Splitting Texture and Depth 28
3.2.2 Implementation of Pixels Rearrange for RGB Channel 29
3.2.3 Implementation of Adjust Pixels Rearrange for YUV420 Channel 30
3.2.4 Implementation of Improved Depth Upsampling 32
3.3 Realization of Depth Image Based Rendering 34
3.3.1 Implementation of Progressive 3D Warping and Hole Filling 34
3.3.2 Implementation of Pixel Rearrange for Joyvision 38
3.4 Proposed Guided Depth Upsampling Network 39
3.4.1 Architecture of Guided Depth Upsampling Network 40
3.4.2 Texture Feature Filtering Module 45
3.4.3 Loss functions 46
Chapter 4 Experimental Results 48
4.1 Environmental Settings and Datasets 48
4.2 Analyze of GPU Kernels Execution Time 50
4.3 Experimental Results and Ablation Studies 51
4.3.1 Training Detail 52
4.3.2 Analyze of Feature Extractor and Improved Modules 52
4.3.3 Analyze of Depth Upsampling and DIBR System 54
Chapter 5 Conclusions 59
Chapter 6 Future Work 60
References 61

[1]T. Hsiao. “Improved depth upsampling and multi-view generation for depacking centralized texture depth depacked 3D videos” M. S. Thesis, National Cheng Kung University, Tainan, Taiwan, July 2019.
[2]W.-J. Yang, J.-F. Yang, G.-C. Chen, P.-C. Chung and M. F. Chung. “An Assigned Color Depth Packing Method with Centralized Texture Depth Packing Formats for 3D VR Broadcasting Services”, Proc. of IEEE Journal on emerging and selected topics in circuits and systems, vol. 9, no. 1, pp. 122-132, March 2019.
[3]Gribbon, Kim T., and Donald G. Bailey. "A novel approach to real-time bilinear interpolation." Proc. of Second IEEE International Workshop on electronic design, test and applications. IEEE, 2004.
[4]Keys, Robert. "Cubic convolution interpolation for digital image processing." proc. of IEEE Transactions on acoustics, speech, and signal processing 29.6 (1981): 1153-1160.
[5]Turkowski, Ken. "Filters for common resampling tasks." Proc. of Graphics Gems I (1990): 147-165.
[6]Durand, Frédo, and Julie Dorsey. "Fast bilateral filtering for the display of high-dynamic-range images." Proc. of The 29th annual conference on computer graphics and interactive techniques. 2002.
[7]Dong, Chao, Chen Change Loy, Kaiming He and Xiaoou Tang. "Learning a deep convolutional network for image super-resolution." Proc. of European conference on computer vision. 2014.
[8]Dong, Chao, Chen Change Loy, and Xiaoou Tang. "Accelerating the super-resolution convolutional neural network." Proc. of European conference on computer vision. 2016.
[9]Juncheng Li, Faming Fang, Jiaqian Li, Kangfu Mei, and Guixu Zhang. "MDCN: Multi-scale Dense Cross Network for Image Super-Resolution." Proc. of IEEE Transactions on circuits and systems for video technology. 2020.
[10]Wei-Sheng Lai, Jia-Bin Huang, Narendra Ahuja, and Ming-Hsuan Yang. "Fast and accurate image super-resolution with deep laplacian pyramid networks." Proc. of IEEE transactions on pattern analysis and machine intelligence. 2018.
[11]Kim, Jiwon, Jung Kwon Lee, and Kyoung Mu Lee. "Accurate image super-resolution using very deep convolutional networks." Proc. of IEEE conference on computer vision and pattern recognition. 2016.
[12]Kim, Jiwon, Jung Kwon Lee, and Kyoung Mu Lee. "Deeply-recursive convolutional network for image super-resolution." Proc. of IEEE conference on computer vision and pattern recognition. 2016.
[13]Wenzhe Shi, Jose Caballero, Ferenc Husz´ar, Johannes Totz, Andrew P. Aitken,
Rob Bishop, Daniel Rueckert, and Zehan Wang. "Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network." Proc. of IEEE conference on computer vision and pattern recognition. 2016.
[14]T Tong Tong, Gen Li, Xiejie Liu, and Qinquan Gao. "Image super-resolution using dense skip connections." Proc. of IEEE international conference on computer vision. 2017.
[15]Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. "Deep residual learning for image recognition." Proc. of IEEE conference on computer vision and pattern recognition. 2016.
[16]Huang, Gao, Zhuang Liu, and Laurens van der Maaten. "Densely connected convolutional networks." Proc. of IEEE conference on computer vision and pattern recognition. 2017.
[17]Yulun Zhang, Yapeng Tian, Yu Kong, Bineng Zhong, and Yun Fu. "Residual dense network for image super-resolution." Proc. of IEEE conference on computer vision and pattern recognition. 2018.
[18]Juncheng Li, Faming Fang, Kangfu Mei, and Guixu Zhang. "Multi-scale residual network for image super-resolution." Proc. of European Conference on Computer Vision (ECCV). 2018.
[19]Ronneberger, Olaf, Philipp Fischer, and Thomas Brox. "U-net: Convolutional networks for biomedical image segmentation." Proc. of International Conference on medical image computing and computer-assisted intervention. 2015.
[20]Bank, Dor, Noam Koenigstein, and Raja Giryes. "Autoencoders." Proc. of arXiv preprint arXiv:2003.05991. 2020.
[21]Jun Fu, Jing Liu, Jie Jiang, Yong Li, Yongjun Bao, and Hanqing Lu. "Scene segmentation with dual relation-aware attention network." Proc. of IEEE Transactions on neural networks and learning systems. 2020.