臺灣博碩士論文加值系統

使用低成本攝影機所拍攝的深度圖總是有大範圍缺少深度資料，且存在著嚴重的雜訊，導致深度圖的不穩定。這些缺陷在3D視訊合成影像上，成像的物體因深度值缺乏、破損或跳動，造成視覺感受度不佳。為了增強深度圖的品質與穩定，本文提出了經由Kinect彩圖資訊以及其深度圖，經由時間、空間上的資訊進行無切割式多重缺陷空洞填補，由大範圍時間性修補至小範圍輪廓性修補，以加強深度穩定性。其修補深度圖能夠提供Kinect彩色影像，合成不錯的虛擬視角，完成有品質之立體視訊顯像。為了驗證本文方法有效性，提出了滿足人眼視覺系統與適合低成本視訊品質的評定方式，以SSIM(Structural Similarity)為基礎，對修補後的深度圖進行有效的評估。模擬結果可呈現良好之深度圖和虛擬視角影像，量化的評估也證實實驗的實用性，使得研究對未來的3D成像系統及發展有極大的貢獻與應用。

Depth maps captured by the low-cost sensor are always full noise. On 3D videos, defective depth maps lead to quality inferior. In this letter, we propose an algorithm to repair depth maps. Its framework consists of two repairing procedures. First, non-segmentation hole filling and color edge guiding achieved depth map reconstruction. Second, used two processing to make depth map stability: temporal prediction-based depth-coherence processing and temporal occlusion-excluded edge dilating-to-tailoring. The simulation result demonstrated that the proposed algorithm obtains acceptable depth map and stable quality. In addition, our algorithm can procure firmware realization. For affirming the performance of proposed scheme, competent quantitative metrics are specified to the appropriated experiments. Based on the proposed depth refinement, stereoscopic 3D approach can be attained by low-cost depth sensor photographing.

摘要 i
Abstract ii
致謝 iii
目錄 iv
圖目錄 vii
表目錄 viii
第一章、 簡介 1
1.1 3D視訊技術相關背景與趨勢 1
1.1.1 3D影像拍攝技術 1
1.1.2 3D影像成像格式 2
1.2 影像品質的評估與測量方法 4
1.3 研究動機與目的 5
1.4 論文架構 5
第二章、 相關研究 7
2.1 Kinect深度攝影機與深度影像繪圖法(Depth Image Based Rendering) 7
2.1.1 Kinect深度攝影機簡介與深度攝影機使用相關研究 7
2.1.2 深度影像繪圖法(Depth Image Based Rendering) 10
2.2 影像品質的評估與測量方法 11
第三章、 低品質連續深度圖修復之研究 14
3.1 Kinect Sensing-Loss Hole(KSLH)判定機制 15
3.2 無需影像切割進行KSLH之修補 16
3.2.1 KSLH之時間性修補 16
3.2.2 KSLH之空間性修補 17
3.3 修補扭曲的變形邊緣 19
3.4 時間上深度一致性 21
3.4.1. 基於時間預測的深度一致性 21
3.4.2. 時間上深度邊緣裁剪一致性 22
3.4.3. 形態學與非線性運算之修補 23
3.5 雙因子濾波器(Bilateral Filter) 24
3.6 Histogram-equalized SSIM與類真深度圖 24
第四章、 實驗結果與討論 27
4.1 修補結果 27
4.2 客觀H-SSIM與主觀定量評估 32
第五章、 結論與未來展望 34
5.1 結論 34
5.2 未來展望 35
參考文獻 36

[1] F. Blais, “Review of 20 years of range sensor development,” Journal of Electronic Imaging, vol. 13, no. 1, pp. 231–243, 2004.
[2] Y. S. Ho and E. K. Lee, “Generation of Multi-view Video using A Fusion Camera System for 3D Displays, ” IEEE Trans. Consumer Electronics, vol. 56, no. 4, pp. 2797-2805, 2010.
[3] D. Ferstl, R. Ranftl, M. Ruther, and H. Bischof, “Multi-Modality Depth Map Fusion using Primal-Dual Optimization,” IEEE Int’l Conf. Computational Photography, pp. 1-8, April 2013.
[4] J. Zhu, L. Wang, R. Yang, J. E. Davis, and Z. Pan, “Reliability Fusion of Time-of-Flight Depth and Stereo Geometry for High Quality Depth Maps,” IEEE Trans. Pattern Anal., vol. 33, no. 7, pp. 1400-1414, July 2011.
[5] W.C. Chiu, U. Blanke, and M. Fritz, “Improving the Kinect by Cross-Modal Stereo,” The 22nd British Machine Vision Conference (BMVC 2011), pp.116.1–116.10, 2010.
[6] K.J. Yoon, I.S. Kweon, “Locally Adaptive Support-Weight Approach for Visual Correspondence Search,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 28, no. 4, pp. 650-656, 2006.
[7] N. Chang, T.H. Tsai, B.H. Hsu, Y.C. Chen, and T.S. Chang, “Algorithm and architecture of disparity estimation with mini-census adaptive support weight,” IEEE Trans. Circuits Syst. Video Technol., vol. 20, no. 6, pp. 792–805, June 2010
[8] J. Fu, S. Wang, Y. Liu, S. Li, and W. Zeng, “Kinect-Like Depth Denoising,” IEEE Int’l Symposium on Circuits and Systems (ISCAS), Seoul Korea, pp. 512-515, May 2012.
[9] C. Fehn, “Depth-image-based rendering (DIBR), compression, and transmission for a new approach on 3D-TV,” Conf. Stereoscopic Displays and Applications, San Jose, Ca., USA, pp. 93-104, Jan. 2004.
[10] R. Dosselmann, X. D. Yang, “Existing and emerging image quality metrics”, Proceedings of the Canadian Conference on Electrical and Computer Engineering, pp. 1906-1913, 2006.
[11] Z. Wang, A. C. Bovik, H. R. Sheikh, and E. P. Simoncelli,“Image quality assessment: from error visibility to structural similarity”, IEEE Trans. on Image Processing, vol. 13,no. 4, pp. 600-612, 2004.
[12] Y. S. Ho and Y. S. Kang, “An Efficient Image Rectification Method for Parallel Multi-camera Arrangement,” IEEE Trans. Consumer Electronics, Toronto, Ont., Canada, vol. 57, no. 3, pp. 1041-1048, Aug. 2011.
[13] L. Corte-Real, H. Goncalves, and J. A. Goncalves, ”Automatic Image Registration through Image Segmentation and SIFT,” IEEE Trans. geoscience and Remote Sensing, vol. 49, no. 7, pp. 2589-2600, July 2011.
[14] W. J. Tam and Z. Liang. “Non-uniform smoothing of depth maps before image-based rendering,” in Proceedings of SPIE Conference on Three Dimensional TV, Video, and Display III (ITCOM’04), vol. 5599, pp. 173–183. Oct. 2004.
[15] K. He, J. Sun, and X. Tang. “Guided image filtering,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 35, no. 6, pp. 1397–1409, Jun. 2013.
[16] J. Liu, X. Gong, J. Liu, “Guided Inpainting and Filtering for Kinect Depth Maps,” IEEE Int’l Conf. Pattern Recognition(ICPR 2012), pp. 2055–2058, 2012.
[17] J. Jiao, R. Wang, W. Wang, D. Li, and W. Gao, “Color Image-Guided Boundary-Inconsistent Region Refinement for Stereo Matching,” IEEE Trans. on circuits and systems for video technology, vol. 27, no. 5, May 2017.
[18] L. Chen, H. Lin, S. Li, “Depth image enhancement for Kinect using region growing and bilateral filter,” 21st Int’l Conf. on Pattern Recognition (ICPR 2012), pp. 3070-3073, Nov. 2012.
[19] S.Y. Kim, J.H. Cho, A. Koschan, M. A. Abidi, “Spatial and temporal enhancement of depth images captured by a time-of-flight depth sensor,” IEEE Int’l Conf. on Pattern Recognition (ICPR), pp. 2358-2361, 2010
[20] S. Matyunin, D. Vatolin, Y. Berdnikov, and M. Smirnov. “Temporal filtering for depth maps generated by kinect depth camera,” 3DTV Conference: The True Vision - Capture, Transmission and Display of 3D Video, pp. 1-4, 2011.
[21] L. Zhang and W. J. Tam, “Stereoscopic image generation based on depth images,” IEEE Trans. On Broadcasting, vol. 51, no. 2, pp. 191-199, June 2005
[22] Z. Wang, E. P. Simoncelli, and A. C. Bovik, “Multi-scale structural similarity for image quality assessment,” IEEE Asilomar Conf. Signals, Syst. Comput., pp. 1398-1402. Nov. 2003.
[23] L. Sheng and K. N. Ngan, “Depth enhancement based on hybrid geometric hole filling strategy,” in Proc. 20th IEEE Int. Conf. Image Process., pp. 2173-2176, Sep. 2013.
[24] P. J. Lee and Effendi, “Nongeometric Distortion Smoothing Approach for Depth Map Preprocessing,” IEEE Trans. Multimedia, vol. 13, no. 2, pp. 246-254, April 2011.