臺灣博碩士論文加值系統

自從第一台光場相機在2012年11月推出後， 光場相機的研究與應用逐漸被重視，與傳統相機不同的是，光場相機不只能記錄光的強度，還能得知光的角度資訊。除此之外，僅僅透過一次拍攝便可以得到足夠的資訊去對影像做重組、深度估測以及不同的對焦。
另一方面，立體匹配也是一個熱門的研究主題，我們可以藉由兩張相同場景、不同角度的影像得出物體的深度，而深度資訊可以進一步地去做許多應用。除此之外，立體匹配當中的局部匹配被廣泛地應用在光場相機的重組和深度估測處理上。
在這篇論文中，我們主要分為三部分。第一部分是藉由局部匹配改良既有的光場影像重組技術。第二部分是提出新的立體匹配，是以影像切割為主，配合超像素的自動調變以及能適用任意切割形狀比對的局部匹配演算法和一些進一步的處理。第三部分是針對光場影像的深度估測，尤其是部分難以藉由立體匹配得到良好深度資訊的影像，我們藉由影像切割以及局部最佳對焦焦距的方式去得到深度資訊。

After the releasing of Plenoptic camera in November 2012, the research of light field camera is getting popular in recent years. The main difference between Plenoptic camera and traditional camera is that the angular information of light ray can be acquired by the former one. With one shot only, we can reconstruct the depth of scene and render the micro images into one final image from different views. We can also change the focal distance to make near or far objects clear. These are the appealing advantages of Plenoptic camera.
Stereo matching is also a popular research topic since we can obtain depth information by two images from left and right views. Many applications can be done if we have the accurate depth information about an image. Besides, the concept of stereo matching can be used in light field image rendering to get better result.
In this thesis, we divide the contents into three parts. The first part is to enhance the original rendering technique used in light field image with better local matching algorithm added. The second part is stereo matching. We use segmentation to help stereo matching and find an auto adjustment method to decide the best number of superpixel for each image. We also find a new local matching algorithm that is efficient especially for stereo matching after segmentation. Some techniques that can further increase the result are also added. The third part is a new depth estimation method used in light field image especially for some light field images that are hard to estimate depth by stereo matching. The method to recover depth information is based on segmentation and images from different focal distance.

誌謝 i
中文摘要 ii
ABSTRACT iii
CONTENTS iv
LIST OF FIGURES vii
LIST OF TABLES xii
Chapter 1 Introduction 1
1.1 Stereo matching 1
1.2 Light Field Camera 4
1.3 Organization 6
Chapter 2 Implementation of Light Field Camera Rendering 8
2.1 Introduction 8
2.2 Raw Data and Rendering Method 9
2.3 Simulation Result 10
Chapter 3 Stereo Matching Methods 13
3.1 Introduction 13
3.2 Feature-based Methods 14
3.2.1 Scale Invariant Feature Transform [10] 14
3.2.2 RANSAC[15] 24
3.2.3 Simulation Result and Discussion 25
3.3 Window-based Methods 27
3.3.1 Local Matching Algorithms 27
3.3.2 Adaptive Support-Weight Approach for Correspondence Search [19] 29
3.3.3 Segmentation-Based Adaptive Support for Accurate Stereo Correspondence [20] 33
3.3.4 A Two-Stage Correlation Method for Stereoscopic Depth Estimation [22] 39
3.3.5 Local Disparity Estimation With Three-Moded Cross Census and Advanced Support Weight [23] 43
3.3.6 Summation 49
3.4 Global-based Methods 49
3.4.1 Segment-Based Stereo Matching Using Belief Propagation and a Self-Adapting Dissimilarity Measure [27] 49
3.4.2 Summation 53
Chapter 4 Proposed Methods for Stereo Matching 54
4.1 Introduction 54
4.2 Stereo Images Dataset 56
4.3 Gradient Map 57
4.4 ERS and Merging 60
4.4.1 ERS (Entropy rate superpixel) [31] 60
4.4.2 Merging [32] 62
4.5 Superpixel Auto Adjustment 64
4.6 Local Matching Algorithm 69
4.6.1 Introduction 69
4.6.2 WSAD and WNCC 70
4.6.3 BWSAD and upper limit 72
4.7 Dilation and Background Limitation 74
4.8 Simulation and Result 76
Chapter 5 Proposed method for light field images 88
5.1 Introduction 88
5.2 Patch Size and Focal Distance 90
5.3 Proposed method 91
5.4 Simulation result 93
Chapter 6 Conclusion and Future Work 95
6.1 Conclusion 95
6.2 Future Work 95
REFERENCE 97

[1]M. J. Hannah, "Computer matching of areas in stereo images," DTIC Document1974.
[2]R. Szeliski, Computer vision: algorithms and applications. Springer Science & Business Media, 2010.
[3]O. Faugeras, Q.-T. Luong, and T. Papadopoulo, The geometry of multiple images: the laws that govern the formation of multiple images of a scene and some of their applications. MIT press, 2004.
[4]C. Loop and Z. Zhang, "Computing rectifying homographies for stereo vision," in Computer Vision and Pattern Recognition, 1999. IEEE Computer Society Conference on., 1999, vol. 1: IEEE.
[5]D. Marr, "Vision: A computational investigation into the human representation and processing of visual information," 1982.
[6]E. H. Adelson and J. R. Bergen, The plenoptic function and the elements of early vision. Vision and Modeling Group, Media Laboratory, Massachusetts Institute of Technology, 1991.
[7]M. Levoy and P. Hanrahan, "Light field rendering," in Proceedings of the 23rd annual conference on Computer graphics and interactive techniques, 1996, pp. 31-42: ACM.
[8]R. Ng, M. Levoy, M. Brédif, G. Duval, M. Horowitz, and P. Hanrahan, "Light field photography with a hand-held plenoptic camera," Computer Science Technical Report CSTR, vol. 2, no. 11, pp. 1-11, 2005.
[9]A. Lumsdaine and T. Georgiev, "Full resolution lightfield rendering," Indiana University and Adobe Systems, Tech. Rep, 2008.
[10]T. Georgiev and A. Lumsdaine, "Focused plenoptic camera and rendering," Journal of Electronic Imaging, vol. 19, no. 2, pp. 021106-021106-11, 2010.
[11]D. G. Lowe, "Object recognition from local scale-invariant features," in Computer vision, 1999. The proceedings of the seventh IEEE international conference on, 1999, vol. 2, pp. 1150-1157: Ieee.
[12]D. G. Lowe, "Distinctive image features from scale-invariant keypoints," International journal of computer vision, vol. 60, no. 2, pp. 91-110, 2004.
[13]K. Mikolajczyk, "Detection of local features invariant to affine transformations," Citeseer, 2011.
[14]C. Schmid and R. Mohr, "Local grayvalue invariants for image retrieval," IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 19, no. 5, pp. 530--534, 1997.
[15]M. A. Fischler and R. C. Bolles, "Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography," Communications of the ACM, vol. 24, no. 6, pp. 381-395, 1981.
[16]D. Scharstein and R. Szeliski, "A taxonomy and evaluation of dense two-frame stereo correspondence algorithms," International journal of computer vision, vol. 47, no. 1-3, pp. 7-42, 2002.
[17]D. Scharstein and R. Szeliski, "High-accuracy stereo depth maps using structured light," in Computer Vision and Pattern Recognition, 2003. Proceedings. 2003 IEEE Computer Society Conference on, 2003, vol. 1, pp. I-195-I-202 vol. 1: IEEE.
[18]J.-J. Ding and S.-C. Chuang, "Adaptive preprocessing and combination techniques for light field image rendering," in Consumer Electronics-Taiwan (ICCE-TW), 2015 IEEE International Conference on, 2015, pp. 244-245: IEEE.
[19]K.-J. Yoon and I. S. Kweon, "Adaptive support-weight approach for correspondence search," IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 28, no. 4, pp. 650-656, 2006.
[20]F. Tombari, S. Mattoccia, and L. Di Stefano, "Segmentation-based adaptive support for accurate stereo correspondence," in Pacific-Rim Symposium on Image and Video Technology, 2007, pp. 427-438: Springer.
[21]M. Gerrits and P. Bekaert, "Local stereo matching with segmentation-based outlier rejection," in The 3rd Canadian Conference on Computer and Robot Vision (CRV''06), 2006, pp. 66-66: IEEE.
[22]N. Einecke and J. Eggert, "A two-stage correlation method for stereoscopic depth estimation," in Digital Image Computing: Techniques and Applications (DICTA), 2010 International Conference on, 2010, pp. 227-234: IEEE.
[23]Z. Lee, J. Juang, and T. Q. Nguyen, "Local disparity estimation with three-moded cross census and advanced support weight," IEEE Transactions on Multimedia, vol. 15, no. 8, pp. 1855-1864, 2013.
[24]R. Khoshabeh, S. H. Chan, and T. Q. Nguyen, "Spatio-temporal consistency in video disparity estimation," in 2011 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2011, pp. 885-888: IEEE
[25]K. Zhang, J. Lu, and G. Lafruit, "Cross-based local stereo matching using orthogonal integral images," IEEE Transactions on Circuits and Systems for Video Technology, vol. 19, no. 7, pp. 1073-1079, 2009.
[26]A. Desolneux, L. Moisan, and J.-M. Morel, From gestalt theory to image analysis: a probabilistic approach. Springer Science & Business Media, 2007.
[27]A. Klaus, M. Sormann, and K. Karner, "Segment-based stereo matching using belief propagation and a self-adapting dissimilarity measure," in 18th International Conference on Pattern Recognition (ICPR''06), 2006, vol. 3, pp. 15-18: IEEE.
[28]L. Hong and G. Chen, "Segment-based stereo matching using graph cuts," in Computer Vision and Pattern Recognition, 2004. CVPR 2004. Proceedings of the 2004 IEEE Computer Society Conference on, 2004, vol. 1, pp. I-74-I-81 Vol. 1: IEEE.
[29]M. Bleyer and M. Gelautz, "Graph-based surface reconstruction from stereo pairs using image segmentation," in Electronic Imaging 2005, 2005, pp. 288-299: International Society for Optics and Photonics.
[30]P. F. Felzenszwalb and D. P. Huttenlocher, "Efficient belief propagation for early vision," International journal of computer vision, vol. 70, no. 1, pp. 41-54, 2006.
[31]M.-Y. Liu, O. Tuzel, S. Ramalingam, and R. Chellappa, "Entropy rate superpixel segmentation," in Computer Vision and Pattern Recognition (CVPR), 2011 IEEE Conference on, 2011, pp. 2097-2104: IEEE.
[32]High Accuracy and High Robust Natural Image Segmentation Algorithm without Parameter Adjusting Copyright (c) 2015, I-Fan Lu, *Jian-Jiun Ding, and Hsuan-Yi Ko @ Graduate Institute of Communication Engineering, National Taiwan University
[33]J.-J. Ding, N.-C. Wang, S.-C. Chuang, and R. Y. Chang, "Morphology-based disparity estimation and rendering algorithm for light field images," in Consumer Electronics-Taiwan (ICCE-TW), 2016 IEEE International Conference on, 2016, pp. 1-2: IEEE.
[34]J. Han, Z. Wu, L. Li, and Y. Ji, "FPGA Implementation for Binocular Stereo Matching Algorithm Based on Sobel Operator," International Journal of Database Theory and Application, vol. 9, no. 4, pp. 221-230, 2016.
[35]M. W. Tao, S. Hadap, J. Malik, and R. Ramamoorthi, "Depth from combining defocus and correspondence using light-field cameras," in Proceedings of the IEEE International Conference on Computer Vision, 2013, pp. 673-680.
[36]H.-G. Jeon et al., "Accurate depth map estimation from a lenslet light field camera," in 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2015, pp. 1547-1555: IEEE.
[37]J. Lu, K. Shi, D. Min, L. Lin, and M. N. Do, "Cross-based local multipoint filtering," in Computer Vision and Pattern Recognition (CVPR), 2012 IEEE Conference on, 2012, pp. 430-437: IEEE.
[38]D. Min, J. Lu, and M. N. Do, "Depth video enhancement based on weighted mode filtering," IEEE Transactions on Image Processing, vol. 21, no. 3, pp. 1176-1190, 2012.
[39]Q. Yang, R. Yang, J. Davis, and D. Nistér, "Spatial-depth super resolution for range images," in 2007 IEEE Conference on Computer Vision and Pattern Recognition, 2007, pp. 1-8: IEEE.
[40]C. Rhemann, A. Hosni, M. Bleyer, C. Rother, and M. Gelautz, "Fast cost-volume filtering for visual correspondence and beyond," in Computer Vision and Pattern Recognition (CVPR), 2011 IEEE Conference on, 2011, pp. 3017-3024: IEEE.
[41]L. Wang, Z. Liu, and Z. Zhang, "Feature based stereo matching using two-step expansion," Mathematical Problems in Engineering, vol. 2014, 2014.
[42]J. Čech, J. Sanchez-Riera, and R. Horaud, "Scene flow estimation by growing correspondence seeds," in Computer Vision and Pattern Recognition (CVPR), 2011 IEEE Conference on, 2011, pp. 3129-3136: IEEE