由於硬體以及影像處理技術的快速提升，光場相機的研究以及應用如同雨後春筍一般快速的被提出。有別於傳統相機只能捕捉場景中的位置資訊，光場相機能紀錄空間中的光線位置與角度資訊。因此，光場相機可以做到重新對焦、改變視角以及3D 立體圖重建的功能。然而，目前的光場相機常常受到影像模糊以及解析度不高的問題，使得最後重建影像品質不佳。有鑑於此，在本篇論文中我們針對光場影像設計一套包含影像重建、影像解模糊以及影像內插的系統，並分別針對這三個部份分別提出新的方法；其中，解模糊與影像內插的部分甚至可以應用在任何自然影像上。
在取像的部分，由於我們的光場影像受到衰減效應的影響，導致既有的方法無法順利產生高品質的影像。因此，我們提出先利用白場影像擷取相機的衰減曲線並估測出補償模型來還原無衰減的影像。不過由於白場影像的光源與實際影像的光源不同，會造成部分影像有過度補償的情況，導致最終影像仍有些許失真。針對這個問題，我們也定義了一個適應性的函數來針對不同的影像調整其補償模型。在最後的實驗結果指出我們所提出來的方法相比於既有的方法可以重建出較高品質的影像。
在影像解模糊當中，由於許多現有的方法以效率換取了效果，使得這些演算法雖然效果優良但是效率極差。在本篇論文中，我們提出了雙L0模型來估測模糊函數，由於模糊函數一般具有稀疏性，比起L2或是L0模型，利用雙L0模型可以更精準的估測模糊函數並減少雜訊。為了更進一步減少雜訊，在估測模糊函數時我們也引進了影像結構抽取的概念，將模糊函數本身作為影像結構而雜訊當作是紋理來得到更精確的函數。此外，我們在估測中繼影像時所採用的輸入為影像結構的梯度而非影像梯度本身，以減少影像中雜訊對於模糊函數估測時的干擾。在最後模糊影像還原的部分，我們也提出了高效率的L0-L2 反捲積演算法，讓最終的影像能夠在短時間內被重建出來。實驗結果顯示，我們提出的方法相比於既存的演算法可以在最短的時間內還原出最高分數的影像。
為了讓低解析度的光場影像能夠在放大到高解析度的同時不流失過多的影像細節，我們也提出了一個改良版的方向三次捲積內插演算法。此演算法會先計算欲內插像素的梯度方向，當某個方向的梯度值大於一個臨界值，則沿著垂直於梯度方向的方向進行三次捲積內插。若梯度值都很小或是很大，則將沿著兩個互相正交的方向的內插結果利用線性組合算出像素值。為了使方法更穩定，我們引入了最小誤差臨界值演算法來對於每個影像產生其較佳的臨界值。在保留影像細節的部分，我們也引進了影像結構抽取方法來分離影像細節與結構。對於影像結構的內插是採用上述的方法；而影像細節的部份我們則另外提出一個以區域梯度為導向的內插演算法。模擬結果顯示，提出的方法相比於現有的方法可以獲得較高的分數，並且也提供最佳的視覺效果。

Due to the fast developments in hardware and image processing techniques, the researches and applications of light field camera are springing up like mushrooms. Different from the conventional camera which can only capture position information in the scene, light field camera is able to record both position information and angular information of light rays. As a result, light field camera can realize the refocusing, changing perspective, and 3D stereo image reconstruction. However, light field camera usually suffers from the problems of blur and lack of resolution which yield the poor quality in the final image reconstruction. Based on this viewpoint, we design a system for the light field camera which consists of image rendering, deblurring, and interpolation. For each part of this system, we propose new methods respectively where the deblurring and interpolation parts can even be used to any natural image.
In the image rendering part, existing methods are not able to generate the high quality rendering results due to our light field images suffer from the attenuation effect. Hence, we suggest to record the attenuation model of light field camera from the white image and estimate the compensation model to recover the original image. Due to the difference between the light source of raw image data and white image, the phenomenon of overcompensation will occur and cause some artifact in the final image. To address this problem, we define an adaptive function to adjust the compensation model for different images. Simulation results indicate that the proposed method can render images with higher quality than existing methods.
For the image deblurring, due to many existing methods sacrifice the efficiency for higher performance, these methods have good performance but poor efficiency. In this thesis, we propose a dual-L0 model to estimate the blur function. Since the blur function usually have high sparsity, compared to L2 or L0 model, using dual-L0 can not only estimate the blur function more precisely but also reduce the noise. To further reduce the noise in the blur kernel, we employ the structure extraction to our blur kernel estimation which views the blur function as image structure and the noise as texture. In addition, we use the gradient of structure image to estimate the interim image instead of the gradient of original image which can reduce the interference from the noise in the images. In the final blurring image reconstruction part, we also propose a high efficiency L0-L2 deconvolution algorithms so that the final image can be reconstructed in a short time. The experimental results show that the proposed method can recover the highest score image in the least computation time.
To zoom a low resolution light field image into high resolution one without losing too many details, we also propose an improved directional cubic convolution interpolation algorithm. This algorithm will first estimate the gradient direction of the missing pixel and interpolate the missing pixel along the direction that orthogonal to its gradients direction by the cubic convolution interpolation. If two of the orthogonal gradient are either very small or very large, the missing pixel is interpolated by the linear combination of the interpolation results along these two orthogonal directions. To stabilize the performance, we apply the minimum error thresholding to determine the better thresholding for each image. In order to preserve the image details, we also apply the image structure extraction to separate the details and structure of an image. For the interpolation of image structure, we use the above method, while we propose another interpolation scheme based on the local gradient to interpolate the image details. Simulation results show that the proposed method gives higher performance then existing methods and also have the best visual quality.

口試委員會審定書 #
誌謝 i
中文摘要 ii
ABSTRACT iv
CONTENTS vii
LIST OF FIGURES xi
LIST OF TABLES xvii
Chapter 1 Introduction 1
1.1 Motivation 1
1.2 Primary Contributions of This Thesis 3
Chapter 2 Review of the Existing Image Rendering Methods for Light Field Camera 6
2.1 Light Field Camera 6
2.1.1 Pinhole light field camera 6
2.1.2 Plenoptic Camera 1.0 [6] 8
2.1.3 Plenoptic Camera 2.0 [9] 9
2.2 Preliminary of Light Field Image Rendering 11
2.2.1 Sub-image Extraction from the Raw Image 11
2.2.2 Disparity Estimation 12
2.2.3 Image Blending 13
2.3 Disparity Estimation Methods 14
2.3.1 Disparity Estimation Based on Feature Point Matching 14
2.3.2 Sum of Absolute Difference Method 16
2.3.3 Weighted Sum of Absolute Difference Method 16
2.4 Image Blending Methods 17
2.4.1 Linear Blending 17
2.4.2 Pyramid Blending 19
2.4.3 Multi-Band Separation Image Blending 21
2.5 Summary 22
Chapter 3 Proposed Image Rendering Method Based on White Image Compensation 23
3.1 Overview of the Proposed Framework 23
3.2 White Image Compensation 24
3.3 Modification function 27
3.4 Proposed Light Field Image Rendering 29
3.5 Simulation Results 30
3.5.1 Visual Comparison 30
3.5.2 Refocusing Rendering 35
3.6 Summary 36
Chapter 4 Review of Existing Image Deblurring Algorithms 37
4.1 Basic Model of the Image Deblurring Problem 37
4.2 Non-Blind Image Deconvolution 38
4.2.1 Direct Inverse Filter 38
4.2.2 Wiener Filter [14] 40
4.2.3 Richardson-Lucy Deconvolution Method [34][35] 43
4.2.4 Maximum a Posterior Based Deconvolution [15] 46
4.2.5 Fast Image Deconvolution Using Hyper-Laplacian Priors [29] 48
4.2.6 Discussion 51
4.3 Blind Deconvolution 52
4.3.1 Blind Deconvolution Using a Normalized Sparsity Measure[20] 53
4.3.2 Kernel Estimation from the salient structure 54
4.3.3 Deblurring Text Images via L0-Regularized Intensity and Gradient Prior [21] 57
4.3.4 Blind Image Deblurring Using the Dark Channel Prior[16] 63
4.3.5 Discussion 66
4.4 Summary 67
Chapter 5 Proposed Blind Image Deblurring Algorithm Using Dual-L0 Kernel Estimation and L0-L2 Deconvolution 68
5.1 Relative Total Variation Structure Extraction 68
5.2 Proposed Kernel Estimation Algorithm 70
5.2.1 Estimate Blur Kernel from the Image Structure 70
5.2.2 Kernel Refinement by Extracting Kernel Structure 72
5.2.3 Image restoration from the L0 73
5.3 Proposed Non-Blind L0-L2 Deconvolution Scheme 73
5.4 Proposed Dual-L0 Kernel Constraint 77
5.5 Simulation Results 80
5.5.1 Performance on proposed blind image deblurring method without Dual-L0 kernel constraint 81
5.5.2 Performance on proposed blind image deblurring method with Dual-L0 kernel constraint 83
5.5.3 Visual Comparison 85
5.5.4 Comparison on Estimated Blur Kernel 92
5.6 Summary 94
Chapter 6 Proposed Image Interpolation Algorithm 95
6.1 Related Work 95
6.1.1 Bilinear Interpolation 95
6.1.2 Cubic Convolution Method 96
6.1.3 New Edge-Directed Interpolation 98
6.1.4 Directional Cubic Convolution Interpolation 100
6.1.5 Discussion 103
6.2 Proposed Image Interpolation Algorithm 104
6.2.1 Modified Edge detection 104
6.2.2 Kittler-Illingworth Minimum Error Thresholding 105
6.2.3 Multilayer Structure-Texture Decomposition 108
6.2.4 Local gradient based interpolation 110
6.3 Proposed Non-integer Scaling interpolation method 115
6.3.1 Estimate the Edge direction from gradient 115
6.3.2 Construct the edge map by the spline interpolation 116
6.3.3 Edge-oriented interpolation method 117
6.3.4 Select the strong edge 118
6.4 Simulation Results 119
6.5 Evaluation on the Non-integer scaling method 125
6.6 Summary 127
Chapter 7 Conclusion and Future Work 128
7.1 Conclusion 128
7.2 Future Work 130
REFERENCE 132

[1]M. Brown, R. Szeliski and S. Winder. “Multi-image matching using multi-scale oriented patches.” In Computer Vision and Pattern Recognition, vol. 1, pp. 510-517, 2005.
[2]D. G. Lowe, "Distinctive image features from scale-invariant keypoints," Int. J. Computer Vision, vol. 60, issue 2, pp. 91-110, 2004.
[3]J. J. Ding, N. C. Wang, S. C. Chuang and R. Y. Chang. “Morphology-based disparity estimation and rendering algorithm for light field images.” IEEE International Conference on Consumer Electronics - Taiwan, pp. 1-2 , 2016.
[4]P. J. Burt and E. H. Adelson. “A multiresolution spline with application to image mosaics.” in ACM Transaction on Graph, vol.2, pp 217-236, 1983.
[5]J. J. Ding and S. C. Chuang, “Adaptive preprocessing and combination techniques for light field image rendering,” IEEE International Conference on Consumer Electronics - Taiwan, pp. 244-245, 2015.
[6]Introduction of fundamental light field concept available from:
http://www.tgeorgiev.net/Asia2009/
[7]Lumsdaine, Andrew, and Todor Georgiev. "The focused plenoptic camera.", IEEE International Conference on Computational Photography, 2009.
[8]Adelson, Edward H., and James R. Bergen. “The plenoptic function and the elements of early vision.” Vision and Modeling Group, Media Laboratory, Massachusetts Institute of Technology, 1991.
[9]Georgiev, Todor, and Andrew Lumsdaine. "Superresolution with plenoptic camera 2.0,"Adobe Systems Incorporated Tech., 2009.
[10]Burt, Peter J., and Edward H. Adelson. "A multiresolution spline with application to image mosaics," in ACM Transactions on Graphics, vol.2, pp. 217-236, 1983.
[11]M. Brown, R. Szeliski, and S. Winder, “Multi-image matching using multi-scale oriented patches,” in IEEE Conference on Computer Vision and Pattern Recognition, vol. 1, pp.510-517, 2005.
[12]D. G. Lowe, “Distinctive image features from scale-invariant keypoints,” International Journal of Computer Vision, vol.60, pp.91-110, 2004.
[13]D. Kundur and D. Hatzinakos, “Bind image deconvolution,” IEEE Signal Processing Magazine, vol. 13, issue 3, pp. 43-64, 1996.
[14]N. Wiener, “Extrapolation, interpolation, and smoothing of stationary time series: with engineering applications,” MIT press, vol. 8, 1964.
[15]A. Levin, Y. Weiss, F. Durand, and W.T. Freeman, “Understanding and evaluating blind deconvolution algorithms,” in IEEE Conference on Computer Vision and Pattern Recognition, pp.1964-1971, 2009.
[16]J. Pan, Z. Hu, H. Pfister, and M. H. Yang, “Blind Image Deblurring Using Dark Channel Prior,” in IEEE Conference on Computer Vision and Pattern Recognition, 2016.
[17]L. Xu, Q. Yan, Y. Xia, and J.Jia, “Structure extraction from texture via relative total variation,” in ACM Transactions on Graphics, vol. 31, issue 6, 2012.
[18]L. Xu, C. Lu, Y. Xu, and J. Jia. “Image smoothing via l0 gradient minimization,” in ACM Transaction on Graphics. vol. 30, no. 174, 2011.
[19]C.T. Shen, W.L. Hwang and S.C. Pei. “Spatially-varying out-of-focus image deblurring with L1-2 optimization and a guided blur map,” in IEEE International Conference on Acoustics, Speech and Signal Processing, pp.1067-1072, 2012.
[20]D. Krishnan, T. Tay, and R. Fergus. “Blind deconvolution using a normalized sparsity measure,” in IEEE Conference on Computer Vision and Pattern Recognition, pp. 233-240, 2011.
[21]J. Pan, Z. Hu, Z. Su, and M. H. Yang. “Deblurring text images via L0-regularized intensity and gradient prior,” in IEEE Conference on Computer Vision and Pattern Recognition, pp. 2901-2908, 2014.
[22]L. Sun and J. Hays. “Super-resolution from internet-scale scene matching,” in IEEE Conference on Computational Photography, pp. 1-12, 2012.
[23]L. Xu and J. Jia. “Two-phase kernel estimation for robust motion deblurring,” in European Conference on Computer Vision , pp. 157-170, 2010.
[24]J. Pan, R. Liu, Z. Su and X. Gu. “Kernel estimation from salient structure for robust motion deblurring,” in Signal Processing: Image Communication, vol.28, pp. 1156-1170, 2013.
[25]L. Xu, S. Zheng, J. Jia. “Unnatural L0 sparse representation for naturalimage deblurring,” in IEEE Conference on Computer Vision and Pattern Recognition, pp.1107-1114, 2013.
[26]A. Kheradmand and P. Milanfar. “A general framework for regularized similarity-based image restoration,” in IEEE Transactions on Image Processing , vol.23, pp. 5136-5151, 2014.
[27]O. Whyte, J. Sivic, and A. Zisserman. "Deblurring Shaken and Partially Saturated Images," in IEEE Conference on Computer Vision Workshop, pp.745-752, 2011.
[28]O. Whyte, J. Sivic, A. Zisserman, and J. Ponce. "Non-uniform Deblurring for Shaken Images," in International Journal of Computer Vision, vol.98, pp.168-186, 2012.
[29]D. Krishnan and R. Fergus. "Fast Image deconvolution using hyper-Laplacian priors." in Advances in Neural Information Processing Systems, pp.1033-1041, 2009.
[30]Y. Chen, J. J. Ding, W. S. Lai, Y. J. Cheng, C. W. Chang and C. C. Chang. “High Quality image deblurring scheme using the pyramid hyper-Laplacian L2 norm priors algorithm,” in Pacific-Rim Conference on Multimedia, pp. 134-145, 2013.
[31]J. Pan, R. Liu, Z. Su and G. Liu. “Motion blur kernel estimation via salient edges and low rank prior,” in IEEE International Conference Multimedia and Expo, pp.1-6, 2014.
[32]G. Xu, G. Zheng, X. Xie and K. Fan. “Kernel optimization based on salient region detection for image deblurring,” in International Conference on Wireless, Mobile and Multi-Media, 2015.
[33]L. Liu and D. R. Chen. “Convergence of ℓ 2/3 Regularization for Sparse Signal Recovery,” in Asia-Pacific Journal of Operational Research, vol.32, issue4, 2015.
[34]W. H. Richardson, “Bayesian-based iterative method of image restoration,” JOSA, vol. 62, no. 1, pp.745-754, 1974.
[35]L.B. Lucy, “An iterative technique for the rectification of observed distributions.” The Astronomical Journal, vol. 79, pp. 745-754, 1974.
[36]Y. Wang, J. Yang, W.Yin, and Y. Zhang, “A new alternating minimization algorithm for total variation image reconstruction,” Society for Industrial and Applied Mathematics Journal on Imaging Sciences, vol.1, pp.248-272, 2008.
[37]R. Keys, “Cubic Convolution Interpolation for Digital Image Processing,” IEEE Transaction Acoustics, Speech, Signal Process, vol. 29, no.6, pp. 1153-1160, 1981.
[38]X. Li and M. Orchard, “New edge-directed interpolation,” in Proc. Int. Conf. Image Process, vol. 2, pp. 311-314, Sept. 2000.
[39]X. Zhang, S. Ma, Y. Zhang, L. Zhang, and W. Gao “Nonlocal edge-directed interpolation” Proc. 10th Pacific Rim Conf. Multimedia-Advances in Multimedia Information Processing, vol. 5879, pp. 1197-1207, 2009.
[40]H. H. Chen and J. J. Ding, “Structural similarity-based nonlocal edge-directed image interpolation,” in Proc. Picture Coding Symposium, pp. 289-292, 2013.
[41]D. Zhou, X. Shen, and W. Dong, “Image zooming using directional cubic convolution interpolation,” IET image processing, vol. 6, no. 6, pp. 627-634, 2012.
[42]S. Ousguine, F. Essannouni, L.Essannouni , and D. Aboutajdine, “A new image interpolation using gradient-orientation and cubic spline interpolation.” in International Journal of Innovation and Applied Studies, vol.5, no.3, pp.215-221, 2014.
[43]J. Kittler and J. Illingworth, “Minimum error thresholding,” Pattern Recognition, vol. 19, no. 1, pp. 41-47, 1986.
[44]A. Giachetti, and N. Asuni, “Fast Artifacts-Free Image Interpolation,” In British Machine Vision Conference ,pp. 1-10 , 2008.
[45]Kodak lossless image suite:
http://r0k.us/graphics/kodak/
[46]S. M. E. Harb, N.A.M. Isa, S. Salamah, “New adaptive interpolation scheme for image upscaling,” in Multimedia Tools and Application, pp.1–33, 2015.
[47]M. Jing, J. Wu, “Fast image interpolation using directional inverse distance weighting for real-time applications,” in Optics Communications , no.286, pp.111–116, 2013.
[48]M. J. Fadili, and E. T. Bullmore, “Wavelet-generalized least squares: a new BLU estimator of linear regression models with 1/f errors,” In NeuroImage, vol.15, pp.217-232, 2002.
[49]S. M. E. B. Harb, N. A. M. Isa, and S. A. Salamah, “An improved image magnification algorithm for color images,” in IEEE Region 10 Symposium, pp. 190-195, 2014.