在這篇論文中，我們提出了一個基於紋理生成的光場影像編輯之方法。我們從
光場影像修補這個問題出發，在解決光場影像修補這個問題之後，接著將我們的方
法應用到光場影像的縮放與光場影像的物件重新排列。在我們的方法中，光場影像
被視為是多視角的二維影像的集合，而不是由原始資料來解決這個問題。在給定一
個四維的光場影像之後，使用者將會在某的視角圈選需要被移除的物件，接著我們
把這個物件拓展到其它的視角，而所有的視角的這個物件也就是我們所需要修補
的四維空洞。我們的方法是利用二維影像的紋理生成演算法，拓展到四維空間，並
且在四維空間中定義補丁之間的距離函數來衡量四維空間中的相似度。換句話說，
我們利用這個方法一次生成整個四維影像，而非獨立的生成每一個視角的二維影
像。另外，我們也利用了深度圖以及極平面影像來做為輔助資料，幫助更準確的生
成四維光場影像資料。最後再將方法擴展到光場影像縮放與光場影像重新排列的
問題中。總結來說，我們提出了一個紋理生成的方法來達到光場影像編輯的目標。
此外，即使我們的深度資料非常粗糙，也足夠幫助我們更準確的生成四維光場影像。

We present an approach to light-field editing using patch-based method. We start from
the light-field completion problem, and then apply our method to light-field retargeting
and light-field reshuffling. In our work, we deal with light-field data at multi-view level
rather than raw data level because raw data varies with different light-field cameras.
Given a 4D light-field data, user specifies an object to be removed in a specific view, and
then we transfer the contour of this region to all the other view. After obtaining the 4D
region, we fill the 4D region based on patch-based synthesis. Compared to traditional 2D
image completion by patch-based method, we extend 2D PatchMatch algorithm to 4D
light-field PatchMatch. Our method aims to find the nearest neighbor field by measuring
distances between 4D patches. In other words, we synthesize 4D light-field instead of
independently synthesizing each 2D image. Besides, in order to retain spatial and angular
consistency when we synthesizing target light-field, we introduce the depth map to help
us finding similar patch. We also leverage depth maps to retain view consistency based
on the concept of epipolar plane image (EPI). In our method, we further define a patch
distance function based on color, depth map and EPI, and use the patch distance function
to measure patch similarity. This framework is applied to light-field retargeting and
reshuffling problem based on bidirectional similarity. To our knowledge, it is a first try
on light-field editing. In summary, we propose a method to edit light-field by measuring
4D patch similarity and it is not necessary to have a good depth map. In our experiment,
it works well even if we have only a rough depth maps.

誌謝 .................................................................................................................................. ii
摘要 ................................................................................................................................. iii
ABSTRACT .................................................................................................................... iv
CONTENTS ...................................................................................................................... v
LIST OF FIGURES ........................................................................................................ vii
Chapter 1 Introduction ............................................................................................... 1
Chapter 2 Related Work ............................................................................................. 4
2.1 Image Editing ................................................................................................ 4
2.2 Stereoscopic Image Inpainting ...................................................................... 5
2.3 Application on Light-field ............................................................................. 6
Chapter 3 Patch-based Light-field Completion .......................................................... 8
3.1 Contour Transfer ............................................................................................ 9
3.2 Depth Estimation Based on EPI .................................................................. 11
3.3 Bidirectional Similarity and PatchMatch on Light-field ............................. 14
3.3.1 Bidirectional Similarity .................................................................. 14
3.3.2 PatchMatch ..................................................................................... 17
3.3.3 Light-field Completion Using Patch-based Synthesis .................... 19
3.4 Patch Distance Function .............................................................................. 24
Chapter 4 Application .............................................................................................. 27
4.1 Light-field Retargeting ................................................................................ 27
4.2 Light-field Reshuffling ................................................................................ 29
Chapter 5 Result and Conclusion ............................................................................. 31
v
REFERENCE .................................................................................................................. 35
vi

[1] Barnes, C., Shechtman, E., Finkelstein, A. and Goldman, D. B. PatchMatch: A
randomized correspondence algorithm for structural image editing. In Proc. of ACM
SIGGRAGH. 2009.
[2] Birklbauer, C., Bimber, O. Light-field retargeting. Comput Graph Forum (Proc
Eurographics), pp. 295–303. 2012.
[3] Cho, T. S., Butman, M., Avidan, S. and Freeman, W. T. The patch transform and its
applications to image editing. In CVPR. 2008.
[4] Criminisi, A., Perez, P. and Toyama. K. Region filling and object removal by
exemplar-based image inpainting. IEEE Trans. Image Process. no. 9, pp.1200-1212.
2004.
[5] Darabi, S., Shechtman, E., Barnes, C., Goldman, D. B. and Sen, P. Image melding:
combining inconsistent images using patch-based synthesis. ACM Transactions on
Graphics (TOG), v.31 n.4, p.1-10. July 2012.
[6] Hervieu, A., Papadakis, N., Bugeau, A., Gargallo, P. and Caselles, V. Stereoscopic
image inpainting: Distinct depth maps and images inpainting. Proc. 20th Int. Conf.
Pattern Recognit. (ICPR), pp.4101-4104. 2010.
[7] Kim, C., Zimmer, H., Pritch, Y., Sorkine-Hornung, A. and Gross, M. Scene
reconstruction from high spatio-angular resolution light fields, ACM Transactions
on Graphics (TOG), v.32 n.4. July 2013.
[8] Li, N., Ye, J., Ji, Y., Ling, H. and Yu, J. Saliency Detection on Light Field. In IEEE
Conference on Computer Vision and Pattern Recognition (CVPR). 2014.
[9] Lowe, D. G. Distinctive image features from scale-invariant keypoints. IJCV,
60(2):91-110, 2004.
[10] Maeno, K., Nagahara, H., Shimada, A. and Taniguchi, R. Light field distortion
feature for transparent object recognition. Computer vision foundation, IEEE CVPR.
2013.
[11] Pritch, Y., Kav-Venaki, E. and Peleg, S. Shift-map image editing. In ICCV, 151--158.
2009.
[12] Simakov, D., Caspi, Y., Shechtman, E. and Irani, M. Summarizing visual data using
bidirectional similarity. In CVPR. 2008.
[13] Wang, L., Jin, H., Yang, R. and Gong, M. Stereoscopic inpainting: joint color and
depth completion from stereo images. IEEE CVPR. 2008.
[14] Wanner, S. and Goldluecke, B. Globally consistent depth labeling of 4D light fields.
In Proc. International Conference on Computer Vision and Pattern Recognition
(CVPR), pages 41-48. 2012.
[15] Wanner, S., Straehle, C. and Goldluecke, B. Globally consistent multi-label
assignment on the ray space of 4D light fields. In Proc. International Conference on
Computer Vision and Pattern Recognition (CVPR). 2013.
[16] Wilburn, B., Joshi, N., Vaish, V., Talvala, EV., Antunez, E., Barth, A., Adams, A.,
Horowitz, M. and Levoy, M. High performance imaging using large camera arrays,
ACM Trans. Graphic. 24(3), 765–776. July 2005.