臺灣博碩士論文加值系統

本論文提出一由影像集之照片重構之系統。傳統上如果想對一張已存在的照片，重拍出另一張相同場景、角度，甚至建築物之立體透視角度也完全一樣，所必需要包含的條件有照相的位置、角度以及鏡頭種類等。然而，這對於一位專業攝影師來說，是非常困難的一件事情，更何況是業餘攝影師。我們提供一種讓使用者能夠較容易重構照片之系統。使用者只需要到已存在之相片場景中，隨意在該相片之相機角度附近拍幾張照片，本系統利用Structure from motion (SfM)分析該照片集之相對關係，估算每張照片的相機參數以及場景的3D資訊。接著，藉由使用者給的對應資訊，本系統可以自動估算出老照片之相機參數並且產生初始的照片重構結果，接著利用Inpainting以及Content-preserving warping產生出最後的結果，達到照片重構之目標。此外，若使用者無法重回現場拍照，本系統還提供使用者可以收集網路上的照片，達成相同的目的。

This thesis proposes a novel system that ``rephotographs'' a historical photograph with a collection of images.
Rather than finding the accurate viewpoint of the historical photo, users only need to take a number of photographs around the target scene.
We adopt the structure from motion technique to estimate the spatial relationship among these photographs, and construct a 3D point cloud.
Based on the user-specified correspondences between the projected 3D point cloud and historical photograph, the camera parameters of the historical photograph can be estimated.
We then combine forwarding and backward warping images to render an initial result.
Finally, inpainting and content-preserving warping are used to refine it, and
the photograph at the same viewpoint of the historical one can be produced by this photo collection.

Contents
口試委員會審定書i
致謝iii
中文摘要v
Abstract vii
1 Introduction 1
2 RelatedWork 3
2.1 Rephotography . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
2.2 Structure from Motion . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
2.3 Image-Based Rendering . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
2.4 Multi-View Reconstruction . . . . . . . . . . . . . . . . . . . . . . . . . 5
3 System Overview 7
4 Scene Reconstruction 9
4.1 Camera Calibration and 3D Point Cloud Construction . . . . . . . . . . . 9
4.2 Segmentation-based Depth Map Estimation . . . . . . . . . . . . . . . . 11
5 Reprojection 15
5.1 Camera Calibration for the Historical Photo . . . . . . . . . . . . . . . . 15
5.2 Pixel-based Historical View Generation . . . . . . . . . . . . . . . . . . 18
5.2.1 Forward Warping . . . . . . . . . . . . . . . . . . . . . . . . . . 18
5.2.2 Backward Warping . . . . . . . . . . . . . . . . . . . . . . . . . 19
5.3 Gradient-based Historical View Generation . . . . . . . . . . . . . . . . 22
5.3.1 Gradient Map Generation for Historical View . . . . . . . . . . . 22
5.3.2 Historical View Generation . . . . . . . . . . . . . . . . . . . . . 23
6 Holes Filling and Refine Perspective 29
6.1 Blending . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30
6.2 Image Inpainting . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30
6.3 Content-preserving Warping . . . . . . . . . . . . . . . . . . . . . . . . 31
7 Result 33
7.1 Depth Map Generation . . . . . . . . . . . . . . . . . . . . . . . . . . . 33
7.2 Render Historical View Photograph . . . . . . . . . . . . . . . . . . . . 38
7.2.1 Pixel-based Method . . . . . . . . . . . . . . . . . . . . . . . . 38
7.2.2 Gradient-based Method . . . . . . . . . . . . . . . . . . . . . . . 39
8 Conclusion 45
Bibliography 47

[1] S. Bae, A. Agarwala, and F. Durand. Computational rephotography. ACM TOG,
29(3):24:1–24:15, 2010.
[2] S. E. Chen and L. Williams. View interpolation for image synthesis. In Proc. ACM
SIGGRAPH, pages 279–288, 1993.
[3] C. M. Christoudias, B. Georgescu, P. Meer, and C. M. Georgescu. Synergism in low
level vision. In In International Conference on Pattern Recognition, pages 150–155,
2002.
[4] D. Comaniciu and P. Meer. Mean shift: A robust approach toward feature space
analysis. IEEE TPAMI, 24(5):603–619, 2002.
[5] A. Criminisi, P. Perez, and K. Toyama. Object removal by exemplar-based inpaint-
ing. In Proc. IEEE CVPR, volume 2, pages 721–728, 2003.
[6] P. Debevec, Y. Yu, and G. Borshukov. Efficient view-dependent image-based ren-
dering with projective texture-mapping. In Proc. EGWR, pages 105–116, 1998.
[7] P. E. Debevec, C. J. Taylor, and J. Malik. Modeling and rendering architecture
from photographs: a hybrid geometry- and image-based approach. In Proc. ACM
SIGGRAPH, pages 11–20, 1996.
[8] C. H. Esteban and F. Schmitt. Silhouette and stereo fusion for 3d object modeling.
CVIU, 96(3):367–392, 2004.
[9] M. Goesele, B. Curless, and S. M. Seitz. Multi-view stereo revisited. In Proc. IEEE
CVPR, volume 2, pages 2402–2409, 2006.
[10] M. Goesele, N. Snavely, B. Curless, H. Hoppe, and S. M. Seitz. Multi-view stereo
for community photo collections. In Proc. IEEE ICCV, pages 1–8, 2007.
[11] S. J. Gortler, R. Grzeszczuk, R. Szeliski, and M. F. Cohen. The lumigraph. In Proc.
ACM SIGGRAPH, pages 43–54, 1996.
[12] R. I. Hartley. Estimation of relative camera positions for uncalibrated cameras. In
Proc. ECCV, pages 579–587, 1992.
[13] R. I. Hartley. In defense of the eight-point algorithm. IEEE TPAMI, 19(6):580–593,
1997.
[14] A. Hornung and L. Kobbelt. Hierarchical volumetric multi-view stereo reconstruction
of manifold surfaces based on dual graph embedding. In Proc. IEEE CVPR,
volume 1, pages 503–510, 2006.
[15] P. Labatut, J.-P. Pons, and R. Keriven. Efficient multi-view reconstruction of largescale
scenes using interest points, delaunay triangulation and graph cuts. In Proc.
IEEE ICCV, pages 1–8, 2007.
[16] M. Levoy and P. Hanrahan. Light field rendering. In Proc. ACM SIGGRAPH, pages
31–42, 1996.
[17] A. Lippman. Movie-maps: An application of the optical videodisc to computer
graphics. ACM SIGGRAPH CG, 14(3):32–42, 1980.
[18] C. Lipski, C. Linz, K. Berger, A. Sellent, and M. Magnor. Virtual video camera:
Image-based viewpoint navigation through space and time. CGF, 29(8):2555–2568,
2010.
[19] F. Liu, M. Gleicher, H. Jin, and A. Agarwala. Content-preserving warps for 3D
video stabilization. ACM TOG, 28(3):44:1–44:9, 2009.
[20] D. G. Lowe. Distinctive image features from scale-invariant keypoints. IJCV,
60(2):91–110, 2004.
[21] D. Mahajan, F.-C. Huang, W. Matusik, R. Ramamoorthi, and P. Belhumeur. Moving
gradients: a path-based method for plausible image interpolation. ACM Trans.
Graph., 28:42:1–42:11, July 2009.
[22] P. Meer and B. Georgescu. Edge detection with embedded confidence. IEEE TPAMI,
23(12):1351–1365, 2001.
[23] J.-P. Pons, R. Keriven, and O. Faugeras. Modelling dynamic scenes by registering
multi-view image sequences. In Proc. IEEE CVPR, volume 2, pages 822–827, 2005.
[24] G. Schindler, F. Dellaert, and S. B. Kang. Inferring temporal order of images from
3D structure. In Proc. IEEE CVPR, pages 1–7, 2007.
[25] S. M. Seitz, B. Curless, J. Diebel, D. Scharstein, and R. Szeliski. A comparison and
evaluation of multi-view stereo reconstruction algorithms. In Proc. IEEE CVPR,
volume 1, pages 519–528, 2006.
[26] S. M. Seitz and C. R. Dyer. View morphing. In Proc. ACM SIGGRAPH, pages
21–30, 1996.
[27] J. Shade, S. Gortler, L.-w. He, and R. Szeliski. Layered depth images. In Proc. ACM
SIGGRAPH, pages 231–242, 1998.
[28] H.-Y. Shum and L.-W. He. Rendering with concentric mosaics. In Proc. ACM
SIGGRAPH, pages 299–306, 1999.
[29] H.-Y. Shum and S. B. Kang. A review of image-based rendering techniques. In
Proc. SPIE VCIP, pages 2–13, 2000.
[30] N. Snavely, S. M. Seitz, and R. Szeliski. Photo tourism: exploring photo collections
in 3D. ACM TOG, 25(3):835–846, 2006.
[31] C. Strecha, R. Fransens, and L. Van Gool. Combined depth and outlier estimation
in multi-view stereo. In Proc. IEEE CVPR, volume 2, pages 2394–2401, 2006.
[32] S. Tran and L. Davis. 3D surface reconstruction using graph cuts with surface constraints.
In Proc. ECCV, pages 219–231, 2006.
[33] K. C. Zheng, A. Colburn, A. Agarwala, M. Agrawala, D. Salesin, B. Curless, and
M. F. Cohen. A consistent segmentation approach to image-based rendering. 2009.
[34] C. L. Zitnick, S. B. Kang, M. Uyttendaele, S. Winder, and R. Szeliski. High-quality
video view interpolation using a layered representation. ACM TOG, 23(3):600–608,
2004.