臺灣博碩士論文加值系統

在本研究中，我們嘗試開發一個Google街景圖用的影像內插方法，改善Google街景圖不連續的問題。為了達成目標，首先，我們透過SuperPoint演算法做特徵點的提取，並用SuperGlue演算法做特徵匹配，找到同一個特徵點在兩張圖的位置，並以這些匹配點為基準，計算每個像素的移動距離與位置的比例，我們稱作移動比例，x與y座標分開計算，x座標以匹配點切割成複數個區塊，每個區塊會以前後兩個匹配點作為x座標的內插比例標準，並用兩個匹配點計算出x座標距離與移動比例的關係，而y座標則是取上下各一個最接近的匹配點，根據與兩個匹配點的距離計算出y座標的移動比例，接著以內插的張數與要內插的是第幾張內插圖做比例的計算，就能得到內插後的位置，將前進後的街景圖縮小，並將前進前的街景圖放大，合成兩張圖便能內插出中間缺少的街景圖，最後，將原本的街景圖，加上內插後的圖片，進而製作出一段流暢的街景導覽影片，目前在直線路段上已經有很好的成果。

In this paper, for providing a continuous and plausible navigation experience on a virtual route, we aim to develop an image interpolation method based on the image resource of Google Street View. To achieve this goal, first, we extracted the feature points from two consecutive street images by using SuperPoint, and matched the correlated feature points by using SuperGlue. Based on these matching points, we calculated the ratio of the moving distance to the position called moving ratio. To interpolate x-coordinate, we used matching points to segment the image into several blocks. Then, we calculated the relation between the distance of x-coordinate and the moving ratio with two matching points in each block. To interpolate y-coordinate, we find two closest matching points for every pixel. We calculated the average moving ratio with the distance of the pixel and the two matching points. By these results, we can zoom in the first source image and zoom out the second source image. Synthesizing these two images to get the interpolation image. Finally, we integrated all these results to synthesize more seamless and continuous image sequences. The current results showed our method has good performances on relatively straight roads.

目次
致謝辭 i
摘要 ii
Abstract iii
目次 iv
圖目次 v
第一章 緒論 1
第一節 研究動機與背景 1
第二節 研究目的 2
第二章 相關研究 3
第一節 特徵提取 3
第二節 特徵匹配 3
第三節 內插 4
第三章 研究方法 5
第一節 特徵提取 6
第二節 特徵匹配 7
第三節 內插 8
第四章 成果展示 15
第五章 結論與未來展望 42
參考文獻 43




圖目次
圖1.街景導覽影片生成系統 1
圖2.路線上的所有街景圖 2
圖3.前進前的街景圖(Image A) 2
圖4.前進後的街景圖(Image B) 2
圖5.街景圖影像內插流程圖 5
圖6.Image A特徵點提取 6
圖7.Image B特徵點提取 6
圖8.紅色部分為使用Floodfill演算法進行路面提取，黃色部分為找到的路面邊界線。 7
圖9.使用SuperGlue演算法[12]做特徵匹配的結果 7
圖10.匹配點的位置矩陣，每個矩陣前兩個數字為MPB的位置，後面兩個數字為MPA的位置 8
圖11.匹配特徵點的位置 9
圖12.與現在為基準的匹配點跟上一個為基準的匹配點的距離 10
圖13.與上下兩個最接近的匹配點的距離 12
圖14.縮小圖Image ZI(以內插29張之第15張為例) 13
圖15.放大圖Image ZO(以內插29張之第15張為例) 13
圖16.內插29張之第15張內插圖 14
圖17.街景圖(1)與內插圖(2~6) 16
圖18.內插圖(7~12) 17
圖19.內插圖(13~18) 18
圖20.內插圖(19~24) 19
圖21.內插圖(25~30) 20
圖22.街景圖(31) 21
圖23.街景圖與內插成果圖-1(test0為Google街景圖) 21
圖24.街景圖與內插成果圖-2(test30為Google街景圖) 22
圖25.街景圖與內插成果圖-3 23
圖26.街景圖與內插成果圖-4(test60為Google街景圖) 24
圖27.街景圖與內插成果圖-5(test90為Google街景圖) 25
圖28.街景圖與內插成果圖-6(test120為Google街景圖) 26
圖29.街景圖與內插成果圖-7(test150為Google街景圖) 27
圖30.大阪街景圖與內插成果圖-1(test0為Google街景圖) 28
圖31.大阪街景圖與內插成果圖-2(test30為Google街景圖) 29
圖32.大阪街景圖與內插成果圖-3(test60為Google街景圖) 30
圖33.大阪街景圖與內插成果圖-4 31
圖34.大阪街景圖與內插成果圖-5(test90為Google街景圖) 32
圖35.大阪街景圖與內插成果圖-6(test120為Google街景圖) 33
圖36.大阪街景圖與內插成果圖-7(test150為Google街景圖) 34
圖37.往日月潭街景圖與內插成果圖-1(test0為Google街景圖) 35
圖38.往日月潭街景圖與內插成果圖-2(test30為Google街景圖) 36
圖39.往日月潭街景圖與內插成果圖-3(test60為Google街景圖) 37
圖40.往日月潭街景圖與內插成果圖-4(test90為Google街景圖) 38
圖41.往日月潭街景圖與內插成果圖-5 39
圖42.往日月潭街景圖與內插成果圖-6(test120為Google街景圖) 40
圖43.往日月潭街景圖與內插成果圖-7(test150為Google街景圖) 41

[1] Lieu-Hen Chen, Hao-Ming Hung, Cheng-Yu Sun, Eric Hsiao-Kuang Wu, Yasufumi Takama, "CANavi: Synthesizing Cartoon-Like Animation for Street Navigation Based on Google Maps", IEEE Intelligent Transportation Systems Magazine (2019 accepted).
[2] Cheng Yu Sun, Lieu-Hen Chen, Yasufumi Takama (2019, Nov). "Synthesizing NPR styled Street View Animation Based on Deep Learning. " 2019 International Conference on Technologies and Applications of Artificial Intelligence.
[3] D. Lowe, "Distinctive image features from scale-invariant keypoints," International Journal of Computer Vision, vol. 60, no. 2, pp. 91-110, 2004.
[4] H. Bay, T. Tuytelaars, and L. Goo!. SURF: Speeded Up Robust Features. In: A. Leonardis, H. Bischof, A. Pinz (Eds. ): ECCV 2006, Part I, LNCS 3951: 404-417, Springer 2006.
[5] D. Neven, B. De Brabandere, S. Georgoulis, M. Proesmans, and L. Van Gool, “Towards End-to-End Lane Detection: An Instance Segmentation Approach,” arXiv, 2018.
[6] K. Behrendt and J. Witt, “Deep learning lane marker segmentation from automatically generated labels,” in 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS),
[7] J. Zaragoza, T. Chin, M. S. Brown and D. Suter, "As-Projective-As-Possible Image Stitching with Moving DLT," 2013 IEEE Conference on Computer Vision and Pattern Recognition, 2013, pp. 2339-2346, doi: 10.1109/CVPR.2013.303.
[8] J. Ishii, S. Sakai, K. Ito, T. Aoki, T. Yanagi, and T. Ando, “3D reconstruction of urban environments using in-vehicle fisheye camera,” Proc. IEEE Int’l Conf. Image Processing, pp. 2145–2148, Sept. 2013.
[9] Muja, M., and Lowe, D.G.: ‘Scalable nearest neighbor algorithms for high dimensional data’, IEEE Trans. Pattern Anal. Mach. Intell., 2014, 36, (11), pp. 2227–2240.
[10] C. B. Choy, J. Gwak, S. Savarese, and M. Chandraker. Universal Correspondence Network. In NIPS. 2016.
[11] Kwang Moo Yi, Eduard Trulls, Yuki Ono, Vincent Lepetit, Mathieu Salzmann, and Pascal Fua. Learning to find good correspondences. In CVPR, 2018. 2, 5, 6, 11, 12
[12] P.-E. Sarlin, D. DeTone, T. Malisiewicz, and A. Rabinovich, “Superglue: Learning feature matching with graph neural networks,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4938–4947, 2020.
[13] Heechang Kim, Sangjun Park, Jin Wang, Yonghoon Kim, and Jechang Jeong, " Advanced bilinear image interpolation based on edge features," in 2009 First International Conference on Advances in Multimedia, Aug. 2009, voL 3, pp. 33-36.
[14] X. Li and M. T. Orchard, “New edge-directed interpolation,” IEEE Trans. Image Process., vol. 10, no. 10, pp. 1521–1527, Oct. 2001.
[15] Z. Liu, R. Yeh, X. Tang, Y. Liu, and A. Agarwala, “Video frame synthesis using deep voxel flow,” arXiv preprint arXiv:1702.02463, 2017.
[16] S. Meyer, A. Djelouah, B. McWilliams, A. SorkineHornung, M. Gross, and C. Schroers. Phasenet for video frame interpolation. In CVPR, June 2018.
[17] Daniel DeTone, Tomasz Malisiewicz, and Andrew Rabinovich. SuperPoint: Self-supervised interest point detection and description. In CVPR Workshop on Deep Learning for Visual SLAM, 2018. 2, 4, 5, 6, 9 [18] Y. Nagasugi, R. Harakawa and M. Iwahashi, "Naturalness-Preserving Image Stitching Based on Optimal Seam Estimation Considering Parallax," 2020 IEEE 2nd Global Conference on Life Sciences and Technologies (LifeTech), 2020, pp. 115-116, doi: 10.1109/LifeTech48969.2020.1570619238.
[19] V. Badrinarayanan, A. Kendall and R. Cipolla, "SegNet: A Deep Convolutional Encoder-Decoder Architecture for Image Segmentation,"in IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 39, no. 12, pp. 2481-2495, 1 Dec. 2017, doi: 10.1109/TPAMI.2016.2644615