臺灣博碩士論文加值系統

本文提出一種適用於架設在P/T(Pan/Tilt) 移動雲台的數支監控攝影機之即時視訊拼接系統，使監視人員能夠看到即時且廣闊的監控畫面，可避免魚眼鏡頭產生全景畫面之扭曲失真現象，而與傳統滙整多部攝影機之監控畫面相比，拼接系統所產生單一且寬廣的全景監控畫面會更加清楚簡潔，也改善管理人員觀看時對於影像內容或目標行為理解得到更好的監控效果; 此外，拼接畫面也能作為移動物偵測、異常偵測、物件辨識等相關電腦視覺之後續處理。本文方法主要理念是從經影像校正後的個別畫面之交集重疊區找出接縫線，再進行影像縫合以產生拼接畫面。目前市場上的影像拼接產品大多為單張影像的拼接產品，而即時視訊拼接產品則皆使用魚眼鏡頭來進行單張影像拼接藉以達到即時應用，但也造成畫面重疊區有扭曲、物體變形等結果;而一般拼接攝影機之拼接更是透過逐幀處理而導致耗時甚鉅，也產生重疊區之移動物體扭變現象。實際上，目前並無適用於數支移動攝影機之即時視訊拼接技術方法與產品，因此本文開發一種即時視訊拼接系統，以利於後續電腦視覺智慧偵測與分析處理，可應用於大廣場公共場地或交通路口的監錄系統並配合移動雲台選擇所需監控場景以擴大監控範圍與減少視野死角。

This thesis presents a realtime video stitching system for several cameras mounted on a pan-tilt platform. This system can provide real-time larger surveillance scope. It can avoid the distortion problem caused by the fisheye lens to produce a panoramic picture. Compared with the traditional monitoring wall composed of multiple images captured by multiple cameras, the large monitoring frame generated by the stitching system can provide more concise and intelligible surveillance image and also make the security person to be more understandable for contents of surveillance frames. Besides, the stitched video is also allowable for further processing of computer vision, e.g., motion detection, object segmentation and recognition. The main idea of the proposed method is to find the seam line within the overlapping region of two individual rectified images, and then such two images are stitched based on the seam. Most previous methods of image stitching are only applied to single image stitching rather than video stitching. For those methods, single image stitching always requires a large number of computations and hence it is not suitable for realtime video-stitching. In the current markets, most image stitching products are based on image-by-image stitching and all video stitching products are based on fisheye cameras. Those products will generate the distortion problem in the overlapping region, especially for the moving objects passing through the overlapping region. In practice, those products are not suitable for the high-quality surveillance applications. Therefore, this research is aimed at developing a realtime video stitching system that can enlarge the surveillance scope and also for the subsequent processing of computer vision.

目錄
摘要 III
ABSTRACT IV
第一章、 緒論 1
1.1 研究動機 1
1.2 系統架構與流程 2
1.3 論文大綱 5
第二章、 相關習知技術與知識 6
2.1 特徵點偵測 6
2.1.1 尺度不變特徵轉換SIFT(Scale-invariant feature transform) 6
2.1.2 加速穩健特徵SURF(Speeded Up Robust Features) 7
2.1.3 ORB特徵偵測(Oriented FAST and rotated BRIEF) 8
2.1.4 LATCH(Learned Arrangements of Three Patch Codes) 9
2.1.5 特徵點偵測總結 10
2.2 影像校正 11
2.2.1 仿射轉換 (Affine Transform) 11
2.2.2 透視變換 (Perspective Transformation) 12
2.2.3 單應性矩陣 13
2.2.4 影像校正總結 14
2.3 影像縮減取樣 14
2.3.1 最近相鄰內插法(Nearest Neighbor Interpolation) 14
2.3.2 雙線性內插法(Bilinear Interpolation) 16
2.3.3 雙立方內插法(Bicubic Interpolation) 18
2.3.4 影像縮減取樣總結 20
2.4 影像混合演算法(Image Blending) 21
2.4.1 線性混合演算法(Linear Blending) 21
2.4.2 多頻帶混合演算法(Multi-Band Blending) 21
2.4.3 影像混合演算法總評 22

第三章、 相關方法探討 23
3.1 圖像拼接相關方法 23
3.1.1 Automatic panoramic image stitching using invariant features 24
3.1.2 Seam-Driven Image Stitching 29
3.1.3 As-Projective-As-Possible Image Stitching with Moving DLT 32
3.2 視訊拼接相關方法 36
3.2.1 Panoramic Video Stitching in Multi-Camera Surveillance System 37
3.2.2 Video stitching with spatial-temporal content-preserving warping 39
3.2.3 Real-time Video Stitching Based on Distribution of Feature Points 43
3.3 接縫搜尋相關演算法 48
3.3.1 Seam Carving for Content-Aware Image Resizing 48
第四章、 本系統與方法 51
4.1 特徵匹配模組 54
4.1.1 SIFT-LATCH特徵計算 54
4.1.2 KNN特徵匹配 57
4.1.3 RANSAC篩選 & 計算單應性矩陣 61
4.2 透視矩陣篩選模組 63
4.2.1 計算候選矩陣誤差 63
4.3 影像校正模組 65
4.3.1 影像透視變換校正 66
4.4 接縫搜尋模組 67
4.4.1 重疊區影像縮減取樣 68
4.4.2 建立重疊區Sobel影像 70
4.4.3 重疊區Sobel影像疊加 73
4.4.4 建立約束累加能量影像 75
4.4.5 搜尋最小能量路徑 78
4.5 影像縫合模組 80
4.5.1 重疊區影像遮罩建立 80
4.5.2 多頻帶混合 82
4.5.3 線性混合 85
第五章、 實驗結果 91
5.1 環境架設與實驗影片拍攝 91
5.2 輸出視訊拼接結果 95
5.3 實驗結果與分析 107
5.3.1 主觀評估 107
5.3.2 客觀評估 118
第六章、 結論與未來方向 138
6.1 結論 138
6.2 未來方向 139
參考文獻 140

[1] David G. Lowe, "Distinctive Image Features from Scale-Invariant Keypoints," International Journal of Computer Vision, vol. 60, no. 2, p. 91–110, Nov 2004.
[2] Bay, Herbert, Tinne Tuytelaars, and Luc Van Gool, "Surf: Speeded up Robust Features," in European Conference on Computer Vision, ECCV, Graz, Austria, May 2006, pp. 404-417.
[3] Ethan Rublee, Vincent Rabaud, Kurt Konolige and Gary Bradski, "ORB: An efficient alternative to SIFT or SURF," in 2011 International Conference on Computer Vision, ICCV, Barcelona, Spain, May 2011, pp.2564-2571.
[4] Gil Levi, Tal Hassner, "LATCH: Learned Arrangements of Three Patch Codes," in IEEE winter conference on applications of computer vision, WACV, NY, USA, Mar 2016, pp.1-9.
[5] Matthew Brown, David G. Lowe, "Recognising panoramas," in Proceedings Ninth IEEE International Conference on Computer Vision, ICCV, Nice, France, Oct 2003, pp.1218-1225.
[6] Matthew Brown, David G. Lowe, "Automatic panoramic image stitching using invariant features," International journal of computer vision, vol. 74, no. 1, pp. 59-73, Aug 2007.
[7] Junhong Gao, Seon Joo Kim, and Michael S. Brown, "Constructing image panoramas using dual-homography warping," in CVPR 2011, Colorado, USA, Jun 2011, pp.1218.
[8] Wen-Yan Lin, Siying Liu, Yasuyuki Matsushita, Tian-Tsong Ng, and Loong-Fah Cheong, "Smoothly varying affine stitching," in CVPR 2011, Colorado, USA, Jun 2011, pp.345-352.
[9] Junhong Gao, Yu Li, Tat-Jun Chin, and Michael S. Brown, "Seam-Driven Image Stitching," in Eurographics Conferences, EG, Girona, Spain, May 2013, pp.45-48.
[10] Fan Zhang, Feng Liu, "Parallax-Tolerant Image Stitching," in 2014 IEEE Conference on Computer Vision and Pattern Recognition, CVPR, OH, USA, Jun 2014, pp.3262-3269.
[11] Julio Zaragoza, Tat-Jun Chin, Quoc-Huy Tran, Michael S. Brown,and David Suter, "As-Projective-As-Possible Image Stitching with Moving DLT," IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 36, no. 7, pp. 1285 - 1298, Jul 2014.
[12] Fei Lei, Wenxue Wang, "A fast method for image mosaic based on SURF," in 2014 9th IEEE Conference on Industrial Electronics and Applications,ICIEA, Hangzhou, China, Jun 2014, pp.79-82.
[13] Che-Han Chang, Yoichi Sato,and Yung-Yu Chuang, "Shape-Preserving Half-Projective Warps for Image Stitching," in 2014 IEEE Conference on Computer Vision and Pattern Recognition, CVPR, OH, USA, Jun, 2014, pp.3254-3261.
[14] Chung-Ching Lin, Sharathchandra U. Pankanti, Karthikeyan Natesan Ramamurthy, and Aleksandr Y. Aravkin, "Adaptive as-natural-as-possible image stitching," in 2015 IEEE Conference on Computer Vision and Pattern Recognition, CVPR, MA, USA, Jun 2015, pp.1155-1163.
[15] Xueqi Li, Shangyang He, Xiaoyu Gao,and Junzhen Zhou, "Research on Container Panorama Image Stitching Method," in 2018 Chinese Automation Congress, CAC, Xi'an, China, Nov, 2018, pp. 661-664.
[16] Martin A. Fischler, Robert C. Bolles, "Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography," Communications of the ACM, vol. 24, no. 6, Jun 1981.
[17] Bin He, Gang Zhao,and Qifang Liu, "Panoramic video stitching in multi-camera surveillance system," in 2010 25th International Conference of Image and Vision Computing New Zealand, IVCNZ, Queenstown, New Zealand, Nov 2010, pp.1-6.
[18] Linqiang Chen, Xiaoqiang Wang,and Xu Liang, "An effective video stitching method," in 2010 International Conference On Computer Design and Applications, ICCDA, Qinhuangdao, China, Jun 2010, pp.297-301.
[19] Wei Jiang, Jinwei Gu, "Video stitching with spatial-temporal content-preserving warping," in IEEE Conference on Computer Vision and Pattern Recognition Workshops, CVPRW, MA, USA, June 2015, pp.42-48.
[20] Chao-Ho Chen, Tsong-Yi Chen, Wu-Chih Hu, Deng-Yuan Huang, Shuo-Wen Yu, and Cheng-Kang Wen, "Real-time Video Stitching Based on Distribution of Feature Points," in International Conference on Engineering Science and Applications, ICESA, Tokyo, Japan, Aug, 2018, pp.39.
[21] Hanyu Liu, Chong Tang , Shaoen Wu , and Honggang Wang, "Real-time video surveillance for large scenes," in International Conference on Wireless Communications and Signal Processing, WCSP, Nanjing, China, Nov 2011,pp.1-4.
[22] Jing Li, Wei Xu, Jianguo Zhang, Maojun Zhang, Zhengming Wang, and Xuelong Li, "Efficient Video Stitching Based on Fast Structure Deformation," IEEE Transactions on Cybernetics, vol. 45, no. 12, pp. 2707-2719, Oct 2015.
[23] Shai Avidan, Ariel Shamir, "Seam carving for content-aware image resizing," ACM Transactions on Graphics, vol. 26, no. 3, Jul 2007.