動作辨識在電腦視覺領域中向來是一項主要的研究議題，並且有著非常廣泛的應用，像是居家照護、公共場所的安全監控等等。傳統上以深度影像為基礎的方法經常不能識別出在空間上相似的姿勢。而另一方面，以骨架為基礎的方法雖然能以關節點位置，描述人體在時空中的運動方式，然而複雜動作/姿勢下所偵測出的關節點位置並不可靠。在本論文中，我們提出了混合式的動作辨識方法，結合深度圖與三維關節點位置等資訊，並且使用新的特徵以及影像序列對齊法來提昇結果表現。實驗結果顯示，我們所提出的方法效果良好且具有效率。

Human action recognition has been a major research topic in computer vision with wide applications like homecare or surveillance. Traditional depth-map-based approaches are usually unable to distinguish the actions that share similar postures on the spatial domain. On the other hand, skeleton-based approaches describe the spatio-temporal movements of human body joints. However, joint positions are not reliable enough for complex actions/postures. In this work, we present a hybrid-based method for human action recognition by combining the information of depth maps and 3D joint positions. New features and sequence alignment scheme are proposed to enhance the performance. Our experimental result demonstrates that the proposed method is effective and efficient.

目錄
摘要 I
ABSTRACT II
誌謝 III
目錄 IV
圖目錄 V
表目錄 VII
第一章 緒論 1
1.1 研究動機 1
1.2 研究目的 2
1.3 論文架構 3
第二章 相關文獻 4
2.1以深度圖為基礎的辨識方法 7
2.2以關節點位置為基礎的辨識方法 13
2.3混合式辨識方法 17
第三章 研究方法 19
3.1深度影像前處理 19
3.2時序對齊(TEMPORAL ALIGNMENT)及取樣(SAMPLING) 25
3.3深度影像特徵擷取 26
3.4方向梯度直方圖(HISTOGRAM OF ORIENTED GRADIENTS) 29
3.5關節點特徵擷取 30
3.6 SVM(SUPPORT VECTOR MACHINE) 33
第四章 實驗結果與分析 36
4.1實驗環境與測試資料 36
4.2時序對齊實驗結果 40
4.3動作分類實驗結果 42
4.4實驗討論 48
第五章 結論與未來研究方向 50
參考文獻 51

圖目錄
圖2-1三種深度感應器[3] 5
圖2-2深度影像序列範例-網球發球(MSR Action 3D Dataset)[4] 6
圖2-3兩個相似的動作在2D上與3D上的梯度示意圖[5] 7
圖2-4三種動作的動作圖形[7] 8
圖2-5深度影像3D點的取樣示意圖[4] 9
圖2-6 STOP時空網格示意圖[8] 10
圖2-7 ROP架構示意圖[9] 10
圖2-8 DMM-HOG架構示意圖[10] 11
圖2-9 HON4D架構示意圖[5] 12
圖2-10特徵擷取示意圖[11] 13
圖2-11結合彩色與深度資訊的強度反應影像[11] 13
圖2-12 KINECT人體關節圖[15] 14
圖2-13 Wang等人方法的架構示意圖[16] 15
圖2-14 EigenJoints架構示意圖[17] 15
圖2-15架構示意圖[18] 16
圖2-16輪廓的架構圖[19] 17
圖2-17架構示意圖[20] 18
圖3-1系統架構圖 19
圖3-2 (a)原始深度影像；(b)去除地板的深度影像 20
圖3-3 (a)去除背景跟地板的深度影像；(b)ROI的深度影像 21
圖3-4 Yang等人[10]提出的DMM架構示意圖 22
圖3-5 (a)XY投影影像；(b)ZY投影影像；(c)XZ投影影像 23
圖3-6透過KINECT、連續錄下50個畫面的深度分布統計資料[21] 23
圖3-7 (a)XY投影影像；(b)ZY投影影像；(c)XZ投影影像 24
圖3-8 (a)XY投影ROI影像；(b)ZY投影ROI影像；(c)XZ投影ROI影像 25
圖3-9深度變化累加的影像 27
圖3-10經過侵蝕與膨脹後的影像 28
圖3-11縮小尺寸與高斯模糊化後的影像 28
圖3-12不同人的梯度圖 29
圖3-13 20個關節點與參考點(第4個關節點)的距離直方圖 31
圖3-14 20個關節點與參考點的距離變化直方圖 31
圖3-15 20個關節點與參考點的距離變化量直方圖 32
圖3-16 20個關節點分別在X(紅)、Y(綠)、Z軸(藍)上的角度變化直方圖 32
圖3-17 20個關節點的角度變化直方圖[23] 33
圖3-18 SVM在線性上的超平面分割[24] 34
圖3-19 SVM在非線性上的超平面分割示意圖[25] 34
圖4-1包含20個關節點的深度影像 36
圖4-2揮手動作的訓練正樣本 38
圖4-3揮手動作的訓練負樣本 39
圖4-4走路動作的深度影像序列範例 41
圖4-5走路動作的深度影像序列範例 42
圖4-6順時針畫圈與逆時針畫圈的深度變化累加影像 44

表目錄
表2-1各種方法在MSR Action3D Dataset的辨識率 18
表4-1 14種動作經過Alignment之後的深度影像序列範例 37
表4-2四種方法的實驗結果(Precision/Recall) 43
表4-3單獨使用DMM的confusion matrix 45
表4-4單獨使用Joint Feature的confusion matrix 46
表4-5使用DMM+Joint的confusion matrix 47

[1] Jamie Shotton, Andrew Fitzgibbon, Mat Cook, Toby Sharp, Mark Finocchio, “Real-Time Human Pose Recognition in Parts from Single Depth Images,” IEEE Conference on Computer Vision and Pattern Recognition, pp. 1297-1304, 2011
[2] Mao Ye, Qing Zhang, Liang Wang, Jiejie Zhu, Ruigang Yang, and Juergen Gall, “A Survey on Human Motion Analysis from Depth Data,” Time-of-Flight and Depth Imaging. Sensors, Algorithms, and Applications, Lecture Notes in Computer Science, vol. 8200, pp. 149-187, 2013.
[3] Lulu Chen, Hong Wei, James Ferryman, “A survey of human motion analysis using depth imagery,” Pattern Recognition Letters, vol. 34, no. 15, pp. 1995–2006, 2013.
[4] Wanqing Li, Zhengyou Zhang, Zicheng Liu, “Action Recognition Based on A Bag of 3D Points,” IEEE Conference on Computer Vision and Pattern Recognition Workshops, pp. 9-14, 2010.
[5] Omar Oreifej, Zicheng Liu, “HON4D: Histogram of Oriented 4D Normals for Activity Recognition from Depth Sequences,” IEEE Conference on Computer Vision and Pattern Recognition, pp. 716-723, 2013.
[6] Piotr Dollar, Vincent Rabaud, Garrison Cottrell, Serge Belongie, “Behavior Recognition via Sparse Spatio-Temporal Features,” IEEE International Workshop on Visual Surveillance and Performance Evaluation of Tracking and Surveillance, pp. 65-72, 2005.
[7] Wanqing Li, Zhengyou Zhang, Zicheng Liu, “Expandable Data-Driven Graphical Modeling of Human Actions Based on Salient Postures,” IEEE Transactions on Circuits and Systems for Video Technology, vol. 18, pp. 1499-1510, 2008.
[8] Antonio W. Vieira, Erickson R. Nascimento, Gabriel L. Oliveira, Zicheng Liu, Mario M. Campos, “STOP: Space-Time Occupancy Patterns for 3D Action Recognition from Depth Map Sequences,” Pattern Recognition, Image Analysis, Computer Vision, and Applications, Lecture Notes in Computer Science, vol. 7441, pp. 252-259, 2012.
[9] Jiang Wang, Zicheng Liu, Jan Chorowski, Zhuoyuan Chen, Ying Wu, “Robust 3D Action Recognition with Random Occupancy Patterns,” European Conference on Computer Vision, Lecture Notes in Computer Science, vol.7573, pp. 872-885, 2012.
[10] Xiaodong Yang, Chenyang Zhang, YingLi Tian, “Recognizing Actions Using Depth Motion Maps-based Histograms of Oriented Gradients,” ACM International Conference on Multimedia, pp. 1057-1060, 2012.
[11] Hao Zhang, Lynne E. Parker, “4-Dimensional Local Spatio-Temporal Features for Human Activity Recognition,” IEEE International Conference on Intelligent Robots and Systems, pp. 2044-2049, 2011.
[12] David M. Blei, Andrew Y. Ng, Michael I. Jordan, “Latent Dirichlet Allocation,” Journal of Machine Learning Research, vol. 3, pp. 993-1022, 2003.
[13] Thomas L. Griffiths, Mark Steyvers, “Finding scientific topics,” National Academy of Sciences of the United States of America, vol. 101, no. 1, pp. 5228-5235, 2004.
[14] Clement Menier, Edmond Boyer, Bruno Raffin, “3D Skeleton-Based Body Pose Recovery,” Third International Symposium on 3D Data Processing, Visualization, and Transmission, pp. 389-396, 2006.
[15] http://msdn.microsoft.com/zh-tw/hh367958.aspx
[16] Jiang Wang, Zicheng Liu, Ying Wu, Junsong Yuan, “Mining Actionlet Ensemble for Action Recognition with Depth Cameras,” IEEE Conference on Computer Vision and Pattern Recognition, pp. 1290-1297, 2012.
[17] Xiaodong Yang, Yingli Tian, “EigenJoints-based Action Recognition Using Naïve-Bayes-Nearest-Neighbor,” IEEE Conference on Computer Vision and Pattern Recognition Workshops, pp. 14-19, 2012.
[18] Xinbo Jiang, Fan Zhong, Qunsheng Peng, Xueying Qin, “Robust Action Recognition Based on a Hierarchical Model,” IEEE International Conference on Cyberworlds, pp. 191-198, 2013.
[19] Alexandros Andre Chaaraoui, José Ramón Padilla-López, Francisco Flórez-Revuelta, ”Fusion of Skeletal and Silhouette-based Features for Human Action Recognition with RGB-D Devices,” IEEE International Conference on Computer Vision Workshops, pp. 91-97, 2013.
[20] Hossein Rahmani, Arif Mahmood, Du Q. Huynh, Ajmal Mian, “Real Time Action Recognition Using Histograms of Depth Gradients and Random Decision Forests,” IEEE Winter Conference on Applications of Computer Vision, pp. 626-633, 2014.
[21]http://kheresy.wordpress.com/2011/12/27/depth_value_of_kinect_in_openni/comment-page-1/
[22] Navneet Dalal and Bill Triggs, “Histograms of oriented gradients for human detection,” IEEE Computer Society Conference on Computer Vision and Pattern Recognition, vol. 1, pp. 886-893, 2005.
[23] Navneet Dalal, “Finding People in Images and Videos,” PhD thesis, 2006.
[24] Christopher J. C. Burges, “A Tutorial on Support Vector Machines for Pattern Recognition”, Data Mining and Knowledge Discovery, vol. 2, pp. 121-167, 1998.
[25]https://cg2010studio.wordpress.com/2012/05/20/%E9%9D%9E%E7%B7%9A%E6%80%A7%E6%94%AF%E6%8C%81%E5%90%91%E9%87%8F%E6%A9%9F%E5%99%A8-non-linear-svms/
[26] Chih-Chung Chang, Chih-Jen Lin, “LIBSVM: a library for support vector machines”, ACM Transactions on Intelligent Systems and Technology, vol. 2, no. 27, 2011. Software available at http://www.csie.ntu.edu.tw/~cjlin/libsvm