臺灣博碩士論文加值系統

近年來，動作辨識是影像視覺領域熱門的研究主題。為了使系統能夠以最貼近人類，以及最自然的方式來解讀精細且複雜的動作，我們透過電腦視覺來設計此動作辨識系統。雖然近年來也有許多關於活動辨識上的研究，但若要提高系統的實用性，則資料中群內相異性與群間相似性的問題便日趨重要。因此，在本篇論文中，我們的目標為建造出一個以電腦視覺為基礎的動作辨識系統，藉由深度影像追蹤其骨架資訊，藉由分析其動作特性，建立具有鑑別力的特徵描述模型。
為了達到此目的，我們將獲取的骨架關節位置，以一自身為中心的球狀座標系統重新描述，以達到無關視點的活動辨識，在空間上的執行變異問題，利用高斯模糊削弱關節位置的絕對性，建立姿態的特徵模型。在執行時間的變異上，我們結合動態時間校正演算法，來解決每個人執行速度快慢不一的問題。除了姿態上靜態的描述，動態的動作變化也在特徵描述模型中加以考慮，藉由記錄關節的角度變化，分析出各關節在不同的活動中隱含的資訊熵及物理意義，然後結合各關節自身的速度資訊，形成動態特徵描述模型。最後，結合姿態與動作特徵描述模型訓練出一支持向量機，並以兩組具代表性的公開動作數據集加以驗證，實驗結果顯示，本論文所提出來的動作辨識法能有效的處理群內相異性與群間相似性的問題，這將有助於未來實際應用於人機互動的介面上。

Human activity recognition has become one of the most important areas of research in computer vision. Without extra markers attached to human body, the interaction between machine and human can be natural in a vision-based recognition system. However, it still remains some challenges such as intra-class variability and inter-class similarity. In order to solve the problems, this thesis presents a novel skeleton-based activity recognition methodology with depth sensors.
Due to the variation of execution styles and speed for different individuals, a Gaussian blur mask is applied to model the joint position variation while dynamic time warping (DTW) is employed to align the temporal sequences. Besides encoding the structure, the motion is also characterized. Through recording the joint angle trajectory, the entropy of each joint can be evaluated and then be combined with the projected velocity feature. Moreover, the support vector machine (SVM) is applied to acquire the classification results. In our implementation, two challenging public datasets are used to simulate real situations in our daily living. The experimental results show that the proposed approach is discriminative for human activity recognition and performs better than state-of-the-arts. This approach can benefit to several applications such as human-machine interaction (HMI).

誌謝 i
摘要 ii
ABSTRACT iii
CONTENTS iv
LIST OF FIGURES vi
LIST OF TABLES ix
Chapter 1 Introduction 1
1.1 Motivation 2
1.2 Literature Review 5
1.3 Contribution 8
1.4 Thesis Organization 9
Chapter 2 Preliminaries 11
2.1 Joint Position Estimation 11
2.1.1 Body Part Labeling 12
2.1.2 Depth Image Features 13
2.1.3 Joint Position Proposals 15
2.2 Support Vector Machine 17
2.2.1 Linear SVM 17
2.2.2 Soft Margin SVM 21
2.2.3 Nonlinear SVM 22
Chapter 3 Methodology 24
3.1 Posture-related Feature 25
3.1.1 Person-centric Coordinate 25
3.1.2 Dynamic Time Warping 29
3.1.3 Temporally-aligned Posture Descriptor 32
3.2 Motion-related Feature 36
3.2.1 Joint Information Analysis 36
3.2.2 Entropy-based Motion Descriptor 44
3.3 SVM-based Classification 48
Chapter 4 Experiments 51
4.1 Environmental Description 51
4.2 Datasets 52
4.2.1 UTKinect Dataset 52
4.2.2 Florence 3D Dataset 56
4.3 Experimental Results 57
4.3.1 The Result of UTKinect Dataset 58
4.3.2 The Result of Florence 3D Dataset 61
Chapter 5 Conclusion and Future Work 67
REFERENCES 69

[1]J. K. Aggarwal and M. S. Ryoo, "Human activity analysis: A review," ACM Computer Survey, vol. 43, pp. 1-43, 2011.
[2]J. K. Aggarwal and L. Xia, "Human activity recognition from 3D data: A review," Pattern Recognition Letters, vol. 48, pp. 70-80, 2014.
[3]L. Wanqing, Z. Zhengyou, and L. Zicheng, "Expandable Data-Driven Graphical Modeling of Human Actions Based on Salient Postures," IEEE Transactions on Circuits and Systems for Video Technology, vol. 18, pp. 1499-1510, 2008.
[4]T. D. Albright and G. R. Stoner, "Visual motion perception," Proceedings of the National Academy of Sciences of the United States of America, vol. 92, pp. 2433-2440, 1995.
[5]J. Shotton, A. Fitzgibbon, M. Cook, T. Sharp, M. Finocchio, R. Moore, et al., "Real-time human pose recognition in parts from single depth images," in IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2011, pp. 1297-1304.
[6]A. Yao, J. Gall, and L. V. Gool, "Coupled Action Recognition and Pose Estimation from Multiple Views," International Journal of Computer Vision, vol. 100, pp. 16-37, 2012.
[7]T. B. Moeslund, A. Hilton, and V. Krüger, "A survey of advances in vision-based human motion capture and analysis," Computer Vision and Image Understanding, vol. 104, pp. 90-126, 2006.
[8]R. Poppe, "A survey on vision-based human action recognition," Image Vision Computer, vol. 28, pp. 976-990, 2010.
[9]M. E. Hussein, M. Torki, M. A. Gowayyed, and M. El-Saban, "Human action recognition using a temporal hierarchy of covariance descriptors on 3D joint locations," in International Conference on Artificial Intelligence, Beijing, China, 2013, pp. 2466-2472.
[10]E. Ohn-Bar and M. M. Trivedi, "Joint Angles Similarities and HOG2 for Action Recognition," in IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), 2013, pp. 465-470.
[11]G. Evangelidis, G. Singh, and R. Horaud, "Skeletal Quads: Human Action Recognition Using Joint Quadruples," in International Conference on Pattern Recognition (ICPR), 2014, pp. 4513-4518.
[12]Y. Chen and G. Medioni, "Object modeling by registration of multiple range images," in IEEE International Conference Proceedings on Robotics and Automation, 1991, pp. 2724-2729 vol.3.
[13]F. Arman and J. K. Aggarwal, "Model-based object recognition in dense-range images--a review," ACM Comput. Surv., vol. 25, pp. 5-43, 1993.
[14]C. Hesher, A. Srivastava, and G. Erlebacher, "A novel technique for face recognition using range imaging," in International Symposium on Signal Processing and Its Applications, 2003, pp. 201-204 vol.2.
[15]S. Malassiotis, N. Aifanti, and M. G. Strintzis, "A gesture recognition system using 3D data," in International Symposium on 3D Data Processing Visualization and Transmission, 2002, pp. 190-193.
[16]B. Sabata and J. K. Aggarwal, "Surface Correspondence and Motion Computation from a Pair of Range Images," Computer Vision and Image Understanding, vol. 63, pp. 232-250, 1996.
[17]L. Wanqing, Z. Zhengyou, and L. Zicheng, "Action recognition based on a bag of 3D points," in IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), 2010, pp. 9-14.
[18]J. Wang, Z. Liu, J. Chorowski, Z. Chen, and Y. Wu, "Robust 3d action recognition with random occupancy patterns," in European Conference on Computer Vision, Florence, Italy, 2012, pp. 872-885.
[19]A. W. Vieira, E. R. Nascimento, G. L. Oliveira, Z. Liu, and M. F. M. Campos, "On the improvement of human action recognition from depth map sequences using Space–Time Occupancy Patterns," Pattern Recognition Letters, vol. 36, pp. 221-227, 2014.
[20]O. Oreifej and Z. Liu, "HON4D: Histogram of Oriented 4D Normals for Activity Recognition from Depth Sequences.," in IEEE Conference on Computer Vision and Pattern Recognition(CVPR), 2013.
[21]X. Lu and J. K. Aggarwal, "Spatio-temporal Depth Cuboid Similarity Feature for Activity Recognition Using Depth Camera," in IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2013, pp. 2834-2841.
[22]H. Fujiyoshi and A. J. Lipton, "Real-time human motion analysis by image skeletonization," in IEEE Workshop on Applications of Computer Vision, 1998, pp. 15-21.
[23]J. Ben-Arie, W. Zhiqian, P. Pandit, and S. Rajaram, "Human activity recognition using multidimensional indexing," IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 24, pp. 1091-1104, 2002.
[24]C. Zhang and Y. Tian, "RGB-D Camera-based Daily Living Activity Recognition," Journal of Computer Vision and Image Processing, vol. 2, 2012.
[25]W. Di and S. Ling, "Leveraging Hierarchical Parametric Networks for Skeletal Joints Based Action Segmentation and Recognition," in IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2014, pp. 724-731.
[26]L. Xia, C.-C. Chen, and J. K. Aggarwal, "View invariant human action recognition using histograms of 3D joints," in IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), 2012, pp. 20-27.
[27]C. Wang, Y. Wang, and A. L. Yuille, "An Approach to Pose-Based Action Recognition," in IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2013, pp. 915-922.
[28]R. Chaudhry, F. Ofli, G. Kurillo, R. Bajcsy, and R. Vidal, "Bio-inspired Dynamic 3D Discriminative Skeletal Features for Human Action Recognition," in IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), 2013, pp. 471-478.
[29]A. A. Chaaraoui, J. R. Padilla-López, P. Climent-Pérez, and F. Flórez-Revuelta, "Evolutionary joint selection to improve human action recognition with RGB-D devices," Expert Systems with Applications, vol. 41, pp. 786-794, 2014.
[30]X. Yang and Y. Tian, "Effective 3D action recognition using EigenJoints," Journal of Visual Communication and Image Representation, vol. 25, pp. 2-11, 2014.
[31]F. Ofli, R. Chaudhry, G. Kurillo, R. Vidal, and R. Bajcsy, "Sequence of the most informative joints (SMIJ): A new representation for human skeletal action recognition," Journal of Visual Communication and Image Representation, vol. 25, pp. 24-38, 2014.
[32]Pham Chinh Huu, Le Quoc Khanh, and L. T. Ha, "Human Action Recognition Using Dynamic Time Warping and Voting Algorithm," VNU Journal of Science, vol. 30, pp. 22-30, 2014.
[33]R. Vemulapalli, F. Arrate, and R. Chellappa, "Human Action Recognition by Representing 3D Skeletons as Points in a Lie Group," presented at the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2014.
[34]W. Jiang, L. Zicheng, W. Ying, and Y. Junsong, "Mining actionlet ensemble for action recognition with depth cameras," in IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2012, pp. 1290-1297.
[35]Z. Yu, C. Wenbin, and G. Guodong, "Fusing Spatiotemporal Features and Joints for 3D Action Recognition," in IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), 2013, pp. 486-491.
[36]Y. Yacoob and M. J. Black, "Parameterized modeling and recognition of activities," in International Conference on Computer Vision, 1998, pp. 120-127.
[37]L. Seidenari, V. Varano, S. Berretti, A. Del Bimbo, and P. Pala, "Recognizing Actions from Depth Cameras as Weakly Aligned Multi-part Bag-of-Poses," in IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), 2013, pp. 479-485.
[38]"Florence 3D action dataset http://www.micc.unifi.it/vim/datasets/3dactions/," ed, 2013.
[39]R. Okada and S. Soatto, "Relevant Feature Selection for Human Pose Estimation and Localization in Cluttered Images," in European Conference on Computer Vision, 2008, pp. 434-445.
[40]P. F. Felzenszwalb and D. P. Huttenlocher, "Pictorial Structures for Object Recognition," International Journal of Computer Vision, vol. 61, pp. 55-79, 2005.
[41](2010). OPENNI, http://www.openni.org/.
[42]B. E. Boser, I. M. Guyon, and V. N. Vapnik, "A training algorithm for optimal margin classifiers," in Proceedings of the fifth annual workshop on Computational learning theory, Pittsburgh, Pennsylvania, USA, 1992, pp. 144-152.
[43]C. Cortes and V. Vapnik, "Support-Vector Networks," Machine Learning, vol. 20, pp. 273-297, 1995.
[44]V. M. Zatsiorsky, Kinematics of Human Motion: Human Kinetics, 1998.
[45]P. Senin, Dynamic Time Warping Algorithm Review. University of Hawaii at Manoa, USA, 2008.
[46]S. Sempena, N. U. Maulidevi, and P. R. Aryan, "Human action recognition using Dynamic Time Warping," in International Conference on Electrical Engineering and Informatics, 2011.
[47]C. E. Shannon, "A mathematical theory of communication," SIGMOBILE Mob. Comput. Commun. Rev., vol. 5, pp. 3-55, 2001.