電腦視覺的相關研究已經進行多年，並徹底地改變了每個人的生活。幸虧科技的進步，使我們進入巨量資料與智慧型裝置的時代。有了機器學習演算法的幫助，電子產品能夠自動從網路等巨量資料庫學習有用的知識，進行自我校正與進步。電腦視覺與機器學習的結合帶來許多不同的應用，使我們的生活更加迅捷與方便。電腦視覺的終極目標是發明一個智慧型機器人，使得此機器人能夠和具有與一般人無異的感知與互動。而要達到此終極目標的第一步則是：使得機器能夠解讀動態影片背後所代表的實質意義。

比起靜態的影像，擁有時空資訊的動態影片往往蘊含更多的知識。因此，人體動作辨識的應用則成為機器人視覺最重要的基礎之一。動態影片所包含的各種變化大幅增加了分析的難度，使得許多研究學者專注於提高動作辨識的準確度。然而，在過去的研究中，從動態影片中取出特徵值的演算法依然太過複雜以致於難以達到即時。

在此論文中，我們首先介紹一些電腦視覺的基礎與應用。這些角點偵測和特徵選取的演算法非常重要，是所有視覺辨識工作的根基。將平面的辨識工作擴展到具有時空的三維空間裡，我們將面臨額外的挑戰。藉由比較和分析各種不同的演算法之優缺點，我們決定使用區域性時空特徵來處理動作辨識。考慮到系統的效率以及準確度，我們使用MoFREAK特徵來穩固地描述動作影片。MoFREAK分別以FREAK來描述靜態資訊、MIP來描述動態資訊，通過許多資料組的驗證，可以證明是一種表現良好的影片描述子。藉由分析整個系統之中每個步驟所花的時間，我們做實驗來證明此系統可以即時運作。接續的是以不同的資料組來驗證整個系統的效率與準確度之實驗結果，從中亦可以看出此系統之實用性。最後，將此系統應用在實際的影片之結果亦描述在最後一章。我們開發了一種創新的滑動式累計直方圖架構來因應動作會持續改變的影片，並探討各種參數的影響。

總結而言，我們設計了一個即時線上的人體動作辨識系統。此系統利用快速的特徵選取、匹配演算法，以及線上的架構讓它可以立即地判別人體動作。

Computer vision has been developed for decades, and has totally changed our lives. Thanks to the progress of technologies, we have entered the era of big data and smart devices. With the help of machine learning algorithms, electronic devices are able to learn knowledge from big data such as the Internet. The combination of computer vision and machine learning has also brought a large amount of applications, making our lives more convenient. The ultimate goal of computer vision is to invent a brilliant robot which perceives and interacts just like human-beings. Understanding the semantic meaning behind videos is the first step toward this goal.
Rather than still images, videos including spatial-temporal information imply richer knowledge. Therefore, human action recognition becomes a basic application that can be implemented in the vision of robots. The fact that different variations in videos increases the difficulty of analysis, leading many researchers to develop better algorithms aiming at raising the recognition accuracy on datasets. However, the computation complexity of feature extraction in videos is still too complicated to be real-time in past researches.
In the thesis, we first introduce some applications and fundamental functions of computer vision. These algorithms, such as corner detection and feature extraction, are very important since they construct the basis of recognition task. Adapting the 2D successful object recognition framework into 3D videos, we face additional challenges. Comparing several related algorithms and examining the pros and cons of each method, we choose to use space-time local features in our approach.
Considering both the efficiency and accuracy, MoFREAK feature is extracted to generate robust descriptors of action videos. MoFREAK is a feature combining the appearance model and motion model independently. We characterize static information by FREAK and dynamic information by MIP, and show good performance through datasets. Analyzing the computation time of entire procedure, it is feasible for real-time applications. Then the experiment results of proposed action recognition system is described thoroughly to prove the robustness and efficiency. Finally, to adapt to high resolution and online situations, an innovative sliding histogram scheme is developed.
To sum up, a real-time online action recognition system is designed. We can recognize different actions instantly due to the proposed fast feature extraction and matching algorithm.

1 Introduction 1
1.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
1.2 The Applications of Computer Vision . . . . . . . . . . . . . 2
1.3 Motivation of Action Recognition . . . . . . . . . . . . . . . 4
1.4 The Fundamental of Computer Vision . . . . . . . . . . . . 7
1.4.1 Detection and Recognition Task in CV . . . . . . . . 7
1.4.2 Corner Detection . . . . . . . . . . . . . . . . . . . . 9
1.4.3 Feature Extraction . . . . . . . . . . . . . . . . . . . 16
1.5 Thesis Organization . . . . . . . . . . . . . . . . . . . . . . . 20
2 Challenges of Action Recognition Systems 21
2.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . 21
2.2 Overview of Action Recognition Methods . . . . . . . . . . . 23
2.2.1 Single-layered Approaches . . . . . . . . . . . . . . . 23
2.2.2 Hierarchical Approaches . . . . . . . . . . . . . . . . 30
2.3 Space-time Features . . . . . . . . . . . . . . . . . . . . . . . 31
2.3.1 Appearance Model . . . . . . . . . . . . . . . . . . . 32
2.3.2 Motion Model . . . . . . . . . . . . . . . . . . . . . . 34
2.4 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35
3 Proposed Robust Action Recognition System 37
3.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . 37
3.2 System Overview . . . . . . . . . . . . . . . . . . . . . . . . 38
3.3 Robust Feature Extraction . . . . . . . . . . . . . . . . . . . 41
3.3.1 Static Feature . . . . . . . . . . . . . . . . . . . . . . 44
3.3.2 Dynamic Feature . . . . . . . . . . . . . . . . . . . . 48
3.4 Recognition System . . . . . . . . . . . . . . . . . . . . . . . 50
3.4.1 BoW Representation . . . . . . . . . . . . . . . . . . 51
3.4.2 Descriptor Matching . . . . . . . . . . . . . . . . . . 53
3.5 Experiment Results . . . . . . . . . . . . . . . . . . . . . . . 53
3.5.1 Proposed Keypoints Removal . . . . . . . . . . . . . 55
3.5.2 Running Time . . . . . . . . . . . . . . . . . . . . . . 56
3.5.3 Accuracy . . . . . . . . . . . . . . . . . . . . . . . . 57
3.6 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . 58
4 Proposed Online Framework for Real-time System 61
4.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . 61
4.2 Realistic Videos . . . . . . . . . . . . . . . . . . . . . . . . . 62
4.2.1 UCF101 dataset . . . . . . . . . . . . . . . . . . . . . 62
4.2.2 Our Dataset . . . . . . . . . . . . . . . . . . . . . . . 64
4.3 Online Recognition Scheme . . . . . . . . . . . . . . . . . . 65
4.3.1 Impact of Parameters . . . . . . . . . . . . . . . . . . 67
4.3.2 Sliding Histogram . . . . . . . . . . . . . . . . . . . . 69
4.3.3 Experiment Results . . . . . . . . . . . . . . . . . . . 70
4.4 Architecture Design . . . . . . . . . . . . . . . . . . . . . . . 71
4.5 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . 73
5 Conclusion 75
Bibliography 77

[1] C. Gu, P. Arbelaez, Y. Lin, K. Yu, and J. Malik, "Multi-component models for object detection," in Computer Vision-ECCV 2012, pp. 445-458, Springer, 2012.
[2] S. Cangeloso, "Self-driving cars could save lives, gas." http://www.geek.com/geek-cetera/self-driving-cars-1447453/, 2011.
[On-line; accessed 15-April-2014].
[3] C. Arthur, "Augmented reality: it''s like real life, but better." http://www.theguardian.com/technology/2010/mar/21/augmented-reality-iphone-advertising, 2010. [Online; accessed 15-April-2014].
[4] R. Shapovalov, "Object detection vs. semantic segmentation." http://computerblindness.blogspot.tw/2010/06/object-detection-vs-semantic.html, 2014. [Online; accessed 15-April-2014].
[5] W. Japan, "Leave your chores to a robot." http://web-japan.org/trends/09_sci-tech/sci090327.html, 2009. [Online; accessed 15-April-2014].
[6] W. Choi, Y.-W. Chao, C. Pantofaru, and S. Savarese, "Understanding indoor scenes using 3d geometric phrases," in Computer Vision and Pattern Recognition (CVPR), 2013 IEEE Conference on, pp. 33-40,IEEE, 2013.
[7] Y. Hu, L. Cao, F. Lv, S. Yan, Y. Gong, and T. S. Huang, "Action detection in complex scenes with spatial and temporal ambiguities," in Computer Vision, 2009 IEEE 12th International Conference on, pp. 128-135, IEEE, 2009.
[8] C. Harris and M. Stephens, "A combined corner and edge detector.,"in Alvey vision conference, vol. 15, p. 50, Manchester, UK, 1988.
[9] C. R. Lab, "Sift object recognition algorithm." http://eecs.vanderbilt.edu/cis/crl/wm.shtml, 2014. [Online; accessed 22-April-2014].
[10] H. Bay, A. Ess, T. Tuytelaars, and L. Van Gool, "Speeded-up robustfeatures (surf)," Computer vision and image understanding, vol. 110,no. 3, pp. 346-359, 2008.
[11] J. Aggarwal and M. S. Ryoo, "Human activity analysis: A review,"ACM Computing Surveys (CSUR), vol. 43, no. 3, p. 16, 2011.
[12] Y. Sheikh, M. Sheikh, and M. Shah, "Exploring the space of a humanaction," in Computer Vision, 2005. ICCV 2005. Tenth IEEE Interna-tional Conference on, vol. 1, pp. 144-149, IEEE, 2005.
[13] I. Laptev, "On space-time interest points," International Journal ofComputer Vision, vol. 64, no. 2-3, pp. 107-123, 2005.
[14] P. Dollar, V. Rabaud, G. Cottrell, and S. Belongie, "Behavior recog-nition via sparse spatio-temporal features," in Visual Surveillance andPerformance Evaluation of Tracking and Surveillance, 2005. 2nd JointIEEE International Workshop on, pp. 65-72, IEEE, 2005.79
[15] K. Soomro, A. R. Zamir, and M. Shah, "Ucf101: A dataset of101 human actions classes from videos in the wild," arXiv preprintarXiv:1212.0402, 2012.
[16] S. Wu and M. Lew, "Evaluation of salient point methods," in Proceed-ings of the 21st ACM international conference on Multimedia, pp. 685-688, ACM, 2013.
[17] H. Jhuang, T. Serre, L. Wolf, and T. Poggio, "A biologically inspiredsystem for action recognition," in Computer Vision, 2007. ICCV 2007.IEEE 11th International Conference on, pp. 1-8, IEEE, 2007.
[18] P. Scovanner, S. Ali, and M. Shah, "A 3-dimensional sift descriptorand its application to action recognition," in Proceedings of the 15thinternational conference on Multimedia, pp. 357-360, ACM, 2007.
[19] A. Fathi and G. Mori, "Action recognition by learning mid-level motionfeatures," in Computer Vision and Pattern Recognition, 2008. CVPR2008. IEEE Conference on, pp. 1-8, IEEE, 2008.
[20] M.-y. Chen and A. Hauptmann, "Mosift: Recognizing human actionsin surveillance videos," 2009.
[21] S. Sadanand and J. J. Corso, "Action bank: A high-level representa-tion of activity in video," in Computer Vision and Pattern Recognition(CVPR), 2012 IEEE Conference on, pp. 1234-1241, IEEE, 2012.
[22] L. Wang, Y. Qiao, and X. Tang, "Motionlets: Mid-level 3d parts for hu-man motion recognition," in Computer Vision and Pattern Recognition(CVPR), 2013 IEEE Conference on, pp. 2674-2681, IEEE, 2013.
[23] Wikipedia, "Outline of object recognition." http://en.wikipedia.org/wiki/Outline_of_object_recognition, 2014. [Online; accessed16-April-2014].80
[24] Wikipedia, "Intelligent vehicle technologies." http://en.wikipedia.org/wiki/Intelligent_vehicle_technologies, 2014. [Online; ac-cessed 16-April-2014].
[25] S. Edelstein, "Bmw activeassist system lets self-driving cars get side-ways and keep you on the road." http://www.digitaltrends.com/cars/bmw-activeassist-introduced-at-ces-2014/#!EkGS4, 2014.[Online; accessed 16-April-2014].
[26] Wikipedia, "Google driverless car." http://en.wikipedia.org/wiki/Google_driverless_car, 2014. [Online; accessed 16-April-2014].
[27] Google TM , "Glass." http://www.google.com/glass/start/, 2014.[Online; accessed 17-April-2014].
[28] S. Izadi, D. Kim, O. Hilliges, D. Molyneaux, R. Newcombe, P. Kohli,J. Shotton, S. Hodges, D. Freeman, A. Davison, et al., "Kinectfusion:real-time 3d reconstruction and interaction using a moving depth cam-era," in Proceedings of the 24th annual ACM symposium on User in-terface software and technology, pp. 559-568, ACM, 2011.
[29] R. A. Newcombe, A. J. Davison, S. Izadi, P. Kohli, O. Hilliges, J. Shot-ton, D. Molyneaux, S. Hodges, D. Kim, and A. Fitzgibbon, "Kinect-fusion: Real-time dense surface mapping and tracking," in Mixed andaugmented reality (ISMAR), 2011 10th IEEE international symposiumon, pp. 127-136, IEEE, 2011.
[30] J. Xiao and L. Quan, "Multiple view semantic segmentation for streetview images," in Computer Vision, 2009 IEEE 12th International Con-ference on, pp. 686-693, IEEE, 2009.
[31] G. J. Brostow, J. Shotton, J. Fauqueur, and R. Cipolla, "Segmen-tation and recognition using structure from motion point clouds," inComputer Vision-ECCV 2008, pp. 44-57, Springer, 2008.81
[32] W. Garage, "Pr2 overview." http://www.willowgarage.com/pages/pr2/overview, 2014. [Online; accessed 16-April-2014].
[33] J. Maitin-Shepard, M. Cusumano-Towner, J. Lei, and P. Abbeel,"Cloth grasp point detection based on multiple-view geometric cueswith application to robotic towel folding," in Robotics and Automa-tion (ICRA), 2010 IEEE International Conference on, pp. 2308-2315,IEEE, 2010.
[34] klaer78, "Antonio espingardeiro develops p37 s65 elderly care bot."https://www.youtube.com/watch?v=EU4pTB6JeEY, 2013. [Online;accessed 16-April-2014].
[35] Wikipedia, "Big data." http://en.wikipedia.org/wiki/Big_data,2014. [Online; accessed 17-April-2014].
[36] Wikipedia, "Cloud computing." http://en.wikipedia.org/wiki/Cloud_computing, 2014. [Online; accessed 17-April-2014].
[37] Wikipedia, "List of most popular websites." http://en.wikipedia.org/wiki/List_of_most_popular_websites, 2014. [Online; accessed17-April-2014].
[38] R. M. Haralock and L. G. Shapiro, Computer and robot vision.Addison-Wesley Longman Publishing Co., Inc., 1991.
[39] D. G. Lowe, "Object recognition from local scale-invariant features,"in Computer vision, 1999. The proceedings of the seventh IEEE inter-national conference on, vol. 2, pp. 1150-1157, Ieee, 1999.
[40] D. G. Lowe, "Distinctive image features from scale-invariant key-points," International journal of computer vision, vol. 60, no. 2, pp. 91-110, 2004.82
[41] H. Bay, T. Tuytelaars, and L. Van Gool, "Surf: Speeded up robust fea-tures," in Computer Vision-ECCV 2006, pp. 404-417, Springer, 2006.
[42] E. Rosten and T. Drummond, "Machine learning for high-speed cornerdetection," in Computer Vision-ECCV 2006, pp. 430-443, Springer,2006.
[43] E. Rosten, R. Porter, and T. Drummond, "Faster and better: A ma-chine learning approach to corner detection," Pattern Analysis andMachine Intelligence, IEEE Transactions on, vol. 32, no. 1, pp. 105-119, 2010.
[44] H. Bannour, L. Hlaoua, and B. el Ayeb, "Survey of the adequate de-scriptor for content-based image retrieval on the web: Global versuslocal features.," in CORIA, pp. 445-456, 2009.
[45] Y. RAOUI, E. H. BOUYAKHF, M. DEVY, and F. REGRAGUI,"Global and local image descriptors for content based image retrievaland object recognition," Applied Mathematical Sciences, vol. 5, no. 42,pp. 2109-2136, 2011.
[46] S. Leutenegger, M. Chli, and R. Y. Siegwart, "Brisk: Binary robustinvariant scalable keypoints," in Computer Vision (ICCV), 2011 IEEEInternational Conference on, pp. 2548-2555, IEEE, 2011.
[47] A. Alahi, R. Ortiz, and P. Vandergheynst, "Freak: Fast retina key-point," in Computer Vision and Pattern Recognition (CVPR), 2012IEEE Conference on, pp. 510-517, IEEE, 2012.
[48] C. Cortes and V. Vapnik, "Support-vector networks," Machine learn-ing, vol. 20, no. 3, pp. 273-297, 1995.83
[49] J. Friedman, T. Hastie, and R. Tibshirani, "Special invited paper. ad-ditive logistic regression: A statistical view of boosting," Annals ofstatistics, pp. 337-374, 2000.
[50] L. Breiman, "Random forests," Machine learning, vol. 45, no. 1, pp. 5-32, 2001.
[51] Wikipedia, "Decision tree learning." http://en.wikipedia.org/wiki/Decision_tree_learning, 2014. [Online; accessed 18-April-2014].
[52] H. P. Moravec, "Obstacle avoidance and navigation in the real worldby a seeing robot rover.," tech. rep., DTIC Document, 1980.
[53] Wikipedia, "Corner detection." http://en.wikipedia.org/wiki/Corner_detection, 2014. [Online; accessed 18-April-2014].
[54] S. M. Smith and J. M. Brady, "Susana new approach to low levelimage processing," International journal of computer vision, vol. 23,no. 1, pp. 45-78, 1997.
[55] J. R. Quinlan, "Induction of decision trees," Machine learning, vol. 1,no. 1, pp. 81-106, 1986.
[56] K. Mikolajczyk and C. Schmid, "An a&;#64259;ne invariant interest point de-tector," in Computer VisionECCV 2002, pp. 128-142, Springer, 2002.
[57] H. Bay, T. Tuytelaars, and L. Van Gool, "Surf: Speeded up robust fea-tures," in Computer Vision-ECCV 2006, pp. 404-417, Springer, 2006.
[58] R. Poppe, "A survey on vision-based human action recognition," Imageand vision computing, vol. 28, no. 6, pp. 976-990, 2010.
[59] A. F. Bobick and J. W. Davis, "The recognition of human movementusing temporal templates," Pattern Analysis and Machine Intelligence,IEEE Transactions on, vol. 23, no. 3, pp. 257-267, 2001.84
[60] Y. Ke, R. Sukthankar, and M. Hebert, "Event detection in crowdedvideos," in Computer Vision, 2007. ICCV 2007. IEEE 11th Interna-tional Conference on, pp. 1-8, IEEE, 2007.
[61] M. Rodriguez, J. Ahmed, and M. Shah, "Action mach a spatio-temporal maximum average correlation height
lter for action recogni-tion," in Computer Vision and Pattern Recognition, 2008. CVPR 2008.IEEE Conference on, pp. 1-8, June 2008.
[62] A. Yilmaz and M. Shah, "Actions sketch: A novel action representa-tion," in Computer Vision and Pattern Recognition, 2005. CVPR 2005.IEEE Computer Society Conference on, vol. 1, pp. 984-989, IEEE,2005.
[63] M. Blank, L. Gorelick, E. Shechtman, M. Irani, and R. Basri, "Actionsas space-time shapes," in Computer Vision, 2005. ICCV 2005. TenthIEEE International Conference on, vol. 2, pp. 1395-1402, IEEE, 2005.
[64] A. A. Efros, A. C. Berg, G. Mori, and J. Malik, "Recognizing actionat a distance," in Computer Vision, 2003. Proceedings. Ninth IEEEInternational Conference on, pp. 726-733, IEEE, 2003.
[65] J. Yamato, J. Ohya, and K. Ishii, "Recognizing human action in time-sequential images using hidden markov model," in Computer Visionand Pattern Recognition, 1992. Proceedings CVPR''92., 1992 IEEEComputer Society Conference on, pp. 379-385, IEEE, 1992.
[66] N. Oliver, E. Horvitz, and A. Garg, "Layered representations for hu-man activity recognition," in Multimodal Interfaces, 2002. Proceedings.Fourth IEEE International Conference on, pp. 3-8, IEEE, 2002.
[67] Y. A. Ivanov and A. F. Bobick, "Recognition of visual activities andinteractions by stochastic parsing," Pattern Analysis and Machine In-telligence, IEEE Transactions on, vol. 22, no. 8, pp. 852-872, 2000.85
[68] M. S. Ryoo and J. K. Aggarwal, "Recognition of composite human ac-tivities through context-free grammar based representation," in Com-puter Vision and Pattern Recognition, 2006 IEEE Computer SocietyConference on, vol. 2, pp. 1709-1718, IEEE, 2006.
[69] M. A. Goodale and A. D. Milner, "Separate visual pathways for per-ception and action," Trends in neurosciences, vol. 15, no. 1, pp. 20-25,1992.
[70] M. A. Giese and T. Poggio, "Neural mechanisms for the recognitionof biological movements," Nature Reviews Neuroscience, vol. 4, no. 3,pp. 179-192, 2003.
[71] C. Schmid, R. Mohr, and C. Bauckhage, "Evaluation of interest pointdetectors," International Journal of computer vision, vol. 37, no. 2,pp. 151-172, 2000.
[72] K. Mikolajczyk and C. Schmid, "A performance evaluation of localdescriptors," Pattern Analysis and Machine Intelligence, IEEE Trans-actions on, vol. 27, no. 10, pp. 1615-1630, 2005.
[73] K. Schindler and L. Van Gool, "Action snippets: How many framesdoes human action recognition require?," in Computer Vision and Pat-tern Recognition, 2008. CVPR 2008. IEEE Conference on, pp. 1-8,IEEE, 2008.
[74] A. Klaser and M. Marszalek, "A spatio-temporal descriptor based on3d-gradients," 2008.
[75] O. Kliper-Gross, Y. Gurovich, T. Hassner, and L. Wolf, "Motion in-terchange patterns for action recognition in unconstrained videos," inComputer Vision-ECCV 2012, pp. 256-269, Springer, 2012.86
[76] L. Fei-Fei and P. Perona, "A bayesian hierarchical model for learningnatural scene categories," in Computer Vision and Pattern Recogni-tion, 2005. CVPR 2005. IEEE Computer Society Conference on, vol. 2,pp. 524-531, IEEE, 2005.
[77] C. Whiten, R. Laganiere, and G.-A. Bilodeau, "Efficient action recog-nition with mofreak," in Computer and Robot Vision (CRV), 2013 In-ternational Conference on, pp. 319-325, IEEE, 2013.
[78] R. W. Hamming, "Error detecting and error correcting codes," BellSystem technical journal, vol. 29, no. 2, pp. 147-160, 1950.
[79] J. J. Gibson, "The perception of the visual world.," 1950.
[80] E. Shechtman and M. Irani, "Matching local self-similarities across im-ages and videos," in Computer Vision and Pattern Recognition, 2007.CVPR''07. IEEE Conference on, pp. 1-8, IEEE, 2007.
[81] S. Maji, A. C. Berg, and J. Malik, "Classi
cation using intersectionkernel support vector machines is e&;#64259;cient," in Computer Vision andPattern Recognition, 2008. CVPR 2008. IEEE Conference on, pp. 1-8,IEEE, 2008.
[82] Wikipedia, "k-means clustering." http://en.wikipedia.org/wiki/K-means_clustering, 2014. [Online; accessed 24-May-2014].
[83] C. Schuldt, I. Laptev, and B. Caputo, "Recognizing human actions: alocal svm approach," in Pattern Recognition, 2004. ICPR 2004. Pro-ceedings of the 17th International Conference on, vol. 3, pp. 32-36,IEEE, 2004.
[84] G. Bradski Dr. Dobb''s Journal of Software Tools.