臺灣博碩士論文加值系統

人們通常會對一些語意事件（semantic-event）感興趣且希望實現自動化偵測， 所以特定行為 （specific-behavior） 偵測已成為影像監視系統（video surveillance system）中相當重要的一門研究。本論文應用一個基於隱型馬可夫模型（Hidden Markov Model）於游泳池環境偵測泳者特定行為的方法。本研究中，泳者行為由時間軸上一連串靜態影像框（image-frame）所描述，運用包體結構（Convexity-Structure）逼近每張影像框中的泳者像素（swimmer-blob），以取出包體結構的三個重要性質－寬高關係、角度比例、頂點的水平座標比例，來描述該泳者的靜態姿勢。此法使原影像串列轉型為三維向量串列。本研究預先取了2000個於任行為中可能存在之姿勢且取出上述特徵向量，用k-mean法分為6群，編製碼書（codebook）以定義6類blob-type，供訓練/辨識時觀測之swimmer-blob可歸屬到最鄰近的族群，達對照轉碼之目的；此法使上述三維向量串列轉型成一簡單編碼（族碼）串列。於訓練階段，使用代表某行為的影片轉出上述的編碼串列，訓練該行為的隱馬可夫模型；測試階段時再將未知行為影片轉成編碼串列輸入訓練好的隱馬可夫模型，檢視匹配度以推論該片段所屬行為。
本研究針對泳者四種正常特定行為（仰泳、蛙泳、捷泳、蝶泳）與兩種異常行為（水道中掙扎、抓繩求救）進行驗證，實作系統以36段影片訓練後對60段影片驗證，對雙向雙水道正常行為泳者同時判別輸出泳姿，達90%的辨識率，對異常行為亦能有效偵測且辨識。

Semantic-event detection is one of the most important parts in video surveillance system, because people usually interest in some specific-behaviors and desire to detect them automatically. This study apply a HMM (Hidden Markov model)-based methodology to detect some specific-behaviors of a swimmer in a swimming pool. In this study, a behavior is composed of a series of the static image-frames. For each frame, we use a Convexity-Structure to enclose the swimmer’s shape after segmenting the swimmer-blob. With the Convexity-Structure, this study clearly defines the feature parameters as a three-dimensional vector. Firstly, more than 2000 swimmer-blobs are analyzed from the possible static postures captured from each interested behaviors. Then, they are clustered into 6 feature-postures by using k-mean method, and a codebook is created for mapping a swimmer-blob into one of 6 feature blob-types in training/recognition stage. Consequently, the time-sequential blobs are converted to a feature-vector sequence and transformed into a symbolic-sequence by the codebook. Thus a learned HMM can be obtained by this symbolic-sequence from a given specific-behavior. After that, the learned HMM is used to detect the swimmer’s behavior which is normal or abnormal.
We implement a system to automatically recognize four different swimming styles (Backstroke, Breaststroke, Freestyle, and Butterfly) and two abnormal behaviors (Rope grasping for help and struggling in waterway). The system is trained by using 36 known video clips and has been verified in two cases. The first case is the classification of 60 video clips, each containing a specified behavior on 1 or 2 waterways. A 90% recognition rate is obtained. Second case is a more realistic situation which combined both abnormal behaviors.

LIST OF CONTENTS
中文摘要......................................................................................................................................I
ABSTRACT.......................................................................................................................II
LIST OF CONTENTS....................................................................................................III
Figure Index.....................................................................................................................IV
Table Index........................................................................................................................V


1. INTRODUCTION…………………………………………………………………..1
1.1 Background……....................................................................................................1
1.2 Motivation..............................................................................................................2
1.3 Objectives of this Study.........................................................................................3
1.4 Organization of This Thesis...................................................................................5
2. Literature Review and Related Work…….……………………………………......6
2.1 The Highlight Detection in Sports Activity.………………………...…………6
2.2 Detection of Behaviors in Swimming Pools……………..…………...………….8
2.3 Detection of Specific/Non-Specific Behaviors....................................................12
2.4 Fundamentals of Hidden Markov Model (HMM)…...........................................14
2.4.1 Markov Chains and Extension to Hidden Models……..............................18
2.4.2 HMM Elements and Symbol Description……...........................................20
2.4.3 Learning Process of HMM……..................................................................22
2.4.4 Recognition Process of HMM……............................................................27
2.5 Feature Extraction for human behavior……………...........................................30
3. SYSTEM IMPLEMENTATION…............................................................................32
3.1 System Overview……………….........................................................................32
3.2 Feature Definition and the Convexity-Structure…………………………........37
3.3 HMM Modeling…...............................................................................................38
4. EXPERIMENTAL RESULTS................................................................47
5. CONCLUSION AND FEATURE WORK………....................................................65
5.1 Conclusion...........................................................................................................65
5.2 Future Work...................................................................................66
6. REFERENCES.............................................................................................................67





III

[Aggarwal1999] J. Aggarwal and Q. Cai, “Human motion analysis: A review,” Comput. Vis. Image Understanding, vol. 73, no. 3, pp. 428–440, 1999.
[Bobick1995] A. Bobick and A. Wilson, “A state-based technique for the summarization and recognition of gesture,” in Proc. Int. Conf. on Computer Vision, 1995, pp. 382–388.
[Bobick1996] J. Davis and A. Bobick, “Real-time recognition of activity using temporal templates,” in Proc. IEEE Computer Soc. Workshop Applications on Computer Vision, 1996, pp. 928–934.
[Chang2002] P. Chang, M. Han and Y. Gong, “Extract Highlights from Baseball Game Video with Hidden Markov Models,“ in Proc. IEEE Int. Conf. Image Processing, vol. 1, Rochester, NY, Sep. 22–25, 2002, pp. 609–612.
[Chang2008] Chao-Yang Chang, “Trajectory-Based Event Detection and Discovery for Surveillance Videos”. Department of Computer Science, National Tsing Hua University, Taiwan, June 2008.
[Chen2006] Hsuan-Sheng Chen, “Human Action Recognition Using Star Skeleton”. Institute of Computer Science and Information Engineering National Chiao-Tung University, 2006
[Cheng2004] F.X. Cheng, W. Christmas, and J. Kittler, “Periodic human motion description for sports video databases”, ICPR 2004.
[Cheng2006] C.-C. Cheng and C.-T. Hsu, "Fusion of Audio and Motion Information on HMM-Based Highlight Extraction for Baseball Games," IEEE Trans. Multimedia, vol. 8, Issue 3, June 2006, pp. 585 – 599.
[Chu2005] W.-T. Chu and J.-L. Wu, “Integration of Rule-based and Model-based Decision Methods for Baseball Event Detection,” in Proc. IEEE Int. Conf. Multimedia and Expo, Amsterdam, 6-8 July 2005
[Cutler2000] R. Cutler and L. Davis, “Robust real-time periodic motion detection, analysis, and applications”, IEEE Trans. on PAMI, 22(8), 2000, pp.781-796.
[Darrel1993] T. Darrel and A. Pentland, “Space-time gesture,” Proc. Computer Vision and Pattern Recognition, pp. 335–340, 1993.
[Duan2004] L.Y Duan, M. Xu, Q. Tian, and C. S. Xu, “Nonparametric motion model with application to camera motion pattern classification”, Proc. of ACM Multimedia, 2004.
[Eng2003] How-Lung Eng, “An automatic drowning detection surveillance system for challenging outdoor pool environments”. Proceedings of the Ninth IEEE International Conference on Computer Vision (ICCV’03).
[Fablet2002] R. Fablet, P. Bouthemy and P. Perez, “Nonparametric motion characterization using causal probabilistic models for video indexing and retrieval”, IEEE Trans. on Image Processing, 11(4), 2002, pp.393-407.
[Ferrando2006] S. Ferrando, G. Gera, and C. Regazzoni, “Classification of Unattended and Stolen Objects in Video-Surveillance System,” in Proc. AVSS’06, 2006, pp. 21-26.
[Fitzgibbon95] A. W. Fitzgibbon and R. B. Fisher, “A buyer’s guide to conic fi tting,” Proceedings of the 5th British Machine Vision Conference (pp. 513–522), Birmingham, 1995.
[Fujiyoshi1998] H. Fujiyoshi and A. J. Lipton. "Real-Time Human Motion Analysis by Image Skeletonization." Proceedings of the Fourth IEEE Workshop on Applications of Computer Vision, pp. 15-21, 1998.
[Gavrila1999] D. Gavrila, “The visual analysis of human movement: A survey,” Comput. Vis. Image Understanding, vol. 73, no. 1, pp. 82–89, 1999.
[Gueziec2002] A. Gueziec, “Tracking Clipsfor Broadcast Television,” in IEEE Computer, vol. 35, Mar 2002, pp. 38-43
[Horii2008] Yu Horii, Hiroaki Kawashima, Takashi Matsuyama, "Speaker Detection Using the Timing Structure of Lip Motion and Sound", IEEE CVPR Workshop on Human Communicative Behavior Analysis (CVPR4HB), 2008.
[Hu2006] W. Hu, X. Xiao, Z. Fu, D. Xie, T. Tan, and S. Maybank, “A System for Learning Statistical Motion Patterns,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 28, no. 9, pp. 1450-1464, Sep. 2006.
[Kale2004] A. Kale, A. Sundaresan, A. N. Rajagopalan, N. P. Cuntoor, A. Roy-Chowdhury, V. Kruger and R. Chellappa. "Identification of Humans Gait," IEEE Transactions on Image Processing, pp. 1163-1173, 2004.
[Kawashima2007] Hiroaki Kawashima and Takashi Matsuyama, "Interval-Based Linear Hybrid Dynamical System for Modeling Cross-Media Timing Structures in Multimedia Signals", International Conference on Image Analysis and Processing (ICIAP 2007), pp.789-794, 2007.
[Leo2004] M. Leo, T. D'Orazio, I. Gnoni, P. Spagnolo and A. Distante. "Complex Human Activity Recognition for Monitoring Wide Outdoor Environments," Proceedings of the 17th International Conference on Pattern Recognition, Vol.4, pp. 913-916, 2004.
[Li2006] Jia-Xu Li, “Baseball Pitch Recognition for Broadcast Television”, Department of Computer Science, National Chung Cheng University, Taiwan, June 2006.
[Li2007] Tsung-Chan Li, “Video Event Detection Using Color and Motion Temporal Histogram”. Department of Computer Science, National Tsing Hua University, Taiwan, June 2007.
[Liang2005] C.-H. Liang, W.-T. Chu, J.-H.-Kuo, J.-L.- Wu and W.-H.-Cheng, “Baseball Event Detection Using Game-Specific Feature Sets and Rules,” IEEE International Symposium on Circuits and Systems, vol. 4, 23-26 May 2005, pp. 3829- 3832
[Liao2003] Wen-Hung Liao, Zhung-Xun Liao , Ming-Je Liu, “Swimming Style Classification from Video Sequences”. 16th IPPR Conference on Computer Vision, Graphics and Image Processing (CVGIP 2003) 2003/8/17~19, Kinmen.
[Lu2004] Wenmiao Lu, Yap-Peng Tan (M’97), “A Vision-Based Approach to Early Detection of Drowning Incidents in Swimming Pools”. IEEE TRANSACTIONS ON CIRCUITS AND SYSTEMS FOR VIDEO TECHNOLOGY, VOL. 14, NO. 2, FEBRUARY 2004.
[Lynch1990] T. E. Lynch, “Apparatus and Method for Detecting Swimmers,” U.S. Patent 4 932 009, June 5, 1990.
[Ma2000] Y.F. Ma, H.J. Zhang, “Motion Texture: A New Motion Based Video Representation”, ICPR 2000, pp. 548-551.
[Meneses2005] Y. L. de Meneses, P. Roduit, F. Luisier and J. Jacot, “Trajectory Analysis for Sport and Video Surveillance,” Electronic Letters on Computer Vision and Image Analysis, 2005, pp. 148-156
[Meniere2000] J. Meniere, “System for Monitoring a Swimming Pool to Prevent Drowning Incidents,” U.S. Patent 6 133 838, Oct. 17, 2000.
[Menoud1999] E. Menoud, “Alarm and Monitoring Device for the Presumption of Bodies in Danger in a Swimming Pool,” U.S. Patent 5 886 630, Mar. 23, 1999.
[Miller1999] B. K. Miller, J. E. Halwachs, and A. J. Farstad, “Swimmer Location Monitor,” US Patent 5 907 281, May 25, 1999.
[Niu2004] F, Niu and M. Abdel-Mottaleb. "View-Invariant Human Activity Recognition Based on Shape and Motion Features," Proceedings of IEEE Sixth International Symposium on Multimedia Software Engineering, pp. 546-556, 2004.
[Oliver2000] N. Oliver, B. Rosario, and A. Pentland, “A bayesian computer vision system for modeling human interactions,” IEEE Trans. Pattern Anal. Machine Intell., vol. 22, pp. 831–843, Aug. 2000.
[Pia1994] F.Pia, “Reflections on lifeguarding surveillance programs,” in Proc. Reflections on Lifeguarding Conf., 1994.
[Rabiner1989] Lawrence R. Rabiner. "A tutorial on Hidden Markov Models and selected applications in speech recognition". Proceedings of the IEEE 77 February 1989 (2): 257-286.
[Rui2000] Y. Rui, A. Gupta and A.Acero, “Automatically Extracting Highlights for Baseball Programs, “in Proc. ACM Multimedia, pp105-115, Oct. 2000.
[Shum2004] H. Shum and T. Komura, “A Spatiotemporal Approach to Extract the 3D Trajectory of the Baseball from a Single View Video Sequence,” in Proc. IEEE Int. Conf. Multimedia and Expo, vol. 3, 27-30 June 2004, pp. 1583- 1586
[Starner1995] T. Starner and A. Pentland, “Visual recognition of american sign language using hidden markov model,” in Proc. Int. Workshop on Automatic Face and Gesture Recognition, 1995, pp. 189–194.
[Sugano2004] M. Sugano, K. Uemura, Y. Nakajima and H. Yanagihara, “High-Level Soccer Indexing on Low-Level Feature Space,” in Proc. IEEE Int. Conf. Image Processing, vol. 3, Singapore, 24-27 Oct. 2004, pp. 1625- 1628.
[Sung2005] Based on CS570 Class Note of Year 2004, Dr. Sung Jung Cho’s tutorial (Year 2005)
[Tong2006] Xiaofeng Tong, Lingyu Duan, Changsheng Xu, Qi Tian, Hanqing Lu1, “Local Motion Analysis and Its Application in Video based Swimming Style Recognition” . The 18th International Conference on Pattern Recognition (ICPR'06)
[Toussaint1983] G. Toussaint. “Solveing geometric problems with the rotating calipers”. In Proceedings of IEEE MELECON’83, Los Alamitos, CA, pp.A10.2/1-4. IEEE Presss, New York,1983.
[Vailaya1999] A.K. Jain, A. Vailaya, and W. Xiong, “Query by video clip”, Multimedia Systems, 7, 1999, pp.369-384.
[WIKIPEDIA2009] http://en.wikipedia.org/wiki/Hidden_Markov_model, July-2009.
[Yamato1992] J. Yamato, J. Ohya, and K. Ishii, “Recognizing human action in timesequential images using hidden markov model,” in Proc. IEEE Conf. Computer Vision and Pattern Recognition, 1992, pp. 379–385.
[Yu2003] X. Yu, C Xu, H.-W. Leong, Q. Tian, Q. Tang and K. W. Wan, “Trajectory-Based Ball Detection and Tracking with Applications to Semantic Analysis of Broadcast Soccer Video,” in Proc. ACM Multimedia, Berkeley, 2003, pp. 11-20
[Zhang2002] D. Zhang and S.-F. Chang, “Event Detection in Baseball Video Using Superimposed Caption Recognition,” in Proc. ACM Multimedia, Juan-les-Pins, 2002, France, pp. 315 – 318
[Zhang96] Z. Zhang, “Parameter estimation techniques: A tutorial with application to conic fitting,” Image and Vision Computing 15 (1996): 59–76.