臺灣博碩士論文加值系統

在人機介面互動方面，透過使用者的情緒與認知表現來瞭解其感受以便回饋是相當重要的研究課題，本論文研究目的主要在於使用多重生理訊號來發展情緒辨識系統；本研究首先建構受測者獨立操作之測量與分析系統，然後建立受測者情緒認知相關多重生理訊號資料庫，其中輸入訊號包含應用非侵入式穿戴裝置，可以經由身體表面擷取與自律神經相關之影響情緒的反射訊號。情緒辨識實驗應用國際情感圖庫系統(IAPS, International Affective Picture System)誘發三十位受試者好笑、愉悅、噁心、害怕等四類情緒表現，同時利用生理訊號感測器量測與記錄末梢血流量、肌電圖、心電圖、膚電反應及體表溫度等生理訊號。所記錄之多重生理訊號經過正規化、生理參數擷取及特徵值選取過程後，將19個生理參數輸入支援向量機(SVM , Support vector machines)分類器進行分類，以達到辨識情緒的目的。從研究結果顯示，利用國際情感圖庫系統作為影片刺激，以成對T檢定為特徵值選取，其辨識率分別為76.2%、66.7%、71.4%、69%；另一方面以變異數分析為特徵值選取，分類辨識率則為82.9%、71.4%、81.4%、78.6%。本研究最後對於研究情緒辨識時所碰到的難題加以討論，並對未來進行受測者情緒辨識系統的研究提供方向與策略。

In multimodal human-computer interaction, to understand emotion and cognition expression of users for feedback control is an important issue. The purpose of this study is to develop emotion recognition system using multiple physiological signals. In this study, a stand-alone measurement and analysis system for subject operation is firstly constructed, and then an emotion related multiple physiological signals database is built. In which, the input signals that may reflect the autonomic nervous system associated with influence of emotion are acquired using non-invasive and wearable devices from the body surface. The IAPS (International Affective Picture System) is employed to elicit the affective responses of happiness, pleasure, disgust, and fear from thirty healthy subjects. The multiple physiological signals including photoplethysmography, electromyography, electrocardiogram, galvanic skin response, and skin temperature signal are measured and recorded simultaneously. After signal normalization, signal preprocessing, feature extraction, and feature selection, nineteen parameters are input to the support vector machine classifier for the corresponding emotional response classification. From the experimental results of using IAPS to elicit the emotion, it is shown that the accuracies of emotion recognition rate are 76.2%, 66.7%, 71.4%, and 69% based on the t-test, and are 82.9%, 71.4%, 81.4%, and 78.6% based on the ANOVA. Finally, the difficulties associated with the investigation of emotion recognition system are discussed, and the future direction and research suggestions are also provided.

CHINESE ABSTRACT I
ABSTRACT II
ACKNOWLEDGEMENT III
CONTENTS IV
LIST OF TABLES VI
LIST OF FIGURES VII

Chapter 1 Introduction 1
1.1 Background 1
1.2 Understanding Emotion 2
1.2.1 What is Emotion 2
1.2.2 Autonomic Nervous System Corresponds to Emotion 3
1.2.3 Physiological Differentiation of Emotions 5
1.3 Literature Review 7
1.4 Motivation and Purpose 9
1.5 The Organization of the Thesis 9

Chapter 2 Materials and Methods 10
2.1 Research Framework 10
2.2 Physiological Signal Development 12
2.2.1 PPG Measuring Device 12
2.2.2 EMG Measuring Device 12
2.2.3 ECG Measuring Device 14
2.2.4 GSR Measuring Device 15
2.2.5 Skin Temperature Measuring Device 16
2.2.6 Multiple Physiological Signals Measuring Device 17
2.2.7 Affective Films of the Stimulation 17
2.3 Parameters Analysis 19
2.3.1 Multiple Physiological Signals Measurement 19
2.3.2 Physiological Parameter 20
2.3.3 Graphical User Interface 27
2.3.4 Empirical Mode Decomposition 29
2.4 Experimental Design 34
2.4.1 Experimental Procedure 34
2.4.2 Subject Selection 36
2.4.3 Task 37
2.5 Statistical Analysis 38
2.5.1 Paired T-test 39
2.5.2 Analysis of Variance 42
2.6 Classification 47
2.6.1 Support Vector Machines 47
2.6.2 Kernel Function 48
2.6.3 Proposed Procedure 49
2.6.4 Cross-Validation 50

Chapter 3 Experimental Results 51
3.1 System Testing 51
3.2 Statistical Analysis of Physiological Parameters 54
3.3 Emotion Recognition 64

Chapter 4 Discussion and Conclusion 69
4.1 Discussion 69
4.2 Conclusion and Prospects 73
References 75

[1] R.W. Picard, Affective Computing, MIT Media Laboratory Perceptual Computing Section Technical Report No. 321, 1995.
[2] R. W. Picard, E. Vyzas, and J. Healey, “Toward machine emotional intelligence: analysis of affective physiological state,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 23, pp. 1175-1191, Oct. 2001.
[3] A. Haag, S. Goronzy, P. Schaich, and J. Williams, “Emotion recognition using bio-sensors: first steps towards and automatic system,” Affective Dialogue Systems, LNAI 3068, pp: 36-48, 2004.
[4] K. Mera and T. Ichimura, “Emotion analyzing method using physiological state,” KES 2004, Springer-Verlag Berlin Heidelberg, LNAI 3214, pp: 195-201, 2004.
[5] J. Z. Zhou and X. H. Wang, “Multimodal Affective User Interface Using Wireless Devices for Emotion Identification,” IEEE Engineering in Medicine and Biology Society, pp: 7155-7157, 2005.
[6] F. H. Wilhelma, M. C. Pfaltza, and P. Grossman, “Continuous electronic data capture of physiology, behavior and experience in real life: towards ecological momentary assessment of emotion,” Interacting with Computers, vol. 18, pp. 171-186, Mar. 2006.
[7] H. Gunes, M. Piccardi, and M. Pantic, “From the Lab to the real world: affect recognition using multiple cues and modalities,” Affective computing, Focus on Emotion Expression, Synthesis and Recognition, pp. 185-218, 2008.
[8] J. Kim and E. André, “Emotion recognition based on physiological changes in music listening,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 30, pp. 2067-2083, Dec. 2008.
[9] K. Oatley and J. Jenkins, Understanding Emotions, Blackwell Publishers Ltd, Oxford, UK, 1996.
[10] A. J. Vander, J. Sherman, D. S. Luciano, E. P. Widmaier, H. Raff, and K. T. Strang, Human Physiology: The Mechanisms of Body Function, McGraw-Hill Education, 10th edition, 2003.
[11] R. P. Kleinginna, and M. A. Kleinginna, “A Categorized List of Emotion Definitions with Suggestions for a Consensual Definition,” Motivation and Emotion, vol. 5, pp.345-355, 1981.
[12] E. N. Marieb, J. Mallatt,and P. B. Wilhelm, Human Anatomy, Benjamin Cummings, 4th edition, 2004.
[13] R. W. Levenson, P. Ekman, and W. V. Friesen, “Voluntary facial action generates emotion-specific autonomic nervous system activity,” Psychophysiology, vol. 27, pp. 363–384, 1990.
[14] P. Ekman, R. W. Levenson, and W. V. Friesen, “Autonomic Nervous System Activity Distinguishes among Emotions,” Science, New Series, vol. 221, no. 4616, pp. 1208-1210, 1983.
[15] J. J. Gross and R. W. Levenson, “Hiding Feelings: The Acute Effects of Inhibiting Negative and Positive Emotions,” Journal of Abnormal Psychology, vol 106, pp: 95-103, 1997.
[16] J.T. Cacioppo and L.G. Tassinary, “Inferring Psychological Significance from Physiological Signals,” Am. Psychologist, vol. 45, pp. 16-28, Jan. 1990.
[17] J. T. Cacioppo, D. J. Klein, G. G. Bemston, and E.Hateld, “The psychophysiology of emotion,” in Handbook of Emotions, M. Lewis and J. Haviland, Eds. New York: Guilford Press, pp. 119–142, 1993.
[18] G. Stemmler, M. Heldmann, C. A. Pauls, and T. Scherer, “Constraints for emotion specificity in fear and anger: The context counts,” Psychophysiology, vol. 38, pp. 275–291, 2001.
[19] J. J. Gross and R. W. Levenson, “Emotion elicitation using films,” Cognition and Emotion, vol. 9, no. 1, pp. 87–108, 1995.
[20] C. L. Lisetti and F. Nasoz, “Using Noninvasive Wearable Computers to Recognize Human Emotion from Physiological Signal,” EURASIP Journal on Applied Signal Processing, no. 11, pp. 1672-1687, 2004.
[21] P.J. Lang, M.M. Bradley, and B.N. Cuthbert, “International Affective Picture System (IAPS):Technical Manual and Affective Ratings,” NIMH Center for the Study of Emotion and Attention 1997.
[22] D. M. Sloan, “Emotion regulation in action: emotional reactivity in experiential avoidance,” Behavior Research and Therapy, vol. 4, pp.1257–1270, 2004.
[23] B. Melin and U. Lundberg, “A biopsychosocial approach to work-stress and musculoskeletal disorders,” Journal of Psychophysiology, vol. 11, no. 3, pp. 238–247, 1997.
[24] M.A. Garcia-Gonzalez and R. Pallas-Areny, “A novel robust index to assess beat-to-beat variability in heart rate time-series analysis,” IEEE Trans., Biomedical Engineering, vol. 48, pp. 617-621, 2001.
[25] F. Wang, K. Sagawa, and H. Inooka, “Time domain heart rate variability index for assessment of dynamicstress,” Computers in Cardiology, pp. 97-100, 1998.
[26] P. D. Drummond and S. H. Quah, “The effect of expressing anger on cardiovascular reactivity and facial blood flow in Chinese and Caucasians,” Psychophysiology, vol. 38, pp.190-196, 2001.
[27] R. McCraty, M. Atkinson, W. A. Tiller, G. Rein, and A. D. Watkins, “The effects of emotions on short-term power spectrum analysis of heart rate variability,” Am. J. Cardiol., vol.76 ,pp. 1089-1093, 1995.
[28] H. P. Huang, “DSP-Based Controller for a Muti-Degree Prosthetic Hand,” Proceedings of IEEE International Conference on Robotics and Automation, pp.1378-1383, 2000.
[29] K. H. Kim, S. W. Bang, and S. R. Kim, “Emotion recognition system using short-term monitoring of physiological signals,” Medical & Biological Engineering & Computing, vol. 42, pp. 419-427, 2004.
[30] V. SHUSTERMAN and O. BARNEA,“Analysis of skin-temperature variability compared to variability of blood pressure and heart rate,” IEEE Ann. Conf. Engineering Medicine Biology Society, pp. 1027–1028, 1995.
[31]N. E. Huang, Z. Shen, S. R. Long, M. C. Wu, H. H. Shih, Q. Zheng, N. Yen, C. C. Tung and H. H Liu, “The Empirical Mode Decomposition and the Hilbert Spectrum for Nonlinear and Non-Stationary Time Series Analysis, Proceedings,’’ in Proc. R. Soc. London A, vol. 454, pp. 903–995, 1998.
[32] F. Nasoz, K. Alvarez, C. L. Lisetti, and N. Finkelstein, “Emotion Recognition from Physiological Signals for Presence Technologies,” Cognition, Technology & Work, vol. 6, no. 1, pp. 4-14, 2004.
[33] http://www.csie.ntu.edu.tw/~cjlin/libsvm/
[34] L. Salahuddin, J. Cho, M. G. Jeong, and D. Kim, “Ultra Short Term Analysis of Heart Rate Variability for Monitoring Mental Stress in Mobile Settings,” IEEE, Engineering in Medicine and Biology Society, pp. 4656-4659, 2007.
[35] R. W. Levenson, Emotion and the autonomic nervous system: A prospectus for research on autonomic specificity, In H. L. Wagner, editor, Social Psychophysiology and Emotion: Theory and Clinical Applications, John Wiley & Sons, Inc., pp. 17–42, 1988.