跳到主要內容

臺灣博碩士論文加值系統

(100.26.176.111) 您好!臺灣時間:2024/07/16 14:34
字體大小: 字級放大   字級縮小   預設字形  
回查詢結果 :::

詳目顯示

: 
twitterline
研究生:Pushpavalli Mohan
研究生(外文):Pushpavalli Mohan
論文名稱:基於MobileNet深度學習方法之面部情緒識別
論文名稱(外文):Face Emotion Recognition Using a MobileNet-Based Deep Learning Approach
指導教授:張賢宗張賢宗引用關係
指導教授(外文):H. T. Chang
口試委員:李權明張賢宗趙一平
口試委員(外文):C. M. LeeH. T. ChangY. P. Chao
口試日期:2023-06-20
學位類別:碩士
校院名稱:長庚大學
系所名稱:資訊工程學系
學門:工程學門
學類:電資工程學類
論文種類:學術論文
論文出版年:2023
畢業學年度:111
語文別:英文
論文頁數:55
外文關鍵詞:Face Emotion RecognitionArtificial Neural Networks (ANNs)MobileNetDeep Learningartificial intelligence
相關次數:
  • 被引用被引用:0
  • 點閱點閱:59
  • 評分評分:
  • 下載下載:16
  • 收藏至我的研究室書目清單書目收藏:0
Face Emotion Recognition has become a fascinating and crucial area of research in Artificial Intelligence in recent years, with applications in various fields. Despite advancement, archiving high recognition rates in facial emotion recognition remains a challenging task. Existing approaches rely on static image frames and neutral facial expressions, which limits their effectiveness. To address this issue, we conducted a study where we utilized various deep learning techniques to identify seven emotions (happiness, anger, disgust, fear, sadness, surprise, and neutrality) from human facial images. This included both static and dynamic images from the CK+ and FER datasets. Our goal was to develop a facial emotion recognition system capable of achieving high accuracy rates in real-time face emotion recognition. We trained and tested our system using different models, including the Lightweight MobileNet and Artificial Neural Networks, and compared their effectiveness in accurately recognizing emotional expressions. Our findings demonstrate the potential of deep learning methods to enhance the accuracy of facial emotion recognition systems. Through our research, we developed a visualization technique that effectively identifies crucial facial regions for detecting distinct emotions by analyzing classifier outputs. The results highlight that various emotions exhibit sensitivity to different parts of the face. The research outcomes underscore the potential of targeted, deep learning approaches in the field of face emotion recognition.
Table of Contents

Contents

English Abstract i
Table of Contents ii
List of Figures iv
List of Tables v
Chapter 1 1
Introduction 1
1.1. Background of Face Recognition 1
1.2. How it works? 2
1.3. Face Emotion Recognition works in Artificial Neural Networks (ANNs) 4
1.4. Objectives on Face Emotion Recognition 5
1.5. Overview of Face Emotion Recognition works on ANN 6
1.6. Contribution 7
1.7. Thesis Composition 8
Chapter 2 9
Related Work 9
2.1. Literature Review 9
2.2. Classification 11
2.2.1 Haar Cascade 12
2.2.2. Local Binary Pattern (LBP) 13
2.3. System Requirements 14
2.3.1 Tensor Flow 15
2.3.2 OpenCV 15
Chapter 3 17
Proposed Method 17
3.2. My Proposed Model 17
3.1. The Consequence of the Proposed Model 19
3.3. Artificial Neural Network (ANN) 20
3.4. Lightweight MobileNetV2 22
3.4.1. FeatureMap global pooling 2D layer 23
3.4.2. Dense layer 24
3.5. Difference between MobileNet and Lightweight MobileNet 25
Chapter 4 27
Evaluation Results 27
4.1. Experimental Setup 27
4.2. Data Sets 27
4.3. Data Preprocessing 29
4.4. Data Visualization 31
4.4. Evaluation and Comparison 32
4.5. Live Test 36
4.5.1. Image Based Result 36
4.5.2. Video Based Result 38
Chapter 5 40
Conclusion and Future Work 40
5.1. Conclusion 40
5.2. Future Work 41
References 42






List of Figures
Figure 1: Various steps are involved in Face Emotion Recognition ..3
Figure 2: Example of seven face emotion from FER2013 dataset[1]. ..........8
Figure 3: Seven Universal Human Facial Emotions from FER2013 dataset .........10
Figure 4: Basic Face Emotion (Anger, Disgust, Fear, Happy, Sad, Surprise, and
Neutral) images from CK+ Dataset. ......10
Figure 5: The Architecture of the Proposed Model demonstrates the integration of
feature maps with the how integration of feature map with the lightweight
MobileNetV2, followed by a connection to the classification stage. 18
Figure 6: Artificial Neural Network (ANN) example model 21
Figure 7: Architecture of Lightweight MobileNetV2[29].....23
Figure 8: Sample images face of images by fine tuning MobileNet with FER2013
dataset.........29
Figure 9: Sample images of Predicted emotions. .......32
Figure 10: Evaluation of Model Performance on the FER2013 Dataset using a
Confusion Matrix. .33
Figure 11: Evaluation of Model Performance on the CK+ Dataset using a
Confusion Matrix .34
Figure 12: The First Image shows the pre-predicted RGB image, while the Second
images shows the emotion predicted from a static image by the emotion of
happiness....37
Figure 13: Live Test while webcam detect the emotion from face. Right side
image shows the FPS Range.......39

List of Tables

Table 1: Dataset Distribution of Training, Testing and Validation for FER 2013 and CK+ datasets. 28

Table 2: Presents a Comparison of classification accuracy results on FER 2013 dataset. 35

Table 3: Presents a Comparison of classification accuracy results on dataset. 36
References


[1]R. Cowie et al., "Emotion recognition in human-computer interaction," IEEE Signal Processing Magazine, vol. 18, no. 1, pp. 32-80, 2001, doi: 10.1109/79.911197.
[2]C. Clavel, I. Vasilescu, L. Devillers, G. Richard, and T. Ehrette, "Fear-type emotion recognition for future audio-based surveillance systems," Speech Communication, vol. 50, no. 6, pp. 487-503, 2008/06/01/ 2008, doi: https://doi.org/10.1016/j.specom.2008.03.012.
[3]S. T. Saste and S. M. Jagdale, "Emotion recognition from speech using MFCC and DWT for security system," in 2017 International conference of Electronics, Communication and Aerospace Technology (ICECA), 20-22 April 2017 2017, vol. 1, pp. 701-704, doi: 10.1109/ICECA.2017.8203631.
[4]J. Edwards, H. J. Jackson, and P. E. Pattison, "Emotion recognition via facial expression and affective prosody in schizophrenia: A methodological review," Clinical Psychology Review, vol. 22, no. 6, pp. 789-832, 2002/07/01/ 2002, doi: https://doi.org/10.1016/S0272-7358(02)00130-7.
[5]H.-C. Chu, W. W.-J. Tsai, M.-J. Liao, and Y.-M. Chen, "Facial emotion recognition with transition detection for students with high-functioning autism in adaptive e-learning," Soft Computing, vol. 22, no. 9, pp. 2973-2999, 2018/05/01 2018, doi: 10.1007/s00500-017-2549-z.
[6]D. Aneja, A. Colburn, G. Faigin, L. Shapiro, and B. Mones, "Modeling Stylized Character Expressions via Deep Learning," in Computer Vision – ACCV 2016, Cham, S.-H. Lai, V. Lepetit, K. Nishino, and Y. Sato, Eds., 2017// 2017: Springer International Publishing, pp. 136-153.
[7]Y. LeCun, "Generalization and network design strategies," Connectionism in perspective, vol. 19, no. 143-155, p. 18, 1989.
[8]E. Tran, M. B. Mayhew, H. Kim, P. Karande, and A. D. Kaplan, "Facial Expression Recognition Using a Large Out-of-Context Dataset," in 2018 IEEE Winter Applications of Computer Vision Workshops (WACVW), 15-15 March 2018 2018, pp. 52-59, doi: 10.1109/WACVW.2018.00012.
[9]P. Ekman and W. V. Friesen, "Facial action coding system," Environmental Psychology & Nonverbal Behavior, 1978.
[10]E.-J. Cheng et al., "Deep Sparse Representation Classifier for facial recognition and detection system," Pattern Recognition Letters, vol. 125, pp. 71-77, 2019/07/01/ 2019, doi: https://doi.org/10.1016/j.patrec.2019.03.006.
[11]P. S. Aleksic and A. K. Katsaggelos, "Automatic facial expression recognition using facial animation parameters and multistream HMMs," IEEE Transactions on Information Forensics and Security, vol. 1, no. 1, pp. 3-11, 2006, doi: 10.1109/TIFS.2005.863510.
[12]A. Krizhevsky, I. Sutskever, and G. E. Hinton, "Imagenet classification with deep convolutional neural networks," Communications of the ACM, vol. 60, no. 6, pp. 84-90, 2017.
[13]K. He, X. Zhang, S. Ren, and J. Sun, "Deep residual learning for image recognition," in Proceedings of the IEEE conference on computer vision and pattern recognition, 2016, pp. 770-778.
[14]J. Long, E. Shelhamer, and T. Darrell, "Fully convolutional networks for semantic segmentation," in Proceedings of the IEEE conference on computer vision and pattern recognition, 2015, pp. 3431-3440.
[15]K. He, G. Gkioxari, P. Dollár, and R. Girshick, "Mask r-cnn," in Proceedings of the IEEE international conference on computer vision, 2017, pp. 2961-2969.
[16]S. Minaee, A. Abdolrashidiy, and Y. Wang, "An experimental study of deep convolutional features for iris recognition," in 2016 IEEE signal processing in medicine and biology symposium (SPMB), 2016: IEEE, pp. 1-6.
[17]C. Shan, S. Gong, and P. W. McOwan, "Facial expression recognition based on local binary patterns: A comprehensive study," Image and vision Computing, vol. 27, no. 6, pp. 803-816, 2009.
[18]C. Liu and H. Wechsler, "Gabor feature based classification using the enhanced fisher linear discriminant model for face recognition," IEEE Transactions on Image processing, vol. 11, no. 4, pp. 467-476, 2002.
[19]S. M. Mavadati, M. H. Mahoor, K. Bartlett, P. Trinh, and J. F. Cohn, "Disfa: A spontaneous facial action intensity database," IEEE Transactions on Affective Computing, vol. 4, no. 2, pp. 151-160, 2013.
[20]M. Abadi et al., "Tensorflow: Large-scale machine learning on heterogeneous distributed systems," arXiv preprint arXiv:1603.04467, 2016.
[21]G. Bradski, "The openCV library," Dr. Dobb's Journal: Software Tools for the Professional Programmer, vol. 25, no. 11, pp. 120-123, 2000.
[22]G. Coelho, E. Kougianos, S. P. Mohanty, P. Sundaravadivel, and U. Albalawi, "An IoT-enabled modular quadrotor architecture for real-time aerial object tracking," in 2015 IEEE International Symposium on Nanoelectronic and Information Systems, 2015: IEEE, pp. 197-202.
[23]R. Manoharan and S. Chandrakala, "Android OpenCV based effective driver fatigue and distraction monitoring system," in 2015 International Conference on Computing and Communications Technologies (ICCCT), 2015: IEEE, pp. 262-266.
[24]G. Bradski and A. Kaehler, Learning OpenCV: Computer vision with the OpenCV library. " O'Reilly Media, Inc.", 2008.
[25]P. Viola and M. Jones, "Robust real-time object detection [J]," International journal of computer vision, vol. 4, pp. 34-47, 2001.
[26]R. Lienhart and J. Maydt, "An extended set of haar-like features for rapid object detection," in Proceedings. international conference on image processing, 2002, vol. 1: IEEE, pp. I-I.
[27]P. Domingos, "A few useful things to know about machine learning," Commun. ACM, vol. 55, no. 10, pp. 78–87, 2012, doi: 10.1145/2347736.2347755.
[28]M. Sandler, A. Howard, M. Zhu, A. Zhmoginov, and L.-C. Chen, "Mobilenetv2: Inverted residuals and linear bottlenecks," in Proceedings of the IEEE conference on computer vision and pattern recognition, 2018, pp. 4510-4520.
[29]J. Talahua, J. Buele, P. Calvopiña, and J. Varela Aldás, "Facial Recognition System for People with and without Face Mask in Times of the COVID-19 Pandemic," Sustainability, vol. 13, p. 6900, 06/18 2021, doi: 10.3390/su13126900.
[30]K. Dong, C. Zhou, Y. Ruan, and Y. Li, "MobileNetV2 model for image classification," in 2020 2nd International Conference on Information Technology and Computer Application (ITCA), 2020: IEEE, pp. 476-480.
[31]Q. Tang, J. Li, Z. Shi, and Y. Hu, "Lightdet: A lightweight and accurate object detection network," in ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2020: IEEE, pp. 2243-2247.
[32]T. Emara, H. E. A. E. Munim, and H. M. Abbas, "LiteSeg: A Novel Lightweight ConvNet for Semantic Segmentation," in 2019 Digital Image Computing: Techniques and Applications (DICTA), 2-4 Dec. 2019 2019, pp. 1-7, doi: 10.1109/DICTA47822.2019.8945975.
[33]I. G. Dumitru, Will Cukierski, Yoshua Bengio.,. "Challenges in representation learning: Facial expression recognition challenge (2013)." www.kaggle.com/c/challenges-in-representation-learning-facial-expression-recognition-challenge (accessed.
[34]P. Lucey, J. F. Cohn, T. Kanade, J. Saragih, Z. Ambadar, and I. Matthews, "The Extended Cohn-Kanade Dataset (CK+): A complete dataset for action unit and emotion-specified expression," in 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition - Workshops, 13-18 June 2010 2010, pp. 94-101, doi: 10.1109/CVPRW.2010.5543262.
[35]J. F. C. Patrick Lucey, Takeo Kanade, Jason Saragih, & Zara Ambadar. "CK+ Dataset." https://www.kaggle.com/datasets/davilsena/ckdataset (accessed.
[36]N. Chakraborty, A. Dan, A. Chakraborty, and S. Neogy, "Effect of Dropout and Batch Normalization in Siamese Network for Face Recognition," in International Conference on Innovative Computing and Communications, Singapore, A. Khanna, D. Gupta, S. Bhattacharyya, V. Snasel, J. Platos, and A. E. Hassanien, Eds., 2020// 2020: Springer Singapore, pp. 21-37.
[37]M. D. Zeiler and R. Fergus, "Visualizing and Understanding Convolutional Networks," in Computer Vision – ECCV 2014, Cham, D. Fleet, T. Pajdla, B. Schiele, and T. Tuytelaars, Eds., 2014// 2014: Springer International Publishing, pp. 818-833.
[38]M. I. Georgescu, R. T. Ionescu, and M. Popescu, "Local Learning With Deep and Handcrafted Features for Facial Expression Recognition," IEEE Access, vol. 7, pp. 64827-64836, 2019, doi: 10.1109/ACCESS.2019.2917266.
[39]P. Giannopoulos, I. Perikos, and I. Hatzilygeroudis, "Deep Learning Approaches for Facial Emotion Recognition: A Case Study on FER-2013," in Advances in Hybridization of Intelligent Methods: Models, Systems and Applications, I. Hatzilygeroudis and V. Palade Eds. Cham: Springer International Publishing, 2018, pp. 1-16.
[40]A. Mollahosseini, D. Chan, and M. H. Mahoor, "Going deeper in facial expression recognition using deep neural networks," in 2016 IEEE Winter Conference on Applications of Computer Vision (WACV), 7-10 March 2016 2016, pp. 1-10, doi: 10.1109/WACV.2016.7477450.
[41]R. T. Ionescu, M. Popescu, and C. Grozea, "Local learning to improve bag of visual words model for facial expression recognition," in Workshop on challenges in representation learning, ICML, 2013: Citeseer.
[42]S. Rifai, Y. Bengio, A. Courville, P. Vincent, and M. Mirza, "Disentangling Factors of Variation for Facial Expression Recognition," in Computer Vision – ECCV 2012, Berlin, Heidelberg, A. Fitzgibbon, S. Lazebnik, P. Perona, Y. Sato, and C. Schmid, Eds., 2012// 2012: Springer Berlin Heidelberg, pp. 808-822.
[43]M. Liu, S. Li, S. Shan, R. Wang, and X. Chen, "Deeply Learning Deformable Facial Action Parts Model for Dynamic Expression Analysis," in Computer Vision -- ACCV 2014, Cham, D. Cremers, I. Reid, H. Saito, and M.-H. Yang, Eds., 2015// 2015: Springer International Publishing, pp. 143-157.
[44]S. Han, Z. Meng, A.-S. Khan, and Y. Tong, "Incremental boosting convolutional neural network for facial action unit recognition," Advances in neural information processing systems, vol. 29, 2016.
[45]Z. Meng, P. Liu, J. Cai, S. Han, and Y. Tong, "Identity-Aware Convolutional Neural Network for Facial Expression Recognition," in 2017 12th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2017), 30 May-3 June 2017 2017, pp. 558-565, doi: 10.1109/FG.2017.140.
[46]H. Jung, S. Lee, J. Yim, S. Park, and J. Kim, "Joint fine-tuning in deep neural networks for facial expression recognition," in Proceedings of the IEEE international conference on computer vision, 2015, pp. 2983-2991.
連結至畢業學校之論文網頁點我開啟連結
註: 此連結為研究生畢業學校所提供,不一定有電子全文可供下載,若連結有誤,請點選上方之〝勘誤回報〞功能,我們會盡快修正,謝謝!
QRCODE
 
 
 
 
 
                                                                                                                                                                                                                                                                                                                                                                                                               
第一頁 上一頁 下一頁 最後一頁 top
無相關期刊