臺灣博碩士論文加值系統

如今，自動人臉表情辨識在人機介面及監控系統為一項極重要的技術，在模式識別及電腦視覺領域已經吸引大量的關注。
自動人臉表情辨識系統會接收一個輸入資料(靜態人臉影像或動態人臉序列)並且將其辨識為一個基本表情(生氣、難過、驚訝、開心、厭惡、恐懼、中立…等)， 我們的目標在著重於靜態人臉影像，並且辨識為七種表情狀態。在這篇論文中，我們提出使用人臉翻正演算法及自適性指數加權組合架構的卷積神經網路的人臉表情辨識系統。翻正演算法用於對齊小角度人臉旋轉(平面上及平面外)並且利用人臉偵測方法來去除多餘的背景雜訊達到資料歸一化，而自適性指數加權組合架構能夠藉由模型本身優劣程度找出適當的加權參數及組合方式強化自動表情辨識系統的穩定性。因此，根據我們提出的系統，在一些常見的資料庫進行實驗，模擬結果顯示以上提出的方法對於人臉表情辨識皆比過往的表情辨識算法結果要好。
關鍵字: 人臉表情;卷機神經網路;電腦視覺;人臉翻正化;階層式架構。

Nowadays, Automatic Facial Expression Recognition (FER) is an important technique in human-computer interfaces and surveillance systems, has attracted significant attention in pattern recognition and computer vision.
Automatic systems for facial expression recognition receive the input (a static facial image or a facial image sequence) and classify it into one of the basic expressions (anger, sad, surprise, happy, disgust and fear, neutral and so on). Our work will focus on methods based on facial static images and it will consider the seven basic expressions. In this paper, we proposed a CNN based system with face frontalization and Hierarchical architecture for FER. The frontalized algorithm can align the small angle rotation (in-of-plane or out-of-plane) and use the face detection to remove the background noise, the adaptive exponentially weighted average ensemble rule can search the optimal weight according to the efficiency of classifier to improve the robust FER system.
As a result, we perform the proposed system on some popular databases, the simulation results show that it is very effective for facial expression recognition, we achieve an accuracy rate surpassing the state-of-the-art system.

Keyword: facial expression; convolutional neural networks; computer vision; face frontalization; hierarchical structure.

口試委員會審定書 #
誌謝 i
中文摘要 ii
ABSTRACT iii
CONTENTS iv
LIST OF FIGURES vii
LIST OF TABLES x
Chapter 1 Introduction 1
Chapter 2 Fundamentals of Neural Networks 4
2.1 Historical Background 4
2.2 Neurons 5
2.2.1 The Model of Neurons 5
2.2.2 The Activation Function and Weights 7
2.2.3 Role of the Bias in Neural Networks 8
2.3 Feedforward Neural Networks 11
2.3.1 Introduction of feedforward neural network 11
2.3.2 Backpropagation 13
2.4 Convolutional Neural Networks 20
2.4.1 Introduction 20
2.4.2 Convolution Layer 21
2.4.3 Pooling Layer 25
2.4.4 Batch Normalization 26
2.4.5 Fully Connected layer 29
2.4.6 Overfitting and Underfitting 29
2.4.7 Some Famous CNNs Architectures 31
Chapter 3 Fundamentals of Facial Expression Recognition 35
3.1 Definition 35
3.2 Introduction of Framework 35
3.2.1 Face Acquisition 37
3.2.2 Pre-processing 37
3.2.3 Feature Extraction 38
3.2.4 Face Recognition 39
3.2.5 Post-processing 41
3.3 Challenges 41
3.3 Related Work 43
Chapter 4 Proposed Method 48
4.1 Motivation 48
4.2 Introduction 48
4.3 Face Frontalization 49
4.3.1 Introduction 49
4.3.2 Camera Matrix 50
4.3.3 Implement 52
4.4 Recognition 59
4.4.1 The Network Architecture of Deep CNN 59
4.4.2 Hierarchical Structure 61
4.4.3 Individual Members of Ensemble Model 62
4.4.4 Adaptive Exponentially-Weighted Average Ensemble Rule 63
4.4.5 The Supplementary Models 67
Chapter 5 Simulation Result 70
5.1 Database 70
5.2 Experiments and Discussion 72
5.3 Compare with existing algorithm 80
5.4 Conclusion and Future Work 83
REFERENCE 85

[1]Ekman, Paul. "Facial expression and emotion." American psychologist 48.4 (1993): 384.
[2]P. Liu, S. Han, Z. Meng, and Y. Tong, “Facial expression recognition via a boosted deep belief network,” in 2014 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2014, pp. 1805–1812.
[3]C. Shan, S. Gong, and P. W. McOwan, “Facial expression recognition based on local binary patterns: A comprehensive study,” Image and Vision Computing, vol. 27, no. 6, pp. 803–816, 2009.
[4]Y.-H. Byeon and K.-C. Kwak*, “Facial expression recognition using 3d convolutional neural network,” in International Journal of Advanced Computer Science and Applications, vol. 5, no. 12, 2014.
[5]J.-J. J. Lien, T. Kanade, J. Cohn, and C. Li, “Detection, tracking, and classification of action units in facial expression,” Journal of Robotics and Autonomous Systems, 1999.
[6]S. Z. Li and A. K. Jain, Handbook of Face Recognition. Springer Science & Business Media, 2011.
[7]Z. Zhang, M. Lyons, M. Schuster, and S. Akamatsu, “Comparison between geometry-based and gabor-wavelets-based facial expression recognition using multi-layer perceptron,” in Third IEEE International Conference on Automatic Face and Gesture Recognition, 1998. Proceedings, 1998, pp. 454–459.
[8]P. Yang, Q. Liu, and D. Metaxas, “Boosting coded dynamic features for facial action units and facial expression recognition,” in IEEE Conference on Computer Vision and Pattern Recognition, 2007. CVPR ’07, 2007, pp. 1–6.
[9]Y. Lin, M. Song, D. T. P. Quynh, Y. He, and C. Chen, “Sparse coding for flexible, robust 3d facial-expression synthesis,” IEEE Computer Graphics and Applications, vol. 32, no. 2, pp. 76–88, 2012.
[10]S. Lawrence, C. Giles, A. C. Tsoi, and A. Back, “Face recognition: a convolutional neural-network approach,” IEEE Transactions on Neural Networks, vol. 8, no. 1, pp. 98–113, 1997.
[11]Neelam, Manisha Dr Jagjit Singh Dr, and R. Prakash. "Facial Expression Recognition Using Neural Network."
[12]Hebb, Donald Olding. The organization of behavior: A neuropsychological theory. Psychology Press, 2005.
[13]Werbos, Paul. "Beyond regression: New tools for prediction and analysis in the behavior science." Unpublished Doctoral Dissertation, Harvard University (1974).
[14]Simonyan, Karen, and Andrew Zisserman. "Very deep convolutional networks for large-scale image recognition." arXiv preprint arXiv:1409.1556 (2014).
[15]Pandya, Jigar M., Devang Rathod, and Jigna J. Jadav. "A survey of face recognition approach." International Journal of Engineering Research and Applications (IJERA) 3.1 (2013): 632-635.
[16]C.-R. Chen, W.-S. Wong, and C.-T. Chiu, “A 0.64 mm real-time cascade face detection design based on reduced two-field extraction,” IEEE Transactions on Very Large Scale Integration (VLSI) Systems, vol. 19, no. 11, pp. 1937–1948, 2011.
[17] C. Garcia and M. Delakis, “Convolutional face finder: a neural architecture for
fast and robust face detection,” IEEE Transactions on Pattern AnalysisandMachineIntelligence,vol.26,no.11,pp.1408–1423,2004.

[18]Z. Zhang, D. Yi, Z. Lei, and S. Li, “Regularized transfer boosting for face detection across spectrum,” IEEE Signal Processing Letters, vol. 19, no. 3, pp. 131–134, 2012.
[19]M.Bartlett,G.Littlewort,M.Frank,C.Lainscsek,I.Fasel,andJ.Movellan, “Recognizing facial expression: machine learning and application to spontaneous behavior,” in IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2005. CVPR 2005, vol. 2, 2005, pp. 568–573 vol. 2.
[20]Nair, Prathap, and Andrea Cavallaro. "3-D face detection, landmark localization, and registration using a point distribution model." IEEE Transactions on multimedia 11.4 (2009): 611-623
[21]Kato, Takekazu, Takeshi Kurata, and Katsuhiko Sakaue. "Face registration using wearable active vision systems for augmented memory." Proc. Digital Image Computing: Techniques and Applications (DICTA2002). 2002.
[22]Z. Zhang, M. Lyons, M. Schuster, and S. Akamatsu, “Comparison between geometry-based and gabor-wavelets-based facial expression recognition using multi-layer perceptron,” in Third IEEE International Conference on Automatic Face and Gesture Recognition, 1998. Proceedings, 1998, pp. 454–459.
[23]P. Yang, Q. Liu, and D. Metaxas, “Boosting coded dynamic features for facial action units and facial expression recognition,” in IEEE Conference on Computer Vision and Pattern Recognition, 2007. CVPR ’07, 2007, pp. 1–6.
[24]S. Jain, C. Hu, and J. Aggarwal, “Facial expression recognition with temporal modeling of shapes,” in 2011 IEEE International Conference on Computer Vision Workshops (ICCV Workshops), pp. 1642–1649.
[25]Y. Lin, M. Song, D. T. P. Quynh, Y. He, and C. Chen, “Sparse coding for flexible, robust 3d facial-expression synthesis,” IEEE Computer Graphics and Applications, vol. 32, no. 2, pp. 76–88, 2012.
[26] Zeiler, Matthew D., and Rob Fergus. "Visualizing and understanding convolutional networks." European conference on computer vision. Springer, Cham, 2014.
[27]M. Pantic, L.J.M. Rothkrantz, "Automatic Analysis of Facial Expressions: the State of the Art", IEEE Trans. Pattern Analysis and Machine Intelligence, Vol. 22, No. 12, pp. 1424-1445, 2000
[28] G. Donato, M.S. Bartlett, J.C. Hager, P. Ekman, T.J. Sejnowski, "Classifying Facial Actions", IEEE Trans. Pattern Analysis and Machine Intelligence, Vol. 21, No. 10, pp. 974-989, 1999
[29] P. Ekman, Emotion in the Human Face, Cambridge University Press, 1982
[30]P. Ekman, W. Friesen, Facial Action Coding System: A Technique for the Measurement of Facial Movement,Consulting Psychologists Press, 1978
[31]Chibelushi, Claude C., and Fabrice Bourel. "Facial expression recognition: A brief tutorial overview." CVonline: On-Line Compendium of Computer Vision 9 (2003).
[32]S. Z. Li and A. K. Jain, Handbook of Face Recognition. Springer Science & Business Media, 2011
[33]Liu, Ping, et al. "Facial expression recognition via a boosted deep belief network." Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2014.
[34]Shan, Caifeng, Shaogang Gong, and Peter W. McOwan. "Facial expression recognition based on local binary patterns: A comprehensive study." Image and Vision Computing 27.6 (2009): 803-816.
[35]Chao, Wei-Lun, Jian-Jiun Ding, and Jun-Zuo Liu. "Facial expression recognition based on improved local binary pattern and class-regularized locality preserving projection." Signal Processing 117 (2015): 1-10.
[36]M. Dahmane and J. Meunier, “Emotion recognition using dynamic grid-based HoG features,” in 2011 IEEE International Conference on Automatic Face Gesture Recognition and Workshops (FG 2011), 2011, pp. 884–888.
[37]M. Matsugu, K. Mori, Y. Mitari, and Y. Kaneda, “Subject independent facial expression recognition with robust face detection using a convolutional neural network,” Neural Networks: The Official Journal of the International Neural Network Society, vol. 16, no. 5, pp. 555–559, 2003.
[38]Y.-H. Byeon and K.-C. Kwak*, “Facial expression recognition using 3d convolutional neural network,” in International Journal of Advanced Computer Science and Applications, vol. 5, no. 12, 2014
[39]Kim, Bo-Kyeong, et al. "Hierarchical committee of deep convolutional neural networks for robust facial expression recognition." Journal on Multimodal User Interfaces 10.2 (2016): 173-189.

B. Background Knowledge
[40]Hassner, Tal, et al. "Effective face frontalization in unconstrained images." Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.
[41]Zhang, Kaipeng, et al. "Joint face detection and alignment using multitask cascaded convolutional networks." IEEE Signal Processing Letters 23.10 (2016): 1499-1503
[42]Huang, Gao, et al. "Densely connected convolutional networks." Proceedings of the IEEE conference on computer vision and pattern recognition. Vol. 1. No. 2. 2017.
[43]Ioffe, Sergey, and Christian Szegedy. "Batch normalization: Accelerating deep network training by reducing internal covariate shift." arXiv preprint arXiv:1502.03167 (2015).
[44]Cire¸san D, Meier U, Schmidhuber J (2012b) Multi-column deep neural networks for image classification. In: Computer Vision and Pattern Recognition(CVPR),2012IEEEConferenceon,IEEE,pp 3642–3649
[45]Kim, Bo-Kyeong, et al. "Hierarchical committee of deep convolutional neural networks for robust facial expression recognition." Journal on Multimodal User Interfaces 10.2 (2016): 173-189.
C. Databases
[46]Lyons, Michael J., et al. "The Japanese female facial expression (JAFFE) database." Proceedings of third international conference on automatic face and gesture recognition. 1998.
[47]P. Lucey, J. Cohn, T. Kanade, J. Saragih, Z. Ambadar, and I. Matthews, “The extended cohn-kanade dataset (CK+): A complete dataset for action unit and emotion-specified expression,” in 2010 IEEE Computer Society Conference on Computer
[48]P.-L. Carrier, A. Courville, I. J. Goodfellow, M. Mirza, and Y. Bengio. FER-2013 Face Database. Technical report, 1365, Université de Montréal, 2013. http://www.kaggle.com/c/challenges-in-representation-learn ing-facialexpression-recognition-challenge

D. Compared Algorithms
[49]Guo, Guodong, and Charles R. Dyer. "Simultaneous feature selection and classifier training via linear programming: A case study for face expression recognition." Computer Vision and Pattern Recognition, 2003. Proceedings. 2003 IEEE Computer Society Conference on. Vol. 1. IEEE, 2003.
[50]Happy, S. L., and Aurobinda Routray. "Automatic facial expression recognition using features of salient facial patches." IEEE transactions on Affective Computing 6.1 (2015): 1-12.
[51]Khan, Sajid Ali, Ayyaz Hussain, and Muhammad Usman. "Reliable facial expression recognition for multi-scale images using weber local binary image based cosine transform features." Multimedia Tools and Applications 77.1 (2018): 1133-1165.
[52]Kamarol, Siti Khairuni Amalina, et al. "Spatiotemporal feature extraction for facial expression recognition." IET Image Processing 10.7 (2016): 534-541.
[53]H. Jung, S. Lee, J. Yim, S. Park, and J. Kim, “Joint fine-tuning in deep
neural networks for facial expression recognitin,” in Proc. IEEE Int.Conf. Comput. Vis. (ICCV), Dec. 2015, pp. 2983–2991.
[54]Liu, Mengyi, et al. "Deeply learning deformable facial action parts model for dynamic expression analysis." Asian conference on computer vision. Springer, Cham, 2014.
[55]Uçar, Ayşegül, Yakup Demir, and Cüneyt Güzeliş. "A new facial expression recognition based on curvelet transform and online sequential extreme learning machine initialized with spherical clustering." Neural Computing and Applications 27.1 (2016): 131-142.
[56]Sun, Wenyun, Haitao Zhao, and Zhong Jin. "An efficient unconstrained facial expression recognition algorithm based on Stack Binarized Auto-encoders and Binarized Neural Networks." Neurocomput
ing 267 (2017): 385-395.v
[57]T. Devries, K. Biswaranjan, and G. W. Taylor. Multi-task learning of facial landmarks and expression. In 2014 Canadian Conference on Computer and Robot Vision, 2014.
[58]Y. Tang. Deep Learning with Linear Support Vector Machines, In ICML Workshops, 2013
[59]R. T. Ionescu, M. Popescu, and C. Grozea. Local learning to improve bag of visual words model for facial expression recognition. In ICML Workshops, 2013
[60] Lopes, A. T., de Aguiar, E., & Oliveira-Santos, T. (2015, August). A facial expression recognition system using convolutional networks. In Graphics, Patterns and Images (SIBGRAPI), 2015 28th SIBGRAPI Conference on (pp. 273-280). IEEE.
[61]Anders, S., Lotze, M., Erb, M., Grodd, W., & Birbaumer, N. (2004). Brain activity underlying emotional valence and arousal: A response‐related fMRI study. Human brain mapping, 23(4), 200-209.
[62]Alshamsi, H., Kepuska, V., & Meng, H. (2017, October). Automated facial expression recognition app development on smart phones using cloud computing. In Ubiquitous Computing, Electronics and Mobile Communication Conference (UEMCON), 2017 IEEE 8th Annual (pp. 577-583). IEEE
[63]Ryu, B., Rivera, A. R., Kim, J., & Chae, O. (2017). Local directional ternary pattern for facial expression recognition. IEEE Transactions on Image Processing, 26(12), 6006-6018
[64]Turan, C., Lam, K. M., & He, X. (2018). Soft Locality Preserving Map (SLPM) for Facial Expression Recognition. arXiv preprint arXiv:1801.03754.
[65]Guo, Y., Tao, D., Yu, J., Xiong, H., Li, Y., & Tao, D. (2016, July). Deep Neural Networks with Relativity Learning for facial expression recognition. In Multimedia & Expo Workshops (ICMEW), 2016 IEEE International Conference on (pp. 1-6). IEEE.
[66]Chang, T., Wen, G., Hu, Y., & Ma, J. (2018). Facial Expression Recognition Based on Complexity Perception Classification Algorithm. arXiv preprint arXiv:1803.00185.
[67]Kuang Liu, M. Zhang, and Z. Pan. FacialExpression Recognition with CNN Ensemble International Conference on Cyberworlds IEEE, pages 163-166, 2016.
[68]Wen, G., Hou, Z., Li, H., Li, D., Jiang, L., & Xun, E. (2017). Ensemble of deep neural networks with probability-based fusion for facial expression recognition. Cognitive Computation, 9(5), 597-610.
E. Others
[69]http://www.psych.utoronto.ca/users/reingold/courses/ai/cache/neural2.html
[70]https://opensourceforu.com/2017/03/neural-networks-in-detail/
[71]https://ujjwalkarn.me/2016/08/09/quick-intro-neural-networks/
[72]https://stackoverflow.com/questions/2480650/role-of-bias-in-neural-networks
[73]http://speech.ee.ntu.edu.tw/~tlkagk/courses_ML16.html
[74]http://blog.csdn.NET/zouxy09/article/details/8775360/
[75]https://www.analyticsvidhya.com/blog/2016/04/deep-learning-computer-vision-introduction-convolution-neural-networks/
[76]https://leonardoaraujosantos.gitbooks.io/artificial-inteligence/content/pooling_layer.html
[77]https://elitedatascience.com/overfitting-in-machine-learning
[78]https://medium.com/@siddharthdas_32104/cnns-architectures-lenet-alexnet-vgg-googlenet-resnet-and-more-666091488df5
[79]https://en.wikipedia.org/wiki/Facial_Action_Coding_System
[80]http://silverwind1982.pixnet.net/blog/post/134551091-pinhole-camera%3A-%E9%87%9D%E5%AD%94%E7%9B%B8%E6%A9%9F%E5%BA%A7%E6%A8%99%E6%88%90%E5%83%8F%E5%8E%9F%E7%90%86
[81]https://blog.csdn.net/TonyShengTan/article/details/43448787