在教學環境中教師取得學習者的學習狀況的方式除了學習成效之外捕捉學習者的學習情緒也是種有效的方式，因為臉部表情是人類表達情緒的主要方式之一且在人們的情緒交流上發揮了重要的作用，藉由補捉學習情緒給教師可以幫助教師了解學習者的學習狀況。

過去相關研究對於辨識的學習情緒是將基本情緒分為正向情緒與負向情緒或者是理解與不理解之情緒，來詮釋學習者於學習中產生的情緒，然而本研究為了得到更完整的學習情緒，使用學業情緒之分類標籤解釋學習情緒，學習情緒除了能得到學習者的正負向情緒外同時也能得知學習者對於教學內容的反應，因此本研究的重點在於如何辨識學習者的學習情緒。

由於有效的臉部表情特徵設計不易，因此本研究使用深度學習領域中的卷積神經網路(Convolutional Neural Networks, CNN)而非傳統特徵工程方式建立預測模型，CNN是種特殊的網路結構，能夠直接輸入原始影像進行訓練，不需要事先手工設計特徵，因為CNN本身就具有對輸入影像自動學習影像特徵的能力，諸多實驗證明CNN在影像辨識上效果出色。

本研究先以FER2013基本表情資料庫建立基本情緒辨識模型來驗證本研究設計的模型架構對於表情的辨識能力，在FER2013提供之測試資料集下的辨識準確率達72.01%，顯示本研究之模型對於表情辨識非常有效。

在學習情緒模型的建立上，使用四十三份學生學習時所錄製之學習影片作為訓練資料，分別以三秒鐘、一秒鐘與半秒的擷取頻率下擷取影片，以90-10的訓練與測試方式建立學習情緒預測模型，實驗結果在三秒鐘、一秒鐘、半秒擷取頻率下預測準確率可達66%、81%、91%，實驗結果說明本研究的模型對於直接辨識學習情緒是有效的，並藉由顯著圖技術視覺化出學習情緒之表情特徵，接著對本實驗室自行建立的表情資料庫進行辨識，辨識準確率可達五成以上，證明本研究的學習情緒預測模型在不同的個體差異下仍具有一定的預測能力。

摘要 i
ABSTRACT ii
目次 iv
表目次 vi
圖目次 vii
第一章 緒論 1
1.1 研究背景 1
1.2 研究動機與目的 2
第二章 文獻探討 4
2.1 學業情緒 4
2.2 情意計算 4
2.3 情意計算方法 5
2.4 深度學習(卷積神經網路) 8
2.5 深度學習在情意計算相關研究 10
2.6 基本情緒資料庫介紹 12
第三章 研究方法 15
3.1 建立表情辨識模型之方法 15
3.1.1 資料預處理 16
3.1.2 模型架構與訓練 18
3.2 基本情緒辨識模型 26
3.2.1 基本情緒辨識結果與分析 26
3.2.2 基本情緒特徵視覺化 29
3.3 學習情緒辨識模型 31
3.3.1 學習情緒資料說明 32
3.3.2 學習情緒辨識模型之建立 32
3.3.3 學習情緒辨識模型之驗證 32
第四章 實驗結果 34
4.1 實驗環境 34
4.2 學習情緒模型實驗結果 34
4.2.1 每三秒擷取一張影像 34
4.2.2 每一秒擷取一張影像 36
4.2.3 每半秒擷取一張影像 37
4.3 學習情緒模型辨識結果與人工標記一致性之比較 41
第五章 結論與建議 43
參考文獻 44

[1] 張盈(方方土)、宋秋美、周啟葶、李雅婷、李懿芳、江芳盛、蔡佳燕、林明地、陳威良、葉連祺、楊家榆, 教育學刊第35期: Educational Review Vol.35. 高等教育出版, 2010.
[2] D. Charles, The Expression of the Emotions in Man and Animals, 1872.
[3] P. Ekman, "Facial Expressions of Emotion: An Old Controversy and New Findings," Philosophical Transactions of the Royal Society (London), p. 63–69, 1992.
[4] R. Picard, "Affective Computing," MIT, Media Laboratory, 1995.
[5] 林敏勤, "臉部微表情辨識系統之開發與應用," 碩士論文，國立中興大學資訊管理學系, 2016.
[6] Corinna Cortes, Vladimir Vapnik, "Support-Vector Networks," Machine Learning, pp. 273-297 , 15 5 1995.
[7] Altman, "An Introduction to Kernel and Nearest-Neighbor Nonparametric Regression," The American Statistician, pp. 175-185, 1992.
[8] Yoav Freund , Robert Schapire, "A Decision-Theoretic Generalization of on-Line Learning and an Application to Boosting," Computer and System Sciences, pp. 119-139, 8 1997.
[9] Alex Krizhevsky, Ilya Sutskever, Geoffrey E. Hinton, "ImageNet classification with deep convolutional neural networks," Proceedings of the 25th International Conference on Neural Information Processing Systems, pp. 1097-1105, 2012.
[10] Karen Simonyan, Andrew Zisserman, "Very Deep Convolutional Networks for Large-Scale Image Recognition," arXiv technical report, 2014.
[11] Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun, "Deep Residual Learning for Image Recognition," Computer Vision and Pattern Recognition , 2016.
[12] Ira Cohen, Nicu Sebe, Ashutosh Garg, Lawrence S. Chen, Thomas S. Huang, "Facial expression recognition from video sequences: temporal and static modeling," Computer Vision and Image Understanding, vol. 91, pp. 160-187, 2003.
[13] Tong Zhang, Mark Hasegawa-Johnson, and Stephen Levinson, "Children''s emotion recognition in an intelligent tutoring scenario," in Proceedings of 8th European Conference on Spoken Language Processing, Korea, 2004.
[14] Jeffrey F. Cohn, Lawrence Ian Reed, Zara Ambadar, Jing Xiao, Tsuyoshi Moriyama, "Automatic analysis and recognition of brow actions and head motion in spontaneous facial behavior," in IEEE International Conference on Systems, Man and Cybernetics, vol. 1, 2004.
[15] Liping Shen, Minjuan Wang, Ruimin Shen, "Affective e-learning: using ‘emotional’ data to improve learning in pervasive learning environment," Educational Technology & Society, vol. 12, no. 2, pp. 176-189, 2009.
[16] Paul Salvador Inventado, Roberto Legaspi, The Duy Bui, Merlin Suare, "Predicting student’s appraisal of feedback in an ITS using previous affective states and continuous affect labels from EEG data".
[17] 潘奕安, “低解析度影像序列之自動化表情辨識系統,” 碩士論文，國立成功大學資訊工程學系, 2004.
[18] B. T. Lau, "Portable real time emotion detection system for the disabled," Expert Systems with Applications, vol. 37, no. 9, pp. 6561-6566, 2010.
[19] Chengjun Liu, Harry Wechsler, "Gabor feature based classification using the enhanced fisher linear discriminant model for face recognition," Image processing, IEEE Transactions, pp. 467-476, 2002.
[20] Caifeng Shan, Shaogang Gong, Peter McOwan, "Facial expression recognition based on local binary patterns patterns: A comprehensive," Image and Vision Computing, pp. 803-816, 2009.
[21] Mohammad Mavadati, Mohammad Mahoor, Kevin Bartlett, Philip Trinh, Jeffrey Cohn, "A spontaneous facial action intensity database," Affective Computing, IEEE Transactions, vol. 4, no. 2, p. 151–160, 2013.
[22] David Hubel, Torsten Wiesel, "Receptive fields of single neurones in the cat''s striate cortex," Journal of Physiology, pp. 574-591, 1959.
[23] K. Fukushima, "Neocognitron: A self-organizing neural network model for a mechanism of pattern recognition unaffected by shift in position," Biological cybernetics,, 1980.
[24] Pooya Khorrami, Tom Le Paine, Thomas S. Huang, "Do Deep Neural Networks Learn Facial Action Units When Doing Expression Recognition?," Proceedings of the 2015 IEEE International Conference on Computer Vision Workshop, pp. 19-27, 7 12 2015.
[25] Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, Andrew Rabinovich, "Going deeper with convolutions," arXiv, 2015.
[26] Ali Mollahosseini, David Chan, Mohammad Mahoor, "Going deeper in facial expression recognition using deep neural networks," IEEE Winter Conference on Applications of Computer Vision, 26 5 2016.
[27] Ian J Goodfellow, Dumitru Erhan, Pierre Luc Carrier, Aaron Courville,Mehdi Mirza, "Challenges in representation learning: Facial expression recognition challenge," 2013. [Online]. Available: https://www.kaggle.com/c/challenges-in-representation-learning-facial-expression-recognition-challenge.
[28] Lundqvist, D., Flykt, A., Öhman, A, "The Karolinska Directed Emotional Faces - KDEF," 1998. [Online]. Available: http://www.emotionlab.se/resources/kdef.
[29] L. M. J., "The Japanese Female Facial Expression (JAFFE) Database [DB]," 1998. [Online]. Available: http://www.mis.atr.co.jp/~mlyons/jaffe.html.
[30] Karen Simonyan, Andrea Vedaldi, Andrew Zisserman, "Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps," CoRR, vol. 1312.6034, 2013.
[31] Dmytro Mishkin, Nikolay Sergievskiy, Jiri Matas, "Systematic evaluation of CNN advances on the ImageNet," Neural and Evolutionary Computing, 1 2016.
[32] Bal E, Harden E, Lamb D, Van Hecke AV, Denver JW, Porges SW., "Emotion recognition in children with autism spectrum disorders: relations to eye gaze and autonomic state.," Journal of Autism and Developmental Disorders, vol. 40, no. 3, pp. 358-370, 2010.
[33] Karen Simonyan, Andrew Zisserman, "Very Deep Convolutional Networks for Large-Scale Image Recognition," arXiv:1409.1556, 4 9 2014.
[34] Xibin Yue, Xiaolin Hu, Liang Tang, "The Influence of the Amount of Parameters in Different Layers on the Performance of Deep Learning Models," Computer Science and Application, pp. 445-453, 10 12 2015.
[35] Xavier Glorot, Yoshua Bengio, "Understanding the difficulty of training deep feedforward neural networks," Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics, PMLR 9, pp. 249-256, 2010.
[36] Yanan Guo, Dapeng Tao, Jun Yu, Hao Xiong, Yaotang Li, Dacheng Tao, "Deep Neural Networks with Relativity Learning for facial expression recognition," IEEE International Conference on Multimedia and Expo Workshops , 11 7 2016.
[37] Ian J. Goodfellow, Dumitru Erhan, Pierre Luc Carrier, Aaron Courville, Mehdi Mirza, Ben Hamner, Will Cukierski, Yichuan Tang, David Thaler, Dong-Hyun Lee, Yingbo Zhou, Chetan Ramaiah, Fangxiang Feng, Ruifan Li, Xiaojie Wang, Dimitris Athanasakis, John Sha, "Challenges in Representation Learning: A report on three machine learning contests," International Conference on Neural Information Processing, pp. 117-124, 2013.
[38] Y. Tang, "Deep learning using linear support vector machines," arXiv preprint, arXiv:1306.0239, 2013.
[39] Hong-Wei Ng, Viet Dung Nguyen, Vassilios Vonikakis, Stefan Winkler, "Deep Learning for Emotion Recognition on Small Datasets using Transfer Learning," Multimodal Interfaces and Machine Learning for Multimodal Interaction, pp. 443-449, 2015.
[40] 蘇莉均, "利用臉部表情特徵辨識學習情緒之研究," 碩士論文，國立中興大學資訊管理學系, 2017.
[41] P. Ekman, Emotions Revealed: Understanding Faces and Feelings, Weidenfeld & Nicolson, 2003.