本論文發展一套應用於機器人互動之人臉辨識與表情辨識之整合設計。人臉影像先經由主動外觀模型(Active Appearance Model, AAM)計算出人臉形狀以及紋理模型；接著對輸入的人臉影像進行改良式Lucas-Kanade 影像校正以找出人臉特徵，再利用AAM的人臉紋理模型建立出人臉紋理的特徵參數，並利用此特徵參數輸入倒傳遞類神經網路做辨識。在辨識過程中，我們提出一種整合設計，先經由人臉辨識找出使用者的身分，接著我們對於辨識出的已知使用者的表情資料庫做個人化表情辨識；由實驗結果可以看出，在使用BU-3DFE人臉表情資料庫做辨識，其人臉辨識成功率可達98.3%。接著使用個人化表情辨識的成功率為83.8%，相對於使用全體的表情辨識器之辨識率僅為69.6%，其成功率可大幅提升。

In this thesis, an integrated design of face and facial expression recognition system has been developed for robotic applications. First, facial image from camera is exacted to compute facial shape and texture model using active appearance model (AAM). Second, we use modified Lucas-Kanade image alignment algorithm to find facial features. Third, the texture model of AAM is used to construct facial texture parameters. These parameters are used to train a back propagation neural network (BPNN) for face and facial expression recognition. In recognition process, we first use face recognition to find user’s identity; then we use recognized user’s facial expression database to recognize his/her facial expression. In experiments based on BU-3DFE database, a face recognition rate of 98.3% has been achieved. The facial expression recognition rate of the proposed integrated method (using a personal facial expression classifier) is 83.8%. It is a great improvement compared with using conventional facial expression classifier of 69.6%.

摘要 i
ABSTRACT ii
誌謝 iii
目錄 iv
圖例 vi
表格 viii
第一章、緒論 1
1.1. 研究動機 1
1.2. 相關文獻回顧 1
1.3. 問題描述 7
1.4. 系統架構與章節說明 8
第二章、人臉偵測與特徵點擷取方法 10
2.1. 人臉偵測演算法 10
2.1.1. 人臉區域決定 11
2.1.2. 人臉偵測成果 13
2.2. 主動外觀模型 15
2.3. 人臉形狀模型 15
2.3.1. 特徵點的設置 15
2.3.2. 形狀對齊 17
2.3.3. 模擬臉部形狀的變化 20
2.4. 人臉紋理模型 22
2.4.1. 人臉的紋理 22
2.4.2. Piecewise Affine Warping 22
2.4.3. 模擬臉部紋理的變化 25
2.5. 影像校正演算法 27
2.5.1. Lucas-Kanade 演算法 27
2.5.2. Inverse Compositional 演算法 30
2.5.3. 整體形狀正規化轉換 38
2.5.4. 梯度影像修正 42
2.5.5. 直方圖等化 45
2.5.6. 整體方法 49
2.6. 結論與討論 50
第三章、人臉辨識與表情辨識演算法 51
3.1. 人臉特徵方法 51
3.2. 倒傳遞類神經網路 52
3.3. 表情辨識策略 54
3.4. 人臉辨識與表情辨識訓練流程 56
3.5. 結論與討論 56
第四章、實驗結果 59
4.1. 影像校正演算法迭代結果 59
4.1.1. 使用資料庫影像 59
4.1.2. 使用自行擷取影像 64
4.2. 人臉辨識結果 66
4.2.1. 使用資料庫影像 66
4.2.2. 使用自行擷取影像 70
4.3. 表情辨識結果 72
4.3.1. 使用資料庫影像 72
4.3.2. 使用自行擷取影像 76
4.4. On-line辨識結果 80
4.5. 結論與討論 82
第五章、結論與未來展望 83
5.1. 結論 83
5.2. 未來展望 83
參考文獻 85

[1] Paul Ekman and Wallace V. Friesen, Unmasking the face, Prentice-Hill Inc. , Englewood Cliffs, New Jersey, 1975.
[2] 洪濬尉, 機器人之表情辨識快速學習法則, 碩士論文, 國立交通大學電機與控制工程學系, 2007.
[3] 鐘仁厚, 基於模糊邏輯之臉部表情辨識, 碩士論文, 國立中央大學電機工程學系, 2008.
[4] D. Cristinacce and T. F. Cootes, “Facial Feature Detection and Tracking with Automatic Template Selection,” in Proc. of 7th IEEE International Conference on Automatic Face and Gesture Recognition, 2006, Southampton, UK , pp. 429-434.
[5] D. Cristinacce and T.F.Cootes, “Feature Detection and Tracking with Constrained Local Models,” in Proc. of British Machine Vision Conference, 2006, Edinburgh, UK, pp.929-938.
[6] T. Hao and T. S. Huang, “3D Facial Expression Recognition Based on Automatically Selected Features,” in Proc. of IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops, 2008, Anchorage, Alaska, USA, pp. 1-8.
[7] Igor S. Pandzic and Robert Forchheimer (Eds), MPEG-4 Facial Animation: The Standard, Implementation and Applications, John Wiley & Sons, Inc., 2002.
[8] S. Kullback, “The Kullback-Leibler distance,” The American Statistician 41:340-341, 1987.
[9] C. Martin, U. Werner and H. M. Gross, “A Real-Time Facial Expression Recognition System Based on Active Appearance Models Using Gray Images and Edge images,” in Proc. of 8th IEEE International Conference on Automatic Face & Gesture Recognition, 2008, Amsterdam, Nederland, pp. 1-6.
[10] P. Viola and M. Jones, “Rapid Object Detection Using a Boosted Cascade of Simple Features,” in Proc. of IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2001, Kauai Marriott, Hawaii, pp. I-511-I-518.
[11] M. Mufti and A. Khanam, “Fuzzy Rule Based Facial Expression Recognition,” in Proc. of International Conference on Computational Intelligence for Modelling, Control and Automation, and International Conference on Intelligent Agents, Web Technologies and Internet Commerce, 2006, Vienna, Austria, pp.57-61.
[12] 許晉懷, 結合影像及語音之雙模情緒辨識系統, 碩士論文, 國立交通大學電機與控制工程學系, 2006.
[13] Rainer Lienhart and Jochen Maydt. “An Extended Set of Haar-like Features for Rapid Object Detection,” in Proc. of International Conference on Image Processing, 2002, Rochester, New York, USA , pp. 900-903
[14] M. Soriano, B. Martinkauppi, S. Huovinen and M. Laaksonen, “Skin Detection in Video Under Changing Illumination Conditions,” in Proc. of 15th International Conference on Pattern Recognition, 2000, Kauai Marriott, Hawaii, pp. 839-842.
[15] C. C. Chiang, W. K. Tai, M. T. Yang, Y. T. Huang and C. J. Huang, “A Novel Method for Detecting Lips, Eyes and Faces in Real Time,” Real-Time Imaging, vol. 9, no. 4, pp. 277-287, 2003.
[16] T. F. Cootes, G. J. Edwards and C. J. Taylor, “Active appearance models.” in Proc. of 5th European Conference on Computer Vision, 1998, Springer, Berlin, pp 484–498.
[17] T. F. Cootes, G. J. Edwards and C. J. Taylor, “Active Appearance Models,” In IEEE Transactions on Pattern Analysis And Machine Intelligence, Vol. 23, No. 6, 2001, pp. 681-685
[18] T. F. Cootes and C. J. Taylor, Technical Report: Statistical Models of Appearance for Computer Vision, The University of Manchester School of Medicine, 2004.
[19] T. F. Cootes, C. J. Taylor, D. H. Cooper and J. Graham, “Active Shape Models - Their Training and Application,” Computer Vision and Image Understanding, Vol. 61. no.1. pp.38-59, 1995.
[20] 王仕傑, 多姿態人臉辨識及其在機器人與人互動之應用, 碩士論文, 國立交通大學電機與控制工程學系, 2008.
[21] http://www2.imm.dtu.dk/~aam/
[22] I. Matthews and S. Baker, “Active Appearance Models Revisited,” International Journal of Computer Vision, vol. 60, no.2, 2004, pp.135-164.
[23] A.M. Martinez and R. Benavente, The AR Face Database, CVC Technical Report #24, June 1998.
[24] C. Goodall, “Procrustes Methods in the Statistical Analysis of Shape,” Journal of the Royal Statistical Society B, Vol 53, no.2, pp.285-339, 1991.
[25] J.R. Shewchuk, “Triangle: Engineering a 2D Quality Mesh Generator and Delaunay Triangulator,” In Applied Computational Geometry, FCRC'96 Workshop, pp. 203-222. Springer-Verlag, 1996.
[26] I. Matthews and S. Baker, “Lucas-Kanade 20 Years On: A Unifying Framework,” International Journal of Computer Vision, Vol. 56, No. 3, 2004, pp. 221 - 255.
[27] P. Jia, “Active Appearance Model,” Online technical report available from http://www.visionopen.com/members/jiapei/publications/pei_aamreport2010.pdf
[28] 羅華強, 類神經網路－MATLAB的應用, 高立圖書, 2005
[29] Lijun Yin, Xiaozhou Wei, Yi Sun, Jun Wang and Matthew J. Rosato, “A 3D Facial Expression Database For Facial Behavior Research,” in Proc. of 7th IEEE International Conference on Automatic Face and Gesture Recognition, 2006, Southampton, UK , pp. 211-216.
[30] X. Jing, et al., “Real-time combined 2D+3D active appearance models,” in Proc. of IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2004, Washington, DC, USA, pp. II-535-II-542 Vol.2.