在這篇論文中我們提出了一個使用卷積神經網路並基於視覺的手語辨識系統，該系統利用膚色模型對影像進行完整的手部區域分割，在膚色偵測之後，我們套入形態學運算來優化偵測結果，並採用區域成長演算法對每個區域標上編號及統計，最後切出屬於手部的區域。此方法在一定程度上不會受到不同背景環境的影響。接著我們透過一個經過微調的卷積神經網路模型來對手部影像萃取特徵，並利用支持向量機來將手勢進行分類。實驗部分，我們透過自己拍攝的中文手語資料集以及公開的美國手語資料集來對系統辨識的效果進行評估，中文手語資料集是由五個人在三種不同環境下 (包含複雜背景) 比出二十六種手勢，並將其錄製而成；而美國手語資料集包含了同一種手勢在不同燈光條件下拍攝而成。實驗結果得到證實該方法比現今其他傳統方法的辨識能力更好。

In this paper, we propose a vision-based sign language recognition system using convolutional neural networks (CNN). We design an skin color modeling method for hand segmentation so that the hand region can be derived accurately even when different users use our system in various background scenes. A feature descriptor is extracted from the fully-connected layer of the fine-tuned CNN model as the representation of a hand gesture, and Support Vector Machine (SVM) is applied to classify hand gestures.
Our recognition method is evaluated by two datasets: (1) The Chinese Sign Language (CSL) dataset collected by ourselves, in which images of 26 gestures are captured from 5 people in three kinds of environments including complex background scenes. (2) The American Sign Language (ASL) dataset, in which images of the same gesture were captured in different lighting conditions. The proposed recognition method achieves high accuracy rates, and is proved to be greater than the results of traditional methods.

Abstract (Chinese) i
Abstract (English) ii
Acknowledgments iii
Table of Contents v
List of Tables vii
List of Figures viii
Chapter 1. Introduction 1
Chapter 2. Related Work 4
Chapter 3. Convolutional Neural Networks 7
3.1 Introduction of NNs and CNNs . . . . . . . . . . . . . . . . . . . . . . 7
3.2 CNN Architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
3.3 Training of CNNs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9
Chapter 4. System Framework 11
4.1 Key Frame Selection . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
4.2 Hand Segmentation . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12
4.2.1 Skin Color Detection . . . . . . . . . . . . . . . . . . . . . . . . 12
4.2.2 Region Growing Algorithm . . . . . . . . . . . . . . . . . . . . 13
4.3 Feature Extraction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14
4.3.1 Traditional Image Features . . . . . . . . . . . . . . . . . . . . . 14
4.3.2 CNN Image Features . . . . . . . . . . . . . . . . . . . . . . . . 18
4.4 SVM Classification . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19
4.4.1 Binary Classification . . . . . . . . . . . . . . . . . . . . . . . . 19
4.4.2 Multiclass Classification . . . . . . . . . . . . . . . . . . . . . . 21
Chapter 5. Experimental Results 22
5.1 Datasets and Experimental Environments . . . . . . . . . . . . . . . . . 22
5.2 Evaluation of CNN Features from Layers . . . . . . . . . . . . . . . . . 24
5.3 Comparison of Re-training and Fine-tuning CNN Models . . . . . . . . 24
5.4 Comparison of Background Subtraction . . . . . . . . . . . . . . . . . . 25
5.5 Comparison of Traditional and CNN Features . . . . . . . . . . . . . . 27
5.6 Comparison of DSAE and CNN Features . . . . . . . . . . . . . . . . . 29
Chapter 6. Conclusions ＆ Future Work 30
References 32

[1] R. Akmeliawati, M. P. L. Ooi, and Y. C. Kuang. Real-time malaysian sign language translation using colour segmentation and neural network. In 2007 IEEE Instrumentation ＆ Measurement Technology Conference IMTC 2007, pages 1–6, 2007.
[2] A. L. C. Barczak, N. H. Reyes, M. Abastillas, A. Piccio, and T. Susnjak. A new 2d static hand gesture colour image dataset for asl gestures. Research Letters in the Information and Mathematical Sciences, 15:12–20, 2011.
[3] J. Bouvrie. Notes on convolutional neural networks. November 2006.
[4] H. Brashear, T. Starner, P. Lukowicz, and H. Junker. Using multiple sensors for mobile sign language recognition. In Wearable Computers, 2003. Proceedings. Seventh IEEE International Symposium on, pages 45–52, 2003.
[5] C.-C. Chang and C.-J. Lin. LIBSVM: A library for support vector machines. ACM Transactions on Intelligent Systems and Technology, 2:27:1–27:27, 2011. Software available at http://www.csie.ntu.edu.tw/~cjlin/libsvm.
[6] K. Chatfield, K. Simonyan, A. Vedaldi, and A. Zisserman. Return of the devil in the details: Delving deep into convolutional nets. CoRR, abs/1405.3531, 2014.
[7] N. Dalal and B. Triggs. Histograms of oriented gradients for human detection. In 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR’05), volume 1, pages 886–893 vol. 1, 2005.
[8] N. H. Dardas and N. D. Georganas. Real-time hand gesture detection and recognition using bag-of-features and support vector machine techniques. IEEE Transactions on Instrumentation and Measurement, 60(11):3592–3607, 2011.
[9] M. Fernando and J. Wijayanayaka. Low cost approach for real time sign language recognition. In 2013 IEEE 8th International Conference on Industrial and Information Systems, pages 637–642, 2013.
[10] H. M. Gamal, H. M. Abdul-Kader, and E. A. Sallam. Hand gesture recognition using fourier descriptors. In Computer Engineering Systems (ICCES), 2013 8th International Conference on, pages 274–279, Nov 2013.
[11] M. Geetha, C. Manjusha, P. Unnikrishnan, and R. Harikrishnan. A vision based dynamic gesture recognition of indian sign language on kinect based depth images. In Emerging Trends in Communication, Control, Signal Processing ＆ Computing Applications (C2SPCA), 2013 International Conference on, pages 1–7, 2013.
[12] Itseez. Open source computer vision library. https://github.com/itseez/opencv, 2015.
[13] Y. Jia, E. Shelhamer, J. Donahue, S. Karayev, J. Long, R. Girshick, S. Guadarrama, and T. Darrell. Caffe: Convolutional architecture for fast feature embedding. arXiv preprint arXiv:1408.5093, 2014.
[14] F. Kai-ping and Y. Fang. Static hand gesture recognition based on hog characters and support vector machines. In Instrumentation and Measurement, Sensor Network and Automation (IMSNA), 2013 2nd International Symposium on, pages 936–938, 2013.
[15] M. Kolsch and M. Turk. Robust hand detection. In Automatic Face and Gesture Recognition, 2004. Proceedings. Sixth IEEE International Conference on, pages 614–619, 2004.
[16] A. Krizhevsky, I. Sutskever, and G. E. Hinton. Imagenet classification with deep convolutional neural networks. In NIPS, pages 1106–1114, 2012.
[17] V. Kumar, G. C. Nandi, and R. Kala. Static hand gesture recognition using stacked denoising sparse autoencoders. In Contemporary Computing (IC3), 2014 Seventh International Conference on, pages 99–104, 2014.
[18] Y. Lecun, L. Bottou, Y. Bengio, and P. Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11):2278–2324, Nov 1998.
[19] Y. Liu, Y. Yin, and S. Zhang. Hand gesture recognition based on hu moments in interaction of virtual reality. In Intelligent Human-Machine Systems and Cybernetics (IHMSC), 2012 4th International Conference on, volume 1, pages 145–148, Aug 2012.
[20] Y. Madhuri, G. Anitha, and M. Anburajan. Vision-based sign language translation device. In Information Communication and Embedded Systems (ICICES), 2013 International Conference on, pages 565–568, 2013.
[21] H. Ming-Kuei. Visual pattern recognition by moment invariants. IRE Transactions on Information Theory, 8(2):179–187, 1962.
[22] J. S. N. A. Abdul Rahim, C. W. Kit. Rgb-h-cbcr skin colour model for human face detection. In MMU International Symposium on Information ＆ Communications Technologies (M2USIC), Petaling Jaya, Malaysia, 2006.
[23] J. Nagi, F. Ducatelle, G. A. D. Caro, D. Cireşan, U. Meier, A. Giusti, F. Nagi, J. Schmidhuber, and L. M. Gambardella. Max-pooling convolutional neural networks for vision-based hand gesture recognition. In Signal and Image Processing Applications (ICSIPA), 2011 IEEE International Conference on, pages 342–347, Nov 2011.
[24] L. Prasuhn, Y. Oyamada, Y. Mochizuki, and H. Ishikawa. A hog-based hand gesture recognition system on a mobile device. In Image Processing (ICIP), 2014 IEEE International Conference on, pages 3973–3977, 2014.
[25] D. Rumelhart, G. Hintont, and R. Williams. Learning representations by backpropagating errors. Nature, 323(6088):533–536, 1986.
[26] C. Sun, T. Zhang, B. K. Bao, C. Xu, and T. Mei. Discriminative exemplar coding for sign language recognition with kinect. IEEE Transactions on Cybernetics, 43(5):1418–1428, 2013.
[27] M. Tariq, A. Iqbal, A. Zahid, Z. Iqbal, and J. Akhtar. Sign language localization: Learning to eliminate language dialects. In Multitopic Conference (INMIC), 2012 15th International, pages 17–22, 2012.
[28] R. Wang, D. Kong, L. Wang, and B. Yin. Gestures’ smooth motion for chinese sign language synthesis based on video joining. In Computational Intelligence and Software Engineering, 2009. CiSE 2009. International Conference
on, pages 1–5, 2009.
[29] T. Yamashita and T. Watasue. Hand posture recognition based on bottom-up structured deep convolutional neural network with curriculum learning. In 2014 IEEE International Conference on Image Processing (ICIP), pages 853–857, Oct 2014.
[30] Q. Yang. Chinese sign language recognition based on video sequence appearance modeling. In Industrial Electronics and Applications (ICIEA), 2010 the 5th IEEE Conference on, pages 1537–1542, 2010.
[31] C. Yixin, B. Jinbo, and J. Z. Wang. Miles: Multiple-instance learning via embedded instance selection. IEEE Transactions on Pattern Analysis and Machine Intelligence, 28(12):1931–1947, 2006.