近年來人機互動(Human-computer interaction)的研究相當受到重視，而最直覺的互動方式就是直接利用攝影機捕捉人的動作影像，再利用電腦視覺的處理來達成互動的效果。但是傳統攝影機的影像品質受到光線的影響較大，在許多光線條件不佳的環境底下容易對使用者的使用上產生限制，為解決此一問題，本論文則是使用微軟所開發的體感攝影機Kinect來當作影像的輸入裝置。
Kinect攝影機與傳統攝影機相比多了深度的資訊，而深度的資訊並不會受到光線的干擾，所以本論文就單純的使用Kinect的深度影像，並在Kinect深度影像中進行指尖的偵測，進而在電腦上彈奏虛擬鋼琴。本系統於指尖彈壓動作判斷之準確率為87%，指尖鬆放動作判斷之準確率為85%，而在鋼琴彈壓之準確率為85%，鋼琴鬆放之準確率為81%。但因為在彈放的判斷上是取決於指尖在各個琴鍵對應區塊內的移動變化，所以會造成使用者在滑琴鍵上的限制。

In recent years, Human-Computer interaction research has been quite popular. The most intuitive way to interact with computers would be using the camera to capture the user’s images, which are then processed with the computer vision techniques to achieve interactive effects. But the quality of the captured images from traditional cameras can be easily affected by lighting conditions. The system accuracy is limited at the environment where light condition is poor. In order to solve the problem, the input device in this study is Microsoft Kinect.
The major difference between a traditional camera and Kinect is the depth information, which cannot be affected by the lighting condition. So only the Kinect depth images are used in this study. The fingertips are then detected in the depth images, and then playing a virtual piano on a computer can be achieved. The accuracies of fingertips pressing and put-off are 87% and 85%, respectively, while the accuracies of piano pressing and put-off are 85% and 81%, respectively. Since the pressing and put-off are determined with the moving changes of fingertips in the keyboard blocks, the way of piano playing represented by keyboard sliping is prohibited in this system.

摘要 i
ABSTRACT ii
誌謝 iii
目錄 iv
圖目錄 vi
表目錄 viii
第一章、 緒論 1
1.1 研究動機 1
1.2 研究目的 1
1.3 研究方法 2
1.3.1 Kinect原理介紹 2
1.3.2 系統流程概述 4
1.4 相關研究 6
第二章、 系統實作 7
2.1 Kinect影像擷取 - OpenNI 7
2.2 手部區域分離 10
2.2.1 結合彩色影像與深度影像分離 10
2.2.2 單一深度影像手部分離 13
2.3 指尖偵測 14
2.3.1 距離轉換 16
2.3.2 手指特徵點 19
2.3.3 掌心與手指中心偵測 20
2.3.4 指尖偵測 22
2.4 虛擬鋼琴彈奏 24
2.4.1 虛擬鋼琴-Virtual MIDI Piano Keyboard 24
2.4.2 琴鍵對應 24
2.4.2.1 鋼琴琴鍵與電腦鍵盤對應 25
2.4.2.2 電腦鍵盤與劃分區塊對應 27
2.4.3 彈奏判斷 28
第三章、 實驗結果分析 29
3.1 系統架構 29
3.2 實驗結果 29
3.3 實驗結果分析 34
第四章、 結論與未來工作 35
4.1 結論 35
4.2 未來工作 35
參考文獻 36

[1]A. Bellarbi, S. Benbelkacem, N. Zenati-Henda, and M.Belhocine, "Hand gesture interaction using color-based method for tabletop interfaces," IEEE 7th International Symposium on Intelligent Signal Processing (WISP), pp.1-6, 2011.
[2] N.W. Kim, D.H. Shin, D.J. Kim, B.G. Lee, and E.S. Kim, "Vision-based 3D fingertip interface for spatial interaction in 3D integral imaging system," Complex International Conference on Intelligent and Software Intensive Systems CISIS ''09, pp.1006-1011, 2009.
[3] K.T. Tseng, W.F. Huang, and C.H. Wu, "Vision-based finger guessing game in human machine interaction," IEEE International Conference on Robotics and Biomimetics ROBIO ''06, pp.619-624, 2006.
[4] M. Van den Bergh, and L. Van Gool, "Combining RGB and ToF cameras for real-time 3D hand gesture interaction," IEEE Workshop on Applications of Computer Vision (WACV), pp.66-72, 2011.
[5] M. Panwar, "Hand gesture recognition based on shape parameters," International Conference on Computing, Communication and Applications (ICCCA), pp.1-6, 2012.
[6] Y. Liu, and P. Zhang, "Vision-based human-computer system using hand gestures," International Conference on Computational Intelligence and Security CIS ''09, pp.529-532, 2009.
[7] D. Lee, and Y. Park, "Vision-based remote control system by motion detection and open finger counting," IEEE Transactions on Consumer Electronics, pp.2308-2313, 2009.
[8] T.H. Tsai, and C.Y. Lin, "Visual hand gesture segmentation using signer model for real-time human-computer interaction application," IEEE Workshop on Signal Processing Systems, pp.567-572, 2007.
[9] X. Wen, and Y. Niu, "A method for hand gesture recognition based on morphology and fingertip-angle," The 2nd International Conference on Computer and Automation Engineering (ICCAE), pp.688-691, 2010.
[10] S.K. Kang, M.Y. Nam, and P.K. Rhee,"Color based hand and finger detection technology for user interaction," ICHIT''08. International Conference on Convergence and Hybrid Information Technology, pp.229-236, 2008.
[11] A.S. Manaf, and R.F. Sari, "Color recognition system with augmented reality concept and finger interaction: Case study for color blind aid system," 9th International Conference on ICT and Knowledge Engineering (ICT &; Knowledge Engineering), pp.118-123, 2011.
 
[12] L. Dung, and M. Mizukawa, "Fast fingertips positioning based on distance-based feature pixels," Third International Conference on Communications and Electronics (ICCE), pp.184-189, 2010.
[13] D.D. Nguyen, T.C. Pham, and J.W. Jeon, "Fingertip detection with morphology and geometric calculation," IROS IEEE/RSJ International Conference on Intelligent Robots and Systems, pp.1460-1465, 2009.
[14] S. Kim, Y. Park, K. Lim, H. Lee, S. Kim, and S. Lee, "Fingertips detection and tracking based on active shape models and an ellipse," IEEE Region 10 Conference TENCON 2009 – 2009, pp.1-6, 2009.
[15] S.A.H. Shah, A. Ahmed, I. Mahmood, and K. Khurshid, "Hand gesture based user interface for computer using a camera and projector," IEEE International Conference on Signal and Image Processing Applications (ICSIPA), pp.168-173, 2011.
[16] K. Oka, Y. Sato, and H. Koike, "Real-time fingertip tracking and
gesture recognition, " IEEE Computer Graphics and Applications, pp.64-71, 2002.
[17] Z. Ren, J. Meng, and J. Yuan, "Depth camera based hand gesture recognition and its applications in human-computer-interaction," 8th International Conference on Information, Communications and Signal Processing (ICICS), pp.1-5, 2011.
[18] N.E. Yang ,Y.G. Kim , and R.H. Park, "Depth hole filling using the depth distribution of neighboring regions of depth holes in the Kinect sensor," IEEE International Conference on Signal Processing, Communication and Computing (ICSPCC), pp.658-661, 2012.
[19] Y. Gao, J. Zhang, L. Zhang, Y. Hu, and J. Zhang, "Finding objects at indoor environment combined with depth information," International Conference on Mechatronics and Automation (ICMA), pp.687-692, 2011.
[20] H. Tao, and Y. Yu, "Finger tracking and gesture interaction with Kinect, " IEEE 12th Computer and Information Technology (CIT), pp.214-218, 2012.
[21] Y. Li, "Hand gesture recognition using Kinect," IEEE 3rd International Conference on Software Engineering and Service Science (ICSESS), pp.196-199, 2012.
[22] L. Xia, C.C. Chen, and J.K. Aggarwal , "Human detection using depth information by Kinect," IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pp.15-22, 2011.
[23] D. Xu, Y.L. Chen, X. Wu, Y. Ou, and Y. Xu, "Integrated approach of skin-color detection and depth information for hand and face localization," IEEE International Conference on Robotics and Biomimetics (ROBIO), pp.952-956, 2011. 
[24] J.L. Raheja, A. Chaudhary, and K. Singal, "Tracking of fingertips and centres of palm using Kinect," Third International Conference on Computational Intelligence, Modelling and Simulation (CIMSiM), pp248-252, 2011.
[25] J. Kovac, P. Peer, and F. Solina, "Human skin colour clustering for face detection," Conference on Computer as a Tool, pp.144-148, 2003.
[26] Sourceforge, Virtual MIDI Piano Keyboard
http://sourceforge.net/projects/vmpk/