(3.238.96.184) 您好!臺灣時間:2021/05/18 16:31
字體大小: 字級放大   字級縮小   預設字形  
回查詢結果

詳目顯示:::

: 
twitterline
研究生:周恒瑋
研究生(外文):Heng-Wei Zhou
論文名稱:基於深度學習之虛擬吉他音樂演奏系統設計
論文名稱(外文):The design of virtual guitar music performance system based on deep learning
指導教授:施國琛施國琛引用關係
指導教授(外文):Timothy K. Shih
學位類別:碩士
校院名稱:國立中央大學
系所名稱:資訊工程學系
學門:工程學門
學類:電資工程學類
論文出版年:2020
畢業學年度:108
語文別:英文
論文頁數:72
中文關鍵詞:人機互動介面深度學習虛擬樂器手部偵測
外文關鍵詞:Human–Computer InteractionDeep LearningVirtual InstrumentHand Detection
相關次數:
  • 被引用被引用:0
  • 點閱點閱:61
  • 評分評分:
  • 下載下載:6
  • 收藏至我的研究室書目清單書目收藏:1
人機互動之研究專注於研發使用者與電腦之溝通介面,而如何讓電腦藉由觀察與追蹤辨識演算法之設計來理解輔助裝置(諸如色彩/深度攝影機、智慧手環、傳感手套)之行為數據,為研究學者很重視的議題。近年來有越來越多體驗裝置之產品(如:虛擬頭盔、智慧眼鏡等)被廣泛應用於體感遊戲或是其他應用中,其中包含音樂演奏表演或是藝術表演,目的為讓玩家能夠在無實體硬體設備下(如:鋼琴、吉他等)以更友善且便利之方式演奏樂器,且讓觀眾能有更好的新科技視覺體驗。
先前人機互動之研究多以人體骨架之行為理解為主,適用於大動作之行為辨別,近年來各學者以機器學習與深度學習做為行為辨識主要之策略,其提升之準確率證明了機器學習與深度學習之可靠性。
為了能夠辨識細部的指尖指法,本論文提出一個以吉他為範例之吉他表演系統,該系統不僅能辨識及他左手之各和絃手勢,且能辨別右手之撥弦演奏行為。經由實驗證明,本系統不僅能被應用在吉他演奏表演系統,亦能應用在其他樂器演奏之表演系統(如:大提琴、小提琴、烏克麗麗等)。除此之外,本論文提出一個手勢辨識之評量機制,能夠用來評估模型與實時演奏之可靠性。
Human-computer interaction (HCI) focused on developing the interfaces between users and computers in computer. Many researchers observe the ways in which user interact with computers and design tracking or recognizing mechanism that let computer realized the input command from user’s behavior by auxiliary sensors (e.g. camera, wisdom bracelet, sensing gloves). More and more experiential device (e.g. virtual reality headset, smart glasses) are widely used for users with somatosensory games but they are also used in other applications, including music or artistic performances, the purpose is to allow users to more conveniently control commands on a small number of devices and give audiences a new visual experience.
In the past, the researchers mainly focused on the observation and analysis of human skeleton movements by using the design of special algorithms, the computer can understand the limb behavior based on the human skeleton. In our studies. In recent years, more and more machine learning and deep learning approach are used in HCI related researches to prove their reliability. In order to be able to recognize the detailed fingering behavior of the finger, this thesis proposed a guitar playing system, which use deep learning as strategy to recognize the finger gesture between all guitar in left-hand, and picking behavior in right-hand. Also, a verification method for discriminating accuracy was proposed in this thesis, which can be used to prove the reliability of the guitar performance system. Experimental results prove that our system based on deep learning approach can effectively identify the fingering behavior, and also the performance system can be used for other musical instruments (e.g. cello, violin or ukulele).
Contents
摘要 i
Abstract ii
Contents iii
List of Figures v
List of Tables viii
Chapter 1. Introduction 1
1.1 Background 1
1.2 Motivation 3
1.3 Thesis Organization 4
Chapter 2. Related work 5
2.1 Music Playing with HCI 5
2.2 Gesture Recognition using Deep Learning 6
2.3 Deep Learning with Gesture Recognition 11
2.4 Digital Music 14
Chapter 3. Proposed Framework 17
3.1 Guitar Structure and Action 19
3.2 Data Acquisition 20
3.3 Data Preprocess 24
3.4 Model Setup 26
3.4.1 Classification 26
3.4.2 Model Detail 27
3.5 Detection 31
Chapter 4. Experiment 34
4.1 Environment Setup 34
4.1.1 Camera 34
4.1.2 Hardware 38
4.1.3 Software 39
4.1.4 Data Recording Tool 42
4.2 Model Evaluation 45
4.2.1 Left-Hand Model Evaluation 45
4.2.2 Right-Hand Model Evaluation 49
4.3 Guitar Performance System 54
Chapter 5. Conclusion and Future Works 56
References 57
References
[1] G. Odowichuk, S. Trail, W. Page, W. Nie and P. Driessen, "Sensor fusion: Towards a fully expressive 3D music control interface," in Proceedings of 2011 IEEE Pacific Rim Conference on Communications, Computers and Signal Processing, Victoria, BC, Canada, 2011.
[2] L. S. Figueiredo, J. Teixeira, A. S. Cavalcanti, V. Teichrieb and J. Kelner, "An Open-Source Framework for Air Guitar Games," in VIII Brazilian Symposium on Games and Digital Entertainment, Rio de Janeiro, Brazil, 2009.
[3] Y. Che and Y. Qi, "Dynamic Projected Segmentation Networks For Hand Pose Estimation," in 2018 24th International Conference on Pattern Recognition (ICPR), Beijing, China, 2018.
[4] S. E. Wei, V. Ramakrishna, T. Kanade and Y. Sheikh, "Convolutional Pose Machines," in 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA, 2016.
[5] C. Zimmermann and T. Brox, "Learning to Estimate 3D Hand Pose from Single RGB Images," in 2017 IEEE International Conference on Computer Vision (ICCV), Venice, Italy, 2017.
[6] Y. Wang, C. Peng and Y. Liu, "Mask-Pose Cascaded CNN for 2D Hand Pose Estimation From Single Color Image," IEEE Transactions on Circuits and Systems for Video Technology, vol. 29, no. 11, pp. 3258-3268, 2018.
[7] S. Baek, K. I. Kim and T.-K. Kim, "Pushing the Envelope for RGB-Based Dense 3D Hand Pose Estimation via Neural Rendering," in 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Long Beach, CA, USA, USA, 2019.
[8] Y. Che, Y. Song and Y. Qi, "A Novel Framework of Hand Localization and Hand Pose Estimation," in ICASSP 2019 - 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Brighton, United Kingdom, United Kingdom, 2019.
[9] L. Ge, H. Liang, J. Yuan and D. Thalmann, "Real-Time 3D Hand Pose Estimation with 3D Convolutional Neural Networks," IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 41, no. 4, pp. 956 - 970, 1 4 2019.
[10] C. R. Naguri and R. C. Bunescu, "Recognition of Dynamic Hand Gestures from 3D Motion Data using LSTM and CNN architectures," in 2017 16th IEEE International Conference on Machine Learning and Applications (ICMLA), Cancun, Mexico, 2018.
[11] R. Azad, M. Asadi-Aghbolaghi, S. Kasaei and S. Escalera, "Dynamic 3D Hand Gesture Recognition by Learning Weighted Depth Motion Maps," IEEE Transactions on Circuits and Systems for Video Technology, vol. 29, no. 6, pp. 1729 - 1740, 12 7 2018.
[12] Y.-J. Son and O. Choi, "Image-based hand pose classification using faster R-CNN," in 2017 17th International Conference on Control, Automation and Systems (ICCAS), Jeju, South Korea, 2017.
[13] M. Abavisani, H. R. Vaezi Joze and V. M. Patel, "Improving the Performance of Unimodal Dynamic Hand-Gesture Recognition With Multimodal Training".
[14] F. Wang, L. Kong, X. Zhang and H. Chen, "Gesture Recognition and Localization Using Convolutional Neural Network," in 2019 Chinese Control And Decision Conference (CCDC), Nanchang, China, China.
[15] N. Dhingra and A. Kunz, "Res3ATN -Deep 3D Residual Attention Network for Hand Gesture Recognition in Videos," in 2019 International Conference on 3D Vision (3DV), Québec City, QC, Canada, Canada, 2019.
[16] O. Köpüklü, A. Gunduz, N. Kose and G. Rigoll, "Real-time Hand Gesture Detection and Classification Using Convolutional Neural Networks," in 2019 14th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2019), Lille, France, France, 2019.
[17] "MIDI - Wikipedia," Wikipedia, [Online]. Available: https://en.wikipedia.org/wiki/MIDI.
[18] C. Szegedy, W. Liu, Y. Jia, P. Sermanet, S. Reed, D. Anguelov, D. Erhan, V. Vanhoucke and A. Rabinovich, "Going deeper with convolutions," in 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Boston, MA, USA, 2015.
[19] K. He, X. Zhang, S. Ren and J. Sun, "Deep Residual Learning for Image Recognition," in 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA, 2016.
連結至畢業學校之論文網頁點我開啟連結
註: 此連結為研究生畢業學校所提供,不一定有電子全文可供下載,若連結有誤,請點選上方之〝勘誤回報〞功能,我們會盡快修正,謝謝!
QRCODE
 
 
 
 
 
                                                                                                                                                                                                                                                                                                                                                                                                               
第一頁 上一頁 下一頁 最後一頁 top
無相關期刊
 
無相關點閱論文