跳到主要內容

臺灣博碩士論文加值系統

(44.192.44.30) 您好!臺灣時間:2024/07/25 08:52
字體大小: 字級放大   字級縮小   預設字形  
回查詢結果 :::

詳目顯示

: 
twitterline
研究生:陳郁萱
研究生(外文):CHEN, YU-HSUAN
論文名稱:通過知識蒸餾提昇輕量級模型:智慧桌球運動系統邊緣運算之應用
論文名稱(外文):Improving Lightweight Models Through Knowledge Distillation: An Edge Computing Application for Smart Table Tennis System
指導教授:陳鏗元陳敦裕陳敦裕引用關係
指導教授(外文):CHEN, KENG-YUANCHEN, DUAN-YU
口試委員:魏志達蘇志文
口試委員(外文):WEI, JYH-DASU, CHIH-WEN
口試日期:2022-07-15
學位類別:碩士
校院名稱:元智大學
系所名稱:電機工程學系甲組
學門:工程學門
學類:電資工程學類
論文種類:學術論文
論文出版年:2022
畢業學年度:110
語文別:英文
論文頁數:57
中文關鍵詞:深度學習知識蒸餾
外文關鍵詞:Deep learningKnowledge distillation
相關次數:
  • 被引用被引用:0
  • 點閱點閱:154
  • 評分評分:
  • 下載下載:29
  • 收藏至我的研究室書目清單書目收藏:0
隨著科技的進步,已有眾多深度神經網路應用在邊緣計算裝置或移動設備上。但鑒於神經網路大量的參數及運算使得邊緣計算裝置運行困難。因此我們採取了不同知識蒸餾的方法並結合現今熱門的智慧運動議題,來使需要在邊緣裝置上運行的球偵測輕量模型有效地提升精度。在實驗中,我們提出的可學習輔助教材對於提升模型的表現是非常好的,相較於知識蒸餾前的原始準確率平均提升了1%,最好的表現能夠達到91.3%。
With the advancement of technology, many deep neural networks have been applied to edge computing devices or mobile devices. However, due to the large number of parameters and operations of the neural network, the operation of edge computing devices is difficult. Therefore, we have adopted different knowledge distillation methods and combined them with the current popular smart motion issues to effectively improve the accuracy of the ball detection lightweight model that needs to run on edge devices. In experiments, the performance of our proposed learnable teaching materials is very effective in improving the model, with an average improvement of 1% compared to the Baseline accuracy before knowledge distillation, and the best performance can achieve 91.3%.
Title Page i
Letter of Approval ii
Abstract in Chinese iii
Abstract in English iv
Acknowledgement v
Table of Contents vi
List of table viii
List of figure ix
Chapter 1. Introduction 1
Chapter 2. Related Work 3
2.1 Objection detection 3
2.2 Computer vision assisted table tennis 4
2.2.1 Keypoint-based Detector 5
2.2.2 Fully Convolutional One-Stage Object Detection (FCOS) 6
2.3 YOLOR 7
2.4 Knowledge Distillation 8
Chapter 3. Proposed Methods 12
3.1 Review 12
3.1.1 FCOS-ResNet-13lite 12
3.1.2 Implicit Knowledge (IK) 13
3.2 Distillation from the knowledge of the output layer 15
3.3 Distillation from the knowledge of the intermediate features 17
3.3.1 Intermediate knowledge acquired with CBAM 18
3.3.2 Intermediate knowledge acquired with multi-head 21
3.4 Distillation with Implicit Knowledge 23
3.5 Distillation with Learnable Teaching Material (LTM) 24
Chapter 4. Experiments 27
4.1 Dataset 27
4.1.1 Private dataset based on CFD method 27
4.1.2 Data Augmentation 30
4.1.3 Establish appraisal standard 31
4.2 Implement detail 32
4.2.1 Training parameters 33
4.3 Results and Ablation study 34
4.3.1 Compare the performance of all methods 34
4.3.2 Ablation study for Multi-head-KD 37
4.3.3 Ablation study for IK-KD 38
4.3.4 Ablation study for LTM-KD 40
Chapter 5. Conclusion 42
Reference 43

[1]H. Li, A. Kadav, I. Durdanovic, H. Samet, and H. P. Graf, "Pruning Filters for Efficient ConvNets," ArXiv, vol. abs/1608.08710, 2017.
[2]Z. Liu, M. Sun, T. Zhou, G. Huang, and T. Darrell, "Rethinking the Value of Network Pruning," p. arXiv:1810.05270. [Online]. Available: https://ui.adsabs.harvard.edu/abs/2018arXiv181005270L
[3]J.-H. Luo, J. Wu, and W. Lin, "ThiNet: A Filter Level Pruning Method for Deep Neural Network Compression," 2017 IEEE International Conference on Computer Vision (ICCV), pp. 5068-5076, 2017.
[4]S. Sen, N. Moha, B. Baudry, and J.-M. Jézéquel, "Meta-model Pruning," in Model Driven Engineering Languages and Systems, Berlin, Heidelberg, A. Schürr and B. Selic, Eds., 2009// 2009: Springer Berlin Heidelberg, pp. 32-46.
[5]M. Zhu and S. Gupta, "To prune, or not to prune: exploring the efficacy of pruning for model compression," ArXiv, vol. abs/1710.01878, 2018.
[6]A. J. Ahumada Jr and H. A. Peterson, "Luminance-model-based DCT quantization for color image compression," in Human vision, visual processing, and digital display III, 1992, vol. 1666: SPIE, pp. 365-374.
[7]H. Peterson, A. Ahumada, and A. Watson, "An Improved Detection Model for DCT Coefficient Quantization," AIAA Computing in Aerospace 9: A Collection of Technical Papers, vol. 1913, 09/08 1993, doi: 10.1117/12.152693.
[8]A. Polino, R. Pascanu, and D. Alistarh, "Model compression via distillation and quantization," ArXiv, vol. abs/1802.05668, 2018.
[9]G. Chen, W. Choi, X. Yu, T. Han, and M. Chandraker, "Learning efficient object detection models with knowledge distillation," presented at the Proceedings of the 31st International Conference on Neural Information Processing Systems, Long Beach, California, USA, 2017.
[10]Y. Kim and A. M. Rush, "Sequence-Level Knowledge Distillation," in EMNLP, 2016.
[11]S. Lee and B. C. Song, "Graph-based Knowledge Distillation by Multi-head Attention Network," p. arXiv:1907.02226. [Online]. Available: https://ui.adsabs.harvard.edu/abs/2019arXiv190702226L
[12]Y. Liu, K. Chen, C. Liu, Z. Qin, Z. Luo, and J. Wang, "Structured Knowledge Distillation for Semantic Segmentation," in 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 15-20 June 2019 2019, pp. 2599-2608, doi: 10.1109/CVPR.2019.00271.
[13]S. I. Mirzadeh, M. Farajtabar, A. Li, N. Levine, A. Matsukawa, and H. Ghasemzadeh, "Improved Knowledge Distillation via Teacher Assistant," in AAAI, 2020.
[14]W. Park, D. Kim, Y. Lu, and M. Cho, "Relational Knowledge Distillation," 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 3962-3971, 2019.
[15]H. Wang, S. Lohit, M. Jones, and Y. R. Fu, "Multi-head Knowledge Distillation for Model Compression," ArXiv, vol. abs/2012.02911, 2020.
[16]J. Yim, D. Joo, J. Bae, and J. Kim, "A Gift from Knowledge Distillation: Fast Optimization, Network Minimization and Transfer Learning," in 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 21-26 July 2017 2017, pp. 7130-7138, doi: 10.1109/CVPR.2017.754.
[17]L. Yuan, F. E. H. Tay, G. Li, T. Wang, and J. Feng, "Revisit Knowledge Distillation: a Teacher-free Framework," ArXiv, vol. abs/1909.11723, 2019.
[18]X. Li, W. Wang, X. Hu, J. Li, J. Tang, and J. Yang, "Generalized Focal Loss V2: Learning Reliable Localization Quality Estimation for Dense Object Detection," 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 11627-11636, 2021.
[19]X. Li et al., "Generalized Focal Loss: Learning Qualified and Distributed Bounding Boxes for Dense Object Detection," ArXiv, vol. abs/2006.04388, 2020.
[20]J. Ba and R. Caruana, "Do Deep Nets Really Need to be Deep?," ArXiv, vol. abs/1312.6184, 2014.
[21]A. Romero, N. Ballas, S. E. Kahou, A. Chassang, C. Gatta, and Y. Bengio, "FitNets: Hints for Thin Deep Nets," CoRR, vol. abs/1412.6550, 2015.
[22]R. B. Girshick, "Fast R-CNN," 2015 IEEE International Conference on Computer Vision (ICCV), pp. 1440-1448, 2015.
[23]R. B. Girshick, J. Donahue, T. Darrell, and J. Malik, "Rich Feature Hierarchies for Accurate Object Detection and Semantic Segmentation," 2014 IEEE Conference on Computer Vision and Pattern Recognition, pp. 580-587, 2014.
[24]S. Ren, K. He, R. B. Girshick, and J. Sun, "Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks," IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 39, pp. 1137-1149, 2015.
[25]A. Bochkovskiy, C.-Y. Wang, and H.-Y. M. Liao, "YOLOv4: Optimal Speed and Accuracy of Object Detection," ArXiv, vol. abs/2004.10934, 2020.
[26]Z. Dong, G. Li, Y. Liao, F. Wang, P. Ren, and C. Qian, "CentripetalNet: Pursuing High-Quality Keypoint Pairs for Object Detection," 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 10516-10525, 2020.
[27]K. Duan, S. Bai, L. Xie, H. Qi, Q. Huang, and Q. Tian, "CenterNet: Keypoint Triplets for Object Detection," 2019 IEEE/CVF International Conference on Computer Vision (ICCV), pp. 6568-6577, 2019.
[28]S. Lan, Z. Ren, Y. Wu, L. S. Davis, and G. Hua, "SaccadeNet: A Fast and Accurate Object Detector," 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 10394-10403, 2020.
[29]H. Law and J. Deng, "CornerNet: Detecting Objects as Paired Keypoints," International Journal of Computer Vision, vol. 128, pp. 642-656, 2019.
[30]W. Liu et al., "SSD: Single Shot MultiBox Detector," in ECCV, 2016.
[31]W. Liu, I. Hasan, and S. Liao, "Center and Scale Prediction: A Box-free Approach for Pedestrian and Face Detection," arXiv: Computer Vision and Pattern Recognition, 2019.
[32]Z. Tian, C. Shen, H. Chen, and T. He, "FCOS: Fully Convolutional One-Stage Object Detection," 2019 IEEE/CVF International Conference on Computer Vision (ICCV), pp. 9626-9635, 2019.
[33]X. Zhou, D. Wang, and P. Krähenbühl, "Objects as Points," ArXiv, vol. abs/1904.07850, 2019.
[34]X. Zhou, J. Zhuo, and P. Krähenbühl, "Bottom-Up Object Detection by Grouping Extreme and Center Points," 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 850-859, 2019.
[35]C. Zhu, Y. He, and M. Savvides, "Feature Selective Anchor-Free Module for Single-Shot Object Detection," 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 840-849, 2019.
[36]R. Voeikov, N. Falaleev, and R. Baikulov, "TTNet: Real-time temporal and spatial video analysis of table tennis," 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pp. 3866-3874, 2020.
[37]C.-Y. Wang, I.-H. Yeh, and H. Liao, "You Only Learn One Representation: Unified Network for Multiple Tasks," ArXiv, vol. abs/2105.04206, 2021.
[38]G. E. Hinton, O. Vinyals, and J. Dean, "Distilling the Knowledge in a Neural Network," ArXiv, vol. abs/1503.02531, 2015.
[39]S. Zagoruyko and N. Komodakis, "Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfer," ArXiv, vol. abs/1612.03928, 2017.
[40]S. Woo, J. Park, J.-Y. Lee, and I.-S. Kweon, "CBAM: Convolutional Block Attention Module," in ECCV, 2018.
[41]管謹中, "超快速基於深度學習之彈跳點偵測於邊緣運算設備之智慧桌球系統," 碩士, 電機工程學系甲組, 元智大學, 桃園縣, 2021. [Online]. Available: https://hdl.handle.net/11296/3q9en7
[42]G. Ghiasi et al., "Simple Copy-Paste is a Strong Data Augmentation Method for Instance Segmentation," 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2917-2927, 2021.
[43]A. Paszke et al., "Pytorch: An imperative style, high-performance deep learning library," Advances in neural information processing systems, vol. 32, pp. 8026-8037, 2019.
[44]D. P. Kingma and J. Ba, "Adam: A method for stochastic optimization," arXiv preprint arXiv:1412.6980, 2014.
[45]P. Goyal et al., "Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour," ArXiv, vol. abs/1706.02677, 2017.

QRCODE
 
 
 
 
 
                                                                                                                                                                                                                                                                                                                                                                                                               
第一頁 上一頁 下一頁 最後一頁 top
無相關期刊