臺灣博碩士論文加值系統

隨著人工智慧的快速發展和肢體辨識技術的革新，智慧校園已不再只是想像。過去，教師往往只能依賴學生的學習成績來評斷課堂教學內容的難易度，對於學生實際吸收的內容，只能透過考試來驗證成果。因此，如果能夠將深度學習技術應用於辨識課堂中學生的行為，將有助於教師更好地了解學生的學習狀況。
本論文提出一個課堂行為辨識系統，該系統結合YOLOv7骨架提取與SVM分類模型，能夠更準確地判斷學生上課時的行為，同時系統還能追蹤人物ID，並紀錄每個人的行為曲線，在辨識結束後，能夠自動校正錯誤的辨識數據，並生成課堂行為報告，透過圖表展示每個時間點學生的行為狀態分佈，以提供教師對課堂動態的深入瞭解。
整個系統只需一台嵌入式裝置、一個觸控螢幕和一個攝影機。此外，我們還為系統設計了簡單易懂的UI介面，並且在不連接網路的情況下也能夠正常運作，實現了高便攜性、高可用性和高安全性。

With the rapid development of artificial intelligence and advancements in gesture recognition technology, the concept of a smart campus is no longer just an imagination. Traditionally, teachers have relied on students' academic performance to assess the difficulty of classroom content, with exams being the primary method to verify the students' understanding. Thus, applying deep learning technology to recognize students' behavior in the classroom can help teachers better understand students' learning status.
This paper proposes a classroom behavior recognition system that combines the YOLOv7 framework for skeleton extraction with an SVM classification model. This system can more accurately identify students' behaviors during class. Additionally, the system can track individual IDs and record each person's behavior trajectory. After recognition, it automatically corrects erroneous data and generates a classroom behavior report, displaying the distribution of students' behaviors at each time point through charts, providing teachers with a comprehensive understanding of classroom dynamics.
The entire system requires only one embedded device, a touch screen, and a camera. Furthermore, we have designed a simple and intuitive UI interface for the system, which can operate normally without an internet connection, achieving high portability, high availability, and high security.

誌謝 i
摘要 ii
Abstract iii
目錄 iv
圖目錄 vi
表目錄 viii
第一章、 緒論 1
1.1 研究背景 1
1.2 研究動機 2
1.3 研究目的 2
1.4 問題定義 2
1.5 研究架構 3
第二章、 文獻探討 5
2.1 課堂行為檢測 5
2.1.1 基於物件偵測的行為檢測方法 5
2.1.2 基於骨架提取的行為檢測方法 7
2.1.3 統整不同檢測方法的優缺點 9
2.2 人體姿態檢測 11
2.2.1 單人/多人姿態檢測 11
2.2.2 姿態檢測方法 12
2.2.3 姿態檢測演算法 13
2.3 學生行為分類 15
2.3.1 SVM[3] 15
第三章、 研究方法 16
3.1 系統架構 16
3.2 行為分類 17
3.2.1 looking 17
3.2.2 asking 18
3.2.3 bowing 18
3.2.4 boring 18
3.3 骨架數據的收集和特徵提取 18
3.3.1 姿態辨識演算法效能比較 18
3.4 SVM分類模型[3] 21
3.5 人物追蹤 23
3.6 校正分析 24
第四章、 實證分析 25
4.1 實驗內容 25
4.1.1 Jetson Xavier NX[22] 25
4.1.2 Jetson AGX Xavier[23] 26
4.1.3 TensorRT 27
4.2 實驗設定 27
4.3 實驗結果 29
4.3.1 姿態辨識演算法效能比較結果 29
4.3.2 行為分類方法及準則 36
4.3.3 行為分類順序比較 39
4.3.4 提出之方法準確率比較結果 40
4.3.5 資料校正分析結果 42
4.4 系統展示 44
4.4.1 系統使用流程 44
4.4.2 系統實際部屬 47
第五章、 問題與討論 50
5.1 研究限制 50
5.1.1 姿態辨識演算法 50
5.1.2 鏡頭擺放位置 50
5.1.3 行為分類 50
5.2 未來展望 51
5.2.1 行為分類 51
第六章、 結論 52
參考文獻 53

[1] Z.-W. Sun, “教室學生行為之即時辨識嵌入式系統,” 逢甲大學 資訊工程學系 碩士論文, 2023. [Online]. Available: https://hdl.handle.net/11296/4sptg6
[2] Z. Cao, T. Simon, S.-E. Wei, and Y. Sheikh, “Realtime Multi-person 2D Pose Estimation Using Part Affinity Fields,” in 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Jul. 2017, pp. 1302–1310. doi: 10.1109/CVPR.2017.143.
[3] C. Cortes and V. Vapnik, “Support-vector networks,” Mach. Learn., vol. 20, no. 3, pp. 273–297, Sep. 1995, doi: 10.1007/BF00994018.
[4] C.-Y. Wang, A. Bochkovskiy, and H.-Y. M. Liao, “YOLOv7: Trainable bag-of-freebies sets new state-of-the-art for real-time object detectors.” arXiv, Jul. 06, 2022. doi: 10.48550/arXiv.2207.02696.
[5] F.-C. Lin, H.-H. Ngo, C.-R. Dow, K.-H. Lam, and H. L. Le, “Student behavior recognition system for the classroom environment based on skeleton pose estimation and person detection,” Sensors, vol. 21, no. 16, p. 5314, 2021.
[6] R. Zheng, F. Jiang, and R. Shen, “Intelligent Student Behavior Analysis System for Real Classrooms,” in ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), May 2020, pp. 9244–9248. doi: 10.1109/ICASSP40776.2020.9053457.
[7] G. Jocher, “YOLOv5 by Ultralytics.” May 2020. doi: 10.5281/zenodo.3908559.
[8] L. Li, M. Liu, L. Sun, Y. Li, and N. Li, “ET-YOLOv5s: Toward Deep Identification of Students’ in-Class Behaviors,” IEEE Access, vol. 10, pp. 44200–44211, 2022, doi: 10.1109/ACCESS.2022.3169586.
[9] S.-Q. Yang, Y.-H. Chen, Z.-Y. Zhang, and J.-H. Chen, “Student in-class behaviors detection and analysis system based on CBAM-YOLOv5,” in 2022 7th International Conference on Intelligent Computing and Signal Processing (ICSP), Apr. 2022, pp. 440–443. doi: 10.1109/ICSP54964.2022.9778630.
[10] M. M. A. Parambil, L. Ali, F. Alnajjar, and M. Gochoo, “Smart Classroom: A Deep Learning Approach towards Attention Assessment through Class Behavior Detection,” in 2022 Advances in Science and Engineering Technology International Conferences (ASET), Feb. 2022, pp. 1–6. doi: 10.1109/ASET53988.2022.9735018.
[11] R. Fu, T. Wu, Z. Luo, F. Duan, X. Qiao, and P. Guo, “Learning Behavior Analysis in Classroom Based on Deep Learning,” in 2019 Tenth International Conference on Intelligent Control and Information Processing (ICICIP), Feb. 2019, pp. 206–212. doi: 10.1109/ICICIP47338.2019.9012177.
[12] K. Simonyan and A. Zisserman, “Very Deep Convolutional Networks for Large-Scale Image Recognition.” arXiv, Apr. 10, 2015. doi: 10.48550/arXiv.1409.1556.
[13] M. H. Le, T. M. Doan, D. D. Nguyen, and M. S. Nguyen, “Smart Desk in Hybrid Classroom: Detecting student’s lack of concentration when studying,” in 2022 9th NAFOSTED Conference on Information and Computer Science (NICS), Oct. 2022, pp. 13–18. doi: 10.1109/NICS56915.2022.10013468.
[14] C. Lugaresi et al., “MediaPipe: A Framework for Building Perception Pipelines.” arXiv, Jun. 14, 2019. doi: 10.48550/arXiv.1906.08172.
[15] M. A. Samir, Y. Maged, and A. Atia, “Exam Cheating Detection System with Multiple-Human Pose Estimation,” in 2021 IEEE International Conference on Computing (ICOCO), Jan. 2021, pp. 236–240. doi: 10.1109/ICOCO53166.2021.9673534.
[16] Z. Cao, G. Hidalgo, T. Simon, S.-E. Wei, and Y. Sheikh, “OpenPose: Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields.” arXiv, May 30, 2019. doi: 10.48550/arXiv.1812.08008.
[17] T. L. Munea, Y. Z. Jembre, H. T. Weldegebriel, L. Chen, C. Huang, and C. Yang, “The Progress of Human Pose Estimation: A Survey and Taxonomy of Models Applied in 2D Human Pose Estimation,” IEEE Access, vol. 8, pp. 133330–133348, 2020, doi: 10.1109/ACCESS.2020.3010248.
[18] C. Zheng et al., “Deep Learning-based Human Pose Estimation: A Survey,” ACM Comput. Surv., vol. 56, no. 1, p. 11:1-11:37, Aug. 2023, doi: 10.1145/3603618.
[19] D. Osokin, “Real-time 2D Multi-Person Pose Estimation on CPU: Lightweight OpenPose.” arXiv, Nov. 29, 2018. doi: 10.48550/arXiv.1811.12004.
[20] A. G. Howard et al., “MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications.” arXiv, Apr. 16, 2017. doi: 10.48550/arXiv.1704.04861.
[21] G. Jocher, A. Chaurasia, and J. Qiu, “Ultralytics YOLO.” Jan. 2023. Accessed: Jan. 23, 2024. [Online]. Available: https://github.com/ultralytics/ultralytics
[22] “適用於嵌入式和邊緣系統的 NVIDIA Jetson Xavier NX,” NVIDIA. Accessed: Feb. 09, 2024. [Online]. Available: https://www.nvidia.com/zh-tw/autonomous-machines/embedded-systems/jetson-xavier-nx/
[23] “部署人工智慧驅動的大規模自動化裝置,” NVIDIA. Accessed: Feb. 09, 2024. [Online]. Available: https://www.nvidia.com/zh-tw/autonomous-machines/embedded-systems/jetson-agx-xavier/
[24] E. Jeong, J. Kim, S. Tan, J. Lee, and S. Ha, “Deep Learning Inference Parallelization on Heterogeneous Processors With TensorRT,” IEEE Embed. Syst. Lett., vol. 14, no. 1, pp. 15–18, Mar. 2022, doi: 10.1109/LES.2021.3087707.