臺灣博碩士論文加值系統

在當前日新月異的科技環境中，人機協作的關鍵作用不可忽視，其對工業發展的影響使生產過程變得更加準確和高效。機械手臂在人機協作中在多個方面體現了其重要性。機械手臂以其卓越的靈活性、精確性和成熟度成為不可或缺的選擇。本論文的主要研究目標是探討機械手臂在動態環境中交付物品給人類情境下之應用。然而，傳統機械手臂在路徑規劃方面存在一些限制，例如難以適應動態環境和複雜的開發過程。因此，本論文提出使用強化學習（Reinforcement Learning, RL）來替代傳統的路徑規劃控制系統。RL模型透過分析RGB影像來判斷手部位置和狀態，充分發揮機器學習和適應的能力。在RL中，首先會在虛擬環境中設置受控體及相關物件並進行訓練，但時常會遇到模擬器中的數據與實際環境存在巨大差距，即所謂的Reality Gap。最小化這種差距並且保留特徵正是本論文另一個相當重要的課題。本論文將使用影像切割（Image Segmentation）技術，將手部特徵與現實環境進行區隔和控制。這一方法有助於提高RL模型在實際環境中的適應性、泛化能力和可控性，從而更安全及有效地應對複雜的人機協作場景。儘管影像切割技術已經大幅減少現實與虛擬環境的差異，但在真實與虛擬的手掌特徵上仍存在不小的差距。因此，本次研究將使用循環對抗網路（CycleGAN），透過將現實手掌轉換成虛擬手掌特徵，來進一步減少訓練環境與真實環境的差異。透過上述方法，能夠使機械手臂在複雜環境中達成人機協作的目標。除了在工廠中的應用，機械手臂在軍事、建築、太空等多個領域也可以透過此技術更方便且快捷地完成任務。

In the rapidly evolving technological environment, the critical role of human-machine collaboration cannot be ignored, and its impact on industrial development makes production processes more accurate and efficient. Robotic arms exemplify their importance in various aspects of human-machine collaboration. With their outstanding flexibility, precision, and maturity, robotic arms have become an indispensable choice. The primary objective of this paper is to explore the application of robotic arms in delivering items to humans in dynamic environments. However, traditional robotic arms face certain limitations in path planning, such as difficulty adapting to dynamic environments and the complexity of the development process. Therefore, thesis paper proposes using Reinforcement Learning (RL) to replace traditional path planning control systems. The RL model leverages the analysis of RGB images to determine the position and state of the hand, fully utilizing the capabilities of machine learning and adaptation.In RL, a controlled agent and relevant objects are initially set up in a virtual environment for training. However, there is often a significant gap between the data in the simulator and the real environment, known as the reality gap. Minimizing this gap is another crucial topic addressed in this paper. This study will use image segmentation technology to separate hand features from the real environment and control them. This approach helps improve the adaptability, generalization, and controllability of the RL model in real environments, thereby more safely and effectively handling complex human-machine collaboration scenarios.

Despite the significant reduction in the differences between the real and virtual environments achieved through image segmentation technology, there still exists a considerable disparity between the real and virtual hand features. Therefore, this study will adapt CycleGAN to further reduce the differences between the training environment and the real environment by transforming real hand features into virtual hand features. Through the methods mentioned above, robotic arms can achieve the goal of human-machine collaboration in complex environments. This technology can facilitate the completion of tasks more conveniently and quickly in various fields, including military, construction, and space, in addition to factory settings.

摘要 I
ABSTRACT III
目錄 V
圖目錄 VIII
表目錄 XI
演算法目錄 XII
第一章 緒論 1
1.1 前言 1
1.2 文獻回顧 2
1.3 論文架構 9
第二章 研究系統架構 11
2.1 硬體設備 11
2.1.1 機械手臂 11
2.1.2 夾爪 14
2.1.3 相機 14
2.2 模擬環境 15
2.2.1 模擬器 15
2.2.2 建置模擬環境 15
第三章 研究方法 18
3.1 系統架構 18
3.2 強化學習(Reinforcement Learning) 20
3.2.2 預測動作 (Action) 24
3.2.3 獎勵函數 (Reward function) 25
3.3 知識蒸餾（Knowledge Distillation） 27
3.4 影像切割(Image Segmentation) 29
3.4.1 U-Net 29
3.4.2 DeepLab V3+ 31
3.5 循環對抗網路(CycleGAN) 33
第四章 實驗結果 37
4.1 影像切割評結果 37
4.1.1 Intersection over Union (IoU) 37
4.1.1 Precision、Recall、F1-score 38
4.1.2 預測結果 39
4.2 CycleGAN 轉換結果 41
4.3 強化學習結果 42
4.3.1 強化學習模型訓練 42
4.3.2 強化學習模型測試 46
4.3.3 知識蒸餾模型測試 56
第五章 結論與未來展望 60
5.1 結論 60
5.2 未來展望 61
參考文獻 62

[1]R. Sultanov, S. Sulaiman, H. Li, R. Meshcheryakov, and E. Magid, “A Review on Collaborative Robots in Industrial and Service Sectors,” in 2022 International Siberian Conference on Control and Communications (SIBCON), IEEE, Nov. 2022, pp. 1–7. doi: 10.1109/SIBCON56144.2022.10003014.
[2]G. Kokotinis, G. Michalos, Z. Arkouli, and S. Makris, “On the quantification of human-robot collaboration quality,” Int J Comput Integr Manuf, vol. 36, no. 10, pp. 1431–1448, Oct. 2023, doi: 10.1080/0951192X.2023.2189304.
[3]S. Ni, L. Zhao, A. Li, D. Wu, and L. Zhou, “Cross-View Human Intention Recognition for Human-Robot Collaboration,” IEEE Wirel Commun, vol. 30, no. 3, pp. 189–195, Jun. 2023, doi: 10.1109/MWC.018.2200514.
[4]M. T. Calcagni, C. Scoccia, G. Battista, G. Palmieri, and M. Palpacelli, “Collaborative Robot Sensorization with 3D Depth Measurement System for Collision Avoidance,” in 2022 18th IEEE/ASME International Conference on Mechatronic and Embedded Systems and Applications (MESA), IEEE, Nov. 2022, pp. 1–6. doi: 10.1109/MESA55290.2022.10004475.
[5]X. Li, Z. Chen, Z. Zhong, and J. Ma, “Human-machine Collaboration Method Based on Key Nodes of Human Posture,” in 2022 IEEE Asia-Pacific Conference on Image Processing, Electronics and Computers (IPEC), IEEE, Apr. 2022, pp. 140–146. doi: 10.1109/IPEC54454.2022.9777570.
[6]J. Marić, L. Petrović, and I. Marković, “Human Intention Recognition in Collaborative Environments using RGB-D Camera,” in 2023 46th MIPRO ICT and Electronics Convention (MIPRO), IEEE, May 2023, pp. 350–355. doi: 10.23919/MIPRO57284.2023.10159985.
[7]A. Franceschetti, E. Tosello, N. Castaman, and S. Ghidoni, “Robotic Arm Control and Task Training Through Deep Reinforcement Learning,” 2022, pp. 532–550. doi: 10.1007/978-3-030-95892-3_41.
[8]K. M. Oikonomou, I. Kansizoglou, and A. Gasteratos, “A Hybrid Reinforcement Learning Approach With a Spiking Actor Network for Efficient Robotic Arm Target Reaching,” IEEE Robot Autom Lett, vol. 8, no. 5, pp. 3007–3014, May 2023, doi: 10.1109/LRA.2023.3264836.
[9]F. Munguia-Galeano, S. Veeramani, J. D. Hernández, Q. Wen, and Z. Ji, “Affordance-Based Human–Robot Interaction With Reinforcement Learning,” IEEE Access, vol. 11, pp. 31282–31292, 2023, doi: 10.1109/ACCESS.2023.3262450.
[10]E. Salvato, G. Fenu, E. Medvet, and F. A. Pellegrino, “Crossing the Reality Gap: A Survey on Sim-to-Real Transferability of Robot Controllers in Reinforcement Learning,” IEEE Access, vol. 9, pp. 153171–153187, 2021, doi: 10.1109/ACCESS.2021.3126658.
[11]P. Xie et al., “Part-Guided 3D RL for Sim2Real Articulated Object Manipulation,” IEEE Robot Autom Lett, vol. 8, no. 11, pp. 7178–7185, Nov. 2023, doi: 10.1109/LRA.2023.3313063.
[12]T. Zhang, K. Zhang, J. Lin, W.-Y. G. Louie, and H. Huang, “Sim2real Learning of Obstacle Avoidance for Robotic Manipulators in Uncertain Environments,” IEEE Robot Autom Lett, vol. 7, no. 1, pp. 65–72, Jan. 2022, doi: 10.1109/LRA.2021.3116700.
[13]O. Ronneberger, P. Fischer, and T. Brox, “U-Net: Convolutional Networks for Biomedical Image Segmentation,” May 2015.
[14]L.-C. Chen, Y. Zhu, G. Papandreou, F. Schroff, and H. Adam, “Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation,” Feb. 2018.
[15]“Collaborative robotic automation | Cobots from Universal Robots.” Accessed: Feb. 23, 2024. [Online]. Available: https://www.universal-robots.com/
[16]“Bullet Real-Time Physics Simulation | Home of Bullet and PyBullet: physics simulation for games, visual effects, robotics and reinforcement learning.” Accessed: Feb. 24, 2024. [Online]. Available: https://pybullet.org/wordpress/
[17]T. Haarnoja et al., “Soft Actor-Critic Algorithms and Applications,” Dec. 2018.