臺灣博碩士論文加值系統

隨著科技發展，機器人不再是只有工業上的應用，有愈來愈多的服務型機器人已逐漸地融入在我們的日常生活當中，為各行各業的工作者提供協助。但是，執行利用視覺感測進行物件操作的任務對於服務型機器人仍是一大難關，因為像這樣的控制演算法通常需要透過非常大量的標註資料進行訓練，而且時常會需要針對模型未見過的物體進行額外標註，這對於一般未接觸過人工智慧的使用者無疑是一大困擾，也間接增加了機器人與潛在消費者市場的隔閡。為了使機器人能夠更妥善地應對日常環境中各種可能遇到的使用情境，並節省掉額外訓練資料的標註成本，勢必需要一個能自動適應任意環境，並且不需要標註訓練資料就能進行學習的通用物件夾取演算法。
有鑑於此，本研究開發一套基於視覺的深度強化學習機械手臂夾取系統，透過深度學習與強化學習的結合，讓機器人能自動根據環境提供的視覺資訊以及回饋值來進行學習，其中不需要任何人工或非人工的標註資料，可以節省掉大量的訓練資料標註人力與時間成本。經實驗測試，本研究所提出的多重動作輸出深度Q網路可以有效的減少訓練時程，並且可以成功夾取未出現在訓練階段的物體。

With the development of technology, robots are no longer confined to industrial applications. An increasing number of service robots have gradually integrated into our daily lives, providing assistance to professionals in various fields. However, performing tasks that involve object manipulation using visual sensing remains a significant challenge for service robots. This is because control algorithms of this nature often require extensive annotated data for training and may need additional annotations for objects unseen by the model. This undoubtedly poses a major inconvenience for users who are unfamiliar with artificial intelligence and creates a gap between robots and potential consumer markets.
To enable robots to handle various scenarios in everyday environments more effectively and to reduce the cost of annotating additional training data, a generic object-grasping algorithm that can adapt automatically to any environment without the need for annotated training data is essential.
In light of this, this research develops a visual-based deep reinforcement learning robotic arm grasping system. By combining deep learning and reinforcement learning, the robot can learn automatically based on the visual information and feedback reward provided by the environment, without requiring any manual or non-manual annotation data. This approach saves a significant amount of human and time costs involved in data annotation for training. Through experimental testing, the proposed multi-action output deep Q network in this research can effectively reduce the training time and successfully grasp objects not seen during the training phase.

摘要 i
Using Vision-Based Deep Reinforcement Learning Algorithm at Robot Arm Grasping Task ii
誌謝 xi
目錄 xii
表目錄 xiv
圖目錄 xv
第1章 緒論 1
1.1 研究動機與目的 1
1.2 文獻探討 1
1.3 整體架構 3
1.4 貢獻 4
1.5 論文架構 4
第2章 系統規格 5
2.1 硬體架構 5
2.2 硬體規格 6
2.2.1 機械手臂 6
2.2.2 夾爪 8
2.2.3 RGB-D 相機 8
2.2.4 電腦 10
2.3 工作空間規格 11
第3章 系統架構 12
3.1 環境設置架構 12
3.1.1 現實環境設置 12
3.1.1.1 通訊網路修改 14
3.1.1.2 電源修改 17
3.1.1.3 攝影機修改 18
3.1.2 模擬環境設置 18
3.1.3 半自動手眼校正系統架構 19
3.2 影像前處理 23
3.3 深度強化學習演算法 24
3.3.1 多重動作輸出深度Q網路 25
3.3.2 深度Q網路推論與訓練流程 26
3.3.3 獎勵塑形 28
3.3.4 動作策略 - ɛ-貪婪策略 29
3.3.5 損失函數 - Smooth L1 Loss 30
第4章 實驗方法 31
4.1 實驗用環境 31
4.2 實驗用代理人 33
4.3 實驗設計 34
4.4 實驗用夾取物件 34
4.5 評量指標 35
第5章 實驗結果 36
5.1 模擬環境實驗 36
5.1.1 不同代理人在夾取任務的表現 36
5.1.2 不同輸入影像尺寸對物件夾取動作的影響 37
5.1.3 不同獎勵值對代理人的學習之影響 37
5.1.4 與VPG演算法的比較 38
5.2 現實環境實驗 39
第6章 結論 40
6.1 總結 40
6.2 未來展望與建議 40
參考文獻 41

[1]Zai, A. & Brown, B. (2021). 深度強化式學習(黃駿譯). 台北市：旗標。
[2]高揚、葉振斌. (2020). 強化學習(RL)：使用PyTorch徹底精通. 台北市：深智數位
[3]Depierre, A., Dellandréa, E., & Chen, L. (2018). Jacquard: a large scale dataset for robotic grasp detection. In IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).
[4]Fang, H.-S., Wang, C., Gou, M., & Lu, C. (2020). GraspNet-1Billion: A Large-Scale Benchmark for General Object Grasping. In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (pp. 11441-11450). Seattle, WA, USA.
[5]Grunnet-Jepsen, A., Sweetser, J. N., Winer, P., Takagi, A., & Woodfill, J. (2023). Whitepaper of Projectors for D400 Series Depth Cameras. Retrieved from https://dev.intelrealsense.com/docs/projectors#1-introduction
[6]Kleeberger, K., Bormann, R., Kraus, W., et al. (2020). A Survey on Learning-Based Robotic Grasping. Current Robot Report, 1, 239–249.
[7]Levine, S., Pastor, P., Krizhevsky, A., & Quillen, D. (2016). Learning hand-eye coordination for robotic grasping with deep learning and large-scale data collection. In International Symposium on Experimental Robotics (ISER).
[8]Lillicrap, T. P. (2015). Continuous control with deep reinforcement learning. Retrieved from arXiv:1509.02971
[9]Hasselt, H., Guez, A., Silver, D.(2015). Deep Reinforcement Learning with Double Q-learning
[10]Mahler, J., Liang, J., Niyaz, S., Laskey, M., Doan R., Liu X., et al. (2017). DexNet 2.0: deep learning to plan robust grasps with synthetic point clouds and analytic grasp Metrics. In: Amato N, Srinivasa S, Ayanian N, Kuindersma S, editors. Robotics: Science and Systems (RSS); July 12–16, 2017; Cambridge, Massachusetts, USA: Robotics Science and Systems Foundation; 2017.
[11]Mnih, V. (2013). Playing Atari with Deep Reinforcement Learning. Retrieved from arXiv:1312.5602
[12]Mnih, V. (2015). Human-level control through deep reinforcement learning. Nature, 518, 529–533. https://doi.org/10.1038/nature14236
[13]Morrison, D., Corke, P., & Leitner, J. (2019). Learning robust, real-time, reactive robotic grasping. The International Journal of Robotics Research (IJRR).
[14]Sutton, R. S., & Barto, A. G. (2020). Reinforcement Learning: An Introduction (2nd ed.). MIT Press.
[15]Tsai R. Y. & Lenz R. K. (1989). A new technique for fully autonomous and efficient 3d robotics hand/eye calibration. In IEEE Transactions on Robotics and Automation, 5(3), 345–358, June.
[16]Zeng, A. (2018). Learning Synergies between Pushing and Grasping with Self-supervised Deep Reinforcement Learning. In IROS 2018. Retrieved from arXiv:1803.09956 [cs.RO]
[17]Cornell University. Cornell Grasping Dataset. http://pr.cs.cornell. edu/grasping/rectdata/data.php. Accessed 1 June 2020.