臺灣博碩士論文加值系統

近年來，深度學習被大量應用於物件識別與機器手臂夾取物件上，如自動化工廠的上下料。然而，隨著現今工廠少量多樣的生產模式，深度學習在面對新的物件時，通常無法直接應用，需針對新的物件重新蒐集資料以訓練模型。為因應此問題，本研究提出一套能針對未知新物件在堆疊場景中的夾取流程。本研究的夾取流程分為兩步驟，第一步驟會先透過與類別無關之實例切割模型(Mask R-CNN)，將物件的遮罩由堆疊場景中辨識出來。其中，為了訓練與類別無關之實例切割模型，本研究透過Blender建立虛擬環境，生成虛擬堆疊資料集；此外，也結合課程學習(Curriculum Learning)，將虛擬堆疊資料集，根據場景中物件的密集程度，分成三種難易度不同的資料集來訓練與類別無關之實例切割模型。而在第二步驟中，會將第一步驟所得到的物件遮罩，擷取其深度資訊輸入夾取點生成卷積類神經網路(Generative Grasping Convolutional Neural Network , GG-CNN2)，得到夾取點。本研究最後透過實際實驗來驗證夾取流程，在5種未知新物件所形成的堆疊場景中，達到92.94%的平均夾取成功率。實驗結果證明了本研究之夾取流程，應用於未知堆疊物件夾取的可行性。

In recent years, methods using deep learning have been widely applied in object recognition and robot grasping, such as loading and unloading in automated factories. However, the deep learning-based model needs to be retrained for new objects due to the low-volume, high-variety production environments in today's factories. In response to this problem, a grasping pipeline containing two steps has been proposed for grasping novel objects in cluttered scenes. In the first step of the grasping pipeline, an object’s mask is identified through a category-agnostic instance segmentation model. A virtual environment is also created using Blender to generate a synthetic dataset for training the instance segmentation model. Furthermore, by using Curriculum Learning to train the instance segmentation model, we generate 3 kinds of synthetic dataset with different density of cluttering. In the second step, the Generative Grasping Convolutional Neural Network (GG-CNN2) which uses the depth information of the object mask obtained in the first step is used to get the grasping points. Finally, we demonstrate the system's ability by using a real robot to grasp 5 unknown new objects in a cluttered environment and achieve a grasp success rate up to 92.94%.

誌謝 i
摘要 ii
Abstract iii
目錄 iv
圖目錄 vii
表目錄 x
第一章
緒論 1
1.1 研究背景 1
1.2 文獻回顧 1
1.2.1 未知單一物件抓取-傳統方法 4
1.2.2 未知單一物件抓取-學習方法 7
1.2.3 未知多重物件抓取 8
1.3 研究目的 11
1.4 本文架構 12
第二章 實例切割 13
2.1 Mask-RCNN 13
2.1.1 特徵提取網路 14
2.1.2 區域提案網路 15
2.1.3 感興趣區域對齊 16
2.1.4 類別、邊界框與遮罩預測 17
2.2 資料蒐集與標註 17
2.2.1 渲染軟體 Blender 17
2.2.2 課程學習 19
2.2.3 資料蒐集方法 22
2.2.4 資料標註方法 24
2.3 模型訓練 25
2.3.1 泛化物件訓練 26
2.4 模型預測結果 27
2.4.1 評判標準 28
2.4.2 Wisdom資料集 29
2.4.3 量化結果 30
第三章 夾取點生成與判定 32
3.1 夾取點生成卷積類神經網路 32
3.1.1 夾取點定義 32
3.1.2 模型架構 34
3.2 模型訓練 35
3.2.1 訓練資料 35
3.2.2 訓練流程 38
3.3 模型預測結果 39
3.3.1 Cornell 夾取資料集 39
3.3.2 Jacquard 資料集 40
3.3.3 深度影像處理 41
3.4 夾取點選擇與干涉判斷 43
3.4.1 夾取點干涉判斷 43
3.4.2 夾取點擴增 45
第四章 實驗系統與驗證 46
4.1 系統 46
4.1.1 系統架構 46
4.1.2 實驗環境 46
4.1.3 夾取系統坐標系 49
4.2 夾取流程驗證 50
4.2.1 夾取流程 50
4.2.2 夾取成功率 51
4.2.3 夾取流程運算時間 52
4.2.4 課程學習驗證 52
4.2.5 GG-CNN2 訓練資料集驗證 53
4.2.6 深度影像處理驗證 55
第五章 結論與未來展望 56
5.1 結論 56
5.2 未來展望 57
參考文獻 58

[1] T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár and C. L. Zitnick, "Microsoft coco: Common objects in context," in Europeon Conference on Computer Vision, Cham, pp. 740-755, 2014.
[2] A. Bicchi and V. Kumar, "Robotic grasping and contact: A review," in IEEE International Conference on Robotics and Automation, vol. 1, pp. 348-353, 2000.
[3] A. T. Miller, S. Knoop, H. I. Christensen and P. K. Allen, "Automatic grasp planning using shape primitives," in IEEE International Conference on Robotics and Automation, vol. 2, pp. 1824-1829, 2003.
[4] A. T. Miller and P. K. Allen, "Graspit! A versatile simulator for robotic grasping," IEEE Robotics & Automation Magazine, vol. 11, no. 4, pp. 110-122, 2004.
[5] J. J. Rodrigues, J. S. Kim, M. Furukawa, J. Xavier, P. Aguiar and T. Kanade, "6D pose estimation of textureless shiny objects using random ferns for bin-picking," in IEEE International Conference on Intelligent Robots and Systems, pp. 3334-3341, 2012.
[6] D. Forsyth and J. Ponce, Computer vision: A modern approach. Prentice Hall, 2003.
[7] J. Baumgartl and D. Henrich, "Fast vision-based grasp and delivery planning for unknown objects," in German Conference on Robotics, pp. 1-5, 2012.
[8] P. V. Hough, "Method and means for recognizing complex patterns," U.S. Patent 3.069.654, 1962.
[9] C. Eppner and O. Brock, "Grasping unknown objects by exploiting shape adaptability and environmental constraints," in IEEE International Conference on Intelligent Robots and Systems, pp. 4000-4006, 2013.
[10] D. Holz, A. J. Trevor, M. Dixon, S. Gedikli, R. B. Rusu and S. Behnke, "Fast segmentation of rgb-d images for semantic scene understanding," in IEEE International Conference on Robotics and Automation, vol. 1, no. 6, 2012.
[11] T. Suzuki and T. Oka, "Grasping of unknown objects on a planar surface using a single depth image," in IEEE International Conference on Advanced Intelligent Mechatronics, pp. 572-577, 2016.
[12] M. A. Fischler and R. C. Bolles, "Random sample consensus: A paradigm for model fitting with applications to image analysis and automated cartography," Communications of the ACM, vol. 24, no. 6, pp. 381-395, 1981.
[13] R. B. Rusu, Semantic 3D object maps for everyday robot manipulation. Springer, 2013.
[14] D. Katz, A. Venkatraman, M. Kazemi, J. A. Bagnell and A. Stentz, "Perceiving, learning, and exploiting object affordances for autonomous pile manipulation," Autonomous Robots, vol. 37, no. 4, pp. 369-382, 2014.
[15] I. Lenz, H. Lee and A. Saxena, "Deep learning for detecting robotic grasps," The International Journal of Robotics Research, vol. 34, no. 4-5, pp. 705-724, 2015.
[16] J. Redmon and A. Angelova, "Real-time grasp detection using convolutional neural networks," in IEEE International Conference on Robotics and Automation, pp. 1316-1322, 2015.
[17] A. Krizhevsky, I. Sutskever and G. E. Hinton, "Imagenet classification with deep convolutional neural networks," Neural Information Processing Systems, vol. 25, 2012.
[18] S. Kumra and C. Kanan, "Robotic grasp detection using deep convolutional neural networks," in IEEE International Conference on Intelligent Robots and Systems, 2017.
[19] K. He, X. Zhang, S. Ren and J. Sun, "Deep residual learning for image recognition," in IEEE Conference on Computer Vision and Pattern Recognition, pp. 770-778, 2016.
[20] L. Pinto and A. K. Gupta, "Supersizing self-supervision: Learning to grasp from 50K tries and 700 robot hours," in IEEE International Conference on Robotics and Automation, pp. 3406-3413, 2016.
[21] J. Mahler and K. Goldberg, "Learning deep policies for robot bin picking by simulating robust grasping sequences," in Conference on robot learning, vol. 78, pp. 515-524, 2017.
[22] E. Coumans. "Bullet physics library." https://pybullet.org/wordpress/ (accessed May 17, 2022).
[23] J. Mahler, J. Liang, S. Niyaz, M. Laskey, R. Doan, X. Liu, J. A. Ojea and K. Goldberg, "Dex-net 2.0: Deep learning to plan robust grasps with synthetic point clouds and analytic grasp metrics," Robotics: Science and Systems, 2017.
[24] A. Zeng, S. Song, K.-T. Yu, E. Donlon, F. Hogan, M. Bauza, D. Ma, O. Taylor, M. Liu, E. Romo Grau, N. Fazeli, F. Alet, N. Dafle, R. Holladay, I. Morena, P. Nair, D. Green, I. Taylor, W. Liu and A. Rodriguez, "Robotic pick-and-place of novel objects in clutter with multi-affordance grasping and cross-domain image matching," in IEEE International Conference on Robotics and Automation, pp. 1-8, 2018.
[25] M. Danielczuk, M. Matl, S. Gupta, A. Li, A. Lee, J. Mahler and K. Goldberg, "Segmenting unknown 3D objects from real depth images using mask r-cnn trained on synthetic data," in International Conference on Robotics and Automation, pp. 7283-7290, 2019.
[26] S. Back, J. Kim, R. Kang, S. Choi and K. Lee, "Segmenting unseen industrial components in a heavy clutter using rgb-d fusion and synthetic data," in IEEE International Conference on Image Processing, pp. 828-832, 2020.
[27] 李佳蓮, "以實例切割與夾取點生成卷積類神經網路應用於隨機堆疊物件之分類夾取," 碩士論文, 國立臺灣大學機械系, 台北市, 2020.
[28] L. Yang. "Bpycv: Computer vision utils for blender." https://github.com/DIYer22/bpycv (accessed May 17, 2022).
[29] Y. Bengio, J. Louradour, R. Collobert and J. Weston, "Curriculum learning," in International Conference on Machine Learning, pp. 41-48, 2009.
[30] K. He, G. Gkioxari, P. Dollár and R. Girshick, "Mask r-cnn," in IEEE International Conference on Computer Vision, pp. 2961-2969, 2017.
[31] B. O. Community. "Blender—A 3D modelling and rendering package." https://www.blender.org/about/ (accessed May 17, 2022).
[32] S. Ren, K. He, R. Girshick and J. Sun, "Faster r-cnn: Towards real-time object detection with region proposal networks," Neural Information Processing Systems, vol. 28, 2015.
[33] T.-Y. Lin, P. Dollár, R. Girshick, K. He, B. Hariharan and S. Belongie, "Feature pyramid networks for object detection," in IEEE Conference on Computer Vision and Pattern Recognition, pp. 2117-2125, 2017.
[34] M. Everingham, L. Van Gool, C. K. Williams, J. Winn and A. Zisserman, "The pascal visual object classes (voc) challenge," International Journal of Computer Vision, vol. 88, no. 2, pp. 303-338, 2010.
[35] B. Calli, A. Singh, A. Walsman, S. Srinivasa, P. Abbeel and A. M. Dollar, "The ycb object and model set: Towards common benchmarks for manipulation research," in International Conference on Advanced Robotics, pp. 510-517, 2015.
[36] T. Hodan, P. Haluza, Š. Obdržálek, J. Matas, M. Lourakis and X. Zabulis, "T-LESS: An rgb-d dataset for 6D pose estimation of texture-less objects," in IEEE Winter Conference on Applications of Computer Vision, pp. 880-888, 2017.
[37] B. Drost, M. Ulrich, P. Bergmann, P. Hartinger and C. Steger, "Introducing mvtec itodd-a dataset for 3d object recognition in industry," in IEEE International Conference on Computer Vision Workshops, pp. 2200-2208, 2017.
[38] S. Koch, A. Matveev, Z. Jiang, F. Williams, A. Artemov, E. Burnaev, M. Alexa, D. Zorin and D. Panozzo, "Abc: A big cad model dataset for geometric deep learning," in IEEE Conference on Computer Vision and Pattern Recognition, pp. 9601-9611, 2019.
[39] D. Morrison, P. Corke and J. Leitner, "Learning robust, real-time, reactive robotic grasping," The International journal of robotics research, vol. 39, no. 2-3, pp. 183-201, 2020.
[40] D. Morrison, P. Corke and J. Leitner, "Closing the loop for robotic grasping: A real-time, generative grasp synthesis approach," Robotics: Science and Systems, 2018.
[41] Cornell University. "Cornell grasping dataset." https://www.kaggle.com/datasets/oneoneliu/cornell-grasp (accessed May 17, 2022).
[42] A. Depierre, E. Dellandréa and L. Chen, "Jacquard: A large scale dataset for robotic grasp detection," in IEEE International Conference on Intelligent Robots and Systems, pp. 3511-3516, 2018.
[43] M. Savva, A. X. Chang and P. Hanrahan, "Semantically-enriched 3D models for common-sense knowledge," in IEEE Conference on Computer Vision and Pattern Recognition Workshops, pp. 24-31, 2015.
[44] J. Ku, A. Harakeh and S. L. Waslander, "In defense of classical image processing: Fast depth completion on the cpu," in Conference on Computer and Robot Vision, pp. 16-22, 2018.
[45] J.-Y. Zhu, T. Park, P. Isola and A. A. Efros, "Unpaired image-to-image translation using cycle-consistent adversarial networks," in IEEE International Conference on Computer Vision, pp. 2223-2232, 2017.
[46] H. Cao, G. Chen, Z. Li, J. Lin and A. Knoll, "Lightweight convolutional neural network with gaussian-based grasping representation for robotic grasping detection," arXiv preprint arXiv:2101.10226, 2021.