臺灣博碩士論文加值系統

在物體操縱系統中的標準機器人手臂應用需要額外的視覺感測器來檢測工作空間內任意位置的物體。然而，需要幾個步驟的計算，諸如相機校準，三維物體姿態估計，以及具有非線性方程式和複雜計算的攝影機到機器人座標系變換。
在本論文中，基於立體視覺的物體操縱系統被開發出來使用裝備有夾具的機器人臂自動地將物體放置在工作區內的任意位置。在這個系統中，眼對手架構的立體相機被設置和校準。然後，它通過影像處理演算法用於檢測3D世界座標中的物件。接著，附著在機器人手臂的末端執行器中的三指夾具可以正確地抓住物體。此外，使用自我調整類神經-模糊推理系統方法對攝影機到機器人手臂座標系進行了校準，以減少計算的難度，保持系統的準確性。
進行了幾項實驗包括立體相機校準、3D物件姿態估計、眼對手校準和使用具有夾持器的6自由度機器人臂的物體拾取任務，以證明所提出的方法的有效性。結果顯示，開發的立體相機校準獲得了三角測量過程的合適的外在和內在參數。3D姿態估計已經成功地檢測了攝影機座標系中的物體位置和姿態。此外，手眼校準將攝影機中的物體姿態轉換為機器人手臂座標系。最後，使用6自由度機器人手臂正確地執行了物體拾取和放置任務。
總之，本研究顯示，使用ANFIS進行眼睛對手校準的基於立體視覺的物件操縱系統可以以可接受的錯誤減少複雜的計算。因此，這些方法可適用於其他物件操縱系統。

The utilization of a standard robot manipulator in object manipulation systems requires additional vision sensors to detect the object in an arbitrary location within the workspace. However, there are several steps involved such as camera calibration, three-dimension (3D) object pose estimation, and transformation of camera to robot coordinate frame which has a non-linear equation and complicated calculation.
In this dissertation, the stereo vision based object manipulation system was developed to pick up and place the object in arbitrary location within the workspace automatically using the robot arm equipped with gripper. In this system, the stereo camera in eye-to-hand configuration was setup and calibrated. Then, it is used to detect the object in 3D world coordinate through the image processing algorithms. In addition, the three-finger gripper attached in the end-effector of robot arm grasps the object properly. Moreover, the camera to robot arm coordinate frame was calibrated using Adaptive Neuro-Fuzzy Inference System (ANFIS) method to reduce the difficulties of computation and keep the accuracy of the system.
Several experiments were performed to demonstrate the effectiveness of the proposed methods included the stereo camera calibration, 3D object pose estimation, eye to hand calibration and the object pick-and-place task using 6 DOF robot arm equipped with gripper. The results revealed that the developed stereo camera calibration obtained suitable extrinsic and intrinsic parameters for triangulation process. The 3D pose estimation has successfully detected the object position and orientation in camera coordinate frame. Furthermore, the eye to hand calibration transformed the object posture in camera to robot arm coordinate frame. Finally, the object pick-and-place task has been performed properly using 6 DOF robot arm.
In conclusion, our study demonstrated that the stereo vision-based object manipulation system with eye to hand calibration using ANFIS can reduce the complicated computation with acceptable errors. Thus, these methods might be applicable to other object manipulation system.

Abstract iv
摘要 v
ACKNOWLEDGMENTS vi
TABLE OF CONTENTS vii
LIST OF TABLES ix
LIST OF FIGURES x
LIST OF SYMBOLS xii
CHAPTER 1: INTRODUCTION 1
1.1. Study background 1
1.2. Objectives of the research 2
1.3. Dissertation outline 3
CHAPTER 2: RELATED LITERATURE REVIEW 4
2.1. Robot manipulator 4
2.1.1. Mitsubishi RV-3SD 5
2.1.2. Robotiq 3-finger gripper 7
2.2. Stereo vision-based object manipulation system 8
2.2.1. Image parameters 9
2.2.2. Pinhole camera model 10
2.2.3. Camera calibration 11
2.2.4. Triangulation 12
2.3. Eye to Hand Calibration 12
2.4. Adaptive neuro-fuzzy inference system 13
2.4.1. ANFIS architecture 14
2.4.2. Learning algorithm 16
CHAPTER 3: MATERIALS AND METHODS 17
3.1. System overview 17
3.2. Stereo camera development and calibration 18
3.2.1. Stereo camera development 19
3.2.2. Stereo camera calibration 20
3.3. Stereo vision-based object detection and pose estimation 21
3.3.1. Stereo vision configuration 23
3.3.2. Object feature extraction and position estimation 25
3.3.3. Object orientation estimation 26
3.4. Eye to Hand Calibration using ANFIS 26
3.4.1. ANFIS structure of eye to hand calibration 27
3.4.2. Training data generation 28
3.5. Object pick and place task 28
CHAPTER 4: EXPERIMENTAL RESULTS AND DISCUSSION 29
4.1. Stereo camera calibration 30
4.2. Object detection and pose estimation 32
4.3. Eye to Hand Calibration 36
4.4. System performance examination 40
CHAPTER 5: CONCLUSIONS AND FUTURE WORKS 44
5.1. Conclusions 44
5.2. Future works 45
REFERENCES 46

[1]J. R. Hagerty, "Meet the New Generation of Robots for Manufacturing," The Wall street Journal, 2015.
[2]P. Ramon Soria, B. Arrue, and A. Ollero, "Detection, Location and Grasping Objects Using a Stereo Sensor on UAV in Outdoor Environments," Sensors, vol. 17, p. 103, 2017.
[3]H. Alzarok, S. Fletcher, and A. Longstaff, "3D Visual Tracking of an Articulated Robot in Precision Automated Tasks," Sensors, vol. 17, p. 104, 2017.
[4]L. Pérez, Í. Rodríguez, N. Rodríguez, R. Usamentiaga, and D. F. García, "Robot Guidance Using Machine Vision Techniques in Industrial Environments: A Comparative Review," Sensors (Basel, Switzerland), vol. 16, p. 335, 03/05 01/13/received 02/26/accepted 2016.
[5]A. Wilson, "3D expands the dimensions of vision systems," Vis. Syst. January 2016, vol. Available online: http://www.vision-systems.com/articles/print/volume-21/issue-1/features/3d-expands-the-dimensions-of-vision-systems.html (accesed on 15 June 2106), 2016.
[6]P. Corke, Robotics, vision and control: fundamental algorithms in MATLAB vol. 73: Springer Science & Business Media, 2011.
[7]T. Borangiu and A. Dumitrache, Robot arms with 3D vision capabilities: INTECH Open Access Publisher, 2010.
[8]B. Fevery, B. Wyns, L. Boullart, J. R. Llata García, and C. Torre Ferrero, "Industrial robot manipulator guarding using artificial vision," Robot Vision, pp. 429-454, 2010.
[9]Point_Grey, "Stereo Vision Introduction and Applications," Technical Application Note (TAN2008005), vol. Revised November 3, 2015.
[10]S. Y. Nof, Springer handbook of automation: Springer Science & Business Media, 2009.
[11]E. S. Kheng, A. H. A. Hassan, and A. Ranjbaran, "Stereo vision with 3D coordinates for robot arm application guide," in Sustainable Utilization and Development in Engineering and Technology (STUDENT), 2010 IEEE Conference on, 2010, pp. 102-105.
[12]J.-Y. Bouguet, "Matlab camera calibration toolbox," http://www.vision.caltech.edu/bouguetj/calib_doc, 2015.
[13]J. Miseikis, K. Glette, O. J. Elle, and J. Torresen, "Automatic Calibration of a Robot Manipulator and Multi 3D Camera System," arXiv preprint arXiv:1601.01566, 2016.
[14]I. Siradjuddin, L. Behera, T. M. McGinnity, and S. Coleman, "Image-Based Visual Servoing of a 7-DOF Robot Manipulator Using an Adaptive Distributed Fuzzy PD Controller," IEEE/ASME Transactions on Mechatronics, vol. 19, pp. 512-523, 2014.
[15]R. Y. Tsai and R. K. Lenz, "A new technique for fully autonomous and efficient 3D robotics hand/eye calibration," IEEE Transactions on Robotics and Automation, vol. 5, pp. 345-358, 1989.
[16]F. Dornaika and R. Horaud, "Simultaneous robot-world and hand-eye calibration," IEEE Transactions on Robotics and Automation, vol. 14, pp. 617-622, 1998.
[17]Z. Tao, "Visual Servoing for a Non-Reactive Industrial Manipulator," Vanderbilt University, 2015.
[18]H. Wu, W. Tizzano, T. T. Andersen, N. A. Andersen, and O. Ravn, "Hand-Eye Calibration and Inverse Kinematics of Robot Arm using Neural Network," in Robot Intelligence Technology and Applications 2, ed: Springer, 2014, pp. 581-591.
[19]K. Daniilidis, "Hand-eye calibration using dual quaternions," The International Journal of Robotics Research, vol. 18, pp. 286-298, 1999.
[20]R. Y. Tsai and R. K. Lenz, "Real time versatile robotics hand/eye calibration using 3D machine vision," in Robotics and Automation, 1988. Proceedings., 1988 IEEE International Conference on, 1988, pp. 554-561.
[21]S. Jafari and R. Jarvis, "Robotic hand-eye coordination: from observation to manipulation," in Hybrid Intelligent Systems, 2004. HIS '04. Fourth International Conference on, 2004, pp. 20-25.
[22]J.-G. Juang, Y.-J. Tsai, and Y.-W. Fan, "Visual Recognition and Its Application to Robot Arm Control," Applied Sciences, vol. 5, p. 851, 2015.
[23]S. Abe, Neural networks and fuzzy systems: theory and applications: Springer Science & Business Media, 2012.
[24]J.-S. R. Jang, "ANFIS: adaptive-network-based fuzzy inference system," Systems, Man and Cybernetics, IEEE Transactions on, vol. 23, pp. 665-685, 1993.
[25]European_Commission, "Robotics," HORIZON 2020 2016.
[26]M. Hägele, K. Nilsson, J. N. Pires, and R. Bischoff, "Industrial robotics," in Springer handbook of robotics, ed: Springer, 2016, pp. 1385-1422.
[27]L. Sciavicco and B. Siciliano, "Modelling and Control of Robot Manipulators," ed: IOP Publishing, 2000.
[28]F. L. Lewis, D. M. Dawson, and C. T. Abdallah, Robot manipulator control: theory and practice: CRC Press, 2004.
[29]M. W. Spong and M. Vidyasagar, Robot dynamics and control: John Wiley & Sons, 2008.
[30]R. N. Jazar, Theory of applied robotics: kinematics, dynamics, and control: Springer Science & Business Media, 2010.
[31]I. R. Mitsubishi, "CRnQ/CRnD Controller INSTRUCTION MANUAL Detailed explanations of functions and operations,", 2010, Tokyo2010.
[32]S. Kucuk and Z. Bingul, Robot kinematics: Forward and inverse kinematics: INTECH Open Access Publisher, 2006.
[33]M. W. Spong, S. Hutchinson, and M. Vidyasagar, Robot modeling and control vol. 3: Wiley New York, 2006.
[34]E. Samson, "Robotiq 3-Finger Adaptive Robot Gripper Instruction Manual," 2013.
[35]M. Abdelhamid, "EXTRACTING DEPTH INFORMATION FROM STEREO VISION SYSTEM, USING A CORRELATION AND A FEATURE BASED METHODS," 2011.
[36]B. Rasolzadeh, M. Björkman, K. Hübner, and D. Kragic, "An active vision system for detecting, fixating and manipulating objects in the real world," The International Journal of Robotics Research, vol. 29, pp. 133-154, 2010.
[37]C. Wöhler, 3D computer vision: efficient methods and applications: Springer Science & Business Media, 2012.
[38]T. Kotthauser and G. F. Mauer, "Vision-based autonomous robot control for pick and place operations," in Advanced Intelligent Mechatronics, 2009. AIM 2009. IEEE/ASME International Conference on, 2009, pp. 1851-1855.
[39]S. Abraham and W. Förstner, "Fish-eye-stereo calibration and epipolar rectification," ISPRS Journal of photogrammetry and remote sensing, vol. 59, pp. 278-288, 2005.
[40]J. Heikkila and O. Silven, "A four-step camera calibration procedure with implicit image correction," in Computer Vision and Pattern Recognition, 1997. Proceedings., 1997 IEEE Computer Society Conference on, 1997, pp. 1106-1112.
[41]Z. Zhang, "A flexible new technique for camera calibration," Pattern Analysis and Machine Intelligence, IEEE Transactions on, vol. 22, pp. 1330-1334, 2000.
[42]R. Tsai, "A versatile camera calibration technique for high-accuracy 3D machine vision metrology using off-the-shelf TV cameras and lenses," IEEE Journal on Robotics and Automation, vol. 3, pp. 323-344, 1987.
[43]T. A. Clarke and J. G. Fryer, "The development of camera calibration methods and models," The Photogrammetric Record, vol. 16, pp. 51-66, 1998.
[44]J. Heikkila, "Geometric camera calibration using circular control points," IEEE Transactions on pattern analysis and machine intelligence, vol. 22, pp. 1066-1077, 2000.
[45]O. Faugeras, Three-dimensional computer vision: a geometric viewpoint: MIT press, 1993.
[46]R. I. Hartley and P. Sturm, "Triangulation," Computer Vision and Image Understanding, vol. 68, pp. 146-157, 1997/11/01 1997.
[47]R. A. Aliev and B. G. Guirimov, Type-2 Fuzzy Neural Networks and Their Applications: Springer, 2014.
[48]M. Buragohain and C. Mahanta, "A novel approach for ANFIS modelling based on full factorial design," Applied Soft Computing, vol. 8, pp. 609-625, 2008/01/01/ 2008.
[49]T. Nguyen, D. Slaughter, N. Max, J. Maloof, and N. Sinha, "Structured Light-Based 3D Reconstruction System for Plants," Sensors, vol. 15, p. 18587, 2015.
[50]Z. Zhang, Y. Matsushita, and Y. Ma, "Camera calibration with lens distortion from low-rank textures," in Computer Vision and Pattern Recognition (CVPR), 2011 IEEE Conference on, 2011, pp. 2321-2328.
[51]Q. Jiadi, Z. Fuhai, F. Yili, and G. Shuxiang, "Approach movement of redundant manipulator using stereo vision," in Robotics and Biomimetics (ROBIO), 2014 IEEE International Conference on, 2014, pp. 2489-2494.
[52]Z. Liu and T. Chen, "Distance Measurement System Based on Binocular Stereo Vision," in Artificial Intelligence, 2009. JCAI '09. International Joint Conference on, 2009, pp. 456-459.
[53]Mathworks. Measure properties of image regions [Online]. Available: https://www.mathworks.com/help/images/ref/regionprops.html
[54]Taryudi and M.-S. Wang, "Eye to hand calibration using ANFIS for stereo vision-based object manipulation system," Microsystem Technologies, pp. 1-13, 2017.