臺灣博碩士論文加值系統

近年因為各種深度學習的神經網路架構出現，與硬體的運算能力的快速進步而帶動電腦視覺的快速發展且將電腦視覺的技術應用到許多不同領域，如人臉辨識、物件偵測、車輛追蹤、街景分析等。在我們的方法中，我們使用Openpose人體骨架節點偵測架構，抓取臉部及五官節點之後，建立資料集，並給入標籤，最後帶入我們建立的神經網路模型，目的是從臉部及五官節點，預測此人物的視角方向，並且在最後加入物件偵測的技術，推算人物正在觀測的物件。在實作此方法之後，發現我們的方法可以正確的推算人物正在觀測的物品，且在多鏡頭可以獲得更多資訊的狀況下，效果會比單鏡頭更好，可以更精準的推算正在被觀測的物件，在距離限制上也比以往的方法更好，約可以偵測距離鏡頭6公尺內人物的視角，且可以進行多人同時偵測。

In recent years, the emergence of various deep learning neural network architectures and the rapid advancement of hardware computing capabilities have driven the rapid development of computer vision and applied computer vision technology to many different fields, such as face recognition, object detection, vehicle tracking, Street view analysis, etc. In this study, we use OpenPose to capture facial and facial features nodes, create a data set and label it, and finally bring into the neural network model we created. The purpose is to predict the direction of the person's line of sight from the face and facial features nodes, and finally add object detection technology to calculate the object that the person is observing. After implementing this method, we found that this method can correctly estimate the objects that the character is observing. In the situation where multiple lenses can obtain more information, the effect will be better than that of a single lens, which can more accurately estimate the objects being observed.

目錄
摘要 ii
Abstract iii
圖目錄 vi
第一章 研究背景 1
第二章 研究動機 3
1.視線方向推算的應用與物品辨識 3
2.與無人機的結合應用 3
3.商店中的應用 3
4.居家安全的應用 3
第三章 文獻回顧 4
1.人體姿態評估 4
1.1 Pose Machine 4
1.2 Convolutional pose machines (CPM) 5
1.3 Openpose 架構 6
2.視線方向檢測 8
2.1 遙測式眼動儀 8
2.2 頭戴式眼動儀 9
2.3 使用一般鏡頭 9
2.4 A Single Camera Eye-Gaze Tracking System with Free Head Motion 10
2.5 Head pose estimation 11
2.6 Head pose estimation(2) 12
2.7 Appearance-based gaze estimation 13
3.物件偵測 14
3.1 R-CNN 14
3.2 Fast R-CNN 14
3.3 Faster R-CNN 15
3.4 You Only Look Once: Unified, Real-Time Object Detection 16
3.5 YOLO9000:Better,Faster,Stronger 19
3.6 YOLOv3: An Incremental Improvement 21
第四章 研究方法 23
1.硬體架構說明 23
2.研究流程與演算法介紹 24
2.1 影像輸入(Image Input) 24
2.2 相機校正(Camera Calibration) 25
2.3 抓取臉部節點(Openpose Face node Output) 26
2.4 訓練神經網路模型(Neural Network) 29
2.5 模型感興趣區域(Region of Interest) 30
2.6 模型結合物件偵測(Gaze predict + YOLO Object detection) 31
2.7 多鏡頭交集(Multiple Camera Intersection) 32
2.8 視角內物件偵測(Gaze Object Detection) 32
3.實驗設計 33
3.1 Openpose距離能力測試 33
3.2 Openpose角度能力測試 38
3.3 神經網路測試 42
3.4 視角偵測模型測試 43
3.5 視角偵測模型結合物件偵測 45
第五章 結論 49
參考文獻 50
附錄 52

1. Fukushima, K., Neocognitron: A self-organizing neural network model for a mechanism of pattern recognition unaffected by shift in position. Biological cybernetics, 1980. 36(4): p. 193-202.
2. Cao, Z., et al., OpenPose: realtime multi-person 2D pose estimation using Part Affinity Fields. arXiv preprint arXiv:1812.08008, 2018.
3. Hennessey, C., B. Noureddin, and P. Lawrence. A single camera eye-gaze tracking system with free head motion. in Proceedings of the 2006 symposium on Eye tracking research & applications. 2006.
4. Tobii Pro Glasses 2 wearable eye tracker. Available from: https://www.tobiipro.com/product-listing/tobii-pro-glasses-2/.
5. Tobii Pro Spectrum remote eye tracker, sync with biometric data. Available from: https://www.tobiipro.com/product-listing/tobii-pro-spectrum/.
6. Ramakrishna, V., et al. Pose machines: Articulated pose estimation via inference machines. in European Conference on Computer Vision. 2014. Springer.
7. Wei, S.-E., et al. Convolutional pose machines. in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2016.
8. Cao, Z., et al. Realtime multi-person 2d pose estimation using part affinity fields. in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2017.
9. Simon, T., et al. Hand keypoint detection in single images using multiview bootstrapping. in Proceedings of the IEEE conference on Computer Vision and Pattern Recognition. 2017.
10. How do Tobii Eye Trackers work? - Learn more with Tobii Pro. Available from: https://www.tobiipro.com/learn-and-support/learn/eye-tracking-essentials/how-do-tobii-eye-trackers-work/.
11. PCEye Plus Track & Learn - Tobii Dynavox. Available from: https://www.tobiidynavox.com/en-us/devices/eye-gaze-devices/pceye-plus-track-learn-vr/.
12. Sewell, W. and O. Komogortsev. Real-time eye gaze tracking with an unmodified commodity webcam employing a neural network. in CHI'10 Extended Abstracts on Human Factors in Computing Systems. 2010. ACM.
13. Meyer, G.P., et al. Robust model-based 3d head pose estimation. in Proceedings of the IEEE international conference on computer vision. 2015.
14. Liu, X., et al. 3D head pose estimation with convolutional neural network trained on synthetic images. in 2016 IEEE International Conference on Image Processing (ICIP). 2016. IEEE.
15. Zhang, X., et al. Appearance-based gaze estimation in the wild. in Proceedings of the IEEE conference on computer vision and pattern recognition. 2015.
16. Girshick, R., et al. Rich feature hierarchies for accurate object detection and semantic segmentation. in Proceedings of the IEEE conference on computer vision and pattern recognition. 2014.
17. Girshick, R. Fast r-cnn. in Proceedings of the IEEE international conference on computer vision. 2015.
18. Ren, S., et al. Faster r-cnn: Towards real-time object detection with region proposal networks. in Advances in neural information processing systems. 2015.
19. Redmon, J., et al. You only look once: Unified, real-time object detection. in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2016.
20. Redmon, J. and A. Farhadi. YOLO9000: better, faster, stronger. in Proceedings of the IEEE conference on computer vision and pattern recognition. 2017.
21. Redmon, J. and A.J.a.p.a. Farhadi, Yolov3: An incremental improvement. 2018.
22. Zhang, Z.J.I.T.o.p.a. and m. intelligence, A flexible new technique for camera calibration. 2000. 22(11): p. 1330-1334.
23. McCulloch, W.S. and W. Pitts, A logical calculus of the ideas immanent in nervous activity. The bulletin of mathematical biophysics, 1943. 5(4): p. 115-133.
24. Machinery, C., Computing machinery and intelligence-AM Turing. Mind, 1950. 59(236): p. 433.