臺灣博碩士論文加值系統

深度學習技術近年來獲得相當大的進步與關注，從影像、語音辨識等甚至能在圍棋上戰勝人類，皆讓我們看到深度學習技術帶來的巨大潛力與未來影響。如今深度學習中的捲積神經網路架構已被大量使用於影像的相關辨識應用上，舉凡是多物件識別與分類、物件追蹤、語義分割等應用皆獲得了優秀的成果。但隨著應用的難度增加，以及網路架構不斷的推陳出新，基於捲積神經網路的新型深度學習網路架構往往為了能提高其識別率，其網路層數相當龐大，造成需要大量標記資料及冗長的訓練運算時間，在嵌入式系統實現上更顯不易。但其實並非所有問題都需要如此深層的神經網路，相對單純環境之辨識問題上，可能在前幾層的神經網路就已經收斂且就可以得到較好的辨識結果，反而會造成硬體空間與運算時間的浪費。故本論文提出一新型具自主彈性伸縮捲積自編碼器之半監督式學習架構，並實際應用於圖像辨識及分類所使用。
本論文提出之具自主彈性伸縮捲積自編碼器網路架構，其主要是基於Actor-Critic強化學習方法，藉由Actor-Critic架構中不斷嘗試並訓練學習，並生成其捲積核進而找到最佳的網路深層架構，以達到可根據問題複雜度使整體網路架構具備自主伸縮功能之重大貢獻。本論文所提出之新型自主彈性伸縮捲積自編碼器網路架構其主要分為四大部分: 自主彈性伸縮單元(auto-elastic unit)、資料生成單元(data generation unit)、記憶單元(memory unit)、學習者單元(learner unit)。首先自主彈性伸縮單元(auto-elastic unit)主要是基於所提出之Actor-Critic編碼(Actor-Critic encoder, ACE)網路架構，其中Actor藉由當前的影像/特徵圖以及過去使用過的捲積核產生一生成編碼的策略(policy)，並透過此策略決定產生的編碼結果，而Critic則負責透過回報評估該策略的優良性，並作為更新策略之依據。接著透過記憶單元(memory unit)以儲存自主彈性伸縮單元所產生的編碼結果及神經網路訓練過程中所需的資訊，如回報資訊、捲積核歷史資訊、特徵圖等；並搭配資料生成單元(data generation unit)以自動生成訓練樣本資料，達到半監督式學習之功效。最後透過學習者單元(learner unit)進行訓練學習，於訓練過程中不斷自主生成捲積核與訓練樣本，並進而達到可自主伸縮神經網路之深度，故可使其根據影像複雜度來自適應獲取出最適合的網路深度架構。最後本論文亦透過大量模擬驗證、量化分析及比較，可有效證明本論文所提出之新穎網路架構之可行性、圖像辨識結果及分類效果。

More and more attention of the deep learning for image and speech recognition, even artificial intelligence (AI) beats humans in ancient game of Go has been attracted with the development of computer and electronic devices. In neural networks, deep learning with convolutional neural network (CNN) has been widely utilized in numerous applications of automatic image recognition, such as multiple object detection and classification, object tracking, sematic segmentation, and most of them have achieved outstanding results. However, with the applications are becoming more complex and various deep learning frameworks are constantly evolving, the neural networks need multiple layers in order to learn more detailed and more abstractions relationships within the data, and then it can enhance the efficiency of the recognition. Some popular types of deep neural networks often requires lots of labeled data as well as huge computing power and for this reason they are not easily to be integrated into embedded system. Indeed, sometimes problems don't need more layer neural networks to solve it, the relatively simple environment may have achieved better results using first few layers merely. Thus, it causes that the excess memory goes to waste and tedious and time-consuming process. Hence, the thesis presents a novel semi-supervised learning framework with an auto-elastic convolutional auto-encoder for image detection and classification.
The proposed an auto-elastic convolutional auto-encoder learning framework is based on the Actor-Critic reinforcement learning approach. The Actor-Critic is the learning of a mapping from situations to actions, which learns from the consequences of its actions, rather than from being unequivocally instructed and it selects its actions on basis of its past experiences and also by new choices. It is a trial and error learning essentially. The main contribution of this study is to repeatedly generate the convolution kernels based on Actor-Critic algorithms and then determine the optimal number of layers of neural network framework so that the whole neural network framework can achieve self-elasticity function depending on task complexity. The novel Auto-Elastic Convolutional Auto-Encoder framework is mainly divided into four parts: auto-elastic unit, data generation unit, memory unit, learner unit. First, the auto-elastic unit is based on the proposed Actor-Critic encoder (ACE) algorithms. The Actor generates a coding policy (action) by using current image/feature map and historical convolution kernels information. The Critic as a value estimator, whereas the actor attempts to select actions (coding policy) based on the value function estimated by the critic. Next, memory unit is used to store the coding results and historical information from the auto-elastic unit, such as corresponding rewards, historical convolution kernels, feature maps, etc., and combine a data generation unit to generate training sample data automatically. Final, the learner unit is proposed as for the proposed semi-supervised learning framework training. Extensively simulation, quantitative analysis and comparison results demonstrate the feasibility and efficiency of the proposed auto-elastic convolutional auto-encoder learning framework for digit recognition and classification.

誌謝 i
中文摘要 ii
Abstract iv
目錄 vi
圖目錄 ix
表目錄 xii
第一章 緒論 1
1.1 研究背景與動機 1
1.2 國內外相關研究 2
1.3 研究貢獻 8
第二章 深度強化學習與方法簡介 10
2.1 深度學習基本介紹 10
2.1.1 捲積層(convolution layer) 10
2.1.2 池化層(pooling layer) 13
2.1.4 全連接層(fully connected layer)與Softmax 14
2.2 遞歸神經網路(Recurrent Neural Networks, RNN) 15
2.3 批量訓練(Batch Training) 18
2.4 強化學習(Reinforcement Learning) 22
2.4.1 馬可夫決策過程(Markov Decision Processes, MDP) 24
2.4.2 策略迭代(Policy Iteration)與價值迭代(Value Iteration) 28
2.5 深度強化學習(Deep Reinforcement Learning) 39
2.5.1 深度強化學習的基本訓練方法 43
2.5.2 近端策略優化算法(Proximal Policy Optimization) 47
2.6 變分自編碼器(Variational Auto-Encoder, VAE)介紹 53
2.6.1自編碼器(Auto-Encoder, AE) 54
2.6.2變分自編碼器(Variational Auto-Encoder, VAE) 60
2.7 生成對抗網路(Generative Adversarial Networks) 65
第三章 具自主彈性伸縮捲積自編碼器之半監督式學習架構於圖像分類設計 69
3.1 系統架構 69
3.2 彈性自主伸縮單元(Auto-Elastic Unit) 71
3.2.1 Actor-Critic編碼網路架構(ACE)設計 73
3.2.2 自主彈性伸縮網路停止邏輯單元 80
3.2.3 解碼器、分類器與判別器設計 82
3.2.4 回報設計 84
3.3 記憶單元(Memory Unit) 85
3.4 資料生成單元(Data Generation Unit) 87
3.5學習者單元(Learner Unit) 88
第四章 實驗結果與分析 93
4.1 學習者單元(Learner Unit)實驗分析 93
4.1.1 策略學習者實驗分析 94
4.1.2 解碼學習者及分類學習者實驗分析 97
4.1.3 過擬合現象分析 98
4.2 二維編碼實驗分析 99
4.3 量化分析 102
4.3.1 歷史資訊觀測網路實驗 103
4.3.2 具自主彈性伸縮功能網路實驗 104
4.3.3 監督式與半監督量化分析 104
4.3.4 資料量增加量化分析 105
4.3.5 其他模型比較 105
第五章 結論與未來工作 106
5.1 結論 106
5.2 未來工作 106
參考文獻 108
附 錄 112
附錄A: Variational Auto-Encoder損失函數推導[27] 112
附錄B: Variational Auto-Encoder 編碼器損失函數推導[27] 113
附錄C: 生成對抗網路證明[28] 114
附錄D: 多維高斯累積分布函數推導與累積分布演算法 115
附錄E: 本論文具自主彈性伸縮捲積自編碼器之偽代碼 121

[1]K. He, X. Zhang, S. Ren, and J. Sun, “Delving deep into rectifiers: Surpassing human-level performance on imagenet classification,” IEEE Conf. Computer Vision (ICCV), pp. 1026-1034, 2015
[2]N. Datal and Bill Triggs, “Histograms of oriented gradients for human detection,” IEEE Conf. Computer Vision and Pattern Recognition (CVPR), pp. 886-893, 2005
[3]D. G. Lowe, “Distinctive image features from scale-invariant keypoints,” Int. J. Computer Vision (IJCV), pp. 91-110, 2004.
[4]C. Adam, and N. Y. Andrew, “Learning feature representations with K-means,” Neural Networks: Tricks of the Trade, Springer, pp. 561-580, 2012.
[5]W. H. Press, S. A. Teukolsky, W. T. Vetterling, and B. P. Flannery, Section 16.1. Gaussian mixture models and K-means clustering, Numerical Recipes: The Art of Scientific Computing 3rd. New York: Cambridge University Press, 2007.
[6]Y. Lecun, L. Bottou, Y. Bengio and P. Haffner, “Gradient-based learning applied to document recognition,” Proc. the IEEE, pp. 2278-2324, 1998.
[7]A. Krizhevsky, I. Sutskever, and G. E. Hinton. “Imagenet classification with deep convolutional neural networks,” in Proc. Conf. Neural Information Processing Syst. (NIPS), pp. 1-9, 2012.
[8]K. Simonyan, and A. Zisserman. “Very deep convolutional networks for large-scale image recognition,” Int. Conf. Learning Representations, pp. 1-14, 2015.
[9]C. Szegedy, W. Liu, Y. Jia, P. Sermanet, S. Reed, D. Anguelov, D. Erhan, V. Vanhoucke, and A. Rabinovich. “Going deeper with convolutions,” IEEE Conf. Computer Vision and Pattern Recognition (CVPR), pp. 1-9, 2015.
[10]J. Redmon, S. Divvala, R. Girshick and A. Farhadi. “You Only Look Once: Unified, Real-Time Object Detection,” IEEE Conf. Computer Vision and Pattern Recognition (CVPR), pp. 779-788, 2016.
[11]W. Liu, D. Anguelov, D. Erhan, S. Christian, S. Reed, C. Y. Fu, and A. C. Berg. “SSD: single shot multibox detector,” in Proc. Eur. Conf. Computer Vision, pp. 1-17, 2016.
[12]S. Ren, K. He, R. Girshick, and J. Sun. “Faster R-CNN: Towards real-time object detection with region proposal networks,” IEEE Trans. Pattern Analysis and Machine Intelligence, pp. 1-14, 2016.
[13]J. Long, E. Shelhamer, and T. Darrell. “Fully convolutional networks for semantic segmentation,” IEEE Conf. Computer Vision and Pattern Recognition (CVPR), pp. 3431-3440, 2015
[14]V. Badrinarayanan, A. Kendall, R. Cipolla, and S. Member. “SegNet: a deep convolutional encoder-decoder architecture for image segmentation,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 39, no. 13, pp. 2481-2495, 2017.
[15]H. Zhao, J. Shi, X. Qi, X. Wang, and J. Jia. “Pyramid scene parsing network,” IEEE Conf. Computer Vision (ICCV), pp. 6230-6239, 2017.
[16]K. He, G. Gkioxari, P. Dollar, and R. Girshick. “Mask R-CNN,” IEEE Conf. Computer Vision (ICCV), pp. 2980-2988, 2017.
[17]A. G. Howard, M. Zhu, B. Chen, D. Kalenichenko, W. Wang, T. Weyand, M. Andreetto, and H. Adam. “Mobilenets: Efficient convolutional neural networks for mobile vision applications,” CoRR, abs/1704.04861, 2017.
[18]V. Mnih, K. Kavukcuoglu, D. Silver, A. A. Rusu, J. Veness, M. G. Bellemare, A. Graves, M. Riedmiller, A. K. Fidjeland, G. Ostrovski, S. Petersen, C. Beattie, A. Sadik, I. Antonoglou, H. King, D. Kumaran, D. Wierstra, S. Legg, and D. Hassabis, “Human-level control through deep reinforcement learning,” Nature, vol. 518, pp. 529-533, 2016.
[19]D. Silver, A. Huang, C. J. Maddison, A. Guez, L. Sifre, G. V. D. Driessche, J. Schrittwieser, I. Antonoglou, V. Panneershelvam, M. Lanctot, S. Dieleman, D. Grewe, J. Nham, N. Kalchbrenner, I. Sutskever, T. Lillicrap, M. Leach, K. Kavukcuoglu, T. Graepel, and D. Hassabis, “Mastering the game of go with deep neural networks and tree search,” Nature, vol. 529, no.1, pp. 484-489, 2016.
[20]S. David, L. Guy, H. Nicolas, D. Thomas, W. Daan, and R. Martin, “Deterministic policy gradient algorithms,” Int. Conf. Machine Learning (ICML), pp. 387-395, 2014.
[21]T. P. Lillicrap, J. J. Hunt, A. Pritzel, and N. Heess, “Continuous control with deep reinforcement learning,” in Proc. Int. Conf. Learning Representations (ICLR), 2016.
[22]J. Schulman, S. Levine, P. Abbeel, M. I Jordan, and P. Moritz, “Trust region policy optimization,” Int. Conf. Machine Learning (ICML), pp. 1889-1897, 2015.
[23]J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and Oleg Klimov, “Proximal policy optimization algorithms,” arXiv preprint arXiv:1707.06347, 2017.
[24]B. Zoph, and Q. V. Le. “Neural architecture search with reinforcement learning. In International Conference on Learning Representations,” in Proc. Int. Conf. Learning Representations (ICLR), 2017.
[25]D. Hand, H. Mannila, and P. Smyth, Principles of Data Mining, The MIT Press, 2001.
[26]D. E. Rumelhart, G. E. Hinton, and R. J. Williams, Learning internal representations by error propagation, The MIT Press, 1986.
[27]D. P. Kingma, and M. Welling, “Auto-encoding variational bayes.” in Proc. Int. Conf. Learning Representations (ICLR), 2014.
[28]I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio, “Generative adversarial nets,” in Proc. Conf. Neural Information Processing Syst. (NIPS), pp. 2672-2680, 2014.
[29]A. Makhzani, J. Shlens, N. Jaitly, and I. Goodfellow, “Adversarial autoencoders,” in Proc. Int. Conf. Learning Representations (ICLR), 2016.
[30]X. Glorot, A. Bordes, and Y. Bengio, “Deep sparse rectifier neural networks,” Int. Conf. Artificial Intelligence and Statistics, pp. 315-323, 2011.
[31]B. Xu, N. Wang, T. Chen, M. Li, “Empirical evaluation of rectified activations in convolutional network,” arXiv preprint arXiv:1505.00853, 2015.
[32]S. Hochreiter, and J. Schmidhuber, “Long short-term memory,” Neural Computation, vol. 9, pp.1735-1780, 1997.
[33]J. Chung, C. Gulcehre, K.Cho and Y. Bengio, “Empirical evalutioan of gated recurrent neural networks on sequence modeling,” arXiv preprint arXiv:1412.3555, 2014.
[34]R. E. Caflisch, "Monte Carlo and quasi-Monte Carlo methods". Acta Numerica. Cambridge University Press, pp. 1-49, 1998.
[35]R. Sutton, and A. Barto, Reinforcement Learning: an Introduction, The MIT Press, 1998.
[36]G. E. P. Box, and Mervin E. Muller, “A note on the generation of random normal deviates,” The Annals of Mathematical Statistics, pp. 610-611, 1958.
[37]J. Zhang, A. Messac, J. Zhang, and S. Chowdhury, “Improving the accuracy of surrogate models using inverse transform sampling,” AIAA/ASME/ASCE/AHS/ASC Structures, Structural Dynamics and Materials Conf. , pp. 1-16, 2012.
[38]J. Schulman, P. Moritz, S. Levine, M. Jordan, and P. Abbeel, “High-dimensional continuous control using generalized advantage estimation,” in Proc. Int. Conf. Learning Representations (ICLR), 2016.
[39]P. Vincent, “A connection between score matching and denoising autoencoders,” J. Neural Computation, vol. 23, no. 7, pp. 1661-1674, 2011.
[40]S. Sabour, N. Frosst, and G. E. Hinton, “Dynamic routing between capsules,” in Proc. Conf. Neural Information Processing Syst. (NIPS), 2017.
[41]I. Tolstikhin, O. Bousquet, S. Gelly, B. Schoelkopf, “Wasserstein auto-encoders,” arXiv preprint arXiv:1711.01558, 2017.
[42]G. Grimmett, D. Stirzaker, Probability and Random Processes, Oxford, England: Oxford University Press, 2009.