臺灣博碩士論文加值系統

在影像辨識處理範疇內，特徵擷取是個重要的課題。然而組內變異性的存在，使得設計特徵擷取器變得更加困難；再者，因人工設計的特徵擷取器難以應付新狀況，因此近來深度學習能自動地從數據裡面學習到特徵而吸引了廣泛的注意。在這篇文章中，我們提出拓增型深度類神經網路的架構，架構中結合了卷積類神經網路的感受域與子空間的概念。將子空間應用在深度網路是個新概念，能提供資料的不同面向。其中子空間是由適應子空間自組織映射圖所訓練出的基底向量所展開，基底向量不僅作為轉換方程式用來取得維度分量，更定義了感受域來擷取資料的基本特徵，轉換過程中遺失的資料稀少且不會扭曲資料間的關聯性。甚者，考慮到現實資料十分地複雜，我們使用到多個平行的框架來實現多個子空間的概念，希望藉由多個子空間提供我們資料多面向且高層次的抽象表示讓網路更佳的穩健，減少組內變異性所造成的問題。最後我們將提出的拓增型深度類神經網路應用在MNIST及COIL-20的資料上，數據結果指出拓增型深度類神經網路能與其它的深度學習演算法匹敵。

In image classification task, feature extraction is always a big issue. Intra-class variability increases the difficulty in designing the extractors. Furthermore, hand-crafted feature extractor cannot simply adapt new situation. Recently, deep learning has drawn lots of attention on automatically learning features from data. In this study, we proposed wide and deep neural network (WDNN) which integrates key components of convolutional neural network (CNN), receptive field, with subspace concept. Associating subspace with deep network is a novel designing, providing various viewpoints of data. Basis vectors trained by adaptive subspace self-organization map (ASSOM) span the subspace, serve as transfer function to access axial components and define the receptive field to extract basic patterns of data without distorting the topology in visual task. Moreover, multiple-subspace strategy is implemented as parallel blocks to adapt real-world data and contribute various interpretations of data hoping to be more robust dealing with intra-class variability issues. To this end, handwritten digit and object image datasets (i.e., MNIST and COIL-20) for classification are employed to validate the proposed WDNN architecture. Experimental results show WDNN is competitive to other state-of-the-art approaches.

Table of Context
Chapter 1. Introduction 9
1.1. Background 9
1.2. Statement of the Problem 9
1.3. Aim of the Study 10
Chapter 2. Background of Neural Network 12
2.1. Adaptive Subspace Self-Organizing Map 12
2.1.1. Learning Scheme of ASSOM 13
2.2. Convolutional Neural Network 14
2.2.1. Convolutional Layer 15
2.2.2. Pooling Layer 15
2.2.3. Fully-connected Layer 16
2.2.4. Output Layer 16
Chapter 3. Wide and Deep Neural Network 17
3.1. Overview 17
3.2. Inner-Product Layer 19
3.3. Pooling Layer 21
3.4. Merging Layer 22
3.5. Fully-Connected Layer 24
3.6. Output Layer 24
3.7. Learning Scheme of WDNN 24
3.7.1. Kernel Initialization 25
3.7.2. Network Initialization 26
3.7.3. WDNN Feedforward 26
3.7.4. WDNN Back Propagation 26
Chapter 4. Experiments – Image Dataset 32
4.1. Overview 32
4.2. MNIST Dataset 32
4.2.1. Overview 32
4.2.2. Network Description 33
4.3. COIL-20 Dataset 34
4.3.1. Overview 34
4.3.2. Network Description 34
Chapter 5. Results and Discussions 35
5.1. Overview 35
5.2. MNIST Dataset 35
5.2.1. Classification Performance 35
5.2.2. Graph Visualization 39
5.2.3. Parameter Analysis 42
5.3. COIL-20 Dataset 47
5.3.1. Classification Performance 47
5.3.2. Graph Visualization 52
5.3.3. Parameter Analysis 53
Chapter 6. Conclusions and Future Work 56
References 57

[1] T. H. Chan, K. Jia, S. Gao, J. Lu, Z. Zeng, and Y. Ma, “PCANet: A Simple Deep Learning Baseline for Image Classification?,” IEEE Trans. Image Process., vol. 24, no. 12, pp. 5017–5032, Dec. 2015.
[2] Y. Gan, J. Liu, J. Dong, and G. Zhong, “A PCA-Based Convolutional Network,” ResearchGate, May 2015.
[3] D. G. Lowe, “Object recognition from local scale-invariant features,” in The Proceedings of the Seventh IEEE International Conference on Computer Vision, 1999, 1999, vol. 2, pp. 1150–1157 vol.2.
[4] K. I. Kim, K. Jung, and H. J. Kim, “Face recognition using kernel principal component analysis,” IEEE Signal Process. Lett., vol. 9, no. 2, pp. 40–42, Feb. 2002.
[5] P. Vincent, H. Larochelle, I. Lajoie, Y. Bengio, and P.-A. Manzagol, “Stacked Denoising Autoencoders: Learning Useful Representations in a Deep Network with a Local Denoising Criterion,” Journal of Machine Learning Research, vol. 11, pp. 3371–3408, Dec. 2010.
[6] Y. Lecun, L. Bottou, Y. Bengio, and P. Haffner, “Gradient-based learning applied to document recognition,” Proc. IEEE, vol. 86, no. 11, pp. 2278–2324, Nov. 1998.
[7] Y. Bengio, P. Lamblin, D. Popovici, and H. Larochelle, “Greedy Layer-Wise Training of Deep Networks,” in Advances in Neural Information Processing Systems 19:Proceedings of the 2006 Conference, MIT Press, 2007, pp. 153–160.
[8] G. E. Hinton, S. Osindero, and Y.-W. Teh, “A Fast Learning Algorithm for Deep Belief Nets,” Neural Comput., vol. 18, no. 7, pp. 1527–1554, May 2006.
[9] A. Krizhevsky, I. Sutskever, and G. E. Hinton, “ImageNet Classification with Deep Convolutional Neural Networks,” in Advances in Neural Information Processing Systems 25, F. Pereira, C. J. C. Burges, L. Bottou, and K. Q. Weinberger, Eds. Curran Associates, Inc., 2012, pp. 1097–1105.
[10] S. Lawrence, C. L. Giles, A. C. Tsoi, and A. D. Back, “Face recognition: a convolutional neural-network approach,” IEEE Trans. Neural Netw., vol. 8, no. 1, pp. 98–113, Jan. 1997.
[11] C. D. Gilbert and T. N. Wiesel, “Columnar specificity of intrinsic horizontal and corticocortical connections in cat visual cortex,” J. Neurosci., vol. 9, no. 7, pp. 2432–2442, Jul. 1989.
[12] Y. Bengio, Y. Lecun, and Y. Lecun, Convolutional Networks for Images, Speech, and Time-Series. 1995.
[13] Y. Bengio, P. Simard, and P. Frasconi, “Learning long-term dependencies with gradient descent is difficult,” IEEE Trans. Neural Netw., vol. 5, no. 2, pp. 157–166, Mar. 1994.
[14] T. Kohonen, “Emergence of invariant-feature detectors in the adaptive-subspace self-organizing map,” Biol. Cybern., vol. 75, no. 4, pp. 281–291, Nov. 1996.
[15] T. Kohonen, S. Kaski, and H. Lappalainen, “Self-Organized Formation of Various Invariant-Feature Filters in the Adaptive-Subspace SOM,” Neural Comput., vol. 9, no. 6, pp. 1321–1344, Aug. 1997.
[16] G. E. Hinton, N. Srivastava, A. Krizhevsky, I. Sutskever, and R. R. Salakhutdinov, “Improving neural networks by preventing co-adaptation of feature detectors,” ArXiv12070580 Cs, Jul. 2012.
[17] G. E. Dahl, T. N. Sainath, and G. E. Hinton, “Improving deep neural networks for LVCSR using rectified linear units and dropout,” in 2013 IEEE International Conference on Acoustics, Speech and Signal Processing, 2013, pp. 8609–8613.
[18] J. Gu, Z. Wang, J. Kuen, L. Ma, A. Shahroudy, B. Shuai, T. Liu, X. Wang, and G. Wang, “Recent Advances in Convolutional Neural Networks,” ArXiv151207108 Cs, Dec. 2015.
[19] J. Bouvrie, “Notes on Convolutional Neural Networks,” 22-Nov-2006. [Online]. Available: http://cogprints.org/5869/. [Accessed: 28-Mar-2016].
[20] S. A. Nene, S. K. Nayar, and H. Murase, “Columbia Object Image Library (COIL-20),” 1996.
[21] D. Erhan, Y. Bengio, A. Courville, P.-A. Manzagol, P. Vincent, and S. Bengio, “Why Does Unsupervised Pre-training Help Deep Learning?,” Journal of Machine Learning Research, vol. 11, pp. 625–660, Mar. 2010.
[22] S. Rifai, Y. N. Dauphin, P. Vincent, Y. Bengio, and X. Muller, “The Manifold Tangent Classifier,” in Advances in Neural Information Processing Systems, 2011, pp. 2294–2302.