臺灣博碩士論文加值系統

在類神經網路領域中，很少研究會同時針對以下三個議題進行研究：
(1) 在學習過程中，神經網路能夠有系統的調整隱藏節點的數量 ;
(2) 使用ReLU作為激發函數，而非使用傳統的tanh ;
(3) 保證能學習所有的訓練資料。
在本研究中會針對上述三點，提出強記暨軟化整合 (Cramming, Softening and Integrating)學習演算法，基於單層神經網路並使用ReLU作為激發函數，解決二元輸入/輸出問題，此外也會進行實驗驗證演算法。在實驗中我們使用SPECT心臟影像資料進行實驗，並且使用張量流(TensorFlow)和圖形處理單元(GPU)進行實作。

Rare Artificial Neural Networks studies address simultaneously the challenges of (1) systematically adjusting the amount of used hidden layer nodes within the learning process, (2) adopting ReLU activation function instead of tanh function for fast learning, and (3) guaranteeing learning all training data. This study will address these challenges through deriving the CSI (Cramming, Softening and Integrating) learning algorithm for the single-hidden layer feed-forward neural networks with ReLU activation function and the binary input/output, and further making the technical justification. For the purpose of verifying the proposed learning algorithm, this study conducts an empirical experiment using SPECT heart diagnosis data set from UCI Machine Learning repository. The learning algorithm is implemented via the advanced TensorFlow and GPU.

摘要 1
Abstract 2
Figure Index 4
Table Index 5
1. Introduction 6
2. Literature Review 9
2.1 Rectified Linear Unit (ReLU) 9
2.2 The Single-hidden Layer Feed-forward Neural Networks (SLFN) with one output node 10
2.3 The Back-Propagation Learning Algorithm associated with SLFN 11
2.4 The Adaptive Single-hidden Layer Feed-forward Neural Networks (ASLFN) 14
2.5 Least Trimmed Squares (LTS) Principle 14
2.6 TensorFlow 15
2.7 Cardiac Single Proton Emission Computed Tomography (SPECT) Heart Diagnosis Data Set 16
3. The Proposed CSI Learning Algorithm and Its Technical Justification 18
4. Experimental Design 29
5. The Performance of the Proposed CSI Learning Algorithm 32
5.1 Evaluate the Efficiency of Four Versions 32
5.2 Total Amount of Adopted Hidden Nodes of Four Versions 34
5.3 The Occurrence Percentages of Step 4, Step 6.1 and Step 6.2 of Four Versions 35
5.4 Evaluate the Cramming Mechanism of Four Versions 37
5.5 Evaluate the Softening and Integrating Mechanisms of Four Versions 39
5.6 Evaluate the Performance of Four Versions 44
6. Conclusion and Future Work 46
Reference 48
Appendix 52

[1] I. C. Yeh, and C. H. Lien, "The comparisons of data mining techniques for the predictive accuracy of probability of default of credit card clients," Expert Systems with Applications, vol. 36(2), pp. 2473-2480, 2009.
[2] J. de Jesús Rubio, E. Lughofer, J. A. Meda-Campaña, L. A. Páramo, J. F. Novoa, and J. Pacheco, “Neural network updating via argument Kalman filter for modeling of Takagi-Sugeno fuzzy models,” Journal of Intelligent & Fuzzy Systems, vol. 35(2), pp. 2585-2596, 2018.
[3] X. L. Meng, F. G. Shi, and J. C. Yao, “An inequality approach for evaluating decision making units with a fuzzy output,” Journal of Intelligent & Fuzzy Systems, vol. 34(1), pp. 459-465, 2018.
[4] J. de Jesús Rubio, “Stable Kalman filter and neural network for the chaotic systems identification,” Journal of the Franklin Institute, vol. 354(16), pp. 7444-7462, 2017.
[5] M. Y. Cheng, D. Prayogo, and Y. W. Wu, “Prediction of permanent deformation in asphalt pavements using a novel symbiotic organisms search-least squares support vector regression,” Neural Computing and Applications, 2018.
[6] J. de Jesús Rubio, “SOFMLS: online self-organizing fuzzy modified least-squares network, “ IEEE Transactions on Fuzzy Systems, vol. 17(6), pp. 1296-1309, 2009.
[7] X. M. Zhang, and Q. L. Han, “State estimation for static neural networks with time-varying delays based on an improved reciprocally convex inequality,” IEEE Transactions on Neural Networks and Learning Systems, vol. 29(4), pp. 1376-1381, 2018.
[8] V. Nair, and G. E. Hinton, “Rectified Linear Units improve restricted boltzman machines,” Proceedings of the 27th international conference on machine learning (ICML-10), pp. 807-814, 2010.
[9] L. Ma, and K. Khorasani, "A new strategy for adaptively constructing multilayer feedforward neural networks," Neurocomputing, vol. 51, pp. 361-385, 2003.
[10] R. R. Tsaih, “An explanation of reasoning neural networks,” Mathematical and Computer Modelling, vol. 28(2), pp 37-44, 1998.
[11] E. Watanabe, and H. Shimizu, “Algorithm for pruning hidden nodes in multi-layered neural network for binary pattern classification problem,” Proceeding of 1993 International Joint Conference on Neural Networks I, pp. 327-330, 1993.
[12] Y. Q. Chen, D. W. Thomas, and M. S. Nixon, "Generating-shrinking algorithm for learning arbitrary classification," Neural Networks, vol. 7(9), pp. 1477-1489, 1994.
[13] M. Mezard, and J. P. Nadal, "Learning in feedforward layered networks: The tiling algorithm," Journal of Physics A: Mathematical and General, vol. 22(12), pp. 2191, 1989.
[14] S. E. Fahlman, and C. Lebiere, "The cascade-correlation learning architecture," Advances in neural information processing systems, pp. 524-532, 1990.
[15] M. Frean, "The upstart algorithm: A method for constructing and training feedforward neural networks," Neural computation, vol. 2(2), pp. 198-209, 1990.
[16] R. R. Tsaih, "The softening learning procedure," Mathematical and computer modelling, vol. 18(8), pp. 61-64, 1993.
[17] R. H. Tsaih, and T. C. Cheng, “A resistant learning procedure for coping with outliers,” Annals of Mathematics and Artificial Intelligence, vol. 57(2), pp. 161-180, 2009.
[18] R. H. Tsaih, B. S. Kuo, T. H. Lin, and C. C. Hsu, “The use of big data analytics to predict the foreign exchange rate based on public media: A machine-learning experiment,” IT Professional, vol. 20(2), pp. 34-41, 2018.
[19] L. A. Kurgan, K. J. Cios, R. Tadeusiewicz, M. Ogiela, and L. S. Goodenday, "Knowledge Discovery Approach to Automated Cardiac SPECT Diagnosis," Artificial Intelligence in Medicine, vol. 23(2), pp. 149-169, Oct 2001.
[20] D. Dua, and E. Karra Taniskidou, (2017). UCI Machine Learning Repository [http://archive.ics.uci.edu/ml]. Irvine, CA: University of California, School of Information and Computer Science.
[21] Y. LeCun, Y. Bengio, and G. Hinton, “Deep Learning,” nature, vol. 521(7553), pp. 436, 2015.
[22] K. Hara, D. Saito, and H. Shouno, “Analysis of function of rectified linear unit used in deep learning,” 2015 International Joint Conference on Neural Networks (IJCNN). IEEE, pp. 1-8, 2015.
[23] B. Xu, N. Wang, T. Chen, and M. Li, “Empirical evaluation of rectified activations in convolutional network,” arXiv preprint, arXiv:1505.00853, 2015.
[24] S. Y. Huang, J. W. Lin, and R. H. Tsaih, “Outlier detection in the concept drifting environment,” IEEE 2016 International Joint Conference Neural Networks, pp.31-37, 2016.
[25] M. Abadi, P. Barham, et al., “Tensorflow: A system for large-scale machine learning,” In 12th {USENIX} Symposium on Operating Systems Design and Implementation ({OSDI} 16), pp.265-283, 2016.
[26] M. Abadi, A. Agarwal et al., “TensorFlow: Large-scale machine learning on heterogeneous distributed systems,” arXiv preprint, arXiv:1603.04467, 2016.
[27] S. Ruder, “An overview of gradient descent optimization algorithms,” arXiv preprint, arXiv:1609.04747.
[28] N. V. Chawla, K. W. Bowyer, L. O. Hall, and W. P. Kegelmeyer, “SMOTE: Synthetic Minority Over-sampling Technique,” Journal of artificial intelligence research, vol. 16, pp. 321-357, 2002.
[29] X. Y. Liu, J. Wu, and Z. H. Zhou, “Exploratory undersampling for class-imbalance learning,” IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics), vol. 39(2), pp. 539-550.