臺灣博碩士論文加值系統

本篇論文將以硬體描述語言 ( Very High Speed Integrated Circuit Hardware Description Language，VHDL)，來設計具有學習能力的小波類神經網路(Wavelet Neural Networks，WNN) ，再使用XILINX ISE 6.2i 將所設計的數位電路實現在FPGA (Field Programmable Gate Array，場效可程式化閘極陣列)硬體上。我們所要做的是將一階微分高斯函數與二階微分高斯函數兩種不同的函數，當作是小波函數。在學習演算法部分，提出一個混合式的學習演算法，包含了架構學習與參數學習。首先，架構學習提出自我分群演算法(Self-Clustering Algorithm，SCA)，可以對輸入空間做一個彈性的分割，並且決定一個較合適的節點數。再配合參數學習演算法使用同步干擾(Simultaneous Perturbation)的演算法，因為它僅需要順向的操作就可以修改參數，不像倒傳遞演算法(Back-Propagation algorithm，BP)那樣複雜。可用來解決分類和預測問題，然後比較其效能的差異。在非線性函數的近似採用了泰勒展開式來做一個比較精確的近似，但不足的地方還是會用查表法（Look-up table，LUT）來做補償。接著我們將介紹粒子群最佳化演算法(Particle swarm optimization，PSO) ，來搜尋一個最佳解，進而提高模擬效能。

This thesis use VHDL (Very High Speed Integrated Circuit Hardware Description Language) to design wavelet neural networks (WNN) with learning ability. There are implemented on an FPGA (Field Programmable Gate Array) using XILINX ISE 6.2i so as to realize the digital circuits. We choose the first derivative of a Gaussian function and the second derivative of a Gaussian function as a wavelet function. At the learning algorithm, the proposed hybrid learning approach consists of structure learning and parameter learning. First, the structure learning algorithm proposes the self-clustering algorithm (SCA). This method can achieve a flexible partitioning of the input space, and find suitable the number of nodes. The parameter learning algorithm use simultaneous perturbation algorithm, because the learning algorithm requires only forward operations of the network to adjust parameters unlike the back-propagation method. To solve the problem of classification and prediction, then compare the difference of its performance. In the approximation of nonlinear activation function, we use Taylor series and Look-up table (LUT) to accomplish a more accurate approximation. In order to improve simulation performance, we adopt a particle swarm optimization (PSO) algorithm to find optimum solution.

Contexts
Abstract in Chinese I
Abstract in English III
Acknowledgements in Chinese V
Contexts VI
List of Figures VIII
List of Tables XIII
Chapter 1 Introduction 1
1.1 Literature Survey 1
1.2 Motivation 6
1.3 Thesis Organization 7
Chapter 2 The Wavelet Neural Network (WNN) 9
Chapter 3 The Self-Clustering Algorithm 14
Chapter 4 The Simultaneous Perturbation Algorithm and Its Hardware Implementation 22
4.1 The Simultaneous Perturbation Algorithm 22
4.2 Hardware Implementation of The Simultaneous Perturbation Algorithm 25
4.2.1 Wavelet Unit 25
4.2.1.1 The Second Derivate of The Gaussian Function 26
4.2.1.2 The First Derivate of The Gaussian Function 30
4.2.2 Learning Unit 32
4.3 Simulation Results 35
4.3.1 The Second Derivate of The Gaussian Function 35
4.3.2 The First Derivate of The Gaussian Function 43
Chapter 5 The Particle Swarm Optimization (PSO) and Its Hardware Implementation 49
5.1 The Particle Swarm Optimization (PSO) 49
5.2 Hardware Implementation of the Particle Swarm Optimization 52
5.3 Simulation Results 56
Chapter 6 Conclusion and Future Works 70
References 72
Appendix 79
Vita 81
Publications 82

References
[1]G.J. Gibson, S. Siu, and C.F.N. Cowan, “Application of multilayer perceptrons as adaptive channel equalizers,” Proc. IEEE Conf. Acoust. Speech Signal Process. pp.1183-1186, 1989.
[2]D. E. Rumelbart, G. E. Hinton, and R. J. Williams, “Learning internal representation by error propagation,” in D. E. Rumelhart and J. L. McClelland, eds. Paralle/Distributed Processing Exploration in yth microstructure of Cognition, MIT press, Cambridge, MA, pp.318-362, 1986.
[3]K. Hornik, “Multilayer Feedforward Networks are Universal Approximators”, Neural Networks, Vol. 2, pp. 359-366, 1989.
[4]A. Ikonomopoulos and A. Endou, “Wavelet decomposition and radial basis function networks for system monitoring”, IEEE Trans. on Nuclear Science, Vol. 45, No. 5, pp. 2293-2301, 1998.
[5]D.W.C. Ho, P. A. Zhang, and J. Xu, “Fuzzy wavelet networks for function learning”, IEEE Trans. on Fuzzy Systems, Vol. 9, No. 1, pp. 200 –211, 2001.
[6]Q. Zhang, “Using Wavelet networks in nonparametric estimation”, IEEE Trans. on Neural Networks, Vol. 8, pp. 227-236, 1998.
[7]F. J. Lin, R. J. Wai, and M. P. Chen, “Wavelet neural network control for linear ultrasonic motor drive via adaptive sliding-mode technique”, IEEE Trans. on Ultrasonics, Ferroelectrics, and Frequency Control, Vol. 50, No.6, pp. 686-698, 2003.
[8]Y. C. Huang and C. M. Huang, “Evolving wavelet networks for power transformer condition monitoring”, IEEE Trans. on Power Delivery, Vol. 17, No. 2, pp. 412-416, 2002.
[9]M. C. Hung and D. L. Yang, “The efficient fuzzy c-means clustering technique,” in Proc. IEEE Int. Conf. Data Mining, pp. 225 -232, Dec. 2001.
[10]J. He, L. Liu and G. Palm, “Speaker identification using hybrid LVQ-SLP networks,” in Proc. IEEE Int. Conf. Neural Networks, vol. 4, pp. 2052 -2055, Dec. 1995.
[11]K. K. Ang, C. Quek, and M. Pasquier, “POPFNN-CRI(S): Pseudo outer product based fuzzy neural network using the compositional rule of inference and singleton fuzzifier,” IEEE Trans. Syst., Man, Cybern., Part B, vol. 33, no. 6, pp. 838–849, Dec. 2003.
[12]W. L. Tung and C. Quek, “GenSoFNN: A generic self-organizing fuzzy neural network,” IEEE Trans. Neural Networks, vol. 13, no. 5, pp. 1075-1086, Sept. 2002.
[13]N. K. Kasabov, and Q. Song, “DENFIS: Dynamic evolving neural-fuzzy inference system and its application for time-series prediction,” IEEE Trans. Fuzzy Syst., vol. 10, no. 2, pp. 144-154, Apr. 2002.
[14]F. J. Lin, C. H. Lin, and P. H. Shen, “Self-constructing fuzzy neural network speed controller for permanent-magnet synchronous motor drive”, IEEE Transactions on Fuzzy Systems, Vol. 9, No. 5, pp. 751 – 759, Oct. 2001.
[15]Y. Maeda, H. Yamashita, and Y. Kanata, “Learning rules for multilayer neural networks using difference approximation,” in Proc. IJCNN, Singapore, vol. 1, pp. 628–633, 1991.
[16]Y. Maeda, “Learning rule of neural networks for inverse systems,” Trans. Inst. Electron., Inform., Commun. Eng., vol. J75-A, pp. 1364–1369, 1992 (in Japanese); English version of this paper is in Electron. Commun. Japan, vol. 76, pp. 17–23, 1993.
[17]J. C. Spall, “A stochastic approximation technique for generating maximum likelihood parameter estimates,” in Proc. 1987 Amer. Contr. Conf., pp. 1161–1167.
[18] J. C. Spall, “Multivariable stochastic approximation using a simultaneous perturbation gradient approximation,” IEEE Trans. Automat. Contr., vol. 37, pp. 332–341, 1992.
[19]J. C. Spall and J. A. Cristion, “Nonlinear adaptive control using neural networks: Estimation with a smoothed form of simultaneous perturbation gradient approximation,” Statistica Sinica, vol. 4, pp. 1–27, 1994.
[20]Y. Maeda and Rui J. P. De Figueiredo, “Learning Rules for Neuro-Controller via Simultaneous Perturbation,” IEEE Transaction on Neural Networks, Vol. 8, No. 5, pp. 1119-1130, September 1997.
[21]Y. Maeda, H. Hirano, and Y. Kanata, “A learning rule of neural networks via simultaneous perturbation and its hardware implementation,” Neural Networks, vol. 8, pp. 251–259, 1995.
[22]Y. Maeda and Y. Kanata, “Learning rules for recurrent neural networks using perturbation and their application to neuro-control,” Trans. Inst. Electr. Eng. Japan, vol. 113-C, pp. 402–408, 1993 (in Japanese).
[23]J. Kennedy, R. Eberhart, “Particle swarm optimization,” Proc. IEEE Int. Conf. on Neural Networks, pp. 1942-1948,1995.
[24]Z. L. Gaing,”A particle swarm optimization approach for optimum design of PID controller in AVR system,” IEEE Trans. on Energy Conversion, vol.19, Issue: 2, pp384 – 391, June 2004.
[25]H. Yoshida, K. Kawata, Y. Fukuyama, S. Takayama, Y. Nakanishi,“A particle swarm optimization for reactive power and voltage control considering voltage security assessment,” IEEE Trans. on Power Systems, vol.15, Issue: 4 , pp1232 – 1239, Nov. 2000.
[26]M.A. Abido,“Optimal design of power-system stabilizers using particle swarm optimization”, IEEE Trans. on Energy Conversion,vol.17, Issue: 3 , pp406 – 413, Sept. 2002.
[27]C. F. Juang, “A hybrid of genetic algorithm and particle swarm optimization for recurrent network design”, IEEE Trans. on Systems, Man and Cybernetics, Part B, vol.34, Issue: 2, pp997 – 1006, April 2004.
[28]R. Mendes, P. Cortez, M. Rocha, and J. Neves, “Particle swarms for feedforward neural network training,” The 2002 International Joint Conference on Neural Networks, pp.1895–1899, 2002.
[29]N.M. Botros and M. Abdul-Aziz, “Hardware implementation of an artificial neural network using field programmable gate arrays (FPGA''s)”, IEEE Transactions on Industrial Electronics, Vol. 41, No. 6, pp. 665 – 667, Dec. 1994.
[30]J. J. Blake, L. P. Maguire, and T. M. McGinnity, B. Roche, L.J. McDaid, “The implementation of fuzzy systems, neural networks and fuzzy neural networks using FPGAs”, Information Sciences, Vol. 112, No. 1-4, pp. 151-168, December, 1998.
[31]H. Faiedh, Z. Gafsi, K. Torki, and K. Besbes, “Digital hardware implementation of a neural network used for classification”, Microelectronics, 2004. ICM 2004 Proceedings. The 16th International Conference on, pp. 551 – 554, Dec. 2004.
[32]Chin Tsu Yen, Wan-de Weng, Yen Tsun Lin, “FPGA realization of a neural-network-based nonlinear channel equalizer”, IEEE Transactions on Industrial Electronics, Vol. 51, No. 2, pp. 472 – 479, April 2004.
[33]Y. Maeda and T. Tada, “FPGA Implementation of a Pulse Density Neural Network With Learning Ability Using Simultaneous Perturbation,” IEEE Transaction on Neural Networks, Vol. 14, No. 3, pp. 688-695, May 2003.
[34]H. Hikawa, “A digital hardware pulse-mode neuron with piecewise linear activation function”, IEEE Transactions on Neural Networks, Vol. 14, No. 5, pp.1028 – 1037, Sept. 2003.
[35]C.J. Lin, C.C. Shih, and P.-Y. Chen, “A Nonlinear Time-Varying Channel Equalizer Using Self-Organizing Wavelet Neural Networks,” IEEE Int’l Joint Conf. on Neural Networks, Budapest, pp. 2089-2094, July 25-29 2004.
[36]Q. Zhang, A. Benveniste, “Wavelet networks”, IEEE Transaction on Neural Networks, Vol. 3, No. 6, pp. 889-898, 1992.
[37]J. H. Chiang and P. Y. Hao, “A new kernel-based fuzzy clustering approach: support vector clustering with cell growing,” IEEE Trans. Fuzzy Syst., vol. 11, no. 4, pp. 518-527, Aug. 2003