臺灣博碩士論文加值系統

本論文係以建立模糊類神經系統在學習中不同階段的學習行為理論。模糊類神經系統為一模糊系統具有利用神經網路的學習能力，系統具有相當優異的學習與建模能力，因此廣泛應用於各種應用領域中。本論文回顧幾種不同的模糊類神經系統架構、結構學習、延遲回授與動態建模。接著討論混合學習法中的遞迴最小平方差法與模糊類神經系統實際結構的交互作用，並提出降低系統計算負擔的折衷方案，且同時保存優異的學習能力，然而優異的學習能力相對帶來的則是過度擬合現象。為了瞭解系統在學習中的實時狀態，接著本論文提出四個學習指標和兩種策略。四個學習指標分別來自訓練階段與測試階段兩大面相，用以將過度擬合更精細的分成在訓練階段中所發生的過度訓練，與在測試階段中所發生的過度擬合。依此將能更瞭解系統在實時學習中的狀態與因應策略。最後本論文提出一種強化局部學習的逼近法，提供模糊類神經系統能以更活躍的更新模糊規則庫，新的演算法逼近法是利用提高前件部學習常數，同時利用模糊度觸發是否達到閥值來選擇更新的模糊規則，以局部學習的方式提高學習速度同時降低系統計算負擔。

This dissertation reports on our study of learning behaviors in neuro-fuzzy systems (NFS) in several different phases. First, we discuss ideas regarding the learning of NFS, such as structure learning techniques, recurrent networks, and the mechanisms and relative learning performance with the use of the recursive least squares algorithm. Due to its outstanding system modeling capability, NFS has been widely applied in various applications. However, the relatively good learning capability is part of the reason for overfitting. We then present four learning indices and two strategies. The four learning indices are for determining the NFS behaviors. This dissertation considers different levels of training and testing phases, and thus a way of determining the study the overfitting situation is observed. It can be found that when overfitting takes place, the three training trajectories and a testing trajectory may exist cross behavior. The reasons for overfitting are due to NFS’s learning capability and the difference between training and testing target. After the analysis of behaviors, two strategies were provided that considered the NFS’s maturity to detect the overfitting. The NFS used in our study are adaptive network-based fuzzy inference system (ANFIS), self-constructing neural fuzzy inference network (SONFIN), and enhanced local learning approach for SONFIN. The simulation results can illustrate the learning behaviors of the system and how to improve the learning.

摘要 I
ABSTRACT II
AKNOWLEGEMENTS III
CONTENTS IV
LIST OF FIGURES V
LIST OF TABLES VI
Chapter 1 Introduction 1
1.1 Motivations 1
1.2 Contributions 6
1.3 Organization of the Dissertation 8
Chapter 2 Neural-Fuzzy Structure 10
2.1 General discussion of Neural-Fuzzy System 11
2.2 Structure Learning for NFS 16
2.3 Delay Feedback and Dynamic Modeling 21
2.4 Analysis of the Use of RLS Algorithm 24
2.5 Concluding Remarks 36
Chapter 3 Learning Behaviors 37
3.1 Learning Target 37
3.2 Stages of Learning 42
3.3 Novel Strategies 51
Chapter 4 Enhanced Local Learning Approach 61
4.1 Proposed Learning Mechanism 61
4.2 Simulations 68
4.3 Concluding Remarks 81
Chapter 5 Conclusions and Future Work 83
5.1 Conclusions 83
5.2 Suggestions for Further Research 84
References 85
VITA 95

[1] C. F. Juang, and C. T. Lin, “An on-line self-constructing neural fuzzy inference network and its applications,” IEEE Transactions on Fuzzy Systems, vol. 6 no. 1, pp. 13–32, 1998.
[2] J. S. R Jang, C.-T. Sun, and E. Mizutani, Neuro-Fuzzy and Soft Computing: A Computational Approach to Learning and Machine Intelligence, Prentice-Hall, 1997.
[3] Y. Zhang, and X. R. Li, “A fast U-D factorization-based learning algorithm with applications to nonlinear system modeling and identification,” IEEE Transactions on Neural Networks, vol. 10, no. 4, pp. 930–938, 1999.
[4] Y. S. Cho, S. B. Kim, and E. J. Powers, “Time-varying spectral estimation using AR models with variable forgetting factors,” IEEE Transactions on Signal Processing, vol. 39, no. 6, pp. 1422–1426, 1991.
[5] J.-W. Yeh, S.-F. Su, and I. Rudas, “Analysis of using RLS in neural fuzzy systems,” IEEE International Conference on Systems, Man, and Cybernetics (SMC), pp. 1831–1836, 2011
[6] R. D. Jones, Y. C. Lee, C. W. Barnes, G. W. Flake, K. Lee, and P. S. Lewis, “Function approximation and time series prediction with neural networks,” IJCNN International Joint Conference on Neural Networks, pp. 649–665, 1990.
[7] M. Grabisch, “The representation of importance and interaction of features by fuzzy measures,” Pattern Recognition Letters, vol. 17, pp. 567–575,1996.
[8] J.-W. Yeh, and S.-F. Su, "Learning analysis for correlation of fuzzy rules in applying RLS for neural fuzzy systems," IEEE International Conference on Granular Computing (GrC), pp. 609–613, 2012.
[9] J.-W. Yeh, S.-F. Su, J.-T. Jeng, and B.-S. Chen, “On learning analysis of neural fuzzy systems,” IEEE International Conference on Fuzzy Systems (FUZZ), pp. 1–6, 2010.
[10] S.-F. Su, and J.-W. Yeh, “On Neural Fuzzy Systems,” International Journal of Fuzzy Logic and Intelligent Systems, vol.14, no.4, pp. 276–287, 2014
[11] J. W. Yeh, and S. F. Su, "Efficient Approach for RLS Type Learning in TSK Neural Fuzzy Systems," IEEE Transactions on Cybernetics, vol.PP, no.99, pp. 1–10, 2016.
[12] Y.-Y. Lin, J.-Y. Chang, and C.-T. Lin, “Identification and prediction of dynamic systems using an interactively recurrent self-evolving fuzzy neural network,” IEEE Transactions on Neural Networks and Learning Systems, vol. 24, no. 2, pp. 310–321, 2013.
[13] Y. Lin, and G. A. Cunningham, III, “A new approach to fuzzy-neural system modeling,” IEEE Transactions on Fuzzy Systems, vol. 3, no. 2, pp. 190–198, 1995.
[14] S. Horikawa, T. Furuhashi, and Y. Uchikawa, “On fuzzy modeling using fuzzy neural networks with the back-propagation algorithm,” IEEE Transactions on Neural Networks, vol. 3, no. 5, pp. 801–806, 1992.
[15] K. Tanaka, M. Sano, and H. Watanabe, “Modeling and control of carbon monoxide concentration using a neuro-fuzzy technique,” IEEE Transactions on Fuzzy Systems, vol. 3, no. 3, pp. 271–279, 1995.
[16] S. F. Su, and K. Y. Chen, “Conceptual discussions and benchmark comparison for neural networks and fuzzy systems,” Differential Equations and Dynamical Systems, vol. 13, no. 1, pp. 35–61, 2005.

[17] J. S. Jang, “ANFIS: adaptive-network-based fuzzy inference system,” IEEE Transactions on Systems, Man, and Cybernetics, vol. 23 pp. 665–685, 1993.
[18] T. Takagi, and M. Sugeno, “Fuzzy identification of systems and its applications to modeling and control,” IEEE Transactions on Systems, Man, and Cybernetics, vol. 15, no. 1, pp. 119–132, 1985.

[19] C.-C. Chuang, S.-F. Su, and S.-S. Chen, “Robust TSK fuzzy modeling for function approximation with outliers,” IEEE Transactions on Fuzzy Systems, vol. 9, no. 6, pp. 810–821, 2001.


[20] S.-F. Su, and F.-Y. P. Yang, “On the dynamical modeling with neural fuzzy networks,” IEEE Transactions on Neural Networks, vol. 13, no. 6, pp. 1548–1553, 2002.

[21] C.-F. Juang, and C.-D. Hsieh, “A locally recurrent fuzzy neural network with support vector regression for dynamic-system modeling,” IEEE Transactions on Fuzzy Systems, vol. 18, no. 2, pp. 261–273, 2010.
[22] Y. Y. Lin, J. Y. Chang, N. R. Pal, and C. T. Lin, "A mutually recurrent interval type-2 neural fuzzy system (MRIT2NFS) with self-evolving structure and parameters," in IEEE Transactions on Fuzzy Systems, vol. 21, no. 3, pp. 492–509, 2013.
[23] J. J. Rubio, and A. Bouchachia, “MSAFIS: an evolving fuzzy inference system,” Soft Computing, vol. 21, no. 9, pp. 2357–2366, 2017.
[24] A. C. Zavoianu, E. Lughofer, G. Bramerdorfer, W. Amrhein, and E. P. Klement, “DECOM2: a robust hybrid and adaptive multi objective evolutionary algorithm,” Soft Computing, vol. 19, no. 12, pp. 3551–3569, 2015.
[25] C.-F. Juang, and P.-H. Chang, “Designing fuzzy-rule-based systems using continuous ant-colony optimization,” IEEE Transactions on Fuzzy Systems, vol. 18, no. 1, pp. 138–149, 2010.
[26] Y.-Y. Lin, J. Y. Chang, and C.-T. Lin, “A TSK-type-based self-evolving compensatory interval type-2 fuzzy neural network (TSCIT2FNN) and its applications,” IEEE Transactions on Industrial Electronics, vol. 61, no. 1, pp. 447–459, 2013.
[27] C.-J. Lin, C.-H. Chen, and C.-T. Lin, “Efficient self-evolving evolutionary learning for neuro fuzzy inference systems,” IEEE Transactions on Fuzzy Systems, vol. 16, no. 6, pp. 1476–1490, 2008.
[28] A. Bouchachia, and C. Vanaret, “GT2FC: an online growing interval type-2 self-learning fuzzy classifier,” IEEE Transactions on Fuzzy Systems, vol. 22, no. 4, pp. 999–1018, 2014.
[29] C.-F. Juang, and Y.-W. Tsao. “A self-evolving interval type-2 fuzzy neural network with online structure and parameter learning,” IEEE Transactions on Fuzzy Systems, vol. 16, no. 6, pp. 1411–1424, 2008.
[30] C.-F. Juang, W.-Y. Chen, and C.-W. Liang, “Speedup of learning in interval type-2 neural fuzzy systems through graphic processing units,” IEEE Transactions on Fuzzy Systems, vol. 23, no. 4, pp. 1286–1298, 2015.
[31] C.-T. Lin, N. R. Pal, S.-L. Wu, Y.-T. Liu, and Y.-Y. Lin, “An interval type-2 neural fuzzy system for online system identification and feature elimination,” IEEE Transactions on Neural Network Learning Systems, vol. 26, no. 7, pp. 1442–1455, 2015.
[32] Y.-Y. Lin, S.-H. Liao, J.-Y. Chang, and C.-T. Lin, “Simplified interval type-2 fuzzy neural networks,” IEEE Transactions on Neural Networks and Learning Systems, vol. 25, no. 5, pp. 959–969, 2014.
[33] W. Zhao, Q. Niu, K. Li, and G. W. Irwin, “A hybrid learning method for constructing compact rule-based fuzzy models,” IEEE Transactions on Cybernetics., vol. 43, no. 6, pp. 1807–1821, 2013.
[34] H. Han, X.-L. Wu, and J.-F Qiao, “Nonlinear systems modeling based on self-organizing fuzzy-neural-network with adaptive computation algorithm,” IEEE Transactions on Cybernetics., vol. 44, no. 4, pp. 554–564, 2014.
[35] Y. Liu, J. A. Starzyk, and Z. Zhu, “Optimized approximation algorithm in neural networks without overfitting,” IEEE Transactions on Neural Networks, vol. 19, no. 6, pp. 983–995, 2008.
[36] L. Prechelt, “Automatic early stopping using cross validation: Quantifying the criteria,” Neural Networks, vol. 11, no. 4, pp. 761–777, 1998.

[37] B. Pizzileo, K. Li, G. W. Irwin, and W. Zhao, “Improved structure optimization for fuzzy-neural networks,” IEEE Transactions on Fuzzy Systems, vol. 20, no. 6, pp. 1076–1089, 2012.
[38] S. Lawrence, C. L. Giles, and A. C. Tsoi, What size neural network gives optimal generalization? Convergence properties of backpropagation, Inst. Adv. Comput. Studies, Univ. Maryland, College Park, MD, Tech. Rep. UMIACS-TR-96-22 and CS-TR-3617, 1996.

[39] J. Espinosa, and J. Vandewalle, “Constructing fuzzy models with linguistic integrity from numerical data-AFRELI algorithm,” IEEE Transactions on Fuzzy Systems, vol. 8, no. 5, pp. 591–600, 2000.
[40] A. Lemos, W. Caminhas, and F. Gomide, “Multivariable Gaussian evolving fuzzy modeling system,” IEEE Transactions on Fuzzy Systems, vol. 19, no. 1, pp. 91–104, 2011.
[41]  C. T. Lin, and C. S. G. Lee, “Neural-network-based fuzzy logic control and decision system,” IEEE Transactions on Computers, vol. 40, no. 12, pp. 1320–1336, 1991.
[42] S. R. Huang, Analysis of model-free estimators: applications to stock market with the use of technical indices, M.S. thesis, National Taiwan University of Science and Technology, Taipei, Taiwan, 1999.
[43] L. X. Wang, and J. M. Mendel, “Generating fuzzy rules by learning from examples,” IEEE Transactions on Systems, Man and Cybernetics, vol. 22, no. 6, pp. 1414–1427, 1992.

[44] T. Kohonen, Self-organization and Associative Memory, 3rd ed., New York, NY: Springer-Verlag, 1989.

[45] C. T. Lin, and C. S. G. Lee, Neural Fuzzy Systems: A Neuro-Fuzzy Synergism to Intelligent Systems, Upper Saddle River, NJ: Prentice Hall PTR, 1996.
[46] R. R. Yager, and D. P. Filev, Essentials of Fuzzy Modeling and Control, New York, NY: Wiley, 1994.

[47] J. A. Dickerson, and B. Kosko, “Fuzzy function approximation with ellipsoidal rules,” IEEE Transactions on Systems, Man, and Cybernetics, Part B: Cybernetics, vol. 26, no. 4, pp. 542–560, 1996.

[48] A. Kroll, “Identification of functional fuzzy models using multidimensional reference fuzzy sets,” Fuzzy Sets and Systems, vol. 80, no. 2, pp. 149–158, 1996.
[49] F. Klawonn, and R. Kruse, “Constructing a fuzzy controller from data,” Fuzzy Sets and Systems, vol. 85, no. 2, pp. 177–193, 1997.

[50] E. Kim, M. Park, S. Ji, and M. Park, “A new approach to fuzzy modeling,” IEEE Transactions on Fuzzy Systems, vol. 5, no. 3, pp. 328–337, 1997.
[51] H. Frigui, and R. Krishnapuram, “A robust competitive clustering algorithm with applications in computer vision,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 21, no. 5, pp. 450–465, 1999.

[52] D. M. Hawkins, Identification of Outliers, New York, NY: Chapman and Hall, 1980.
[53] M. Smith, Neural Networks for Statistical Modeling, New York, NY: Van Nostrand Reinhold, 1993.

[54] W. S. Sarle, “Stopped training and other remedies for overfitting,” Proceedings of the 27th Symposium on the Interface of Computing Science and Statistics, Pittsburgh, PA, pp. 352–360, 1995.
[55] P. L. Bartlett, “For valid generalization, the size of the weights is more important than the size of the network,” Advances in Neural Information Processing Systems, vol. 9, pp. 134–140, 1996.
[56] J. C. Bezdek, Pattern Recognition with Fuzzy Objective Function Algorithms, New York, NY: Plenum Press, 1981.
[57] A. K. Jain, and R. C. Dubes, Algorithms for Clustering Data, Englewood Cliffs, NJ: Prentice Hall, 1988.
[58] R. N. Dave, and R. Krishnapuram, “Robust clustering methods: a unified view,” IEEE Transactions on Fuzzy Systems, vol. 5, no. 2, pp. 270–293, 1997.
[59] A. Cichocki, and R. Unbehauen, Neural Networks for Optimization and Signal Processing, New York, NY: J. Wiley, 1993.
[60] P. J. Rousseeuw, and A. M. Leroy, Robust Regression and Outlier Detection, New York, NY: Wiley, 1987.
[61] D. S. Chen, and R. C. Jain, “A robust backpropagation learning algorithm for function approximation,” IEEE Transactions on Neural Networks, vol. 5, no. 3, pp. 467–479, 1994.
[62] J. T. Connor, R. D. Martin, and L. E. Atlas, “Recurrent neural networks and robust time series prediction,” IEEE Transactions on Neural Networks, vol. 5, no. 2, pp. 240–254, 1994.
[63] K. Liano, “Robust error measure for supervised neural network learning with outliers,” IEEE Transactions on Neural Networks, vol. 7, no. 1, pp. 246–250, 1996.
[64] V.David Sánchez A, “Robustization of a learning method for RBF networks,” Neuro computing, vol. 9, no. 1, pp. 85–94, 1995.
[65] L. Huang, B.-L. Zhang, and Q. Huang, “Robust interval regression analysis using neural networks,” Fuzzy Sets and Systems, vol. 97, no. 3, pp. 337–347, 1998.
[66] C.C. Chuang, S.F. Su, and C.C. Hsiao, “The annealing robust backpropagation (ARBP) learning algorithm,” IEEE Transactions on Neural Networks, vol. 11, no. 5, pp. 1067–1077, 2000.
[67] S. S. Haykin, Neural Networks: A Comprehensive Foundation, 2nd ed., Upper Saddle River, NJ: Prentice Hall, 1999.
[68] C. F. Juang, and C. T. Lin, “A recurrent self-organizing neural fuzzy inference network,” IEEE Transactions on Neural Networks, vol. 10, no. 4, pp. 828–845, 1999.
[69] C. H. Lee, and C. C. Teng, “Identification and control of dynamic systems using recurrent fuzzy neural networks,” IEEE Transactions on Fuzzy Systems, vol. 8, no. 4, pp. 349–366, 2000.

[70] K. S. Narendra, and K. Parthasarathy, “Identification and control of dynamical systems using neural networks,” IEEE Transactions on Neural Networks, vol. 1, no. 1, pp. 4–27, 1990.

[71] P. S. Sastry, G. Santharam, and K. P. Unnikrishnan, “Memory neuron networks for identification and control of dynamical systems,” IEEE Transactions on Neural Networks, vol. 5, no. 2, pp. 306–319, 1994.

[72] S. S. Haykin, Adaptive Filter Theory, 2nd ed., Englewood Cliffs, NJ: Prentice Hall, 1991.

[73] L. A. Zadeh, “The concept of a linguistic variable and its application to approximate reasoning-I,” Information Sciences, vol. 8, no. 3, pp. 199–249, 1975.
[74] L. A. Zadeh, “The concept of a linguistic variable and its application to approximate reasoning-II,” Information Sciences, vol. 8, no. 4, pp. 301–357, 1975.
[75] L. A. Zadeh, “The concept of a linguistic variable and its application to approximate reasoning-III,” Information Sciences, vol. 9, no. 1, pp. 43–80, 1975.
[76] D. E. Rumelhart, G. E. Hinton, and R.J. Williams, “Learning internal representations by error propagation,” Parallel Distribution Processing, vol. 1, pp. 318–362, 1986.
[77] D. S. Yeung, X.-Z. Wang, and E. C. C. Tsang, "Handling interaction in fuzzy production rule reasoning," IEEE Transactions on Systems, Man, and Cybernetics, Part B: Cybernetics, vol.34, no.5, pp. 1979–1987, 2004
[78] J. J. Rubio, “Least square neural network model of the crude oil blending process,” Neural Networks, vol. 78, pp. 88–96, 2016.
[79] Q. Liu, J. Yin, V. C. M. Leung, J.-H. Zhai, Z. Cai and J. Lin., “Applying a new localized generalization error model to design neural networks trained with extreme learning machine,” Neural Computing and Applications, vol. 27, no. 1, pp. 59–66, 2016.
[80] J. J. Rubio, “Adaptive least square control in discrete time of robotic arms,” Soft Computing, vol. 19, vo. 12, pp. 3665–3676, 2015.
[81] J. Viola, and L. Angel, “Identification, control and robustness analysis of a robotic system using fractional control,” IEEE Latin America Trans., vol. 19, vo. 12, pp. 3665–3676, 2015.
[82] C. F. Juang, Y. Y. Lin, and C. C. Tu, “A recurrent self-evolving fuzzy neural network with local feedbacks and its application to dynamic system processing,” Fuzzy Sets and Systems, vol. 161, no. 19, pp. 2552–2568, 2010.
[83] C. J. Lin, and C. C. Chin, “Prediction and identification using wavelet-based recurrent fuzzy neural networks,” IEEE Transactions on Systems, Man, and Cybernetics, Part B: Cybernetics, vol. 34, no. 5, pp. 2144–2154, 2004.
[84] C. F. Juang, “A TSK-type recurrent fuzzy network for dynamic systems processing by neural network and genetic algorithm,” IEEE Transactions on Fuzzy Systems, vol. 10, no. 2, pp. 155–170, 2002.