臺灣博碩士論文加值系統

In this thesis we propose a motor control model inspired by organizational priciples of the cerebral cortex. Specifically the model is based on cortical maps and functional hierarchy in sensory and motor areas of the brain. We introduce observed properties of the F5 area in the macaque monkey brain, an area which combines sensory and motor information, producing actions without high processing information. The properties here observed can be quickly summarized to mdularity and hierarchical processing. These form the basis for the model we propose. We make use of well known computational tools, to put
together a biology imitating model, for action learning and motor control. The Self-Organizing Maps (SOM) have proven to be useful in modeling cortical topological maps. A hierarchical SOM provides a natural way to extract
hierarchical information from the environment, which we propose may in turn be used to select actions hierarchically. We use a neighborhood update version of
the Q-learning algorithm, so the final model maps a continuous input space to a continuous action space in a hierarchical, topology preserving manner. The model is called the Hierarchical Map Forming model (HMF) due to the way in which it forms maps in both the input and output spaces in a hierarchical manner.

In this thesis we propose a motor control model inspired by organizational priciples of the cerebral cortex. Specifically the model is based on cortical maps and functional hierarchy in sensory and motor areas of the brain. We introduce observed properties of the F5 area in the macaque monkey brain, an area which combines sensory and motor information, producing actions without high processing information. The properties here observed can be quickly summarized to mdularity and hierarchical processing. These form the basis for the model we propose. We make use of well known computational tools, to put
together a biology imitating model, for action learning and motor control. The Self-Organizing Maps (SOM) have proven to be useful in modeling cortical topological maps. A hierarchical SOM provides a natural way to extract
hierarchical information from the environment, which we propose may in turn be used to select actions hierarchically. We use a neighborhood update version of
the Q-learning algorithm, so the final model maps a continuous input space to a continuous action space in a hierarchical, topology preserving manner. The model is called the Hierarchical Map Forming model (HMF) due to the way in which it forms maps in both the input and output spaces in a hierarchical manner.

1 Neurophysiology of Visuo-Motor Area F5 7
1.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
1.2 Neurophysiology of the Mirror Neuron System . . . . . . . . . 7
1.2.1 Visuomotor Neurons . . . . . . . . . . . . . . . . . . . 8
1.2.2 Canonical Neurons . . . . . . . . . . . . . . . . . . . . 9
1.2.3 Mirror Neurons . . . . . . . . . . . . . . . . . . . . . . 9
1.2.4 On Action understanding . . . . . . . . . . . . . . . . 12
1.3 In Review . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12
1.4 Brain Models . . . . . . . . . . . . . . . . . . . . . . . . . . . 14
1.4.1 The FARS model . . . . . . . . . . . . . . . . . . . . . 14
1.4.2 The MOSAIC model . . . . . . . . . . . . . . . . . . . 21
2 Theoretical Background 24
2.1 Self-Organization and the Self-Organizing Map . . . . . . . . 24
2.1.1 Motivation Behind Kohonen’s Self-Organizing Map . . 24
2.1.2 Conditions for Self-Organization . . . . . . . . . . . . 25
2.1.3 The Basic SOM . . . . . . . . . . . . . . . . . . . . . . 26
2.1.4 Self-Organizing Map and Brain Models . . . . . . . . 29
2.1.5 Self-Organizing Trees . . . . . . . . . . . . . . . . . . 30
2.1.6 Case Analisys 1 . . . . . . . . . . . . . . . . . . . . . . 33
2.1.7 Case Analisys 2 . . . . . . . . . . . . . . . . . . . . . . 34
2.2 Reinforcement Learning . . . . . . . . . . . . . . . . . . . . . 39
2.2.1 The Reinforcement Learning Problem . . . . . . . . . 39
2.2.2 Policy . . . . . . . . . . . . . . . . . . . . . . . . . . . 39
2.2.3 Value . . . . . . . . . . . . . . . . . . . . . . . . . . . 40
2.2.4 Temporal Difference Learning . . . . . . . . . . . . . . 41
2.2.5 TD(0) Method . . . . . . . . . . . . . . . . . . . . . . 41
2.2.6 Q-Learning . . . . . . . . . . . . . . . . . . . . . . . . 41
2.2.7 Applications of SOM to Reinforcement Learning Problems
. . . . . . . . . . . . . . . . . . . . . . . . . . . . 43
3 The Hierarchical Map Forming Model 45
3.1 Combining TS-SOM and Q-learning . . . . . . . . . . . . . . 45
3.1.1 Neighborhood Q-learning . . . . . . . . . . . . . . . . 47
3.2 Training Methodologies . . . . . . . . . . . . . . . . . . . . . 48
3.2.1 Sequential Training . . . . . . . . . . . . . . . . . . . . 48
3.2.2 Concurrent Training . . . . . . . . . . . . . . . . . . . 48
3.3 Software Implementation . . . . . . . . . . . . . . . . . . . . . 49
3.3.1 M-language version . . . . . . . . . . . . . . . . . . . . 49
3.3.2 Simulink version . . . . . . . . . . . . . . . . . . . . . 49
3.3.3 HMF Toolbox . . . . . . . . . . . . . . . . . . . . . . . 49
3.4 2D Trajectory Mapping Simulation . . . . . . . . . . . . . . . 50
3.5 3D Trajectory Mapping Simulation . . . . . . . . . . . . . . . 59
3.5.1 The 3D arm mapping problem . . . . . . . . . . . . . 59
4 Conclusions 65
4.1 Future Work . . . . . . . . . . . . . . . . . . . . . . . . . . . 66
5 Bibliography 67

1. Fagg, A. H. A computational model of the cortical mechanisms involved
in primate grasping. Ph.D. Dissertation, University of Southern California,
Computer Science Department, 1996.

2. Fagg, A., Arbib, M., Modeling parietal–premotor interactions in primate
control of grasping, Neural Networks, Issue. 11 , 1998.

3. Frith, C. Wolpert, D. M. The Neuroscience of Social Interaction. Oxford
University Press, New York, 2004.

4. Gibson J. J. The optical expansion-pattern in aerial location. American
Journal of Psychology, 68, 1995.

5. Haykin, S. Neural Networks. Prentice-Hall, Inc. Upper Saddle River,
New Jersey, Second Edition,1999.

6. Holroyd, C. B., Coles, M. G. H . The Neural Basis of Human Error Processing:
Reinforcement Learning, Dopamine, and the Error-Related Negativity.
Psychological Review, Vol. 109, No. 4, 2002.

7. Kohler, E., Keysers, C., Umilta, M. A., Fogassi, L., Gallese, V., Rizzolattii,
G., Hearing Sounds, Understanding Actions: Action Representation
in Mirror Neurons, Science, Vol. 297. no. 5582, 2002.

8. Koikkalainen, P., Oja, E. Self-organizing hierarchical feature maps. In
Proceedings of International Joint Conference on Neural Networks (IJCNN’90),
pages 279 284, 1990.

9. Kohonen, T. Self-Organizing Maps. Springer Verlag, Heidelberg, Germany,
2001.

10. Matelli, M., Camarada, R., Glickstein, M., & Rizzolatti, G. Afferent and
efferent projections of the inferior area 6 in the macaque monkey. Journal
of Comparative Neurology, 251 (3), 1986.

11. Muakkassa, K. F., & Strick, P. L. Frontal Lobe Inputs to Primate Motor
Cortex. Evidence for Four Somatotopically Organized ”Premotor” Areas.
Brain Research, 177, 1979.

12. Mulier, F., Cherkassky, V. : In Proc. 12 (ICPR) International Conferenceon
Pattern Recognition, IEEE.

13. Murata, A., Gallese, V., Kaseda, M., Kunimoto, S. & Sakata, H. Handmanipulation-
related neurons of the parietal cortex of the monkey: further
analysis of selectivity in shape, size, and orientation of objects for manipulation.
Proceedings of the Society of Neuroscience Annual Meeting, Washington,
DC, 1993.

14. Murata, A., Fadiga, L., Fogassi, L., Gallese, V., Raos, V., & Rizzolatti,
G. Object representation in the ventral premotor cortex (area F5) of the
monkey. Journal of Neurophysiology(78), 2226-2230, 1997.

15. Oja, E., Kaski, S. editors. Kohonen Maps. Elsevier Science, Amsterdam,
Netherlands, 1999.

16. Palakal, M. J. U, Murthy, S.K. Chittajallu, D. Wong. Tonotopic Representation
of Auditory Responses Using Self-Organizing Maps. Mathematical
Computing Modeling. Vol. 22, No. 2, pp 7-21, 1995.

17. Rizzolatti, G., Camarda, R., Fogassi, L., Gentilucci, M., Luppino, G.,
& Matelli, M. Functional organization of inferior area 6 in the macaque
monkey: II. Area F5 and the control of distal movements. Experimental
Brain Research, 71(3), 491-507, 1988.

18. Rizzolatti, G., & Luppino, G. The cortical motor system. Neuron, 31,
2001.

19. Sakata H., & Kusunoki M. Organization of space perception: neural representation
of three-dimensional space in the posterior parietal cortex. Current
Opinions in Neurobiology, 2 (2), 1992.

20. Smith, J. A. Applications of the self-organizing map to reinforcement
learning. Neural Networks, Vol. 15 (8-9), 2002.

21. Stamenov, M., Gallese, V., Mirror Neuron and the Evolution of Brain
and Language: Advances in Consciousness Research, Vol 42, 2002.

22. Taira M., Mine S., Georgopoulos A. P., Murata A., & Sakata H. Parietal
cortex neurons of the monkey related to the visual guidance of hand
movement. Experimental Brain Research, 83, 1990.

23. Watkins, C.J.C.H., Dayan, P., Technical Note: Q-Learning. Machine
Learning, Vol. 8, 1992.