臺灣博碩士論文加值系統

腦機介面的目標是將神經訊號精準地透過解碼器預測出動作指令，但隨著記錄時間增長，則產生神經記錄情況改變的問題，降低解碼器於長時間動作指令預測的準確度，如神經元的變動或電極與組織間的交互作用。近年來，研究學者提出以局部場電位做為輸入訊號，此方法可能有效降低神經元變動導致記錄情況改變的問題，因為局部場電位是透過電極記錄其周圍數顆神經元的放電反應，但以局部場電位做為輸入訊號的精準度相較於細胞動作電位低，因此研究學者將細胞動作電位結合局部場電位同步輸入至解碼器中，讓解碼器不僅保有精準地解碼，也提升長時間預測的穩定性。不過，不管是以細胞動作電位、局部場電位或兩者一同做為輸入訊號，皆需要長時間蒐集不同的記錄情況資料訓練解碼器，但已有研究指出神經記錄情況的改變會與過往的情況相似，因此，本研究以此理論，提出資料擴增技術擴充資料量進而降低神經記錄的時間，，並發展適用於細胞動作電位與局部場電位可同步輸入的多模態擴增技術，且以兩種解碼器驗證擴增技術的表現-線性回歸及神經網路演算法。本研究以大鼠動作皮質區的神經訊號做為訓練資料及其右前肢按壓給水桿之軌跡作為預測目標，實驗結果顯示相較於其他擴增技術以多模態神經信號擴增技術進行數據增廣，不管是利用線性回歸或神經網路演算法，皆讓資料在較短的記錄時間中，達到高準確度與高穩定性的解碼。

The goal of the brain computer interface (BMI) focused on accuracy of decoder, but neural recording conditions were changed from day to day. It would decrease the long-term prediction, e.g. neuron changes and electrodes/tissue interaction. In recent years, researchers suggest to use local field potentials (LFPs) as the input, because LFPs were the sum of several neurons in the adjacent range of electrodes. Even so, using action potential (spikes) as inputs for decoder shows higher performance than LFPs did. Hence, researchers combined spikes and LFPs as inputs, which keep the decoder to have high accuracy in long-term prediction. However, no matter what neural signals were recorded, long recording days are needed to train decoding model. Due to the recording condition would be similar to the past condition in recent research, we want to propose multimodal augmentation for multi-scale with spikes and LFPs. Moreover, we hope that this method increase the database, decrease the time-consuming in data recording and keep high performance and stability in movement prediction. The linear regression and neural network algorithms were used to evaluate the efficacy of multimodal augmentation. We collected the neural signals from the primary motor cortex (M1) of rat and the right forelimb as prediction target. Results showed that compared with other augmentation methods, multimodal augmentation had better performance in short recording day and it also improved long-term prediction by linear regression and neural network algorithms.

Table of Contents
Acknowledgements i
摘要 ii
Abstract iii
Table of Contents iv
List of Tables vii
List of Figures viii
Chapter 1. Introduction 1
1.1 Overview 1
1.2 Literature Review 5
1.3 Motivations and Objectives 7
Chapter 2. Materials and Methods 9
2.1 Architecture of Neural Decoding in Forelimb Movement 9
2.2 Animal Preparation and Surgical Procedures 10
2.3 Animal Training and Behavioral Task 12
2.4 Data Recording and Preprocessing 13
2.4.1 Instantaneous Spike Rates 14
2.4.2 Local Field Potentials (LFPs) 16
2.5 Trajectory Tracking of Forelimb Movement 17
2.6 Establishment of Augmentation 19
2.6.1 Synchronize Perturbation 22
2.7 Neural Decoding Model 23
2.7.1 Feedforward Neural Network (FNN) 25
2.7.2 Kernel Sliced Inverse Regression (kSIR) 27
2.8 Time-Lags and Temporal Order of the Neural Signals 29
2.9 Decoding Performance Evaluation 30
2.10 Statistical Analysis 31
Chapter 3. Results 32
3.1 Effects of Different Time-Lags and Temporal Orders for Trajectory Decoding 32
3.1.1 Instantaneous Spike Rates 32
3.1.2 Spectral Power 34
3.1.3 Multi-Scale 36
3.2 Neuronal Signal Pattern between Source Data and Augmentation 38
3.2.1 Instantaneous Spike Rates 38
3.2.2 Spectral Power 40
3.3 The Performance with Different Augmentation Scheme 42
3.4 The Comparison of Different Quantities in Decoders 44
3.5 The Stability with Multi-Augmentation Scheme 45
Chapter 4. Discussion 47
4.1 Neuronal Signal Pattern during Behavior Task 48
4.2 Different Augmentation Effected Neuronal Signal Pattern 49
4.3 Effects of Different Quantities in Decoders 51
4.4 Stability of Multi-Augmentation 52
Chapter 5. Conclusion 54
Reference 55
Supplementary 1 – Correlation between Spikes and Powers 60
Supplementary 2 – R2 Results 62
S2.1 Effects of Different Time-Lags and Temporal Orders for Trajectory Decoding 62
S2.1.1 Instantaneous Spike Rates 62
S2.1.2 Spectral Power 63
S2.1.3 Multi-Scale 64
S2.2 The Performance with Different Augmentation Scheme 65
S2.3 Effects of Different Quantities in Decoders 66
S2.4 The Stability with Multi-Augmentation Scheme 67
 
List of Tables
Table 1. Training parameters used for decoder on four individual rats. 18
Table 2. Training parameters used for FNN on four individual rats. 26

 
List of Figures
Figure 1. System architecture of BMI. 1
Figure 2. Architecture of multi-scale neural signals. 4
Figure 3. The framework of data augmentation. 6
Figure 4. Multimodal neural signals augmentation in BMI. 8
Figure 5. Architecture of forelimb movement decoding. 9
Figure 6. A double-layered 8-channels laboratory-made microelectrodes array. 11
Figure 7. Location of microelectrodes array implantation. 11
Figure 8. Side view of behavioral box. 12
Figure 9. Example of forelimb movement over time.. 13
Figure 10. Preprocessing of spike signals.. 15
Figure 11. Preprocessing of LFPs signals. 16
Figure 12. Trajectory tracking. 17
Figure 13. Schematic diagram of the proposed framework for neural signals augmentation. 19
Figure 14. The prediction framework based on multimodal augmentation. 24
Figure 15. FNN topology. 25
Figure 16. Flow chart of kSIR.. 28
Figure 17. Illustration of time-lags and temporal orders. 29
Figure 18. Decoding performance of using instantaneous spike rates data with different time-lags and temporal orders in kSIR model. 33
Figure 19. Decoding Performance of using instantaneous spike rates data with different time-lags and temporal order in FNN model. 33
Figure 20. Decoding Performance of using spectral power data with different time-lags and temporal order in kSIR model. 35
Figure 21. Decoding Performance of using spectral power data with different time-lags and temporal order in FNN model. 35
Figure 22. Decoding Performance of using instantaneous spike rates and spectral power data with different time-lags and temporal order in kSIR model. 37
Figure 23. Decoding Performance of using instantaneous spike rates and spectral power data with different time-lags and temporal order in FNN model. 37
Figure 24. Comparison of the peri-stimulus-time distribution between various methods. 39
Figure 25. Comparison of the spectral power distribution between various methods. 41
Figure 26. Comparison of decoding performance between kSIR and FNN with or without augmentation scheme. 43
Figure 27. Comparison of trajectory reconstruction by kSIR and FNN. 43
Figure 28. Comparison of decoding performance in different quantities. 44
Figure 29. Comparison of decoding performance between kSIR with and without augmentation.. 46
Figure 30. Comparison of decoding performance between FNN with and without augmentation. 46
Figure 31. The correlation between spikes and powers. 61
Figure 32. Decoding R2 of using instantaneous spike rates data with different time-lags and temporal orders in kSIR model. 62
Figure 33. Decoding R2 of using instantaneous spike rates data with different time-lags and temporal order in FNN model. 62
Figure 34. Decoding R2 of using spectral power data with different time-lags and temporal order in kSIR model. 63
Figure 35. Decoding R2 of using spectral power data with different time-lags and temporal order in FNN model. 63
Figure 36. Decoding R2 of using instantaneous spike rates and spectral power data with different time-lags and temporal order in kSIR model. 64
Figure 37. Decoding R2 of using instantaneous spike rates and spectral power data with different time-lags and temporal order in FNN model. 64
Figure 38. Comparison of decoding R2 between kSIR and FNN with or without augmentation scheme. 65
Figure 39. Comparison of decoding R2 in different quantities. 66
Figure 40. Comparison of decoding R2 between kSIR with and without augmentation. 67
Figure 41. Comparison of decoding R2 between FNN with and without augmentation. 67

[1] J. C. Kao, S. D. Stavisky, D. Sussillo, et al., "Information systems opportunities in brain–machine interface decoders," Proceedings of the IEEE, vol. 102, no. 5, pp. 666-682, 2014.
[2] C. E. Vargas-Irwin, G. Shakhnarovich, P. Yadollahpour, et al., "Decoding complete reach and grasp actions from local primary motor cortex populations," Journal of neuroscience, vol. 30, no. 29, pp. 9659-9669, 2010.
[3] I. Stepniewska, T. M. Preuss, and J. H. Kaas, "Architectionis, somatotopic organization, and ipsilateral cortical connections of the primary motor area (M1) of owl monkeys," Journal of Comparative Neurology, vol. 330, no. 2, pp. 238-271, 1993.
[4] J. N. Sanes and J. P. Donoghue, "Plasticity and primary motor cortex," Annual review of neuroscience, vol. 23, no. 1, pp. 393-415, 2000.
[5] A. Ortiz-Rosario and H. Adeli, "Brain-computer interface technologies: from signal to action," Reviews in the Neurosciences, vol. 24, no. 5, pp. 537-552, 2013.
[6] N. Birbaumer, "Brain-computer-interface research: coming of age," Clinical neurophysiology, 2006.
[7] A. Finke, A. Lenhardt, and H. Ritter, "The MindGame: a P300-based brain–computer interface game," Neural Networks, vol. 22, no. 9, pp. 1329-1333, 2009.
[8] G. Pfurtscheller, C. Guger, G. Müller, et al., "Brain oscillations control hand orthosis in a tetraplegic," Neuroscience letters, vol. 292, no. 3, pp. 211-214, 2000.
[9] J. L. Collinger, B. Wodlinger, J. E. Downey, et al., "High-performance neuroprosthetic control by an individual with tetraplegia," The Lancet, vol. 381, no. 9866, pp. 557-564, 2013.
[10] A. K. Bansal, W. Truccolo, C. E. Vargas-Irwin, et al., "Decoding 3D reach and grasp from hybrid signals in motor and premotor cortices: spikes, multiunit activity, and local field potentials," Journal of neurophysiology, vol. 107, no. 5, pp. 1337-1355, 2011.
[11] V. Gilja, C. Pandarinath, C. H. Blabe, et al.,"Clinical translation of a high-performance neural prosthesis," Nature medicine, vol. 21, no. 10, pp. 1142-1145, 2015.
[12] C. E. Bouton, A. Shaikhouni, N. V. Annetta, et al., "Restoring cortical control of functional movement in a human with quadriplegia," Nature, vol. 533, no. 7602, pp. 247-250, 2016.
[13] C. Pandarinath, P. Nuyujukian, C. H. Blabe, et al., "High performance communication by people with paralysis using an intracortical brain-computer interface," eLife, vol. 6, p. e18554, 2017.
[14] Z. Li, "Decoding methods for neural prostheses: where have we reached?," Frontiers in systems neuroscience, vol. 8, p. 129, 2014.
[15] S. H. Yang, Y. Y. Chen, S.H. Lin, et al., "A Sliced Inverse Regression (SIR) decoding the forelimb movement from neuronal spikes in the rat motor cortex," Frontiers in neuroscience, vol. 10, p. 556, 2016.
[16] H. M. Wu, "Kernel sliced inverse regression with applications to classification," Journal of Computational and Graphical Statistics, vol. 17, no. 3, pp. 590-610, 2008.
[17] A. Aertsen, M. Diesmann, and M. O. Gewaltig, "Propagation of synchronous spiking activity in feedforward neural networks," Journal of physiology-Paris, vol. 90, no. 3-4, pp. 243-247, 1996.
[18] D. Sussillo, P. Nuyujukian, J. M. Fan, et al., "A recurrent neural network for closed-loop intracortical brain–machine interface decoders," Journal of neural engineering, vol. 9, no. 2, p. 026027, 2012.
[19] J. M. Carmena, M. A. Lebedev, C. S. Henriquez, et al., "Stable ensemble performance with single-neuron variability during reaching movements in primates," Journal of Neuroscience, vol. 25, no. 46, pp. 10712-10716, 2005.
[20] U. Rokni, A. G. Richardson, E. Bizzi, et al., "Motor learning with unstable neural representations," Neuron, vol. 54, no. 4, pp. 653-666, 2007.
[21] R. D. Flint, Z. A. Wright, M. R. Scheid, et al., "Long term, stable brain machine interface performance using local field potentials and multiunit spikes," Journal of neural engineering, vol. 10, no. 5, p. 056005, 2013.
[22] B. Pesaran, J. S. Pezaris, M. Sahani, et al., "Temporal structure in neuronal activity during working memory in macaque parietal cortex," Nature neuroscience, vol. 5, no. 8, p. 805, 2002.
[23] Z. C. Chao, Y. Nagasaka, and N. Fujii, "Long-term asynchronous decoding of arm motion using electrocorticographic signals in monkey," Frontiers in neuroengineering, vol. 3, p. 3, 2010.
[24] S. D. Stavisky, J. C. Kao, P. Nuyujukian, et al., "A high performing brain–machine interface driven by low-frequency local field potentials alone and together with spikes," Journal of neural engineering, vol. 12, no. 3, p. 036009, 2015.
[25] C. Mehring, J. Rickert, E. Vaadia, et al., "Inference of hand movements from local field potentials in monkey motor cortex," Nature neuroscience, vol. 6, no. 12, p. 1253, 2003.
[26] D. Sussillo, S. D. Stavisky, J. C. Kao, et al., "Making brain–machine interfaces robust to future neural variability," Nature communications, vol. 7, p. 13749, 2016.
[27] J. Simeral, S.-P. Kim, M. Black, et al., "Neural control of cursor trajectory and click by a human with tetraplegia 1000 days after implant of an intracortical microelectrode array," Journal of neural engineering, vol. 8, no. 2, p. 025027, 2011.
[28] J. A. Perge, M. L. Homer, W. Q. Malik, et al., "Intra-day signal instabilities affect decoding performance in an intracortical neural interface system," Journal of neural engineering, vol. 10, no. 3, p. 036004, 2013.
[29] J. A. Perge, S. Zhang, W. Q. Malik, et al., "Reliability of directional information in unsorted spikes and local field potentials recorded in human motor cortex," Journal of neural engineering, vol. 11, no. 4, p. 046007, 2014.
[30] B. Wodlinger, J. Downey, E. Tyler-Kabara, et al., "Ten-dimensional anthropomorphic arm control in a human brain− machine interface: difficulties, solutions, and limitations," Journal of neural engineering, vol. 12, no. 1, p. 016011, 2014.
[31] J. C. Barrese, N. Rao, K. Paroo, et al., "Failure mode analysis of silicon-based intracortical microelectrode arrays in non-human primates," Journal of neural engineering, vol. 10, no. 6, p. 066014, 2013.
[32] E. Fernández, B. Greger, P. A. House, et al., "Acute human brain responses to intracortical microelectrode arrays: challenges and future prospects," Frontiers in neuroengineering, vol. 7, 2014.
[33] A. Jackson, J. Mavoori, and E. E. Fetz, "Long-term motor cortex plasticity induced by an electronic neural implant," Nature, vol. 444, no. 7115, pp. 56-60, 2006.
[34] J. W. Salatino, K. A. Ludwig, T. D. Kozai, et al., "Glial responses to implanted electrodes in the brain," Nature Biomedical Engineering, vol. 1, no. 11, p. 862, 2017.
[35] B. Jarosiewicz, A. A. Sarma, D. Bacher, et al., "Virtual typing by people with tetraplegia using a self-calibrating intracortical brain-computer interface," Science translational medicine, vol. 7, no. 313, pp. 313ra179-313ra179, 2015.
[36] B. Jarosiewicz, N. Y. Masse, D. Bacher, et al., "Advantages of closed-loop calibration in intracortical brain–computer interfaces for people with tetraplegia," Journal of neural engineering, vol. 10, no. 4, p. 046012, 2013.
[37] D. Wang, Q. Zhang, Y. Li, Y. Wang, et al., "Long-term decoding stability of local field potentials from silicon arrays in primate motor cortex during a 2D center out task," Journal of Neural Engineering, vol. 11, no. 3, p. 036009, 2014.
[38] R. D. Flint, E. W. Lindberg, L. R. Jordan, et al., "Accurate decoding of reaching movements from field potentials in the absence of spikes," Journal of neural engineering, vol. 9, no. 4, p. 046006, 2012.
[39] M. D. Bloice, C. Stocker, and A. J. a. p. a. Holzinger, "Augmentor: an image augmentation library for machine learning," 2017.
[40] R. Gonzalez and R. Woods, "Digital image processing: Pearson prentice hall," Upper Saddle River, NJ, vol. 1, 2008.
[41] F. Wang, S.-h. Zhong, J. Peng, et al., "Data augmentation for eeg-based emotion recognition with deep convolutional neural networks," in International Conference on Multimedia Modeling, 2018: Springer, pp. 82-93.
[42] E. A. Pohlmeyer, B. Mahmoudi, S. Geng, et al., "Using reinforcement learning to provide stable brain-machine interface control despite neural input reorganization," PloS one, vol. 9, no. 1, p. e87253, 2014.
[43] H. Ismail Fawaz, G. Forestier, J. Weber, et al., "Data augmentation using synthetic data for time series classification with deep residual networks," arXiv preprint arXiv:1808.02455, 2018.
[44] Y. Bengio, F. Bastien, A. Bergeron, et al., "Deep learners benefit more from out-of-distribution examples," in Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics, pp. 164-172, 2011.
[45] M. M. Krell, A. Seeland, and S. K. Kim, "Data Augmentation for Brain-Computer Interfaces: Analysis on Event-Related Potentials Data," arXiv preprint arXiv:1801.02730, 2018.
[46] M. M. Krell and S. K. Kim, "Rotational data augmentation for electroencephalographic data," in IEEE, pp. 471-474, 2017.
[47] T. T. Um, F. M. J. Pfister, D. Pichler, et al., "Data augmentation of wearable sensor data for Parkinson's disease monitoring using convolutional neural networks," arXiv preprint arXiv:1706.00527, 2017.
[48] R. Mukamel, A. D. Ekstrom, J. Kaplan, et al., "Single-neuron responses in humans during execution and observation of actions," Current biology, vol. 20, no. 8, pp. 750-756, 2010.
[49] A. P. Georgopoulos, J. F. Kalaska, R. Caminiti, et al., "On the relations between the direction of two-dimensional arm movements and cell discharge in primate motor cortex," Journal of Neuroscience, vol. 2, no. 11, pp. 1527-1537, 1982.
[50] B. Khoshkrood Mansoori, "Acute inactivation of the contralesional hemisphere for longer durations improves recovery after cortical injury," Experimental neurology, vol. 254, pp. 18-28, 2014.
[51] J. Zhuang, W. Truccolo, C. Vargas-Irwin, et al., "Decoding 3-D reach and grasp kinematics from high-frequency local field potentials in primate primary motor cortex," IEEE Transactions on Biomedical Engineering, vol. 57, no. 7, pp. 1774-1784, 2010.
[52] J. P. Donoghue, J. N. Sanes, N. G. Hatsopoulos, et al., "Neural discharge and local field potential oscillations in primate motor cortex during voluntary movements," Journal of neurophysiology, vol. 79, no. 1, pp. 159-173, 1998.
[53] M. J. Rasch, A. Gretton, Y. Murayama, et al., "Inferring spike trains from local field potentials," vol. 99, no. 3, pp. 1461-1476, 2008.
[54] J. G. Jefferys, R. D. Traub, and M. A. Whittington, "Neuronal networks for induced ‘40 Hz’rhythms," Trends in neurosciences, vol. 19, no. 5, pp. 202-208, 1996.
[55] H. C. Tuckwell, "Introduction to theoretical neurobiology, " Cambridge University Press, vol. 8, 2005.
[56] A. Zell, "Simulation neuronaler netze," Addison-Wesley Bonn, vol. 1, no. 5.3, 1994.
[57] S. Walczak, "Artificial neural networks," Advanced Methodologies and Technologies in Artificial Intelligence, Computer Simulation, and Human-Computer Interaction: IGI Global, pp. 40-53, 2019.
[58] K. C. Li, "Sliced inverse regression for dimension reduction," Journal of the American Statistical Association, vol. 86, no. 414, pp. 316-327, 1991.
[59] J. K. Chapin, K. A. Moxon, R. S. Markowitz, et al., "Real-time control of a robot arm using simultaneously recorded neurons in the motor cortex," Nature neuroscience, vol. 2, no. 7, pp. 664-670, 1999.
[60] M. M. Churchland, G. Santhanam, and K. V. Shenoy, "Preparatory activity in premotor and motor cortex reflects the speed of the upcoming reach," Journal of neurophysiology, vol. 96, no. 6, pp. 3130-3146, 2006.
[61] J. Rickert, S. C. de Oliveira, E. Vaadia, et al., "Encoding of movement direction in different frequency ranges of motor cortical local field potentials," Journal of Neuroscience, vol. 25, no. 39, pp. 8815-8824, 2005.
[62] A. Jackson and T. M. Hall, "Decoding local field potentials for neural interfaces," IEEE Transactions on Neural Systems and Rehabilitation Engineering, vol. 25, no. 10, pp. 1705-1714, 2017.
[63] I. Asher, E. Stark, M. Abeles, et al., "Comparison of direction and object selectivity of local field potentials and single units in macaque posterior parietal cortex during prehension," Journal of neurophysiology, vol. 97, no. 5, pp. 3684-3695, 2007.
[64] D. Cheyne, S. Bells, P. Ferrari, et al., "Self-paced movements induce high-frequency gamma oscillations in primary motor cortex," Neuroimage, vol. 42, no. 1, pp. 332-342, 2008.
[65] M. T. Jurkiewicz, W. C. Gaetz, A. C. Bostan, et al., "Post-movement beta rebound is generated in motor cortex: evidence from neuromagnetic recordings," Neuroimage, vol. 32, no. 3, pp. 1281-1289, 2006.
[66] G. Pfurtscheller, B. Graimann, J. E. Huggins, et al., "Spatiotemporal patterns of beta desynchronization and gamma synchronization in corticographic data during self-paced movement," Clinical neurophysiology, vol. 114, no. 7, pp. 1226-1236, 2003.
[67] M. M. Mukaka, "A guide to appropriate use of correlation coefficient in medical research," Malawi Medical Journal, vol. 24, no. 3, pp. 69-71, 2012.