跳到主要內容

臺灣博碩士論文加值系統

(100.24.118.144) 您好!臺灣時間:2022/12/06 04:59
字體大小: 字級放大   字級縮小   預設字形  
回查詢結果 :::

詳目顯示

我願授權國圖
: 
twitterline
研究生:郭俊麟
研究生(外文):Jiun-Lin Kuo
論文名稱:運用多階層式電阻式記憶體實現類神經網路運算
論文名稱(外文):The Implementation of Neuromorphic Computing Network By Multiple Levels Resistance Random Access Memory
指導教授:莊紹勳
指導教授(外文):Chung, Shao-Shiun
口試委員:李佩雯郭治群
口試委員(外文):Li, Pei-WenGuo, Jyh-Chyurn
口試日期:2020-01-02
學位類別:碩士
校院名稱:國立交通大學
系所名稱:電子研究所
學門:工程學門
學類:電資工程學類
論文種類:學術論文
論文出版年:2020
畢業學年度:108
語文別:英文
論文頁數:58
中文關鍵詞:類神經網路電阻式記憶體深度學習
外文關鍵詞:neuromorphic computingRRAMdeep learning
相關次數:
  • 被引用被引用:0
  • 點閱點閱:165
  • 評分評分:
  • 下載下載:0
  • 收藏至我的研究室書目清單書目收藏:0
隨著電子科技的大幅進步,大數據服務漸漸成為一個很重要的議題,為了分析大數據,同時快速推動了機器學習的發展。現今主流的機器學習方法是基於GPU的數值運算,然而這種范紐曼架構(von Neumann arch.)的計算機計算速度有上限,這個上限來自於頻繁地對動態記憶體(DRAM)的存取。有一種方法理論上能夠突破這個限制,透過結合儲存及運算於單一元件上來降低資料存取頻率。這種方法稱作類神經運算網路(neuromorphic computing)。另一方面,儲存的浮點數精度也是一個機器學習中的議題,當我們將網路中的變數視為連續的,這個網路會對變數的誤差非常敏感,為了提升精度,可將單精度(single precision)浮點數改為雙精度(double precision)浮點數,但是,這會使儲存的資料大小變大,對讀取資料時的頻寬要求也會上升,不利於終端應用。目前已經有針對低精度變數網路的研究,已經成功證實能夠在將網路中的變數視為不連續時,能夠正常運作,由這個結論推論,當精度下降到幾個位元時,若依然能夠運作,那麼使用前瞻記憶體來組成類神經網路,將會是個適當的選擇。

本文中,吾人透過探討利用多階層式電阻式記憶體(resistance random access memory, RRAM)來實現類神經網路運算,在研究方法上,我們將使用機器學習演算法來建立一個特殊的模型,接著將從吾人所製備的電阻式記憶體上萃取的電性帶入類神經網路的模型,在MNIST資料組中,能夠在2階層時準確度達到98.3%,在4階層時達到98.9%,在CIFAR-10資料組中,4階層時準確度能夠達到83.8%。最後更進一步,我們探討RRAM的變異性誤差會如何影響訓練的過程。

最後,我們成功地說明了, 運用我們的RRAM元件來實現類神經網路運算是可行的。
As the electronics technology advance, big data become a more and more important topic. In order to analyze very large amount of data, it drives machine learning to develop more quickly. Now, the mainstream of the machine learning machine is based on GPU. However, this traditional computing architecture, known as von Neumann architecture, has a limitation of speed. The limitation is due to the latency of memory access, which is known as memory wall. One of solutions is neuromorphic computing, which combines memory storage units and computing units together. On the other hand, the precision of a floating-point variable is another topic in machine learning. If we thought variables are continuous, the network would be sensitive to errors from digitalization. As the result, more bits are needed when we describe variables. Unfortunately, those variables, such as double precision, bring us the requirement of bandwidth and storage. There are many successful researches about network with low-precision weights. Low -precision weights are good for neuromorphic computing, too. It highly reduces the requirements of units so that building a network with emerging memory devices become possible.

In this thesis, we would like to build up a neuromorphic computing circuit by our RRAMs. Based on this framework, we developed our operation method to achieve RRAM devices with the capability of near-linear-conductance-tuning. Our RRAM can achieve 128 levels and levels spread uniformly. Then we applied the device properties to machine learning API and demonstrated the neural network whose weights are comprised by the model of RRAM electrical characteristics. We can achieve an accuracy of 98.3% with 2-level weights and 98.9% with 4-level weights over “MNIST” dataset. If we choose “CIFAR-10” as the dataset, we can get accuracy of 83.8% with 4-level weights. Furthermore, we discussed how the RRAM error affects the test accuracy of the training results.

Finally, in the proposed methodology, we have successfully proven that the quantized-weights network based on the characteristics of few-level storage in RRAM can be utilized in the image-detection during training and inference.
摘要 i
ABSTRACT ii
誌 謝 iv
Contents v
Figure Captions viii
Table Captions x
Chapter 1 Introduction 1
1.1. Background 1
1.2. Motivation 1
1.3. Cross-point Neural Network 3
1.4. The Organization of The Thesis 3
Chapter 2 The Device Level 6
2.1. The Device Structure and Basic Characteristics 6
2.1.1. Setting and Resetting Processes 6
2.1.2. The Influences of Thickness 6
2.1.3. Multiple Levels Characteristic 7
2.2. Pulse Operation 8
2.2.1. Experiment 8
2.2.2. The Operation of Multiple Levels in One RRAM Device 9
2.2.3. A Possible Explanation for Multiple Levels in One RRAM Device 9
2.2.4. Endurance and Stability 9
Chapter 3 The Implementation of Applications to Neural Networks 26
3.1. Introduction to Machine Learning 26
3.1.1. The Concepts of Machine Learning 26
3.1.2. Building a Model 26
3.1.3. Fully Connected Layers 26
3.1.4. Convolution layers 27
3.1.5. Activation Function 27
3.1.6. Training 28
3.1.7. Overfitting and regularization 29
3.1.8. Inference 29
3.2. Low Precision Network 30
3.2.1. Review 30
3.2.2. Rounding 30
3.3. Experiment 31
3.3.1. Environment Setup 31
3.3.2. Introduction to Tensorflow 31
3.3.3. The Build-up of a Framework for Quantized Networks 32
3.3.4. Simulation of Quantized Neural Network 32
3.4. Discussion 33
3.4.1. Training on Few Levels of Neural Networks 33
3.4.2. More Discussions about Few-Level Networks 34
3.4.3. Variation versus Accuracy 34
3.5. Extension and Future Work 36
3.5.1. Problem Encountered with Quantized Operation 36
3.5.2. Assumption - Regularization Effect from Quantization and Noise 37
3.5.3. Assumption – Information Spreading 38
Chapter 4 Conclusions 54
References 56
[1] S. Suthaharan, "Big data classification: Problems and challenges in network intrusion prediction with machine learning," ACM SIGMETRICS Performance Evaluation Review, vol. 41, no. 4, pp. 70-73, 2014.
[2] T. Condie, P. Mineiro, N. Polyzotis, and M. Weimer, "Machine learning on big data," in 2013 IEEE 29th International Conference on Data Engineering (ICDE), 2013: IEEE, pp. 1242-1244.
[3] J. D. Owens, M. Houston, D. Luebke, S. Green, J. E. Stone, and J. C. Phillips, "GPU computing," Proceedings of the IEEE, 2008.
[4] J. Nickolls and W. J. Dally, "The GPU computing era," IEEE micro, vol. 30, no. 2, pp. 56-69, 2010.
[5] S. A. McKee, "Reflections on the memory wall," in Conf. Computing Frontiers, 2004, p. 162.
[6] D. Stamoulis, E. Cai, D.-C. Juan, and D. Marculescu, "HyperPower: Power-and memory-constrained hyper-parameter optimization for neural networks," in 2018 Design, Automation & Test in Europe Conference & Exhibition (DATE), 2018: IEEE, pp. 19-24.
[7] J. Mudigonda, H. M. Vin, and R. Yavatkar, "Overcoming the memory wall in packet processing: hammers or ladders?," in Proceedings of the 2005 ACM symposium on Architecture for networking and communications systems, 2005: ACM, pp. 1-10.
[8] D. Monroe, "Neuromorphic computing gets ready for the (really) big time," Communications of the ACM, vol. 57, no. 6, pp. 13-15, 2014.
[9] G. W. Burr, R. M. Shelby, A. Sebastian, S. Kim, S. Kim, S. Sidler, K. Virwani, M. Ishii, P. Narayanan, and A. Fumarola, "Neuromorphic computing using non-volatile memory," Advances in Physics: X, vol. 2, no. 1, pp. 89-124, 2017.
[10] S. Hamdioui, S. Kvatinsky, G. Cauwenberghs, L. Xie, N. Wald, S. Joshi, H. M. Elsayed, H. Corporaal, and K. Bertels, "Memristor for computing: Myth or reality?," in Proceedings of the Conference on Design, Automation & Test in Europe, 2017: European Design and Automation Association, pp. 722-731.
[11] M. Hu, H. Li, Y. Chen, Q. Wu, G. S. Rose, and R. W. Linderman, "Memristor crossbar-based neuromorphic computing system: A case study," IEEE transactions on neural networks and learning systems, vol. 25, no. 10, pp. 1864-1878, 2014.
[12] E. R. Hsieh, Y. C. Kuo, C.-H. Cheng, J. L. Kuo, M.-R. Jiang, J.-L. Lin, H.-W. Chen, S. S. Chung, C.-H. Liu, and T. P. Chen, "A 14-nm FinFET Logic CMOS Process Compatible RRAM Flash With Excellent Immunity to Sneak Path," IEEE Transactions on Electron Devices, vol. 64, no. 12, pp. 4910-4918, 2017.
[13] K.-C. Chuang, K.-Y. Lin, J.-D. Luo, W.-S. Li, Y.-S. Li, C.-Y. Chu, and H.-C. Cheng, "Effects of electric fields on the switching properties improvements of RRAM device with a field-enhanced elevated-film-stack structure," IEEE Journal of the Electron Devices Society, vol. 6, pp. 622-626, 2018.
[14] H. Lee, P. Chen, T. Wu, Y. Chen, C. Wang, P. Tzeng, C. Lin, F. Chen, C. Lien, and M.-J. Tsai, "Low power and high speed bipolar switching with a thin reactive Ti buffer layer in robust HfO2 based RRAM," in 2008 IEEE International Electron Devices Meeting, 2008: IEEE, pp. 1-4.
[15] Y.-S. Chen, T.-Y. Wu, P.-J. Tzeng, P.-S. Chen, H.-Y. Lee, C.-H. Lin, F. Chen, and M.-J. Tsai, "Forming-free HfO 2 bipolar RRAM device with improved endurance and high speed operation," in 2009 International Symposium on VLSI Technology, Systems, and Applications, 2009: IEEE, pp. 37-38.
[16] E. Alpaydin, Introduction to machine learning. MIT press, 2009.
[17] P. Ramachandran, B. Zoph, and Q. V. Le, "Searching for activation functions," arXiv preprint arXiv:1710.05941, 2017.
[18] U. Köster, T. Webb, X. Wang, M. Nassar, A. K. Bansal, W. Constable, O. Elibol, S. Gray, S. Hall, and L. Hornof, "Flexpoint: An adaptive numerical format for efficient training of deep neural networks," in Advances in neural information processing systems, 2017, pp. 1742-1752.
[19] S. Gupta, A. Agrawal, K. Gopalakrishnan, and P. Narayanan, "Deep learning with limited numerical precision," in International Conference on Machine Learning, 2015, pp. 1737-1746.
[20] A. Krizhevsky, I. Sutskever, and G. E. Hinton, "Imagenet classification with deep convolutional neural networks," in Advances in neural information processing systems, 2012, pp. 1097-1105.
[21] L. Bottou and O. Bousquet, "The tradeoffs of large scale learning," in Advances in neural information processing systems, 2008, pp. 161-168.
[22] P. Micikevicius, S. Narang, J. Alben, G. Diamos, E. Elsen, D. Garcia, B. Ginsburg, M. Houston, O. Kuchaiev, and G. Venkatesh, "Mixed precision training," arXiv preprint arXiv:1710.03740, 2017.
[23] C. Zhu, S. Han, H. Mao, and W. J. Dally, "Trained ternary quantization," arXiv preprint arXiv:1612.01064, 2016.
[24] R. Durrett, Probability: theory and examples. Cambridge university press, 2019.
連結至畢業學校之論文網頁點我開啟連結
註: 此連結為研究生畢業學校所提供,不一定有電子全文可供下載,若連結有誤,請點選上方之〝勘誤回報〞功能,我們會盡快修正,謝謝!
QRCODE
 
 
 
 
 
                                                                                                                                                                                                                                                                                                                                                                                                               
第一頁 上一頁 下一頁 最後一頁 top
無相關期刊