臺灣博碩士論文加值系統

目錄
中文摘要 I
Abstract II
致謝 III
目錄 IV
圖目錄 V
表目錄 VI
第一章 緒論 1
1-1前言 2
1-2研究背景與目的 2
1-3深度神經網路 3
1-3-1監督式學習 4
1-3-2遷移式學習 6
1-3-3神經網路的傳遞 7
1-3-4優化器介紹 8
1-3-5神經網路的激活函式 9
1-3-6批量性標準化 11
第二章 記憶體中運算實現硬體式神經網路運算 12
2-1 記憶體中運算實現硬體式神經網路運算 13
2-2非重疊離子佈植元件之突觸元件介紹 18
2-2-1非重疊離子佈植電晶體之運算模型建立 20
2-2-2 非重疊離子佈植電晶體之運算電路實現 22
2-2-3 非重疊離子佈植記憶體實現硬體式神經網路運算 23
第三章 軟體平台訓練以及硬體式遷移式學習之辨識 25
3-1 資料庫介紹與處理 26
3-2 純軟體網路訓練與比較 27
3-3 神經網路載入非重疊離子佈植元件運算模型之模擬 29
3-4 非重疊離子佈植記憶體辨識電路 31
第四章 平行式誤差修正網路之設計及驗證 34
4-1 平行式誤差修正網路系統介紹以及資料庫的準備 35
4-2 硬體式遷移式學習之誤差分析 36
4-3 網路架構設計及實驗數據分析與討論 37
第五章 結論 41
5-1結論 41
5-2未來展望 42
參考文獻 43

圖目錄
圖1- 1 量化運算示意圖 3
圖1- 2 深度學習蓋括之領域 4
圖1- 3 監督式學習在深度神經網路之意義 5
圖1- 4 監督式學習實際訓練圖 5
圖1- 5 遷移式學習以及其網路應用示意圖 6
圖1- 6 神經網路的前向/倒傳遞模式 7
圖1- 7 隨機梯度下降法 8
圖1- 8 Linear 激活函式 9
圖1- 9 Sigmoid 激活函式 10
圖1- 10 ReLU 激活函式 10
圖1- 11 批量性標準化後之訓練示意圖 11
圖2- 1 馮紐曼架構示意圖 13
圖2- 2 ReRAM 2T2M架構示意圖 15
圖2- 3 MRAM 2T1M 架構示意圖 15
圖2- 4 FeRAM 2T2C架構示意圖 15
圖2- 5 SRAM 12T架構示意圖 16
圖2- 6 ReRAM數位CIM架構示意圖 16
圖2- 7 MRAM數位CIM架構示意圖 17
圖2- 8 NAND-Flash類比CIM架構示意圖 17
圖2- 9 MRAM類比CIM架構示意圖 18
圖2- 10 NOI元件進行熱電子注入示意 19
圖2- 11 NOI元件進行熱電洞抹除示意 20
圖2- 12 生物神經元示意 21
圖2- 13 代表神經元運作的等效模型 21
圖2- 14 NOI矩陣電路 22
圖2- 15 NOI神經網路運算晶片電路區塊圖 23
圖2- 16 NOI矩陣運算電路等校神經網路運算示意圖 24
圖3- 1 MNIST手寫資料庫的範例 26
圖3- 2 圖片標準化示意圖 26
圖3- 3 監督式學習架構 27
圖3- 4 監督式網路輸出神經元數量及時序關係 27
圖3- 5 監督式學習之模型訓練情況 29
圖3- 6 差動對電路 30
圖3- 7 NOI元件ID-VG曲線 30
圖3- 8 NOI元件ISA-VSA曲線 31
圖4- 1 本論文之系統架構 35
圖4- 2 硬硬硬誤差修正網路架構 38
圖4- 3 硬硬硬誤差修正網路時序關係 39

表目錄
表3- 1 監督式學習網路條件 28
表3- 2 監督式學習之辨識率表現 29
表3- 3 遷移式學習之辨識率 32
表3- 4 各層進行硬體遷移式學習之辨識率情形 33
表4- 1 各層網路經遷移後之權重偏移情況 36
表4- 2 硬硬硬誤差修正網路條件表 39
表4- 3 硬硬硬誤差修正網路實驗結果 40
表5- 1 修正網路於不同層進行修正後之結果 41

參考文獻
第一章
[1-1]W. Samek, G. Montavon, S. Lapuschkin, C. J. Anders and K. -R. Müller, "Explaining Deep Neural Networks and Beyond: A Review of Methods and Applications," in Proceedings of the IEEE, vol. 109, no. 3, pp. 247-278, March 2021.
[1-2]R. Y. Choi et al., "Introduction to Machine Learning, Neural Networks, and Deep Learning," Translational Vision Science & Technology, vol. 9, no. 2, pp. 14, Feb. 2020.
[1-3]O. Costilla-Reyes, P. Scully and K. B. Ozanyan, "Deep Neural Networks for Learning Spatio-Temporal Features From Tomography Sensors," in IEEE Transactions on Industrial Electronics, vol. 65, no. 1, pp. 645-653, Jan. 2018.
[1-4]R. Nash et al., "An Introduction to Convolutional Neural Networks," ArXiv, vol. abs/1511.08458, 2015.
[1-5]L. Deng, G. Hinton and B. Kingsbury, "New types of deep neural network learning for speech recognition and related applications: an overview," 2013 IEEE International Conference on Acoustics, Speech and Signal Processing, 2013, pp. 8599-8603.
[1-6]L. M. Q. d. Santana, R. M. Santos, L. N. Matos and H. T. Macedo, "Deep Neural Networks for Acoustic Modeling in the Presence of Noise," in IEEE Latin America Transactions, vol. 16, no. 3, pp. 918-925, March 2018.
[1-7]Z. Yan et al., "Automatic Photo Adjustment Using Deep Neural Networks," Association for Computing Machinery, vol. 35, no. 2, pp. 15, May 2016.
[1-8]B. Liang, H. Li, M. Su, X. Li, W. Shi and X. Wang, "Detecting Adversarial Image Examples in Deep Neural Networks with Adaptive Noise Reduction," in IEEE Transactions on Dependable and Secure Computing, vol. 18, no. 1, pp. 72-85, 1 Jan.-Feb. 2021.
[1-9]G. J. Scott, M. R. England, W. A. Starms, R. A. Marcum and C. H. Davis, "Training Deep Convolutional Neural Networks for Land–Cover Classification of High-Resolution Imagery," in IEEE Geoscience and Remote Sensing Letters, vol. 14, no. 4, pp. 549-553, April 2017.
[1-10]D. C. Cireşan, U. Meier and J. Schmidhuber, "Transfer learning for Latin and Chinese characters with Deep Neural Networks," The 2012 International Joint Conference on Neural Networks (IJCNN), 2012, pp. 1-6.
[1-11]A. Kaya et al., "Analysis of transfer learning for deep neural network based plant classification models," Computers and Electronics in Agriculture, Volume 158, Pages 20-29, 2019.
[1-12]M. H. Sazli, "A brief review of feed-forward neural networks", Communications Faculty of Sciences University of Ankara Series A2-A3 Physical Sciences and Engineering, vol. 50, no. 01, pp. 0-0, Jan. 2006.
[1-13]N. Rajput and S. K. Verma, "Back propagation feed forward neural network approach for Speech Recognition," Proceedings of 3rd International Conference on Reliability, Infocom Technologies and Optimization, 2014, pp. 1-6.
[1-14]F. F. Alkhalid, "The effect of optimizers in fingerprint classification model utilizing deep learning," Indonesian Journal of Electrical Engineering and Computer Science, vol. 20, pp. 1098-1102, 2020.
[1-15]F. Schneider et al., "DeepOBS: A Deep Learning Optimizer Benchmark Suite," ArXiv, March. 2019.
[1-16]X. Jin et al., "Deep Learning with S-Shaped Rectified Linear Activation Units," Thirtieth AAAI Conference on Artificial Intelligence, Vol. 30, no. 1, 2016.
[1-17]V. Thakkar, S. Tewary and C. Chakraborty, "Batch Normalization in Convolutional Neural Networks — A comparative study with CIFAR-10 data," 2018 Fifth International Conference on Emerging Applications of Information Technology (EAIT), 2018, pp. 1-5.
[1-18]O. Fagbohungbe, L. Qian, "Effect of Batch Normalization on Noise Resistant Property of Deep Learning Models," ArXiv, 2022.
第二章
[2-1]Merolla, Paul et al. “A million spiking-neuron integrated circuit with a scalable communication network and interface.” Science 345 (2014): 668 - 673.
[2-2]J. Kim and J. Park, "A Charge-domain 10T SRAM based In-Memory-Computing Macro for Low Energy and Highly Accurate DNN inference," 2021 18th International SoC Design Conference (ISOCC), 2021, pp. 89-90.
[2-3]S. Kim, W. Lee, S. Kim, S. Park and D. Jeon, "An In-Memory Computing SRAM Macro for Memory-Augmented Neural Network," in IEEE Transactions on Circuits and Systems II: Express Briefs, vol. 69, no. 3, pp. 1687-1691, March 2022.
[2-4]Y. Pan et al., "A Multilevel Cell STT-MRAM-Based Computing In-Memory Accelerator for Binary Convolutional Neural Network," in IEEE Transactions on Magnetics, vol. 54, no. 11, pp. 1-5, Nov. 2018, Art no. 9401305.
[2-5]M. Natsui et al., "Dual-Port SOT-MRAM Achieving 90-MHz Read and 60-MHz Write Operations Under Field-Assistance-Free Condition," in IEEE Journal of Solid-State Circuits, vol. 56, no. 4, pp. 1116-1128, April 2021.
[2-6]W. S. Zhao et al., "Embedded MRAM for high-speed computing," 2011 IEEE/IFIP 19th International Conference on VLSI and System-on-Chip, 2011, pp. 37-42.
[2-7]Q. Wu et al., "A Non-volatile Computing-in-Memory ReRAM Macro using Two-bit Current-Mode Sensing Amplifier," 2021 IEEE 10th Non-Volatile Memory Systems and Applications Symposium (NVMSA), 2021, pp. 1-6.
[2-8]J. Li, Y. Cui, C. Gu, C. Wang and W. Liu, "Dynamically Configurable Physical Unclonable Function based on RRAM Crossbar," 2021 IEEE/ACM International Symposium on Nanoscale Architectures (NANOARCH), 2021, pp. 1-6.
[2-9]C. -C. Chou et al., "An N40 256K×44 embedded RRAM macro with SL-precharge SA and low-voltage current limiter to improve read and write performance," 2018 IEEE International Solid - State Circuits Conference - (ISSCC), 2018, pp. 478-480.
[2-10]D. Bankman, J. Messner, A. Gural and B. Murmann, "RRAM-Based In-Memory Computing for Embedded Deep Neural Networks," 2019 53rd Asilomar Conference on Signals, Systems, and Computers, 2019, pp. 1511-1515.
[2-11]S. Slesazeck, T. Ravsher, V. Havel, E. T. Breyer, H. Mulaosmanovic and T. Mikolajick, "A 2TnC ferroelectric memory gain cell suitable for compute-in-memory and neuromorphic application," 2019 IEEE International Electron Devices Meeting (IEDM), 2019, pp. 38.6.1-38.6.4.
[2-12]Y. Luo, Y. -C. Luo and S. Yu, "A Ferroelectric-based Volatile/Non-volatile Dual-mode Buffer Memory for Deep Neural Network Accelerators," in IEEE Transactions on Computers, 2021.
[2-13]Y. Luo, Y. -C. Luc and S. Yu, "A FeRAM based Volatile/Non-volatile Dual-mode Buffer Memory for Deep Neural Network Training," 2021 Design, Automation & Test in Europe Conference & Exhibition (DATE), 2021, pp. 1871-1876.
[2-14]L. Zhang et al., "A high-reliability and low-power computing-in-memory implementation within STT-MRAM," Microelectronics Journal, Volume 81, 2018, Pages 69-75.
[2-15]Y. Luo et al., "Performance Benchmarking of Spin-Orbit Torque Magnetic RAM (SOT-MRAM) for Deep Neural Network (DNN) Accelerators," 2022 IEEE International Memory Workshop (IMW), 2022, pp. 1-4.
[2-16]L. Wang et al., "Efficient and Robust Nonvolatile Computing-In-Memory Based on Voltage Division in 2T2R RRAM With Input-Dependent Sensing Control," in IEEE Transactions on Circuits and Systems II: Express Briefs, vol. 68, no. 5, pp. 1640-1644, May 2021.
[2-17]F. Tan et al., "A ReRAM-Based Computing-in-Memory Convolutional-Macro With Customized 2T2R Bit-Cell for AIoT Chip IP Applications," in IEEE Transactions on Circuits and Systems II: Express Briefs, vol. 67, no. 9, pp. 1534-1538, Sept. 2020.
[2-18]Z. Jing, B. Yan, Y. Yang and R. Huang, "VSDCA: A Voltage Sensing Differential Column Architecture Based on 1T2R RRAM Array for Computing-in-Memory Accelerators," in IEEE Transactions on Circuits and Systems I: Regular Papers, 2022.
[2-19]T. Ravsher, H. Mulaosmanovic, E. T. Breyer, V. Havel, T. Mikolajick and S. Slesazeck, "Adoption of 2T2C ferroelectric memory cells for logic operation," 2019 26th IEEE International Conference on Electronics, Circuits and Systems (ICECS), 2019, pp. 791-794.
[2-20]X. Si et al., "A Twin-8T SRAM Computation-in-Memory Unit-Macro for Multibit CNN-Based AI Edge Processors," in IEEE Journal of Solid-State Circuits, vol. 55, no. 1, pp. 189-202, Jan. 2020.
[2-21]C. -H. Lee, Y. -T. Hsu, T. -T. Liu and T. -D. Chiueh, "Design of an 45nm NCFET Based Compute-in-SRAM for Energy-Efficient Machine Learning Applications," 2020 IEEE Asia Pacific Conference on Circuits and Systems (APCCAS), 2020, pp. 193-196.
[2-22]H. Fujiwara et al., "A 5-nm 254-TOPS/W 221-TOPS/mm2 Fully-Digital Computing-in-Memory Macro Supporting Wide-Range Dynamic-Voltage-Frequency Scaling and Simultaneous MAC and Write Operations," 2022 IEEE International Solid- State Circuits Conference (ISSCC), 2022, pp. 1-3.
[2-23]Z. Jiang, S. Yin, M. Seok and J. Seo, "XNOR-SRAM: In-Memory Computing SRAM Macro for Binary/Ternary Deep Neural Networks," 2018 IEEE Symposium on VLSI Technology, 2018, pp. 173-174.
[2-24]Y. Pan et al., "A Multilevel Cell STT-MRAM-Based Computing In-Memory Accelerator for Binary Convolutional Neural Network," in IEEE Transactions on Magnetics, vol. 54, no. 11, pp. 1-5, Nov. 2018.
[2-25]E. Giacomin, T. Greenberg-Toledo, S. Kvatinsky and P. Gaillardon, "A Robust Digital RRAM-Based Convolutional Block for Low-Power Image Processing and Learning Applications," in IEEE Transactions on Circuits and Systems I: Regular Papers, vol. 66, no. 2, pp. 643-654, Feb. 2019.
[2-26]L. Wang et al., "Efficient and Robust Nonvolatile Computing-In-Memory Based on Voltage Division in 2T2R RRAM With Input-Dependent Sensing Control," in IEEE Transactions on Circuits and Systems II: Express Briefs, vol. 68, no. 5, pp. 1640-1644, May 2021.
[2-27]M. Kim, M. Liu, L. R. Everson and C. H. Kim, "An Embedded nand Flash-Based Compute-In-Memory Array Demonstrated in a Standard Logic Process," in IEEE Journal of Solid-State Circuits, vol. 57, no. 2, pp. 625-638, Feb. 2022.
[2-28]Z. Hao et al., "A Computing-in-memory Scheme with Series Bit-cell in STT-MRAM for Efficient Multi-bit Analog Multiplication," 2021 IEEE/ACM International Symposium on Nanoscale Architectures (NANOARCH), 2021, pp. 1-6.
[2-29]C. S. Hsieh et al., "NVM Characteristics of Single-MOSFET Cells Using Nitride Spacers with Gate-to-Drain NOI," IEEE Trans. Electron Device, vol. 51, no. 11, pp. 1811-1817, Nov. 2004
[2-30]Erik S. Jeng et al., "Investigation of Programming Charge Distribution in Non-overlapped Implantation nMOSFETs," IEEE Trans. Electron Device, vol. 53, no. 10, pp. 2517-2524, Oct. 2006.
[2-31]E. S. Jeng, C. S. Chiu, C. H. Hon, P. C. Kuo, C. C. Fan, C. S. Hsieh, H. C. Hsu, and Y. F. Chen, “Performance improvement and scalability of nonoverlapped implantation nMOSFETs with charge-trapping spacers as nonvolatile memories,” IEEE Trans. Electron Device, vol. 54, no. 12, pp. 3299-3307, Dec. 2007.
[2-32]E. S. Jeng, K. M. Peng, S. W. Chou, H. X. Chen, and Y. L. Chiang, "Non-overlapped implantation (NOI) MOSFET synapse and its implementation on supervised neural network," Neurocomputing, vol. 167, pp. 290-298, Nov. 2015.
第三章
[3-1]F. Chen et al., " Assessing four Neural Networks on Handwritten Digit Recognition Dataset (MNIST)," ArXiv. 2018.
[3-2]E.S. Jeng, H. X. Chen, Y. L. Chiang, J. H. Chang and J. Y. Chen, "A non-overlapped implantation MOSFET differential pair implementation of bidirectional weight update synapse for neuromorphic computing," Microelectronics Journal, Volume:90, pp. 306-314, 2019.
第四章
[4-1]H. Deng et al., "An efficient background calibration technique for analog-to-digital converters based on neural network," Integration, Vol. 74, pp. 63-70, 2020.
[4-2]D. Pedamonti, " Comparison of non-linear activation functions for deep neural networks on MNIST classification task," ArXiv. 2018.