臺灣博碩士論文加值系統

近年，人工智慧已成為學術最為熱門的主題之一，主要歸功於硬體加速能力與方便的收集大量數據資料集，然而越大的網路架構雖能提高準確度，但記憶體使用量、功耗與運算處理時間等成本也隨之增加，如何有效的利用記憶體與速度達到一定的準確率已成為近年熱門的題目之一。
在本篇論文中，我們先介紹卷積神經網路近年發展，再來介紹與探討近年各種二值神經網路與變化，最後我們針對深度殘差網路(Deep Residual Network)提出改良反互斥或網路(Xnor-Net)，主要簡單化大型網路架構並大幅提升準確率與原先網路，其中調整深度殘差網路基本架構、增加輸入層結果可能、並將每個權重乘法利用較簡單的位元計數器，實驗結果在我們提出的網路中記憶體使用量相同於互斥反或網路，卻可大幅提高Cifar10/Cifar100數據集準確率，並不失去互斥反或網路與二值神經網路特色—省電與快速，結果在與其他論文比較也仍能展現出優秀準確率。

In recent years, deep learning has been one of the most popular subject in academia and widely used in many fields such as computer vision, image classification, motion recognition, voice recognition, and big-data analysis tasks. Although the larger neural network architecture can improve accuracy obviously, the cost of memory usage, power consumption and time consumption also increase. How to use memory and speed effectively to achieve a certain accuracy has been the most popular subject in recent years.
In the first part of this thesis, we will introduce the development of convolution neural network in recent years, and then we will introduce and explore the diversification of binary neural network. Finally, we will focus on Deep Residual Network and propose our method to improved XNOR-Net. By adjusting Deep Residual Network basic structure, increasing the possible of input layer and replacing more simply bit counter than multiplier, we can simplify large network architecture and increase accuracy than previous network greatly. The experimental results demonstrate that our design achieves the same performances in memory usage as XNOR-Net. Moreover, it can dramatically increase accuracy in Cifar10/Cifar100 datasets, and achieve the good accuracy result than other binary neural network paper.

第一章、 緒論
1.1 研究背景
1.2 研究動機與方向
1.3 論文組織
第二章、 相關探討
2.1 卷積神經網路(Convolutional Neural Network)
2.2 二值神經網路(Binary Neural Network)
2.3 深度殘差網路(Deep Residual Network)
2.4 參數【0】影響
第三章、 二值化殘差網路訓練
3.1 Residual Net架構與訓練方法
3.2 二值化Residual Net架構與訓練方法
第四章、 提出之演算法與改良
4.1 增加輸入量化結果
4.2 移除激勵函數ReLU層
4.3 Basic Block1優化處理
第五章、 實驗結果
5.1 Cifar10/Cifar100資料庫
5.2 移除ReLU層結果與量化0結果
5.3 與XNOR-Net兩種方法比較
5.4 與近年其他論文比較
5.5 其他數據比較
第六章、 結論與未來工作

[1] Y. LeCun, D. Touresky, G. Hinton, and T. Sejnowski, "A theoretical framework for back-propagation," in Proceedings of the 1988 connectionist models summer school, 1988, vol. 1, pp. 21-28: CMU, Pittsburgh, Pa: Morgan Kaufmann.
[2] H. Sak, A. Senior, and F. Beaufays, "Long short-term memory recurrent neural network architectures for large scale acoustic modeling," in Fifteenth annual conference of the international speech communication association, 2014.
[3] I. Goodfellow et al., "Generative adversarial nets," in Advances in neural information processing systems, 2014, pp. 2672-2680.
[4] S. Han, H. Mao, and W. J. Dally, "Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding," arXiv preprint arXiv:1510.00149, 2015.
[5] I. Hubara, M. Courbariaux, D. Soudry, R. El-Yaniv, and Y. Bengio, "Binarized Neural Networks," in Advances in Neural Information Processing Systems 29, D. D. Lee, M. Sugiyama, U. V. Luxburg, I. Guyon, and R. Garnett, Eds.: Curran Associates, Inc., 2016, pp. 4107-4115.
[6] M. Rastegari, V. Ordonez, J. Redmon, and A. Farhadi, "Xnor-net: Imagenet classification using binary convolutional neural networks," in European Conference on Computer Vision, 2016, pp. 525-542: Springer.
[7] K. He, X. Zhang, S. Ren, and J. Sun, "Deep Residual Learning for Image Recognition," in The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
[8] K. Fukushima and S. Miyake, "Neocognitron: A self-organizing neural network model for a mechanism of visual pattern recognition," in Competition and cooperation in neural nets: Springer, 1982, pp. 267-285.
[9] Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner, "Gradient-based learning applied to document recognition," Proceedings of the IEEE, vol. 86, no. 11, pp. 2278-2324, 1998.
[10] A. Krizhevsky, I. Sutskever, and G. E. Hinton, "Imagenet classification with deep convolutional neural networks," in Advances in neural information processing systems, 2012, pp. 1097-1105.
[11] IMAGENET Large Scale Visual Recognition Challenge 2012. Available: http://www.image-net.org/challenges/LSVRC/2012/
[12] S. Ioffe and C. Szegedy, "Batch normalization: Accelerating deep network training by reducing internal covariate shift," arXiv preprint arXiv:1502.03167, 2015.
[13] N. Srivastava, G. Hinton, A. Krizhevsky, I. Sutskever, and R. Salakhutdinov, "Dropout: a simple way to prevent neural networks from overfitting," The Journal of Machine Learning Research, vol. 15, no. 1, pp. 1929-1958, 2014.
[14] M. Courbariaux, Y. Bengio, and J.-P. David, "Binaryconnect: Training deep neural networks with binary weights during propagations," in Advances in neural information processing systems, 2015, pp. 3123-3131.
[15] C. Zhu, S. Han, H. Mao, and W. J. Dally, "Trained ternary quantization," arXiv preprint arXiv:1612.01064, 2016.
[16] Z. Lin, M. Courbariaux, R. Memisevic, and Y. Bengio, "Neural networks with few multiplications," arXiv preprint arXiv:1510.03009, 2015.
[17] H. Alemdar, V. Leroy, A. Prost-Boucle, and F. Pétrot, "Ternary neural networks for resource-efficient AI applications," in neural networks (IJCNN), 2017 international joint conference on, 2017, pp. 2547-2554: IEEE.
[18] V. Nair and G. E. Hinton, "Rectified linear units improve restricted boltzmann machines," in Proceedings of the 27th international conference on machine learning (ICML-10), 2010, pp. 807-814.
[19] cs231n Lecture4. Available: http://cs231n.stanford.edu/slides/2017/cs231n_2017_lecture4.pdf
[20] Y. Bengio, N. Léonard, and A. Courville, "Estimating or propagating gradients through stochastic neurons for conditional computation," arXiv preprint arXiv:1308.3432, 2013.
[21] Cifar10/100 dataset. Available: https://www.cs.toronto.edu/~kriz/cifar.html
[22] Pytorch. Available: https://pytorch.org/
[23] D. P. Kingma and J. Ba, "Adam: A method for stochastic optimization," arXiv preprint arXiv:1412.6980, 2014.
[24] ImageNet dataset. Available: http://image-net.org/download-images
[25] S. Zhou, Y. Wu, Z. Ni, X. Zhou, H. Wen, and Y. Zou, "Dorefa-net: Training low bitwidth convolutional neural networks with low bitwidth gradients," arXiv preprint arXiv:1606.06160, 2016.
[26] J. Choi, Z. Wang, S. Venkataramani, P. I.-J. Chuang, V. Srinivasan, and K. Gopalakrishnan, "PACT: Parameterized Clipping Activation for Quantized Neural Networks," arXiv preprint arXiv:1805.06085, 2018.
[27] K. Simonyan and A. Zisserman, "Very deep convolutional networks for large-scale image recognition," arXiv preprint arXiv:1409.1556, 2014.
[28] Z. Cai, X. He, J. Sun, and N. Vasconcelos, "Deep learning with low precision by half-wave gaussian quantization," arXiv preprint arXiv:1702.00953, 2017.
[29] L. Deng, P. Jiao, J. Pei, Z. Wu, and G. Li, "GXNOR-Net: Training deep neural networks with ternary weights and activations without full-precision memory under a unified discretization framework," Neural Networks, vol. 100, pp. 49-58, 2018.