本論文之目的為輕量化卷積神經網路模型的參數，藉以改善原本網路模型參數過大，以至於無法在移動式裝置或小運算資源平台無法運算或是運算緩慢的缺點，並且利用了三種不同的方法來解決上述問題。這三種方法可分成兩大類，第一類為修改網路架構的群組卷積神經網路與分離卷積神經網路，第二類為刪減已訓練好的網路模型的權重的網路模型剪枝。其中，群組卷積神經網路運用輸入特徵圖的分群和有效地拆分、分離卷積神經網路利用了輸入特徵圖的拆分在耦合一個逐點式卷積層，再者網路型剪枝利用了不同的權重分析方式進一步刪減特徵圖或是卷積核的權重來達到參數輕量化的效果。而這三種不同的方法也結合其他有名方法例如：擠壓激發區塊與洗牌機制，或是進一步分析網路模型的參數分布來實現網路參數輕量化的目的：減少大量參數，性能依然維持一定水平。實驗結果證實利用不同的方法於不同的網路模型與資料集上面，能達到良好的效果，並且最終做出了詳細的比較來，與尚未進行網路模型參數輕量化的原先模型相比較，以便三種佐證方法皆有達成網路模型參數輕量化之目的。
 

In this thesis, lightweight convolutional neural network (CNN) model is presented to reduce the number of parameters of conventional CNNs because the size of regular CNN is too heavy to perform well on mobile and low-performance devices, there are two types of strategies to design lightweight CNN. One is to redesign network’s architecture, the other is to prune redundant weights such as convolutional feature maps and filter weights. In the first strategy, there are two methods including the group convolutional neural network that reduces number of parameters by splitting the input of feature maps into many small groups efficiently, and separable convolutional neural network that splits input of each feature maps before performing a pointwise convolutional layer. In the second strategy, network model pruning method deletes redundant weights significantly after analyzing different famous network models on different datasets. This thesis also adopts other two operations in the literature to further reduce the number of the parameters including squeeze-and-extraction operation from SE-Net and shuffle operation from ShffleNet. Experimental results show that three methods not only reduce the number of parameters significantly but also maintain the performance of classification. Finally, comparisons between original network model and lightweight network models are made to demonstrate that all of methods achieve the purpose of the design of lightweight CNN model.

中 文 摘 要 I
ABSTRACT II
致謝 III
目錄 IV
表目錄 XII
壹、 緒 論 1
一、 研究背景 1
二、 研究動機 3
三、 論文主題與架構 4
貳、 傳統卷積神經網路 6
一、 簡介 6
(一)、 卷積層 7
(二)、 池化層 8
(三)、 激活函式 10
(四)、 損失函式 13
(五)、 優化演算法 15
(六)、 其他常見方法 18
(七)、 知名的網路模型 21
二、 總結 26
參、 群組卷積神經網路 28
一、 簡介 28
二、 基本架構方塊 28
(一)、 瓶頸架構(Bottleneck) 28
(二)、 基數 31
(三)、 內部結構 32
三、 改進與延伸 34
(一)、 使用洗牌(Shuffle)機制 34
(二)、 使用SE-Block架構 36
四、 實驗 38
(一)、 資料集 38
(二)、 實驗環境與設置 40
(三)、 實驗結果 41
肆、 分離卷積神經網路 58
一、 簡介 58
二、 基本架構方塊 58
(一)、 深度可分離卷積層 58
(二)、 顛倒式瓶頸架構 62
(三)、 激活函式之改進 64
三、 網路模型歸納與探討 66
(一)、 MobileNet系列 66
(二)、 ShuffleNet系列 70
四、 實驗 74
(一)、 資料集 74
(二)、 實驗環境與設置 75
(三)、 實驗結果 76
伍、 網路模型剪枝 86
一、 簡介 86
二、 權重評估方法 86
三、 模型剪枝方法 89
(一)、 基於特徵圖的剪枝方法 90
(二)、 基於卷積核權重的剪枝方法 91
四、 實驗 92
(一)、 基於特徵圖的剪枝 93
(二)、 基於卷積核的剪枝 102
陸、 結論與未來展望 112
一、 結論 112
(一)、 Flower102資料集 112
(二)、 CIFAR-100資料集 113
(三)、 CIFAR-10資料集 114
二、 未來展望 115
參考文獻 116

[1]周秉誼, 淺談Deep Learning原理及應用, 台灣大學計算機及資訊網路中心, 第0038期‧2016.09.20 發行, ISSN 2077-8813,
http://www.cc.ntu.edu.tw/chinese/epaper/0038/20160920_3805.html
[2]Google Trend, https://trends.google.com.tw/trends/?geo=TW
[3]LeCun, Y., Bottou, L., Bengio, Y., & Haffner, P. (1998). Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11), 2278-2324.
[4]Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2012). Imagenet classification with deep convolutional neural networks. In Advances in neural information processing systems (pp. 1097-1105).
[5]Springenberg, J. T., Dosovitskiy, A., Brox, T., & Riedmiller, M. (2014). Striving for simplicity: The all convolutional net. arXiv preprint arXiv:1412.6806.
[6]Nair, V., & Hinton, G. E. (2010). Rectified linear units improve restricted boltzmann machines. In Proceedings of the 27th international conference on machine learning (ICML-10) (pp. 807-814).
[7]Xu, B., Wang, N., Chen, T., & Li, M. (2015). Empirical evaluation of rectified activations in convolutional network. arXiv preprint arXiv:1505.00853.
[8]Ruder, S. (2016). An overview of gradient descent optimization algorithms. arXiv preprint arXiv:1609.04747.
[9]Duchi, J., Hazan, E., & Singer, Y. (2011). Adaptive subgradient methods for online learning and stochastic optimization. Journal of machine learning research, 12(Jul), 2121-2159.
[10]Geoffrey Hinton,”RMSprop”,https://arxiv.org/pdf/1609.04747.pdf
[11]Kingma, D. P., & Ba, J. (2014). Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980.
[12]Srivastava, N., Hinton, G., Krizhevsky, A., Sutskever, I., & Salakhutdinov, R. (2014). Dropout: a simple way to prevent neural networks from overfitting. The journal of machine learning research, 15(1), 1929-1958.
[13]Ioffe, S., & Szegedy, C. (2015). Batch normalization: Accelerating deep network training by reducing internal covariate shift. arXiv preprint arXiv:1502.03167.
[14]Simonyan, K., & Zisserman, A. (2014). Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556.
[15]He, K., Zhang, X., Ren, S., & Sun, J. (2016). Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 770-778).
[16]Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., ... & Rabinovich, A. (2015). Going deeper with convolutions. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 1-9).
[17]Ioffe, S., & Szegedy, C. (2015). Batch normalization: Accelerating deep network training by reducing internal covariate shift. arXiv preprint arXiv:1502.03167.
[18]Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J., & Wojna, Z. (2016). Rethinking the inception architecture for computer vision. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 2818-2826).
[19]Szegedy, C., Ioffe, S., Vanhoucke, V., & Alemi, A. A. (2017, February). Inception-v4, inception-resnet and the impact of residual connections on learning. In Thirty-first AAAI conference on artificial intelligence.
[20]Xie, S., Girshick, R., Dollár, P., Tu, Z., & He, K. (2017). Aggregated residual transformations for deep neural networks. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 1492-1500).
[21]Zhang, X., Zhou, X., Lin, M., & Sun, J. (2018). Shufflenet: An extremely efficient convolutional neural network for mobile devices. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 6848-6856).
[22]Hu, J., Shen, L., & Sun, G. (2018). Squeeze-and-excitation networks. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 7132-7141).
[23]Chollet, F. (2017). Xception: Deep learning with depthwise separable convolutions. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 1251-1258).
[24]Liu, K. H., Yan, S., & Kuo, C. C. J. (2015). Age estimation via grouping and decision fusion. IEEE Transactions on Information Forensics and Security, 10(11), 2408-2423.
[25]Hu, Z., Hu, Y., Liu, J., Wu, B., Han, D., & Kurfess, T. (2018). 3D separable convolutional neural network for dynamic hand gesture recognition. Neurocomputing, 318, 151-161.
[26]Howard, A. G., Zhu, M., Chen, B., Kalenichenko, D., Wang, W., Weyand, T., ... & Adam, H. (2017). Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint arXiv:1704.04861.
[27]Lin, M., Chen, Q., & Yan, S. (2013). Network in network. arXiv preprint arXiv:1312.4400.
[28]Sandler, M., Howard, A., Zhu, M., Zhmoginov, A., & Chen, L. C. (2018). Mobilenetv2: Inverted residuals and linear bottlenecks. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 4510-4520).
[29]Ramachandran, P., Zoph, B., & Le, Q. V. (2017). Searching for activation functions. arXiv preprint arXiv:1710.05941.
[30]Howard, A., Sandler, M., Chu, G., Chen, L.C., Chen, B., Tan, M., Wang, W., Zhu, Y., Pang, R., Vasudevan, V. and Le, Q.V. (2019). Searching for mobilenetv3. In Proceedings of the IEEE International Conference on Computer Vision (pp. 1314-1324).
[31]Ma, N., Zhang, X., Zheng, H. T., & Sun, J. (2018). Shufflenet v2: Practical guidelines for efficient cnn architecture design. In Proceedings of the European Conference on Computer Vision (ECCV) (pp. 116-131).
[32]Liu, B., Wang, M., Foroosh, H., Tappen, M., & Pensky, M. (2015). Sparse convolutional neural networks. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 806-814).
[33]Hu, H., Peng, R., Tai, Y. W., & Tang, C. K. (2016). Network trimming: A data-driven neuron pruning approach towards efficient deep architectures. arXiv preprint arXiv:1607.03250.
[34]Li, H., Kadav, A., Durdanovic, I., Samet, H., & Graf, H. P. (2016). Pruning filters for efficient convnets. arXiv preprint arXiv:1608.08710.