臺灣博碩士論文加值系統

本研究針對藥丸自動視覺偵測與辨識功能進行開發。本系統包含兩個關鍵技術以及一個藥丸資料庫：(1) 多藥丸偵測技術：本技術基於特徵金字塔網路之物件偵測技術建構一藥丸偵測網路模型。在輸入影像中，搜尋一至多顆藥丸的座標位置，並將利用座標位置進行裁切後的結果傳送至藥丸辨識神經網路進行要藥丸種類辨識。本技術藥丸偵測度達95%具高準確度且平均偵測時間為0.07秒，具快速偵測圖片中藥丸位置的效能；(2)藥丸辨識技術：本技術使用卷積神經網路訓練一藥丸辨識網路，將多藥丸偵測技術偵測到的每顆藥丸進行辨識並產生相對應藥丸種類。其藥丸辨識網路的準確度在目前所蒐集的612種類藥丸中達90%，具高辨識率；且平均處理每顆藥丸的時間為0.02秒具快速辨識各種藥丸的效能；(3)藥丸資料庫：本團隊透過一套標準作業程序，蒐集一國內地方醫學中心藥丸資料庫，其中包含612類約7,000筆藥丸拍攝座標與經過資料增集合計約240萬張藥丸拍攝圖片。本技術可應用於廣泛的情況，如家庭醫藥安全，了解未知藥物和回收藥物分析。醫務人員和患者往往難以區分無包裝藥物。藥物知識的改進和向患者提供充足的藥物信息已成為減少藥物浪費和提高藥物安全性的重要問題。

Automatic Multi-Pill Detection and Recognition System includes two Convolution Normal Network (CNN) models, and a pill dataset: (1)Multi-pill detection: The detection model is built based on the Feature Pyramid Network and Convolution Normal Network (CNN). The pill detector estimates coordinates of one to more pills; and the pill images are cropped from the input image, according to the estimated result. The accuracy of pill detection achieves 95% and the execution time is 0.065s; (2)Pill recognition: A CNN model is used to classify the pill images, which are captured by the multi-pill detection task. The accuracy achieves 90% and the execution time is 0.02s for classification task of each pill; (3) Pill dataset: the pill dataset is constructed from images collected in a local medical center. Training, validation and testing datasets are included in the database, which contains 2,429,753 images with data argumentation under 612 drugs categories. The pill detection can be applied to wide range of situations, such as home medicine safety, understand the unknown drug. Medical personnel and patients often have difficulties in distinguishing unpackaged pills. The improvement in medication knowledge and the provision of adequate pill information to patients have become important issues in an effort to increase medicine safety.

中文摘要 I
Abstract II
誌謝 IV
Content V
Table List VI
Figure List VII
Chapter 1 Introduction 1
1.1 Background and Motivation 1
1.2 Objective 2
1.3 Organization 3
1.4 Contribution 3
Chapter 2 Related Work 4
2.1 Automatic Drug Detection 4
2.2 Drug Dataset 5
2.3 Deep Convolution Neural Networks Computer Vision 6
2.3.1 CNN-based Object Detector 6
2.3.2 CNN-based Object Classifier 8
Chapter 3 The Proposed System 11
3.1 System Overview 11
3.2 Enhanced Feature Pyramid Network for Drug Detection 12
3.2.1 Feature Extraction Backbone 12
3.2.2 Enhanced Feature Pyramid Network Structure 14
3.2.3 Regression module and Confidence module 17
3.2.4 Multi-task Loss 19
3.3 Inception-ResNet v2 for Drug Classification 23
3.3.1 Inception-ResNet v2 23
Chapter 4 Experiments 26
4.1 Drug Pills Image Database 26
4.2 Training and Inference 27
4.3 Evaluation and Discussion 28
4.4 Mistaken Cases 35
Chapter 5 Conclusion and Future Work 37
Reference 38

[1] D. Ushizima, A. Carneiro, M. Souza, and F. Medeiros, Investigating pill recognition methods for a new national library of medicine image dataset, in International Symposium on Visual Computing, 2015: Springer, pp. 410-419.
[2]C. L. Bekker, H. Gardarsdottir, T. Egberts, B. J. van de Bemt, and M. Bouvy, Unused medicines returned to community pharmacy: An analysis of medication waste and possibilities for redispensing, International Journal of Clinical Pharmacy, vol. 39, no. 1, p. 240, 2017.
[3]J. Lenzer, US could recycle 10 million unused prescription drugs a year, report says, ed: British Medical Journal Publishing Group, 2014.
[4]B. Hazell and R. Robson, Pharmaceutical waste reduction in the NHS, Report, Version, vol. 1, 2015.
[5] T.-Y. Lin, P. Dollár, R. Girshick, K. He, B. Hariharan, and S. Belongie, Feature pyramid networks for object detection, in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2017, pp. 2117-2125.
[6] C. Peng, X. Zhang, G. Yu, G. Luo, and J. Sun, Large Kernel Matters--Improve Semantic Segmentation by Global Convolutional Network, in Proceedings of the IEEE conference on computer vision and pattern recognition, 2017, pp. 4353-4361.
[7] A. Hartl, Computer-vision based pharmaceutical pill recognition on mobile phones, in Proceedings of CESCG 2010: 14th Central European Seminar on Computer Graphics, 2010, pp. 51-58.
[8] Y.-B. Lee, U. Park, and A. K. Jain, Pill-ID: Matching and retrieval of drug pill imprint images, in 2010 20th International Conference on Pattern Recognition, 2010: IEEE, pp. 2632-2635.
[9] J. J. Caban, A. Rosebrock, and T. S. Yoo, Automatic identification of prescription drugs using shape distribution models, in 2012 19th IEEE International Conference on Image Processing, 2012: IEEE, pp. 1005-1008.
[10] Z. Chen and S.-i. Kamata, A new accurate pill recognition system using imprint information, in Sixth International Conference on Machine Vision (ICMV 2013), 2013, vol. 9067: International Society for Optics and Photonics, p. 906711.
[11]R.-C. Chen, Y.-K. Chan, Y.-H. Chen, and C.-T. Bau, An automatic drug image identification system based on multiple image features and dynamic weights, International Journal of Innovative Computing, Information and Control, vol. 8, no. 5, pp. 2995-3013, May 2012 2012.
[12] J. Yu, Z. Chen, and S.-i. Kamata, Pill recognition using imprint information by two-step sampling distance sets, in 2014 22nd International Conference on Pattern Recognition, 2014: IEEE, pp. 3156-3161.
[13]J. Yu, Z. Chen, S.-i. Kamata, and J. Yang, Accurate system for automatic pill recognition using imprint information, IET Image Processing, vol. 9, no. 12, pp. 1039-1047, 2015.
[14] S. Suntronsuk and S. Ratanotayanon, Automatic text imprint analysis from pill images, in 2017 9th International Conference on Knowledge and Smart Technology (KST), 2017: IEEE, pp. 288-293.
[15] S. Suntronsuk and S. Ratanotayanon, Pill image binarization for detecting text imprints, in 2016 13th International Joint Conference on Computer Science and Software Engineering (JCSSE), 2016: IEEE, pp. 1-6.
[16] M. A. V. Neto, J. W. de Souza, P. P. Reboucas Filho, and W. d. O. Antonio, CoforDes: An Invariant Feature Extractor for the Drug Pill Identification, in 2018 IEEE 31st International Symposium on Computer-Based Medical Systems (CBMS), 2018: IEEE, pp. 30-35.
[17] X. Zeng, K. Cao, and M. Zhang, MobileDeepPill: A small-footprint mobile deep learning system for recognizing unconstrained pill images, in Proceedings of the 15th Annual International Conference on Mobile Systems, Applications, and Services, 2017: ACM, pp. 56-67.
[18] Y. Wang, J. Ribera, C. Liu, S. Yarlagadda, and F. Zhu, Pill recognition using minimal labeled data, in 2017 IEEE Third International Conference on Multimedia Big Data (BigMM), 2017: IEEE, pp. 346-353.
[19] Y.-Y. Ou, A.-C. Tsai, J.-F. Wang, and J. Lin, Automatic Drug Pills Detection based on Convolution Neural Network, in 2018 International Conference on Orange Technologies (ICOT), 2018: IEEE, pp. 1-4.
[20] Z. Yaniv et al., The national library of medicine pill image recognition challenge: An initial report, in 2016 IEEE Applied Imagery Pattern Recognition Workshop (AIPR), 2016: IEEE, pp. 1-9.
[21]Y.-B. Lee, U. Park, and A. K. Jain, Pill-ID: Matching and retrieval of drug pill imprint images, presented at the 20th International Conference on Pattern Recognition, Istanbul, Turkey, Aug. 2010, 2010.
[22]J. J. Caban, A. Rosebrock, and T. S. Yoo, Automatic identification of prescription drugs using shape distribution models, presented at the 19th IEEE International Conference on Image Processing, Orlando, FL, USA Oct. 2012, 2012.
[23]Z. Chen and S.-i. Kamata, A new accurate pill recognition system using imprint information, presented at the Sixth International Conference on Machine Vision, London, UK, Dec. 2013, 2013.
[24]J. Yu, Z. Chen, and S.-i. Kamata, Pill Recognition Using Imprint Information by Two-Step Sampling Distance Sets, presented at the 22nd International Conference on Pattern Recognition, Stockholm, Sweden, Aug. 2014, 2014.
[25]S. Suntronsuk and S. Ratanotayanon, Automatic text imprint analysis from pill images, presented at the 9th International Conference on Knowledge and Smart Technology, Chonburi, Thailand, Feb. 2017, 2017.
[26]S. Suntronsuk and S. Ratanotayanon, Pill image binarization for detecting text imprints, presented at the 13th International Joint Conference on Computer Science and Software Engineering, Khon Kaen, Thailand, Jul. 2016, 2016.
[27]M. A. V. Neto, J. a. W. M. d. Souza, P. P. R. c. Filho, and A. W. d. O. Rodrigues, CoforDes: An Invariant Feature Extractor for the Drug Pill Identification, presented at the IEEE 31st International Symposium on Computer-Based Medical Systems, Karlstad, Sweden, Jun. 2018, 2018.
[28]X. Zeng, K. Cao, and M. Zhang, MobileDeepPill: A small-footprint mobile deep learning system for recognizing unconstrained pill images, presented at the Proceedings of the 15th Annual International Conference on Mobile Systems, Applications, and Services, Niagara Falls, NY, USA, Jun. 2018, 2017.
[29]Y. Wang, J. Ribera, C. Liu, S. Yarlagadda, and F. Zhu, Pill Recognition Using Minimal Labeled Data, presented at the IEEE Third International Conference on Multimedia Big Data, Laguna Hills, CA, USA, Apr. 2017, 2017.
[30]R. Girshick, J. Donahue, T. Darrell, and J. Malik, Region-based convolutional networks for accurate object detection and segmentation, IEEE transactions on pattern analysis and machine intelligence, vol. 38, no. 1, pp. 142-158, 2015.
[31]K. He, X. Zhang, S. Ren, and J. Sun, Spatial pyramid pooling in deep convolutional networks for visual recognition, IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 37, no. 9, pp. 1904-1916, 2014, doi: 10.1109/TPAMI.2015.2389824.
[32]R. Girshick, Fast R-CNN, presented at the IEEE International Conference on Computer Vision, Washington, DC, USA, Dec. 2015, 2015.
[33]S. Ren, K. He, R. Girshick, and J. Sun, Faster r-cnn: Towards real-time object detection with region proposal networks, IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 39, no. 6, pp. 1137-1149, 2015, doi: 10.1109/TPAMI.2016.2577031.
[34] J. Dai, Y. Li, K. He, and J. Sun, R-fcn: Object detection via region-based fully convolutional networks, in Advances in neural information processing systems, 2016, pp. 379-387.
[35]J. R. Uijlings, K. E. Van De Sande, T. Gevers, and A. W. Smeulders, Selective search for object recognition, International journal of computer vision, vol. 104, no. 2, pp. 154-171, 2013.
[36]X. Zhang, Y. Yuan, and Q. Wang, ROI-wise Reverse Reweighting Network for Road Marking Detection, presented at the 29th British Machine Vision Conference, Newcastle, UK, September 3-6, 2018, 2018.
[37]K. Li, G. Cheng, S. Bu, and X. You, Rotation-insensitive and context-augmented object detection in remote sensing images, IEEE Transactions on Geoscience and Remote Sensing, vol. 56, no. 4, pp. 2337-2348, 2018.
[38]K. He, G. Gkioxari, P. Doll´ar, and R. Girshick, Mask r-cnn, presented at the IEEE International Conference on Computer Vision, Venice, Italy, Oct. 2017, 2017.
[39]S. Zhang, L. Wen, X. Bian, Z. Lei, and S. Z. Li, Single-shot refinement neural network for object detection, presented at the Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2018.
[40]Y. Yuan, Z. Xiong, and Q. Wang, VSSA-NET: Vertical Spatial Sequence Attention Network for Traffic Sign Detection, IEEE Transactions on Image Processing, pp. 1-1, 01 February 2019 2019, doi: 10.1109/TIP.2019.2896952.
[41]G. Cheng, J. Han, P. Zhou, and D. Xu, Learning rotation-invariant and fisher discriminative convolutional neural networks for object detection, IEEE Transactions on Image Processing, vol. 28, no. 1, pp. 265-278, 2019.
[42]G. Cheng, P. Zhou, and J. Han, Learning rotation-invariant convolutional neural networks for object detection in VHR optical remote sensing images, IEEE Transactions on Geoscience and Remote Sensing, vol. 54, no. 12, pp. 7405-7415, 2016.
[43] J. Redmon, S. Divvala, R. Girshick, and A. Farhadi, You only look once: Unified, real-time object detection, in Proceedings of the IEEE conference on computer vision and pattern recognition, 2016, pp. 779-788.
[44] W. Liu et al., Ssd: Single shot multibox detector, in European conference on computer vision, 2016: Springer, pp. 21-37.
[45]Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner, Gradient-based learning applied to document recognition, Proceedings of the IEEE, vol. 86, no. 11, pp. 2278-2324, 1998.
[46] A. Krizhevsky, I. Sutskever, and G. E. Hinton, Imagenet classification with deep convolutional neural networks, in Advances in neural information processing systems, 2012, pp. 1097-1105.
[47] X. Glorot, A. Bordes, and Y. Bengio, Deep sparse rectifier neural networks, in Proceedings of the fourteenth international conference on artificial intelligence and statistics, 2011, pp. 315-323.
[48] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei, Imagenet: A large-scale hierarchical image database, in 2009 IEEE conference on computer vision and pattern recognition, 2009: Ieee, pp. 248-255.
[49] M. D. Zeiler and R. Fergus, Visualizing and understanding convolutional networks, in European conference on computer vision, 2014: Springer, pp. 818-833.
[50]K. Simonyan and A. Zisserman, Very deep convolutional networks for large-scale image recognition, presented at the International Conference on Learning Representations, San Diego, CA, USA, May 2015, 2015.
[51] C. Szegedy et al., Going deeper with convolutions, in Proceedings of the IEEE conference on computer vision and pattern recognition, 2015, pp. 1-9.
[52]S. Ioffe and C. Szegedy, Batch normalization: Accelerating deep network training by reducing internal covariate shift, arXiv preprint arXiv:1502.03167, 2015.
[53]K. He, X. Zhang, S. Ren, and J. Sun, Deep residual learning for image recognition, presented at the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA, Jun. 2016, 2016.
[54] J. Hu, L. Shen, and G. Sun, Squeeze-and-excitation networks, in Proceedings of the IEEE conference on computer vision and pattern recognition, 2018, pp. 7132-7141.
[55] G. Huang, Z. Liu, L. Van Der Maaten, and K. Q. Weinberger, Densely connected convolutional networks, in Proceedings of the IEEE conference on computer vision and pattern recognition, 2017, pp. 4700-4708.
[56] R. Girshick, J. Donahue, T. Darrell, and J. Malik, Rich Feature Hierarchies for Accurate Object Detection and Semantic Segmentation, in 2014 IEEE Conference on Computer Vision and Pattern Recognition, 23-28 June 2014 2014, pp. 580-587, doi: 10.1109/CVPR.2014.81.
[57] R. Girshick, Fast R-CNN, in 2015 IEEE International Conference on Computer Vision, 7-13 Dec. 2015 2015, pp. 1440-1448, doi: 10.1109/ICCV.2015.169.
[58]T.-Y. Lin, P. Goyal, R. Girshick, K. He, and P. Dollár, Focal loss for dense object detection, arXiv preprint arXiv:1708.02002, 2017.
[59]C. Szegedy, V. Vanhoucke, S. Ioffe, and J. Shlens, Rethinking the inception architecture for computer vision, presented at the Proceedings of the IEEE conference on computer vision and pattern recognition, Las Vegas, NV, USA, Jul. 2016, 2016.
[60]F. Chollet, Xception: Deep learning with depthwise separable convolutions, presented at the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA, Jul. 2017, 2017.
[61]C. Szegedy, S. Ioffe, V. Vanhoucke, and A. A. Alemi, Inception-v4, inception-resnet and the impact of residual connections on learning, presented at the Proceedings of the Thirty-First AAAI Conference on Artificial Intelligence, San Francisco, CA, USA, Feb. 2017, 2017.
[62]D. P. Kingma and J. L. Ba, Adam: A method for stochastic optimization, presented at the International Conference on Learning Representations, San Diego, CA, USA, May 2015, 2015.