臺灣博碩士論文加值系統

急性盲腸炎是急性腹痛最常見的原因。儘管先前已有許多的研究，急性盲腸炎的診斷仍然具有挑戰性，誤診或延誤診斷會導致併發症與後續治療的困難度，其嚴重者甚至導致死亡。因此，快速且準確地診斷急性盲腸炎對於有效治療急性腹痛至關重要。然而，急性盲腸炎的臨床表現與許多內科疾病相似，即使醫生進行理學檢查和抽血檢查的情況下，診斷急性盲腸炎仍然困難且耗時。因此我們使用深度學習模型進行影像分割任務，以輔助醫生進行影像判讀，透過知識蒸餾的方法進一步提高影像切割模型的準確性，最後在AOCR 2024 AI Challenge的急性盲腸炎電腦斷層影像資料集中，我們的模型對單一案例診斷的F1 score達到了97.67%、切片影像的F1-score達到了92.54%，展現我們的模型能夠對電腦斷層影像中的急性盲腸炎進行高效而準確的預測。

Acute appendicitis is the most common cause of acute abdominal pain. It remains challenging to diagnose despite an extensive amount of prior research. Misdiagnosis or delayed diagnosis can cause problems and make therapy later on more difficult. In the worst cases, it may even be fatal. Therefore, the rapid and accurate diagnosis of acute appendicitis is crucial for the effective treatment of acute abdominal pain. However, the clinical presentation of acute appendicitis is similar to that of many other medical conditions. Even with physical examinations and blood tests conducted by doctors, diagnosing acute appendicitis remains difficult and time-consuming. To address this, we use deep learning models for image segmentation tasks to assist doctors in image interpretation. By employing knowledge distillation, we further enhance the accuracy of the image segmentation model. In the AOCR 2024 AI Challenge acute appendicitis CT image dataset, our model achieved an F1 score of 97.67% for single-case diagnoses and an F1 score of 92.54% for slice images. This demonstrates that our model can efficiently and accurately predict acute appendicitis in CT images.

Title Page i
Approval Page ii
Abstract in Chinese iii
Abstract in English iv
Acknowledgement v
Table of Contents vi
List of Tables viii
List of Figures ix
Chapter 1. Introduction 1
Chapter 2. Related Work 6
2.1 Convolutional Neural Network 6
2.2 Medical Image Segmentation 6
2.3 Volumetric Medical Image Segmentation 8
2.4 Diagnosis of Appendicitis Using Deep Learning 8
2.5 Ensemble Learning 10
2.6 Efficient segmentation model of Acute Appendicitis 11
2.7 Knowledge Distillation 12
2.7.1 Response-Based Knowledge 13
2.7.2 Feature-Based Knowledge 13
2.7.3 Relation-Based Knowledge 13
2.7.4 Knowledge distillation schemes 14
2.7.5 Multi-teacher Distillation 16
Chapter 3. Proposed method 17
3.1 Data preprocessing 18
3.2 Spatial Domain Data Augmentation 19
3.3 Network architecture 20
3.4 Dual-Teacher Knowledge Distillation 22
3.4.1 Target Loss 24
3.4.2 Logit Knowledge Distillation Loss 25
3.4.3 Feature Distillation Loss 26
3.4.4 Similarity Loss 26
3.5 Logit Standardization 27
3.6 Inference with sliding window 27
Chapter 4. Experiment 28
4.1 Experiment Environment 28
4.2 Implementation Details 28
4.3 Evaluation Metric 30
4.4 Result 31
4.4.1 Performance Comparison with Different Loss Combination. 31
4.4.2 Performance Comparison with Different Multi-Teacher KD method 33
4.4.3 Performance Comparison with Different Teacher Model of Similarity Loss 34
4.4.4 Performance Comparison with Different Dice Loss Weight. 35
4.4.5 Performance Comparison of Inference Time 36
4.4.6 Visualization of segmentation result 37
Chapter 5. Conclusion 39
Reference 40

List of Tables
Table 1. HU value for difference substance 5
Table 2. Detail Information about AA Dataset 9
Table 3. Data distribution of AOCR 2024 AI Challenge CT Dataset 28
Table 4. Performance Comparison with Different Loss Combination 31
Table 5. Performance Comparison with Different Multi-Teacher Knowledge Distillation method 33
Table 6. Performance Comparison with Different Teacher Model of Similarity Loss 34
Table 7. Performance Comparison with Different hyperparameter of Dice Loss. 35
Table 8. Performance Comparison of Inference Time(Device: Nvidia A100 80G) 36


List of Figures
Figure 1. Appendix Location Diagram. (https://www.news-medical.net/health/What-is-the-Appendix.aspx) 1
Figure 2. Normal appendix CT image(https://litfl.com/abdominal-ct-appendicitis/) 3
Figure 3. U-Net architecture[10]. 7
Figure 4. The integration methods of ensemble learning. (a)Bagging(b)Boosting(c)Voting(d)Stacking 10
Figure 5.The Pipeline of the first-place competition team’s model ensemble 11
Figure 6. the Knowledge distillation methods (a) Response-Based Knowledge (b) Feature-Based Knowledge(c) Relation-Based Knowledge 12
Figure 7 the illustrate of knowledge distillation schemes 15
Figure 8. Multi-teacher Distillation method(a)output separate distillation(b)output ensemble Distillation 16
Figure 9. data pre-processing flowchart 18
Figure 10. the flowchart of data augmentation 19
Figure 11. Architecture of 3D ResUnet 21
Figure 12. difference of original image and image after colon highlight 22
Figure 13. Overall training Framework of Dual-Teacher Knowledge Distillation. 23
Figure 14. Confusion matrix 30
Figure 15. segmentation result of case A in slice. (a)Ground Truth(b)Teacher w/o colon highlight segmentation result. (c) Teacher w/o colon highlight segmentation result. (d) student’s segmentation result. 37
Figure 16. segmentation slices result of case B in slice. (a) Ground Truth (b)Teacher w/o colon highlight segmentation result. (c) Teacher w/o colon highlight segmentation result. (d) student’s segmentation result. 38
Figure 17. 3D segmentation visualization of the model's predicted results in a specific example of intestinal volvulus. (a) Ground Truth (b)Teacher w/o colon highlight segmentation result. (c) Teacher w/o colon highlight segmentation result. (d) student’s segmentation result. 38

[1]A. Bhangu, K. Søreide, S. Di Saverio, J. H. Assarsson, and F. T. Drake, "Acute appendicitis: modern understanding of pathogenesis, diagnosis, and management," The Lancet, vol. 386, no. 10000, pp. 1278-1287, 2015.
[2]N. J. Carr, "The pathology of acute appendicitis," Annals of diagnostic pathology, vol. 4, no. 1, pp. 46-58, 2000.
[3]F. T. Drake et al., "Time to appendectomy and risk of perforation in acute appendicitis," JAMA surgery, vol. 149, no. 8, pp. 837-844, 2014.
[4]M. Busch, F. S. Gutzwiller, S. Aellig, R. Kuettel, U. Metzger, and U. Zingg, "In-hospital delay increases the risk of perforation in adults with appendicitis," World journal of surgery, vol. 35, pp. 1626-1633, 2011.
[5]K. O'shea and R. Nash, "An introduction to convolutional neural networks," arXiv preprint arXiv:1511.08458, 2015.
[6]J. Schmidhuber, "Deep learning in neural networks: An overview," Neural networks, vol. 61, pp. 85-117, 2015.
[7]K. He, X. Zhang, S. Ren, and J. Sun, "Deep residual learning for image recognition," in Proceedings of the IEEE conference on computer vision and pattern recognition, 2016, pp. 770-778.
[8]S. Xie, R. Girshick, P. Dollár, Z. Tu, and K. He, "Aggregated residual transformations for deep neural networks," in Proceedings of the IEEE conference on computer vision and pattern recognition, 2017, pp. 1492-1500.
[9]K. Simonyan and A. Zisserman, "Very deep convolutional networks for large-scale image recognition," arXiv preprint arXiv:1409.1556, 2014.
[10]O. Ronneberger, P. Fischer, and T. Brox, "U-net: Convolutional networks for biomedical image segmentation," in Medical image computing and computer-assisted intervention–MICCAI 2015: 18th international conference, Munich, Germany, October 5-9, 2015, proceedings, part III 18, 2015: Springer, pp. 234-241.
[11]A. Krizhevsky, I. Sutskever, and G. E. Hinton, "Imagenet classification with deep convolutional neural networks," Advances in neural information processing systems, vol. 25, 2012.
[12]L.-C. Chen, G. Papandreou, I. Kokkinos, K. Murphy, and A. L. Yuille, "Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs," IEEE transactions on pattern analysis and machine intelligence, vol. 40, no. 4, pp. 834-848, 2017.
[13]D. Ciresan, A. Giusti, L. Gambardella, and J. Schmidhuber, "Deep neural networks segment neuronal membranes in electron microscopy images," Advances in neural information processing systems, vol. 25, 2012.
[14]R. Girshick, J. Donahue, T. Darrell, and J. Malik, "Rich feature hierarchies for accurate object detection and semantic segmentation," in Proceedings of the IEEE conference on computer vision and pattern recognition, 2014, pp. 580-587.
[15]P. Isola, J.-Y. Zhu, T. Zhou, and A. A. Efros, "Image-to-image translation with conditional adversarial networks," in Proceedings of the IEEE conference on computer vision and pattern recognition, 2017, pp. 1125-1134.
[16]S. Ren, K. He, R. Girshick, and J. Sun, "Faster r-cnn: Towards real-time object detection with region proposal networks," Advances in neural information processing systems, vol. 28, 2015.
[17]A. Elnakib, G. Gimel’farb, J. S. Suri, and A. El-Baz, "Medical image segmentation: a brief survey," Multi Modality State-of-the-Art Medical Image Segmentation and Registration Methodologies: Volume II, pp. 1-39, 2011.
[18]D. Jha et al., "Resunet++: An advanced architecture for medical image segmentation," in 2019 IEEE international symposium on multimedia (ISM), 2019: IEEE, pp. 225-2255.
[19]A. Norouzi et al., "Medical image segmentation methods, algorithms, and applications," IETE Technical Review, vol. 31, no. 3, pp. 199-213, 2014.
[20]D. D. Patil and S. G. Deore, "Medical image segmentation: a review," International Journal of Computer Science and Mobile Computing, vol. 2, no. 1, pp. 22-27, 2013.
[21]Z. Zhou, M. M. Rahman Siddiquee, N. Tajbakhsh, and J. Liang, "Unet++: A nested u-net architecture for medical image segmentation," in Deep Learning in Medical Image Analysis and Multimodal Learning for Clinical Decision Support: 4th International Workshop, DLMIA 2018, and 8th International Workshop, ML-CDS 2018, Held in Conjunction with MICCAI 2018, Granada, Spain, September 20, 2018, Proceedings 4, 2018: Springer, pp. 3-11.
[22]Z. Zhang, Q. Liu, and Y. Wang, "Road extraction by deep residual u-net," IEEE Geoscience and Remote Sensing Letters, vol. 15, no. 5, pp. 749-753, 2018.
[23]J. Long, E. Shelhamer, and T. Darrell, "Fully convolutional networks for semantic segmentation," in Proceedings of the IEEE conference on computer vision and pattern recognition, 2015, pp. 3431-3440.
[24]F. Milletari, N. Navab, and S.-A. Ahmadi, "V-net: Fully convolutional neural networks for volumetric medical image segmentation," in 2016 fourth international conference on 3D vision (3DV), 2016: Ieee, pp. 565-571.
[25]Ö. Çiçek, A. Abdulkadir, S. S. Lienkamp, T. Brox, and O. Ronneberger, "3D U-Net: learning dense volumetric segmentation from sparse annotation," in Medical Image Computing and Computer-Assisted Intervention–MICCAI 2016: 19th International Conference, Athens, Greece, October 17-21, 2016, Proceedings, Part II 19, 2016: Springer, pp. 424-432.
[26]K. Kamnitsas, L. Chen, C. Ledig, D. Rueckert, and B. Glocker, "Multi-scale 3D convolutional neural networks for lesion segmentation in brain MRI," Ischemic stroke lesion segmentation, vol. 13, p. 46, 2015.
[27]L. Yu, X. Yang, H. Chen, J. Qin, and P. A. Heng, "Volumetric ConvNets with mixed residual connections for automated prostate segmentation from 3D MR images," in Proceedings of the AAAI conference on artificial intelligence, 2017, vol. 31, no. 1.
[28]Q. Dou, H. Chen, Y. Jin, L. Yu, J. Qin, and P.-A. Heng, "3D deeply supervised network for automatic liver segmentation from CT volumes," in Medical Image Computing and Computer-Assisted Intervention–MICCAI 2016: 19th International Conference, Athens, Greece, October 17-21, 2016, Proceedings, Part II 19, 2016: Springer, pp. 149-157.
[29]E. Gibson et al., "Automatic multi-organ segmentation on abdominal CT with dense V-networks," IEEE transactions on medical imaging, vol. 37, no. 8, pp. 1822-1834, 2018.
[30]A. Dosovitskiy et al., "An image is worth 16x16 words: Transformers for image recognition at scale," arXiv preprint arXiv:2010.11929, 2020.
[31]H. Cao et al., "Swin-unet: Unet-like pure transformer for medical image segmentation," in European conference on computer vision, 2022: Springer, pp. 205-218.
[32]A. Hatamizadeh et al., "Unetr: Transformers for 3d medical image segmentation," in Proceedings of the IEEE/CVF winter conference on applications of computer vision, 2022, pp. 574-584.
[33]H.-Y. Zhou, J. Guo, Y. Zhang, L. Yu, L. Wang, and Y. Yu, "nnformer: Interleaved transformer for volumetric segmentation," arXiv preprint arXiv:2109.03201, 2021.
[34]P. Rajpurkar et al., "AppendiXNet: deep learning for diagnosis of appendicitis from a small dataset of CT exams using video pretraining," Scientific reports, vol. 10, no. 1, p. 3958, 2020.
[35]J. J. Park, K. A. Kim, Y. Nam, M. H. Choi, S. Y. Choi, and J. Rhie, "Convolutional-neural-network-based diagnosis of appendicitis via CT scans in patients with acute abdominal pain presenting in the emergency department," Scientific reports, vol. 10, no. 1, p. 9556, 2020.
[36]M. Kim et al., "Development and Validation of Automated Three-dimensional Convolutional Neural Network Model for Acute Appendicitis Diagnosis," 2024.
[37]T. D. Bui, J. Shin, and T. Moon, "3D densely convolutional networks for volumetric segmentation," arXiv preprint arXiv:1709.03199, 2017.
[38]G. Huang, Z. Liu, L. Van Der Maaten, and K. Q. Weinberger, "Densely connected convolutional networks," in Proceedings of the IEEE conference on computer vision and pattern recognition, 2017, pp. 4700-4708.
[39]W. W. Y. Ng et al., "LSSED: A Robust Segmentation Network for Inflamed Appendix from CT Images," in ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2023: IEEE, pp. 1-5.
[40]T. G. Dietterich, "Ensemble methods in machine learning," in International workshop on multiple classifier systems, 2000: Springer, pp. 1-15.
[41]R. Polikar, "Ensemble learning," Ensemble machine learning: Methods and applications, pp. 1-34, 2012.
[42]O. Sagi and L. Rokach, "Ensemble learning: A survey," Wiley interdisciplinary reviews: data mining and knowledge discovery, vol. 8, no. 4, p. e1249, 2018.
[43]T. K. Ho, "Random decision forests," in Proceedings of 3rd international conference on document analysis and recognition, 1995, vol. 1: IEEE, pp. 278-282.
[44]Y. Freund and R. E. Schapire, "A desicion-theoretic generalization of on-line learning and an application to boosting," in European conference on computational learning theory, 1995: Springer, pp. 23-37.
[45]F. Isensee, P. F. Jaeger, S. A. Kohl, J. Petersen, and K. H. Maier-Hein, "nnU-Net: a self-configuring method for deep learning-based biomedical image segmentation," Nature methods, vol. 18, no. 2, pp. 203-211, 2021.
[46]J. Wasserthal et al., "TotalSegmentator: robust segmentation of 104 anatomic structures in CT images," Radiology: Artificial Intelligence, vol. 5, no. 5, 2023.
[47]J. Gou, B. Yu, S. J. Maybank, and D. Tao, "Knowledge distillation: A survey," International Journal of Computer Vision, vol. 129, no. 6, pp. 1789-1819, 2021.
[48]C. Yang, X. Yu, Z. An, and Y. Xu, "Categories of response-based, feature-based, and relation-based knowledge distillation," in Advancements in Knowledge Distillation: Towards New Horizons of Intelligent Systems: Springer, 2023, pp. 1-32.
[49]G. Hinton, O. Vinyals, and J. Dean, "Distilling the knowledge in a neural network," arXiv preprint arXiv:1503.02531, 2015.
[50]C. Buciluǎ, R. Caruana, and A. Niculescu-Mizil, "Model compression," in Proceedings of the 12th ACM SIGKDD international conference on Knowledge discovery and data mining, 2006, pp. 535-541.
[51]K. Kim, B. Ji, D. Yoon, and S. Hwang, "Self-knowledge distillation: A simple way for better generalization," arXiv preprint arXiv:2006.12000, vol. 3, no. 1, 2020.
[52]X. Chen, J. Su, and J. Zhang, "A two-teacher framework for knowledge distillation," in Advances in Neural Networks–ISNN 2019: 16th International Symposium on Neural Networks, ISNN 2019, Moscow, Russia, July 10–12, 2019, Proceedings, Part I 16, 2019: Springer, pp. 58-66.
[53]T. Fukuda, M. Suzuki, G. Kurata, S. Thomas, J. Cui, and B. Ramabhadran, "Efficient Knowledge Distillation from an Ensemble of Teachers," in Interspeech, 2017, pp. 3697-3701.
[54]S. You, C. Xu, C. Xu, and D. Tao, "Learning from multiple teacher networks," in Proceedings of the 23rd ACM SIGKDD international conference on knowledge discovery and data mining, 2017, pp. 1285-1294.
[55]D. Ulyanov, A. Vedaldi, and V. Lempitsky, "Instance normalization: The missing ingredient for fast stylization," arXiv preprint arXiv:1607.08022, 2016.
[56]D. Bahdanau, K. Cho, and Y. Bengio, "Neural machine translation by jointly learning to align and translate," arXiv preprint arXiv:1409.0473, 2014.
[57]A. L. Maas, A. Y. Hannun, and A. Y. Ng, "Rectifier nonlinearities improve neural network acoustic models," in Proc. icml, 2013, vol. 30, no. 1: Atlanta, GA, p. 3.
[58]V. Nair and G. E. Hinton, "Rectified linear units improve restricted boltzmann machines," in Proceedings of the 27th international conference on machine learning (ICML-10), 2010, pp. 807-814.
[59]Q. Zhu, B. Du, B. Turkbey, P. L. Choyke, and P. Yan, "Deeply-supervised CNN for prostate segmentation," in 2017 international joint conference on neural networks (IJCNN), 2017: IEEE, pp. 178-184.
[60]N. Tajbakhsh et al., "Convolutional neural networks for medical image analysis: Full training or fine tuning?," IEEE transactions on medical imaging, vol. 35, no. 5, pp. 1299-1312, 2016.
[61]N. Passalis and A. Tefas, "Learning deep representations with probabilistic knowledge transfer," in Proceedings of the European Conference on Computer Vision (ECCV), 2018, pp. 268-284.
[62]T. Kim, J. Oh, N. Kim, S. Cho, and S.-Y. Yun, "Comparing kullback-leibler divergence and mean squared error loss in knowledge distillation," arXiv preprint arXiv:2105.08919, 2021.
[63]S. Sun, W. Ren, J. Li, R. Wang, and X. Cao, "Logit standardization in knowledge distillation," in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024, pp. 15731-15740.
[64]J. Kiefer and J. Wolfowitz, "Stochastic estimation of the maximum of a regression function," The Annals of Mathematical Statistics, pp. 462-466, 1952.
[65]H. Robbins and S. Monro, "A stochastic approximation method," The annals of mathematical statistics, pp. 400-407, 1951.