臺灣博碩士論文加值系統

在現今科技產品的廣泛運用下，相關電子產業蓬勃發展，半導體晶片封裝的研發也朝向高效能與輕量化，以滿足電子產品的需求。利用機器視覺與神經網路分類的方式來辨別半導體封裝晶片缺陷與種類，將可大量降低人工檢驗產品缺陷的成本，並提升檢測速度和準確率。本研究運用卷積神經網路與Mask R-CNN兩種演算法，及不同分類種類與晶片影像共三種因子，建立探討晶片封裝的缺陷分類與檢測模型，進而探討各因子對模型的影響度。
本研究所建立的檢測模型可分為四個部分，第一部分為影像資料前處理，將蒐集到的影像資料切割成單一晶片；第二部分為影像資料擴增處理，將影像數量過少的缺陷類別，提取缺陷特徵後複製在良品影像上，使良品與不良品資料數量達到平衡；第三部分為訓練資料的前處理，將影像資料整理成演算法可判讀的格式；第四部份為模型訓練與驗證，運用實驗設計，分析實驗因子對分類結果的影響。研究結果發現Mask R-CNN所建立的模型比卷積神經網路所建立的模型更能在較複雜的影像中得到較準確的分類結果，同時因Mask R-CNN的標註特性，判斷缺陷時能顯示出缺陷位置，能夠得到更完整的預測結果。此外透過實驗結果也發現到若分類種類分得越多，則模型的判斷準確度也會跟著下降；晶片影像結構較為簡單的影像，也能得到較準確的檢測結果。

With the widespread use of high-tech products, the related electronic industry is booming, therefore, the development of semiconductor chip packaging is moving toward high performance and light weight to meet the needs of electronic products. Using machine vision and neural network to identify defects and their types of semiconductor package has significantly reduced the manual inspection cost for product defects and improved the inspection speed and accuracy. In this study, two algorithms, convolutional neural network (CNN) and Mask Region-based Convolutional Neural Networks (Mask R-CNN), are used to develop a defect classification and detection model for IC packaging, and then other factors: classification type and chip image are included to investigate the effect of each factor on the model.
There are four stages for developing the classification model. The first stage is pre-processing image data, which cuts the whole image data into a single chip. The second stage is augmenting defect image data, which extracts defect parts from the defect images and copies them on the normal images. So that the number of defect image would increase and could be balanced with the normal images. The third stage is transforming pre-processed images data into a format that can be interpreted by the algorithms. The fourth stage is model training and validation, experimental design is used to analyze factor effects on classification accuracy. It was found that the Mask R-CNN model could get more accurate classification results in more complex images than convolutional neural network model, and the labeling feature of Mask R-CNN showed the defect location when it detected the defects. In such a case, Mask R-CNN generated more complete prediction results. In addition, it was found that the accuracy of the model decreased in more defect categories cases, and for simpler chip patterns, the models may have better classification accuracy.

目錄
摘要...........................................................................................................................................................I
ABSTRACT .................................................................................................................................................II
致謝.........................................................................................................................................................III
目錄.........................................................................................................................................................IV
圖目錄.....................................................................................................................................................VI
表目錄...................................................................................................................................................VIII
第一章 緒論 ............................................................................................................................................1
1.1 研究背景 .......................................................................................................................................1
1.2 研究動機 .......................................................................................................................................2
1.3 研究目的 .......................................................................................................................................2
1.4 研究架構 .......................................................................................................................................3
第二章 文獻探討 ....................................................................................................................................5
2.1 半導體製程介紹............................................................................................................................5
2.2 超音波掃瞄顯微鏡........................................................................................................................6
2.3 影像處理 .......................................................................................................................................7
2.3.1 二值化 ...................................................................................................................................7
2.3.2 侵蝕與膨脹............................................................................................................................8
2.3.3 尋找輪廓 ...............................................................................................................................9
2.3.4 結構相似性............................................................................................................................9
2.4 晶片影像定位與切割.............................................................................................................10
2.5 類神經網路 .................................................................................................................................12
2.5.1 卷積神經網路......................................................................................................................13
2.5.2 R-CNN、Fast R-CNN、Faster R-CNN ...................................................................................14
2.5.3 Mask R-CNN .........................................................................................................................16
2.6 文獻討論 .....................................................................................................................................17
第三章 研究方法 ..................................................................................................................................18
3.1 研究現況與範圍 .........................................................................................................................18
3.2 研究架構 .....................................................................................................................................19
3.3 影像資料前處理..........................................................................................................................20
3.4 影像資料擴增處理......................................................................................................................21
3.5 模型訓練前處理..........................................................................................................................24
3.6 模型訓練與驗證..........................................................................................................................26
3.7 績效指標 .....................................................................................................................................27
3.8 實驗設備與環境介紹..................................................................................................................29
第四章 實驗結果與分析 ......................................................................................................................30
4.1 實驗資料的介紹..........................................................................................................................30
4.2 實驗流程 .....................................................................................................................................32
4.2.1 影像資料前處理..................................................................................................................33
4.2.2 影像資料擴增處理..............................................................................................................33
4.2.3 模型訓練前處理..................................................................................................................34
4.2.4 模型訓練與驗證..................................................................................................................36
4.3 實驗結果分析 .............................................................................................................................36
4.3.1 試驗一 .................................................................................................................................36
4.3.2 試驗二 .................................................................................................................................38
4.3.3 試驗三 .................................................................................................................................39
4.4 試驗分析與比較 .........................................................................................................................41
第五章 結論與未來研究方向 ..............................................................................................................47
5.1 研究結論 .....................................................................................................................................47
5.2 未來方向 .....................................................................................................................................48
參考文獻................................................................................................................................................49
附錄-所有實驗的數據分析...................................................................................................................53

圖目錄
圖 2. 1 尋找輪廓示意圖................................................................................9
圖 2. 2 晶片影像定位與切割流程圖..........................................................10
圖 2. 3 二值化後的影像參考圖..................................................................11
圖 2. 4 收縮膨脹後的影像參考圖..............................................................11
圖 2. 5 晶片切割線影像參考圖..................................................................12
圖 2. 6 類神經網路架構示意圖..................................................................12
圖 2. 7 神經元示意圖..................................................................................13
圖 2. 8 卷積神經網路示意圖......................................................................14
圖 2. 9 R-CNN 示意圖 ................................................................................15
圖 2. 10 Fast R-CNN 示意圖.......................................................................15
圖 2. 11 Faster R-CNN 示意圖....................................................................16
圖 2. 12 Mask R-CNN 示意圖 ....................................................................16
圖 3. 1 研究架構流程圖..............................................................................20
圖 3. 2 SAT 掃描機台掃描完的晶片影像示意圖......................................21
圖 3. 3 影像資料擴增處理流程圖..............................................................22
圖 3. 4 晶片良品影像示意圖......................................................................22
圖 3. 5 晶片不良品影像示意圖..................................................................23
圖 3. 6 缺陷特徵示意圖..............................................................................23
圖 3. 7 差值影像示意圖..............................................................................23
圖 3. 8 不良品擴增影像示意圖..................................................................24
圖 3. 9 Mask R-CNN 訓練資料集示意圖 ..................................................25
圖 3. 10 Mask R-CNN 模型訓練前處理流程圖 ........................................25
圖 3. 11 CNN 訓練模型前處理流程圖 ......................................................26
圖 3. 12 模型訓練流程圖............................................................................27
圖 4. 1 空洞缺陷種類示意圖......................................................................30
圖 4. 2 二次膠缺陷種類示意圖..................................................................31
圖 4. 3 將需要標記影像讀入 labelme........................................................34
圖 4. 4 不良品缺陷輪廓標註......................................................................35
圖 4. 5 重要因子圖......................................................................................44
圖 4. 6 交互作用圖......................................................................................44

表目錄
表 2. 1 侵蝕膨脹舉例說明影像....................................................................8
表 3. 1 二元分類混淆矩陣..........................................................................28
表 3. 2 多元分類混淆矩陣..........................................................................28
表 3. 3 二元分類指標公式..........................................................................29
表 3. 4 多元分類指標公式..........................................................................29
表 4. 1 實驗資料影像介紹..........................................................................32
表 4. 2 類神經網路架構..............................................................................36
表 4. 3 試驗一卷積神經網路實驗資料影像 A 結果.................................37
表 4. 4 試驗一卷積神經網路實驗資料影像 B 結果 .................................37
表 4. 5 試驗一 Mask R-CNN 實驗資料影像 A 結果.................................37
表 4. 6 試驗一 Mask R-CNN 實驗資料影像 B 結果.................................38
表 4. 7 試驗二卷積神經網路實驗資料影像 A 結果.................................38
表 4. 8 試驗二卷積神經網路實驗資料影像 B 結果 .................................39
表 4. 9 試驗二 Mask R-CNN 實驗資料影像 A 結果.................................39
表 4. 10 試驗二 Mask R-CNN 實驗資料影像 B 結果...............................39
表 4. 11 試驗三卷積神經網路實驗資料影像 A 結果 ...............................40
表 4. 12 試驗三卷積神經網路實驗資料影像 B 結果 ...............................40
表 4. 13 試驗三 Mask R-CNN 資料影像 A 結果.......................................41
表 4. 14 試驗三 Mask R-CNN 實驗資料影像 B 結果...............................41
表 4. 15 試驗整理總表................................................................................42
表 4. 16 實驗設計統整................................................................................42
表 4. 17ANOVA 檢定..................................................................................43

[1] David Silver, Aja Huang , Chris J. Maddison, Arthur Guez, Laurent Sifre, George van den Driessche, Julian Schrittwieser, Ioannis Antonoglou, Veda Panneershelvam, Marc Lanctot, Sander Dieleman, Dominik Grewe, John Nham, Nal Kalchbrenner, Ilya Sutskever, Timothy Lillicrap, Madeleine Leach, Koray Kavukcuoglu, Thore Graepel & Demis Hassabis(2016), Mastering the game of Go with deep neural networks and tree search, Nature vol 529.
[2] 上報(2018)。中美貿易戰顯台灣「半導體王國」重要地位。民國110年11月01日，取自：https://www.upmedia.mg/news_info.php?SerialNo=45923
[3] Yimou Lee, Norihiko Shirouzu & David Lague (2021), Taiwan chip industry emerges as battlefront on U.S.-China showdown, Reuters. https://www.reuters.com/investigates/special-report/taiwan-china-chips/
[4] Pat Hudson(2014), The Industrial Revolution, Bloomsbury Publishing.
[5] 工業技術研究院(2020，10月22日)。智慧製造是什麼？工業4.0下，智慧製造發展的4要素與4大挑戰。民國110年11月10日，取自：https://college.itri.org.tw/event/525-smart-machinery2020.html
[6] D.W. Raymond& D.F. Haigh(1997), Why automate optical inspection?, Proceedings International Test Conference 1997. https://doi.org/10.1109/TEST.1997.639723
[7] 彭德保、劉曉薇、張晴晴、洪竟堯、黃國書、陳柏安。側照式 SMD-LED 自動光學檢測系統。2010全國AOI論壇與展覽大會手冊。
[8] 施錦村(無日期)。我國IC 產業與其相關市場廠商競爭行為之研究。公平交易委員會。
[9] 張勁燕(2009)。半導體製程設備。五南圖書有限公司。
[10] 鍾文仁、陳佑任(2010)。IC封裝製程與CAE應用第三版。全華圖書有限公司。
[11] Lawrence W. Kessler& Donald E. Yuhas (1979), Acoustic Microscopy, Proceedings of the IEEE Vol.67.
[12] 閎康科技(無日期)。非破壞性分析-SAT。民國110年11月23日，取自：https://www.ma-tek.com/zh-TW/services/index/SAT
[13] 繆紹綱(2019)。數位影像處理。高立圖書有限公司。
[14] Joan S. Weszka & Azriel Rosenfeld(1978), Threshold Evaluation Techniques, IEEE Transactions on Systems Man and Cybernetics vol8. https://doi.org/10.1109/TSMC.1978.4310038
[15] Joseph (Yossi) Gil & Ron Kimmel(2002), Efficient Dilation, Erosion, Opening and Closing Algorithms, IEEE Transactions On Pattern Analysis And Machine Intelligence Vol.24. https://doi.org/10.1109/TPAMI.2002.1114852
[16] Roy Davies(2018), Computer Vision: Principles, Algorithms, Applications, Learning, Mara Conner.
[17] OpenCV-Python Tutorials. 圖片網址：https://reurl.cc/VElpVR
[18] S. Suzuki, "Topological structural analysis of digitized binary images by border following,"Computer vision, graphics, and image processing, 1985, pp.32-46.
[19] OpenCV-Contours.圖片網址：https://docs.opencv.org/3.4/d4/d73/tutorial_py_contours_begin.html
[20] Zhou Wang, Alan Conrad Bovik, Hamid Rahim Sheikh & Eero P. Simoncelli(2004), Image Quality Assessment: From Error Visibility to Structural Similarity, IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. 13, NO.4. https://doi.org /10.1109/TIP.2011.2173206
[21] 蕭東岳(2021)。兩種半導體封裝檢驗類神經模型績效之比較。中原大學工業與系統工程學所。
[22] Nokibul Islam, Vinayak Pandey, Ming-Che Hsieh, Kang Keon Taek(2017), Fine Pitch Cu Pillar Assembly Challenges for Advanced Flip Chip Package, International Wafer Level Packaging Conference.
[23] Janusz Kacprzyk(2016), Artificial Neural Network Modelling, Studies in Computational Intelligence Volume 628. https://doi.org/10.1007/978-3-319-28495-8
[24] Wikipedia(無日期), Artificial neural network. 民國110年11月23日，取自：https://en.wikipedia.org/wiki/Artificial_neural_network
[25] Saad Albawi, Tareq Abed Mohammed & Saad Al-Zawi(2017), Understanding of a Convolutional Neural Network, The International Conference on Engineering and Technology. https://doi.org/10.1109/ICEngTechnol.2017.8308186
[26] R. Girshick, J. Donahue, T. Darrell, J. Malik(2014), Rich feature hierarchies for accurate object detection and semantic segmentation, in: Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR).
[27] R. Girshick(2015), Fast R-CNN, in: Proc. IEEE International Conference on Computer Vision.
[28] S. Ren, K. He, R. Girshick, J. Sun(2015), Faster R-CNN: towards real-time object detection with region proposal networks, IEEE Transactions on Pattern Analysis & Machine Intelligence vol.39 1137-1149.
[29] K. He, G. Gkioxari, P. Dollar, R. Girshick(2017), Mask R-CNN, in: Proc. IEEE International Conference on Computer Vision.
[30] Matteo Polsinelli ,Luigi Cinque& Giuseppe Placidi(2020), A light CNN for detecting COVID-19 from CT scans of the chest, Pattern Recognition Letters Volume 140. https://doi.org/10.1016/j.patrec.2020.10.001
[31] Jun-Nian Gou1, Xiao-Yuan Wu1& Li Liu(2020), Detection and Segmentation of Defects in Industrial CT Images Based on Mask R-CNN, Journal of Computers Vol. 31 No. 6. https://doi.org/10.3966/199115992020123106012
[32]美信檢測(無日期)。超聲波掃描(C-SAN)檢測分析。民國110年10月25日，取自：https://m.mttlab.com/article.php?tt_id=213