知識庫被廣泛地使用作為儲存大量資訊的庫藏，其儲存資訊所描述之項目稱為實體。所謂實體連結，指的是將文件內文中提及的項目(簡稱提及)，與知識庫中相對應的實體配對連結。透過實體連結可提昇文本文意之理解，是問答系統以及知識推理等方法的基礎。傳統基於字典之實體連結方法，透過實體字典產生提及的候選實體後，根據出現頻率或名稱相似度決定連結，因無法呈現語意導致連結效果差強人意。表示學習之實體連結方法，透過類神經網路模型產生提及與實體之表示向量，因未充分考慮前後文而語意模糊，故仍有改進空間。內文感知實體連結方法，使用如長短期記憶模型與添加更多訓練資料，能進一步提升連結的準確度。不過，現有內文感知實體連結方法中，提及或實體是只具有本身詞意的單一向量表示。在一個詞彙有多個詞意造成的詞彙歧義下，這種以詞意決定連結的方法仍可能有錯誤連結。此外，因人工標註或傳統方法，有時並未將可能的實體與提及連結，這種稱為失踪提及的問題，造成有效實體連結的大幅減少。
因此，本論文提出透過句對消歧義的內文感知實體連結方法(簡稱EL-SP)。用提及句與實體句兩句意形成的句對來避免詞彙歧義，改善現有的內文感知實體連結。主要是先將多種資訊組成兩種句子以構成句對，再透過BERT產生句對的嵌入向量。然後將固定個數的(提及、候選實體)句對嵌入向量輸入全連接網路，計算並配對得分最高之候選實體以達成實體連結。使用CoNLL-2003資料集並與相關方法的實驗結果顯示，EL-SP與基於字典之方法相比，Micro F1分數由80.68提升至89.54；與其他知名的內文感知實體連結方法相比，Micro F1分數由88.91提升至89.54。此外，應用EL-SP方法將可連結之實體與失踪提及配對的實驗結果顯示，其Micro F1分數可達到79.69，可增加有效實體連結。

Knowledge base is widely used as a repository for storing a large amount of factual information. The item described in the factual information is referred to as entity. Entity linking (abbreviated as EL) is to link the mentioned items (abbreviated as mentions) in certain text to their corresponding entities in the knowledge base so as to improve text understanding. Thus, EL is essential to question answering, knowledge inference, and so on. Dictionary-based EL introduces poor links due to the absence of semantics. Representation-learning EL considers partial context information so that the ambiguous semantics improves accuracy with limitation. Context-aware EL uses additional context information but polysemy creates semantic ambiguity and reduces the accuracy for these methods. Additionally, incorrect labelling as nil by human or traditional EL methods bring out the issue of missing mention and impairs the effectiveness of entity linking.
Therefore, in this thesis, we propose a method called entity linking through sentence-pair disambiguation (abbreviated as EL-SP), form a sentence-pair by using mention-sentence and entity-sentence to resolve semantic ambiguity and increase the linking accuracy. EL-SP forms potential sentence pairs, generates embedding vectors with BERT, and determines the candidate entity with the highest score in a fully connected network to link the mention. Experiments using the CoNLL-2003 dataset show that, the micro-F1 score increases from 80.68 to 89.54 in comparison with dictionary-based EL, and raises from 88.91 to 89.54 in comparison with a well-known context-aware EL. The experiment of using EL-SP to solve missing mention has an micro-F1 of 79.69 so that effective entity linking can be achieved.

誌謝 i
摘要 ii
Abstract iii
目錄 iv
圖目錄 vi
表目錄 vii
第一章 緒論 1
1.1 研究背景 1
1.2 研究動機 5
1.3 研究目的 6
1.4 問題定義 7
1.5 研究架構 7
第二章 文獻探討 9
2.1 基於自注意之序列對序列模型 9
2.2 表示學習之實體連結 12
2.3 內文感知實體連結 13
第三章 提出之方法 17
3.1 EL-SP之方法的架構 18
3.2 提及檢測(Mention Detection) 20
3.3 候選實體生成(Candidate Generation) 21
3.4 句對消歧義(Sentence Pair Disambiguation) 23
3.4.1 句對組合(Pair Combination) 23
3.4.2 句對嵌入(Pair Embedding) 25
3.4.3 候選實體排名(Candidate Ranking) 25
第四章 實驗結果 28
4.1 實驗資料 28
4.1.1 CoNLL-2003資料集 28
4.1.2 CoNLL-YAGO資料集 29
4.1.3 PPRforNED實體字典 29
4.1.4 資料特性 30
4.2 實驗環境 30
4.2.1 資料處理 30
4.2.2 欲比較實體連結方法 32
4.2.3 評估指標 32
4.2.4 實驗平台 33
4.2.5 實驗參數設置 33
4.3 連結性能的評估 34
4.3.1 正常提及的連結性能評估 35
4.3.2 失蹤提及的連結性能評估 36
4.3.3 模型訓練時間 37
4.4 BERT的參數評估 38
4.5 實驗總結 40
第五章 結論 42
5.1 論文貢獻 42
5.2 未來研究 42
參考文獻 44

參考文獻
[1]R. Bunescu and M. Pasca, “Using Encyclopedic Knowledge for Named entity Disambiguation,” Proceedings of the 11th Conference of the European Chapter of the Association for Computational Linguistics, pp. 9-16, 2006.
[2]J. Chung, C. Gulcehre, K. Cho, and Y. Bengio, “Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling,” Computing Research Repository, arXiv: 1412.3555v1, 2014.
[3]K. Cho, B. ven Merrienboer, C. Gulcegre, D. Bahdanau, F. Bougares, H. Schwenk, and Y. Bengio, “Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation,” Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing, pp. 1724-1734, 2014.
[4]M. Dredze, P. McNamee, D. Rao, A. Gerber, and T. Finin, “Entity Disambiguation for Knowledge Base Population,” Proceedings of the 23rd International Conference on Computational Linguistics, pp. 277-285, 2010.
[5]J. Devlin, M. Chang, K. Lee, K. Toutanova, “BERT: Pre-trianing of Deep Bidirectional Transformers for Language Understanding,” Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 4171-4186, 2019.
[6]Z. Guo and D. Barbosa, “Robust Entity Linking via Random Walks,” Proceedings of the 23rd ACM International Conference on Conference on Information and Knowledge Management, pp. 499-508, 2014.
[7]S. Guo, M. Chang, and E. Kiciman, “To Link or Not to Link? A Study on End-to-End Tweet Entity Linking,” Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 1020-1030, 2013.
[8]N. Gupta, S. Singh, and D. Roth, “Entity Linking vai Joint Encoding of Types, Descriptions, and Context,” Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pp. 2681-2690, 2017.
[9]Z. He, S. Liu, M. Li, M. Zhou, L. Zhang, and H. Wang, “Learning Entity Representation for Entity Disambiguation,” Proceedings of the 51th Annual Meeting of the Association for Computational Linguistics, Vol. 2, pp. 30-34, 2013.
[10]S. Hochreiter and J. Schmidhunber, “Long short-term memory,” Neural computation, Vol. 9, No. 8, pp. 1735-1780, 1997.
[11]S. Hu, Z. Tan, W. Zeng, B. Fe, and W. Xiao, “Entity Linking via Symmetrical Attention-Based Neural Network and Entity Structural Features,” Symmetry, Vol. 11, 2019.
[12]S. Kulkarni and A. Singh, “Collective annotation of Wikipedia entities in web text,” Proceedings of the 15th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 457-466, 2009.
[13]N. Kolitsas, O. Ganea, and T. Hofmann, “End-to-End Neural Entity Linking,” Proceedings of the 22nd Conference on Computational Natural Language Learning, pp. 519-529, 2018.
[14]Z. Lipton, J. Berkowitz, and C. Elkan, “A critical review of recurrent neural networks for sequence learning,” Computing Research Repository, arXiv: 1506.00019v4, 2015.
[15]T. Mikolov, K. Chen, G. Corrado, and J. Dean, “Efficient Estimation of Word Representations in Vector Space,” Proceedings of the 1st International Conference on Learning Representations, 2013.
[16]M. Nickel, K. Murphy, V. Tresp, and E. Gabrilovich, “A Review of Relational Machine Learning for Knowledge Graphs,” Proceedings of the IEEE, Vol. 104, pp. 11-33, 2015.
[17]M. Peters, M. Neumann, M. Iyyer, M. Gardner, C. Clark, K. Lee, and L. Zettlemoyer, “Deep Contextualized Word Representations,” Proceeds of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Vol. 1, pp. 2227-2237, 2018.
[18]M. C. Phan, A. Sun, T. Tay, and J. Han, “NeuPL: Attention-based Semantic Matching and Pair-Linking for Entity Disambiguation,” Proceedings of the 2017 ACM on Conference on Information and Knowledge Management, pp. 1667-1676, 2017.
[19]A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, and I. Sutskever, “Language Models are Unsupervised Multitask Learners,” Technical report, OpenAI, 2018.
[20]W. Shen, J. Han, J. Wang, X. Yuan, and Z. Yang, “SHINE+: A General Framework for Domain-Specific Entity Linking with Heterogeneous Information Networks,” IEEE Transactions on Knowledge and Data Engineering, Vol. 30, pp. 353-366, 2018.
[21]N. A. Smith, “Contextual Word Representations: A Contextual Introduction,” Computing Research Repository, arXiv: 1902.06006v3, 2019.
[22]I. Sutskever, O. Vinyals, and Q. V. Le, “Sequence to Sequence Learning with Neural Networks,” Processings of the 27th International Conference on Neural Information Processing System, pp. 3104-3112, 2014.
[23]W. Shen, J. Wang, and J. Han, “Entity Linking with a Knowledge Base: Issues, Techniques, and Solutions,” IEEE Transactions on Knowledge and Data Management, Vol. 27, pp. 443-460, 2015.
[24]A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin, “Attention Is All You Need,” Processings of the 30th International Conference on Neural Information Processing System, pp. 5998-6008, 2017.
[25]X. Yin, Y. Huang, B. Zhou, A. Li, L. Lan, and Y. Jia, “Deep Entity Linking via Eliminating Semantic Ambiguity With BERT,” IEEE Access, Vol. 7, pp. 169434-169445, 2019.
[26]S. Zwicklbauer, C. Seifert, and M. Granitzer, “Robust and Collective Entity Disambiguation through Semantic Embeddings,” Proceedings of the 39th International ACM SIFIR Conference on Research and Development in Information Retrieval, pp. 425-434, 2016.
[27]CoNLL-2003 dataset URL(Accessed: 2020/06/22):
https://www.clips.uantwerpen.be/conll2003/ner/
[28]CoNLL-YAGO dataset URL(Accessed: 2020/06/22):
https://www.mpi-inf.mpg.de/departments/databases-and-information-
systems/research/yago-naga/aida/downloads/
[29]Categorizing and Tagging Words URL(Accessed: 2020/06/22):
https://www.nltk.org/book/ch05.html
[30]Extracting Information from Text URL(Accessed: 2020/06/22):
https://www.nltk.org/book/ch07.html
[31]Germany URL(Accessed: 2020/06/22):
https://en.wikipedia.org/wiki/Germany
[32]BERT URL(Accessed: 2020/06/22):
https://github.com/google-research/bert
[33]PPRforNED dataset URL(Accessed: 2020/06/22):
https://github.com/masha-p/PPRforNED
[34]RDF Primer URL(Accessed: 2020/06/22):
https://www.w3.org/TR/rdf-primer/