( 您好!臺灣時間:2021/04/15 08:38
字體大小: 字級放大   字級縮小   預設字形  


研究生(外文):Yu-Jen Wang
論文名稱:利用 Attentive 來改善端對端中文語篇剖析遞迴類 神經網路系統
論文名稱(外文):Using Attentive to improve Recursive LSTM End-to- End Chinese Discourse Parsing
指導教授(外文):Chia-Hui Chang
外文關鍵詞:Deep LearningDiscourse ParsingAttentionRecursive neural network
  • 被引用被引用:0
  • 點閱點閱:108
  • 評分評分:系統版面圖檔系統版面圖檔系統版面圖檔系統版面圖檔系統版面圖檔
  • 下載下載:30
  • 收藏至我的研究室書目清單書目收藏:0
篇章剖析,可以幫助我們以不同角度來理解文句之間的關係與連結,但篇章剖析的資 料結構目前仰賴人工標記,使得這項技術無法直接被利用在任意篇章中。因此至目前為 止,有許多研究著手於讓電腦能夠自動對篇章進行剖析,並建立出一個完整的剖析樹。以 中文語料庫 CDTB 來說,欲建立完整的篇章剖析程式,其問題主要可以被分成四個,分 別是子句分割、剖析樹建立、子句關係辨識、中心關係辨識。
由於深度學習近幾年發展快速,因此針對篇章剖析的建構方法也從傳統的 SVM, CRF 等,進展到目前已遞迴類神經方式來建構剖析篇章。而在本篇論文中,我們也加入了許多 目前最新的深度學習技術,例如 Attentive RvNN、self-attentive、BERT 等方法,來提高模 型的準確度。
最後,我們成功將每一項任務的 F1 都提高了近 10% 左右,達到目前我們所知研究中 最好的效能。
Discourse parser can help us to understand the relationship and connection between sentences from different angles, but the tree structure data still need to rely on manual marking, which makes this technology cannot be directly used in life. So far, there have been many research studies on automatically construct the complete tree structure on the computer. Since deep learning has progressed rapidly in recent years, the construction method for discourse parser has also changed from the traditional SVM, CRF method to the current recursive neural.
In the Chinese corpus tree library CDTB, the parsing analysis problem can be divided into four main problems, including elementary discourse unit (EDU) segmentation, tree structure construction, center labeling, and sense labeling. In this paper, we use many state-of-the-art deep learning techniques, such as attentive recursive neural networks, self-attentive, and BERT to improve the performance.
In the end, we succeed to increase the accuracy by more than 10% of F1 of each task, reaching the best performance we know so far.
摘要 i
Abstract ii
誌謝 iii
目錄 iv
圖目錄 vi
表目錄 vii
壹、 緒論 1
1.1. 問題定義 2
1.2. 研究動機與目標 4
貳、 相關研究 6
2.1. 篇章剖析語料庫 6
2.1.1. 英文語料庫 6
2.1.2. 中文語料庫 7
2.2. 篇章剖析程式 (Discourse Parser) 8
2.2.1. SVM Base Parser 8
2.2.2. DCRF Base Parser 8
2.2.3. Recursive Deep Models Parser 8
2.2.4. RvNN Chinese Discourse Parser 9
2.2.5. Transition-Based Dependency Parser 12
2.3. 深度學習相關技術 13
2.3.1. Recursive Neural Network (RvNN) 13
2.3.2. Attentive RvNN 15
2.3.3. Self-Attentive Sentence Embedding 16
2.3.4. FastText 16
2.3.5. BERT 16
參、 CDTB語料庫 18
3.1. 子句關係 18
3.2. 隱性關係與顯性關係 20
肆、 模型設計 22
4.1. 子句分割 23
4.2. 文字嵌入與注意力機制 25
4.3. 注意力遞迴類神經網路 26
伍、 實驗 28
5.1. 標準子句實驗 29
5.2. 端對端剖析模型實驗 30
5.3. 二元樹與多元樹比較分析實驗 30
5.4. 二元樹分析實驗 32
5.5. 子句關係分析 33
5.6. 個別實驗數據分析 34
5.6.1. 文字嵌入實驗 34
5.6.2. Self-Attentive 子句效能實驗 34
5.6.3. 多層Attentive RvNN 實驗 35
5.6.4. 子句分割實驗 35
5.7. 學習曲線 36
陸、 結論 37
6.1. 未來展望 37
參考 39
[1]. William C Mann and Sandra A Thompson. Rhetorical structure theory: Toward a functional theory of text organization. Text-Interdisciplinary Journal for the Study of Discourse, 8(3):243–281, 1988.
[2]. Lynn Carlson, Daniel Marcu, and Mary Ellen Oku- rowski. 2003. Building a Discourse- tagged Corpus in the Framework of Rhetorical Structure Theory. Springer Netherlands.
[3]. Rashmi Prasad, Bonnie Webber, and Aravind Joshi. Reflections on the penn discourse treebank, comparable corpora, and complementary annotation. Computational Linguistics,
40(4):921–950, 2014.
[4]. Li Yancui, Feng Wenhe, Sung Jing, Kong Fang, *Zhou Guodong. Building Chinese
Discourse Corpus with Connective-driven Dependency Tree Structure[C]. In Proceedings of the 2014 conference on Emporical Methods in Natural Language Processing, pages 2105– 2114,October 25-29, 2014, Doha, Qatar.
[5]. Annie Louis, Aravind Joshi, and Ani Nenkova. Discourse indicators for content selection in summarization. In Proceedings of the SIGDIAL 2010 Conference, pages 147–156, Tokyo, Japan, September 2010. Association for Computational Linguistics. URL http: //www.aclweb.org/anthology/W/W10/W10-4327.
[6]. Christina Lioma, Birger Larsen, and Wei Lu. Rhetorical relations for information retrieval. In Proceedings of the 35th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR’12), pages 931–940, 2012. ISBN 978-1-4503- 1472-5. . URL http://doi.acm.org/10.1145/2348283.2348407.
[7]. Richard Socher, Alex Perelygin, Jean Y. Wu, Jason Chuang, Christopher D. Manning, Andrew Y. Ng and Christopher Potts. Recursive Deep Models for Semantic Compositionality Over a Sentiment Treebank. Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pages 1631–1642, Seattle, Washington, USA, 18-21 October 2013. 2013 Association for Computational Linguistics
[8]. Kai Sheng Tai, Richard Socher*, Christopher D. Manning. Improved Semantic Representations From Tree-Structured Long Short-Term Memory Networks. ACL 2015
[9]. Yangfeng Ji and Noah A. Smith. Neural Discourse Structure for Text Categorization. Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, pages 996–1005 Vancouver, Canada, July 30 - August 4, 2017. 2017 Association for Computational Linguistics
[10]. Jean Maillard, Stephen Clark, Dani Yogatama. Jointly Learning Sentence Embeddings and Syntax with Unsupervised Tree-LSTMs. Submitted to 31st Conference on Neural Information Processing Systems (NIPS 2017).
[11]. Yanyan Jia, Yuan Ye, Yansong Feng, Yuxuan Lai, Rui Yan and Dongyan Zhao. Modeling Discourse Cohesion for Discourse Parsing via Memory Network. Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Short Papers), pages
438–443 Melbourne, Australia, July 15 - 20, 2018. ©2018 Association for Computational Linguistics.
[12]. Yancui Li, Wenhe Feng, Jing Sun, Fang Kong, and Guodong Zhou. Building chinese dis- course corpus with connective-driven dependency tree structure. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP’14), pages 2105–2114, Doha, Qatar, October 2014b. Association for Computational Linguistics. URL http://www.aclweb.org/anthology/D14-1224.
[13]. Tomas Mikolov, Kai Chen, Greg Corrado, Jeffrey Dean .Efficient Estimation of Word Representations in Vector Space. Computation and Language (cs.CL)
[14]. Piotr Bojanowski∗ and Edouard Grave∗ and Armand Joulin and Tomas Mikolov. Enriching Word Vectors with Subword Information. Transactions of the Association for Computational Linguistics, vol. 5, pp. 135–146, 2017. Action Editor: Hinrich Schu ̈tze. Submission batch: 9/2016; Revision batch: 12/2016; Published 6/2017. 2017 Association for Computational Linguistics. Distributed under a CC-BY 4.0 license
[15]. Jeffrey Pennington, Richard Socher, Christopher D. Manning. GloVe: Global Vectors for Word Representation. Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1532–1543, October 25-29, 2014, Doha, Qatar. 2014 Association for Computational Linguistics
[16]. Jacob Devlin, Ming-Wei Chang, Kenton Lee, Kristina Toutanova. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding.
[17]. Pitler, M. Raghupathy, H. Mehta, A. Nenkova, A. Lee, and A. Joshi. 2008. Easily identifiable discourse relations. In Proceedings of the 22nd International Conference on Computational Linguistics (COLING08), short paper.
[18]. A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,” in Advances in Neural Information Processing Systems, 2017, pp. 5998–6008.
[19]. C. Goller and A. Kuchler. Learning task-dependent distributed representations by back- propagation through structure. In Proceedings of the 1996 IEEE International Conference on Neural Networks, volume 1, pages 347–352 vol.1, Jun 1996.
[20]. Chuan-An Lin. A Unified RvNN Framework for End-to-End Chinese Discourse Parsing. Proceedings of the 27th International Conference on Computational Linguistics: System Demonstrations, pages 73–77 Santa Fe, New Mexico, USA, August 20-26, 2018.
[21]. David A. duVerle , Helmut Prendinger .A Novel Discourse Parser Based on Support Vector Machine Classification. Proceedings of the 47th Annual Meeting of the ACL and the 4th IJCNLP of the AFNLP, pages 665–673, Suntec, Singapore, 2-7 August 2009. 2009 ACL and AFNLP.

[22]. Shafiq Joty, Giuseppe Carenini, Raymond Ng, and Yashar Mehdad. 2013. Combining intra- and multi-sentential rhetorical parsing for document-level discourse analysis. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (ACL 2013), pages 486–496, Sofia, Bulgaria, August.
[23]. Jiwei Li, Rumeng Li and Eduard Hovy. Recursive Deep Models for Discourse Parsing. Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2061–2069, October 25-29, 2014, Doha, Qatar. c©2014 Association for Computational Linguistics
[24]. Daniel H Younger. Recognition and parsing of context-free languages in time n3. Infor- mation and control, 10(2):189–208, 1967.
[25]. Samuel R. Bowman, Jon Gauthier, Abhinav Rastogi, Raghav Gupta, Christopher D. Man- ning, and Christopher Potts. A fast unified model for parsing and sentence understanding. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (ACL’16), pages 1466–1477, August 2016. URL http://www.aclweb.org/ anthology/P16- 1139.
[26]. Miguel Ballesteros, Chris Dyer, and Noah A. Smith. 2015. Improved transition-based parsing by modeling characters instead of words with LSTMs. In EMNLP’15, Lisbon, Portugal. 349–359.
[27]. YANYAN JIA, YANSONG FENG, YUAN YE, CHAO LV. Improved Discourse Parsing with Two-Step Neural Transition-Based Model. ACM Trans. Asian Low-Resour. Lang. Inf. Process., Vol. 17, No. 2, Article 11. Publication date: January 2018.
[28]. Emily Pitler, Annie Louis, Ani Nenkova. Automatic sense prediction for implicit discourse relations in text, Proceedings of the 47th Annual Meeting of the ACL and the 4th IJCNLP of the AFNLP, pages 683ñ691, Suntec, Singapore, 2-7 August 2009. c©2009 ACL and AFNLP
[29]. D. Bahdanau, K. Cho, and Y. Bengio, “Neural machine translation by jointly learning to align and translate,” arXiv preprint arXiv:1409.0473, 2014.
[30]. Yao Zhou, Cong Liu, Yan Pan. Modelling Sentence Pairs with Tree-structured Attentive Encoder. 10 pages, 3 figures, COLING2016.
[31]. Mahtab Ahmed, Muhammad Rifayat Samee, Robert E. Mercer. Improving Tree-LSTM with Tree Attention. 8 Pages, 3 figures, Accepted in The 13th IEEE International Conference on Semantic Computing (ICSC 2019)
[32]. Zhouhan Lin , Minwei Feng, Cicero Nogueira dos Santos, Mo Yu,Bing Xiang, Bowen Zhou& Yoshua Bengio. A S TRUCTURED S ELF-ATTENTIVE S ENTENCE EMBEDDING. Published as a conference paper at ICLR 2017.
[33]. Mahtab Ahmed, Muhammad Rifayat Samee, Robert E. Mercer . Improving Tree-LSTM with Tree Attention. 8 Pages, 3 figures, Accepted in The 13th IEEE International Conference on Semantic Computing (ICSC 2019)
[34]. Xiaomian Kang, Haoran Li, Long Zhou, Jiajun Zhang, and Chengqing Zong. An end-to-end chinese discourse parser with adaptation to explicit and non-explicit relation recognition. In Proceedings of the CoNLL-16 shared task, pages 27–32, August 2016. URL http://anthology.aclweb.org/K16-2003.
[35]. Lynn Carlson, Daniel Marcu, and Mary Ellen Okurowski. Building a discourse-tagged corpus in the framework of rhetorical structure theory. In Proceedings of the Second SIGdial Workshop on Discourse and Dialogue (SIGDIAL’01), pages 1–10, 2001 . URL https://doi.org/10.3115/1118078.1118083.
[36]. Jihun Choi, Kang Min Yoo, Sang-goo Lee. Learning to Compose Task-Specific Tree Structures. Copyright 2018, Association for the Advancement of Artificial Intelligence (www.aaai.org). All rights reserved.
[37]. Maillard, J.; Clark, S.; and Yogatama, D. 2017. Jointly learning sentence embeddings and syntax with unsupervised Tree-LSTMs. arXiv preprint arXiv:1705.09189.
註: 此連結為研究生畢業學校所提供,不一定有電子全文可供下載,若連結有誤,請點選上方之〝勘誤回報〞功能,我們會盡快修正,謝謝!
第一頁 上一頁 下一頁 最後一頁 top
系統版面圖檔 系統版面圖檔