跳到主要內容

臺灣博碩士論文加值系統

(44.222.64.76) 您好!臺灣時間:2024/06/15 06:17
字體大小: 字級放大   字級縮小   預設字形  
回查詢結果 :::

詳目顯示

我願授權國圖
: 
twitterline
研究生:黃千芝
研究生(外文):Chien-Chi Huang
論文名稱:利用知識蒸餾於前後文相關之問題改寫以提升對話式問答
論文名稱(外文):Contextual Question Rewriting with Knowledge Distillation for Improving Conversational Question Answering
指導教授:陳縕儂
指導教授(外文):Yun-Nung Chen
口試委員:曹昱李宏毅蔡宗翰
口試委員(外文):Yu TsaoHung-yi LeeTzong-Han Tsai
口試日期:2023-01-19
學位類別:碩士
校院名稱:國立臺灣大學
系所名稱:資訊工程學系
學門:工程學門
學類:電資工程學類
論文種類:學術論文
論文出版年:2023
畢業學年度:111
論文頁數:44
中文關鍵詞:對話式問答不完整語句改寫知識蒸餾
外文關鍵詞:Conversational Question AnsweringIncomplete Utterance RewritingKnowledge Distillation
DOI:10.6342/NTU202300619
相關次數:
  • 被引用被引用:0
  • 點閱點閱:71
  • 評分評分:
  • 下載下載:0
  • 收藏至我的研究室書目清單書目收藏:0
智慧助理越來越普及的環境下,人類與機器的對話互動更趨近人與人之間的口說關係,因此,人類的對話習慣中,省略主詞或使用代名詞使得語意不完整的現象,對機器理解是一項重大的挑戰。機器必須從歷史紀錄中抽取對話中被省略的資訊,將使用者的問題重新組織成完整的問句,再依完整問句來搜尋回答。
本篇論文分析不同模型在對話式問答的問題重寫上的表現,以及在不同的資料集和不同資料特性上的表現與泛化能力等差異。並且提出一種在訓練過程中,利用知識蒸餾的技術加強模型的能力的方法,以提高改寫後的問題品質。
The dialogue between humans and machines is more similar to the oral language with the increasing popularity of intelligent assistants. Therefore, the task of incomplete utterance rewriting (IUR) in multi-turns conversations is a major challenge for machines to understand. To answer user's questions, the machines will extract the omitted information form historical records , reconstruct user's utterances into complete questions and response to the questions. This paper analyzes the performance of different models on incomplete utterance rewriting tasks and compare the differences on generalization ability, data features and data domains. In addition, we propose a knowledge distillation techniques to strengthen the performance of the models, and improve the performance of rewritten questions.
致謝 i
摘要 iii
Abstract v
Contents vii
List of Figures xi
List of Tables xiii
Chapter 1 Introduction 1
1.1 Motivation 1
1.2 Main Contribution 2
1.3 Thesis Structure 3
Chapter 2 Background 5
2.1 DeepLearning Models 5
2.1.1 Transformer 5
2.1.2 GPT2 6
2.1.3 BERT 7
2.2 Question Answering 8
2.2.1 Question Answering 8
2.2.2 ConversationalQuestionAnswering 9
Chapter 3 Related Work 11
3.1 Approaches to Conversational Question Answering 11
3.1.1 Pipeline Approach 11
3.1.2 End-to-End Approach 12
3.2 Incomplete Utterance Rewriting 13
Chapter 4 Preliminary Study 15
4.1 Datasets 15
4.2 Input Format and Sample Strategy 16
4.3 Analysis of Generalization Capability 18
4.4 Analysis of Question Features 18
Chapter 5 Proposed Method 21
5.1 Overview 21
5.2 Pre-training Stage 22
5.2.1 Teacher Model Training 22
5.2.2 Extract Ratio and Region 23
5.3 Knowledge Distillation 23
5.3.1 Encoder Knowledge Distillation Loss 24
5.3.2 Decoder Knowledge Distillation Loss 25
Chapter 6 Experiments 27
6.1 Datasets 27
6.2 Baseline Models 27
6.3Evaluation Metrics 28
6.4 Training Details 28
6.4.1 Hyperparameters 30
6.5 MainResults 30
Chapter 7 Discussion 33
7.1 Hyperparameter Grid Search 33
7.2 Ablation Study 34
7.3 Analysis of QA and Retrieval Results 35
7.4 Case Study 36
Chapter 8 Conclusion 39
References 41
[1] E. Choi, H. He, M. Iyyer, M. Yatskar, W.-t. Yih, Y. Choi, P. Liang, and L. Zettle- moyer. Quac: Question answering in context. arXiv preprint arXiv:1808.07036, 2018.
[2] M. Del Tredici, G. Barlacchi, X. Shen, W. Cheng, and A. de Gispert. Ques- tion rewriting for open-domain conversational qa: Best practices and limitations. In Proceedings of the 30th ACM International Conference on Information & Knowledge Management, pages 2974–2978, 2021.
[3] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805, 2018.
[4] A. Elgohary, D. Peskov, and J. Boyd-Graber. Can you unpack that? learning to rewrite questions-in-context. Can You Unpack That? Learning to Rewrite Questions-in-Context, 2019.
[5] M. Grbovic, N. Djuric, V. Radosavljevic, F. Silvestri, and N. Bhamidipati. Context- and content-aware embeddings for query rewriting in sponsored search. In Proceedings of the 38th international ACM SIGIR conference on research and development in information retrieval, pages 383–392, 2015.
[6] J. Gu, Z. Lu, H. Li, and V. O. Li. Incorporating copying mechanism in sequence-to- sequence learning. arXiv preprint arXiv:1603.06393, 2016.
[7] M. Huang, F. Li, W. Zou, and W. Zhang. Sarg: A novel semi autoregressive gen- erator for multi-turn incomplete utterance restoration. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 13055–13063, 2021.
[8] Y. Ju, F. Zhao, S. Chen, B. Zheng, X. Yang, and Y. Liu. Technical report on conver- sational question answering. arXiv preprint arXiv:1909.10772, 2019.
[9] Q. Liu, B. Chen, J.-G. Lou, B. Zhou, and D. Zhang. Incomplete utterance rewriting as semantic segmentation. arXiv preprint arXiv:2009.13166, 2020.
[10] Y. Ohsugi, I. Saito, K. Nishida, H. Asano, and J. Tomita. A simple but effective method to incorporate multi-turn context with bert for conversational machine com- prehension. arXiv preprint arXiv:1905.12848, 2019.
[11] Y.PapakonstantinouandV.Vassalos.Queryrewritingforsemistructureddata.ACM SIGMOD Record, 28(2):455–466, 1999.
[12] M.Qiu,X.Huang,C.Chen,F.Ji,C.Qu,W.Wei,J.Huang,andY.Zhang.Reinforced history backtracking for conversational question answering. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 13718–13726, 2021.
[13] C. Qu, L. Yang, C. Chen, M. Qiu, W. B. Croft, and M. Iyyer. Open-retrieval conver- sational question answering. In Proceedings of the 43rd International ACM SIGIR conference on research and development in Information Retrieval, pages 539–548, 2020.
[14] C. Qu, L. Yang, M. Qiu, Y. Zhang, C. Chen, W. B. Croft, and M. Iyyer. Attentivehistory selection for conversational question answering. In Proceedings of the 28th ACM International Conference on Information and Knowledge Management, pages 1391–1400, 2019.
[15] A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, I. Sutskever, et al. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9, 2019.
[16] C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y. Zhou, W. Li, P. J. Liu, et al. Exploring the limits of transfer learning with a unified text-to-text transformer. J. Mach. Learn. Res., 21(140):1–67, 2020.
[17] S. Rizvi, A. Mendelzon, S. Sudarshan, and P. Roy. Extending query rewriting tech- niques for fine-grained access control. In Proceedings of the 2004 ACM SIGMOD international conference on Management of data, pages 551–562, 2004.
[18] R. S. Roy. 3.9 conversational question answering over knowledge graphs. Conversational Search, page 45.
[19] H. Su, X. Shen, R. Zhang, F. Sun, P. Hu, C. Niu, and J. Zhou. Improving multi-turn dialogue modelling with utterance rewriter. arXiv preprint arXiv:1906.07004, 2019.
[20] S. Vakulenko, S. Longpre, Z. Tu, and R. Anantha. Question rewriting for conversa- tional question answering. In Proceedings of the 14th ACM International Conference on Web Search and Data Mining, pages 355–363, 2021.
[21] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
[22] T. Wolf, L. Debut, V. Sanh, J. Chaumond, C. Delangue, A. Moi, P. Cistac, T. Rault,R. Louf, M. Funtowicz, et al. Huggingface’s transformers: State-of-the-art natural language processing. arXiv preprint arXiv:1910.03771, 2019.
[23] S.Yu,Z.Liu,C.Xiong,T.Feng,andZ.Liu.Few-shotconversationaldenseretrieval. In Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 829–838, 2021.
[24] Y. Zhang, Z. Li, J. Wang, N. Cheng, and J. Xiao. Self-attention for incomplete utter- ance rewriting. In ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 8047–8051. IEEE, 2022.
連結至畢業學校之論文網頁點我開啟連結
註: 此連結為研究生畢業學校所提供,不一定有電子全文可供下載,若連結有誤,請點選上方之〝勘誤回報〞功能,我們會盡快修正,謝謝!
QRCODE
 
 
 
 
 
                                                                                                                                                                                                                                                                                                                                                                                                               
第一頁 上一頁 下一頁 最後一頁 top