臺灣博碩士論文加值系統

在如今網路發達的時代，客服中心擔任至關重要的角色，它提供了企業與顧客之間的直接溝通渠道。隨著現代的AI技術不斷創新，使用AI工具輔助的智能客服中心已成為業界的新趨勢。目前在中小企業中主流的做法是將商業大型語言模型（LLM）作為AI客服的核心，並結合檢索增強生成（RAG）架構，透過提供專屬的私有數據來賦予LLM生成更精確的回答。這種方法有效解決了由於商業LLM缺乏專業知識或私有訊息而導致的回答不準確或生成幻覺的問題。然而現有的RAG架構除了透過其他科技公司提供技術支持外，當中小企業想要自行建立時，對於檢索能力優化、資料量利用率、系統運行成本高與時間效率低等問題上仍有許多可以改進的空間。
於是本論文提出一套基於知識圖譜與變體三元相似度SBERT的RAG檢索器訓練架構，主要目的為用更小的成本與時間建立更加細緻的QA訓練資料，並訓練出更加優秀的RAG檢索器模型。在本論文的首階段利用CKIP分詞技術從文本中提取關鍵字，接著利用BM25算法計算文本間的相似度，建立文本相似度知識圖譜，此階段透過精確的相似度分析，豐富了知識圖譜的結構，提升了文本間關聯性的可視化和檢索效率。在第二階段本論文利用知識圖譜來確定問題與各文本之間的相似度軟標籤。這一方法不僅增強了問題與文本之間的關聯性，還細化了問題與多個相關文本的相似度關係。這樣的細緻關聯性為模型提供了更豐富的訓練訊號，有助於提升RAG檢索器的精準度和泛化能力。在最終階段本論文透過創新的三元相似度SBERT架構，將訓練模式從單一文本對應擴展到問題與多個文本間的多維配對，這不僅大幅增加了訓練資料的量，也提升了資料利用率。此架構讓問題與文本之間的配對更為精確，有效地強化了模型在處理複雜查詢時的適應性與精準度。
本論文的主要貢獻在於開發了一套高效率且低能耗的訓練架構，不僅提升了私有資料的利用率及文本檢索的準確性，也顯著提升了模型在處理複雜查詢時的適應性與準確度。實驗結果顯示透過本論文方法抓取的參考文本對於進階模型如LLAMA2、GPT4在生成答案上都展示了更佳的效果。

In today's digitally connected world, customer service centers play a crucial role, providing a direct communication channel between businesses and customers. With the continuous innovation of modern AI technologies, AI-assisted intelligent customer service centers have become an industry trend. For small and medium-sized enterprises (SMEs), the mainstream approach involves using large language models (LLMs) as the core of AI customer service, combined with a Retrieval-Augmented Generation (RAG) framework. By integrating proprietary data, this approach allows LLMs to generate more precise answers, effectively addressing issues related to inaccurate responses or hallucinations due to the lack of specialized knowledge or proprietary information in commercial LLMs. However, existing RAG frameworks, aside from being supported by other tech companies, still face several challenges when SMEs attempt to build them independently. These include optimization of retrieval capabilities, efficient data utilization, high system operation costs, and low time efficiency.

To address these issues, this paper proposes a RAG retriever training framework based on knowledge graphs and variant triple similarity SBERT. The primary goal is to establish more refined QA training data at a lower cost and in less time, thereby training a more effective RAG retriever model. In the first phase of this study, CKIP segmentation technology is used to extract keywords from text, followed by the BM25 algorithm to calculate text similarity and construct a text similarity knowledge graph. This phase enriches the structure of the knowledge graph through precise similarity analysis, enhancing the visualization of text correlations and retrieval efficiency. In the second phase, this paper uses the knowledge graph to determine the soft labels of similarity between the query and various texts. This method not only strengthens the relevance between the query and the texts but also refines the similarity relationships among multiple related texts. Such detailed associations provide the model with richer training signals, aiding in improving the precision and generalization capability of the RAG retriever. In the final phase, this paper introduces an innovative triple similarity SBERT architecture, expanding the training model from single-text matching to multi-dimensional matching between the query and multiple texts. This not only significantly increases the amount of training data but also improves data utilization. This architecture makes the pairing between queries and texts more accurate, effectively enhancing the model's adaptability and precision when handling complex queries.

The primary contribution of this paper lies in developing an efficient and low-power training framework that not only enhances the utilization of proprietary data and the accuracy of text retrieval but also significantly improves the model's adaptability and precision in handling complex queries. Experimental results show that the reference texts retrieved through this approach demonstrate better performance in generating answers with advanced models like LLAMA2 and GPT-4.

目錄

誌謝 I
目錄 VI
圖目錄 IX
表目錄 X
第一章 簡介 1
第二章 相關研究 8
2.1 傳統文本檢索技術 8
2.1.1 傳統統計方法: 8
2.1.2 基於淺層詞嵌入: 9
2.1.3 基於深度學習: 11
2.2 RAG檢索器增強技術 13
2.2.1 遞迴式檢索: 13
2.2.2 混合式檢索: 15
2.2.3 檢索器微調: 19
2.3 總覽 23
第三章 背景知識 25
3.1 Transformer 25
3.2 BERT 27
3.3 SBERT 28
3.4 知識圖譜 30
第四章 系統設計 32
4.1 整體架構 32
4.2 資料前處理與問題集生成 33
4.2.1 資料蒐集 33
4.2.2 資料前處理(資料清洗) 34
4.2.3 資料前處理(問題生成) 36
4.3 知識圖譜群構建 36
4.3.1 文本相識度比對 37
4.3.2 知識圖譜建立 39
4.4 問題軟標籤生成 40
4.5 向量檢索模型訓練 43
4.5.1 三元相似度SBERT 45
4.5.2 三元相似度SBERT訓練資料建立與模型訓練 47
第五章 實驗分析 49
5.1 資料集 49
5.2 環境與系統參數設定 50
5.3 實驗結果 50
5.3.1 模型參數優化實驗 52
5.3.2 SBERT消融實驗 59
5.3.3 TOP-N檢索率 65
第六章 結論 69
參考文獻 71

圖目錄
圖 1:系統架構圖 5
圖 2:Transformer架構 26
圖 3:BERT架構 28
圖 4:SBERT架構 30
圖 5:簡易知識圖譜圖 31
圖 6:系統設計整體架構 33
圖 7 :文本相識度比對 37
圖 8:知識圖譜建立 40
圖 9:問題軟標籤生成 40
圖10:PBMSR做法 42
圖 11:三元相似度SBERT 44
圖12:參數優化實驗- HR指標 55
圖13:參數優化實驗- MRR、ARP指標 58
圖14: SBERT消融實驗- HR、MRR指標 62
圖15:軟標籤消融實驗- HR、MRR指標 64
圖16:模型對比實驗- HR、MRR、ARP指標 68

表目錄
表 1:相關研究比較表 23
表2:文本完全對應問題資料集 49
表3:本研究系統實驗環境 50
表4:參數優化實驗- HR指標 54
表5 :參數優化實驗- MRR、ARP指標 57
表6 :SBERT消融實驗 61
表7:軟標籤消融實驗 63
表8 :對比實驗-HR指標 65
表9:對比實驗-MRR指標 65
表10:對比實驗-ARP指標 66

[1]Vaswani, Ashish, et al., "Attention is all you need." Advances in neural information processing systems 30, 2017.
[2]Devlin, Jacob, et al., "Bert: Pre-training of deep bidirectional transformers for language understanding." arXiv preprint arXiv:1810.04805, 2018.
[3]Brown, Tom, et al., "Language models are few-shot learners." Advances in neural information processing systems 33, 2020, pp. 1877-1901.
[4]Ouyang, Long, et al., "Training language models to follow instructions with human feedback." Advances in neural information processing systems 35, 2022, pp. 27730-27744.
[5]Lewis, Patrick, et al., "Retrieval-augmented generation for knowledge-intensive nlp tasks." Advances in Neural Information Processing Systems 33, 2020, pp. 9459-9474.
[6]S. Yao, J. Zhao, D. Yu et al., "React: Synergizing reasoning and acting
in language models, " in ICLR, 2023.
[7]Sun, Jiashuo, et al., "Think-on-graph: Deep and responsible reasoning of large language model with knowledge graph." arXiv preprint arXiv:2307.07697, 2023.
[8]T. Pouplin, H. Sun, S. Holt, and M. Van der Schaar, "Retrieval-augmented thought process as sequential decision making, " arXiv:2402.07812, 2024.
[9]W. Wang, Y. Wang et al., "Rap-gen: Retrieval-augmented patch gener-ation with codet5 for automatic program repair, " in ESEC/FSE, 2023.
[10]K. Sawarkar, A. Mangal et al., "Blended rag: Improving rag (retriever-augmented generation) accuracy with semantic search and hybrid query-based retrievers, " arXiv:2404.07220, 2024.
[11]S. Lu, N. Duan, H. Han et al., "Reacc: A retrieval-augmented code completion framework, " in ACL, 2022.
[12]W. Shi, S. Min, M. Yasunaga et al., "Replug: Retrieval-augmented black-box language models, " arXiv:2301.12652, 2023.
[13]Zhang, Tianjun, et al., "Raft: Adapting language model to domain specific rag." arXiv preprint arXiv:2403.10131, 2024.
[14]J. Li, Y. Li, G. Li et al., "Editsum: A retrieve-and-edit framework for source code summarization, " in ASE, 2021.
[15]S. Robertson and H. Zaragoza., "The probabilistic relevance framework: BM25 and beyond, " Foundations and Trends in Information Retrieval, vol. 3, No. 4, pp. 333–389. 2009.
[16]Mikolov, Tomas, et al., "Efficient estimation of word representations in vector space." arXiv preprint arXiv:1301.3781, 2013.
[17]Pennington, Jeffrey, Richard Socher, and Christopher D. Manning. "Glove: Global vectors for word representation." Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP). 2014.
[18]R. Nils, and I. Gurevych. "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks," Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing (EMNLP-IJCNLP), Hong Kong, China, pp. 3982-3992, Nov. 2019.
[19]Karpukhin, Vladimir, et al., "Dense passage retrieval for open-domain question answering." 2020 Conference on Empirical Methods in Natural Language Processing, EMNLP 2020. Association for Computational Linguistics (ACL), 2020.
[20]L. Wang, N. Yang, X. Huang, L. Yang, R. Majumder, and F. Wei, "Multilingual e5 text embeddings: A technical report, " arXiv preprint arXiv:2402.05672, Feb, 2024.
[21]Yan, Shi-Qi, et al. "Corrective retrieval augmented generation." arXiv preprint arXiv:2401.15884, 2024.
[22]Huang, Wenyu, et al., "Retrieval augmented generation with rich answer encoding." Proceedings of the 13th International Joint Conference on Natural Language Processing and the 3rd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics (Volume 1: Long Papers). 2023.
[23]Wang, Hongru, et al., "Unims-rag: A unified multi-source retrieval-augmented generation for personalized dialogue systems." arXiv preprint arXiv:2401.13256, 2024.
[24]Siriwardhana, Shamane, et al., "Fine-tune the Entire RAG Architecture (including DPR retriever) for Question-Answering." arXiv preprint arXiv:2106.11517, 2021.
[25]Wang, Xiaohua, et al., "Searching for Best Practices in Retrieval-Augmented Generation." arXiv preprint arXiv:2407.01219, 2024.
[26]Asai, Akari, et al., "Self-rag: Learning to retrieve, generate, and critique through self-reflection." arXiv preprint arXiv:2310.11511, 2023.
[27]Dodgson, Jennifer, et al., "Establishing performance baselines in fine-tuning, retrieval-augmented generation and soft-prompting for non-specialist llm users." arXiv preprint arXiv:2311.05903, 2023.
[28]Vaswani, A., Shazeer, N.M., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, L., & Polosukhin, I, "Attention is All you Need," Neural Information Processing Systems, 2017.