臺灣博碩士論文加值系統

近年來生成式模型越來越受到重視,尤其是基於 Transformer 或是 Attention的模型在各個領域都有不少的成果,像是文章、音樂、圖片、影片等等。與此同時,在生成帶文字標籤的圖結構(如知識圖譜、心智圖等)上並沒有太多發展,由於該問題同時牽扯到圖結構的生成與文字標籤的生成,以往的方法大致上會分成兩種,一種是將文字與圖結構分別用兩個不同的模型,另一種則是將圖拆解成一段段的文字序列並使用序列模型來處理。然而,使用兩個模型的方法容易缺少圖結構與文字之間交互的資訊,而將圖拆解成序列的方法則是會損失部分的圖結構資訊並且將低生成效率。本論文提出了一種結構標記,能夠將圖結構與文字共同轉成單一的表示法。透過這種表示法,模型可以更有效率的學習以及生成圖結構與文字,在此之上我們也提出了一種預訓練的方法。為了證明方法的有效性,我們在兩個公開的資料集上做測試,並且結果顯示我們的方法可以用更少的參數量達到跟過去模型可比的分數。

Transformer layer has been proved to work well in several domains beyond text, like audio, image, and even multi-modal. The idea behind these models is that we can treat different kind of input as a series of tokens. Recent research also shown that with carefully designed input token, a pure transformer encoder can also be a powerful graph encoder. Taking steps further in this direction, we propose a new kind of input representation called ”Structure Token”. With structure token, we can represent graph with text label as a sequence of tokens. By converting both graph and text into structure token, we train a pure transformer encoder-decoder that learn a unified representation and generate both graph and text with the same model. We also propose a new pretrain method similar to mBART pre-training but with the structure token. In this paper, we show that with the proposed method, we are able to train a smaller model that has performance comparable to the T5 variants on text-to-graph and graph-to-text tasks.

Acknowledgements i
摘要 iii
Abstract v
Contents vii
List of Figures xi
List of Tables xiii
Chapter 1 Introduction 1
1.1 Graph Structure in Natural Language Processing 2
1.2 Common Methods for Generating Text Graph 3
1.2.1 Multi-stage Approach 3
1.2.2 Graph Linearization 4
1.3 Motivation 5
1.4 Contribution 6
1.5 Thesis Organization 6
Chapter 2 Preliminaries 9
2.1 Introduction to Generative Model 9
2.1.1 Basics of Text Generation 10
2.1.2 Basics of Graph Generation 11
2.2 Introduction to Transformer Model 11
2.2.1 The Core Module: Attention Mechanism 12
2.2.2 Handling Sequential Data: Position Embedding 14
2.3 Transformer for Graph Data 15
2.4 Improve Model Performance: Pre-Training and Fine-Tuning 17
2.5 Evaluate Generation Result 18
2.5.1 Metrics for Text Generation 19
2.5.2 Metrics for Text Graph Generation 20
Chapter 3 Method 21
3.1 Model Overview 21
3.2 The Core Design: Representing Graph via Structure Token 24
3.2.1 Problem Setup 25
3.2.2 Convert Graph/Text to Structure Token 27
3.2.3 Convert Structure Token to Vector Representation 31
3.2.4 Text Generation through Structure Token 36
3.2.5 Graph Generation through Structure Token 37
3.2.6 Efficiency of Structure Token 38
3.3 Transformer Model Training 39
Chapter 4 Experiments 41
4.1 Experiment Setup 41
4.1.1 Training Setup 41
4.1.2 Model Parameters 42
4.1.3 Datasets 42
4.1.4 Data Processing 46
4.2 Effectiveness of Structure Token on Graph-to-Text Generation 47
4.3 Effectiveness of Structure Token on Text-to-Graph Generation 48
4.4 Ablation Study 53
Chapter 5 Conclusions and Future Work 55
5.1 Summary 55
5.2 Discussion and Future Work 56
5.3 Conclusions 61
References 63

O. Agarwal, H. Ge, S. Shakeri, and R. Al-Rfou. Knowledge graph based synthetic corpus generation for knowledge-enhanced language model pre-training. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 3554–3565, Online, June 2021. Association for Computational Linguistics.
O. Agarwal, M. Kale, H. Ge, S. Shakeri, and R. Al-Rfou. Machine translation aided bilingual data-to-text generation and semantic parsing. In Proceedings of the 3rd International Workshop on Natural Language Generation from the Semantic Web (WebNLG+), pages 125–130, Dublin, Ireland (Virtual), 12 2020. Association for Computational Linguistics.
S. Auer, C. Bizer, G. Kobilarov, J. Lehmann, R. Cyganiak, and Z. Ives. Dbpedia: A nucleus for a web of open data. In Proceedings of the 6th International The Semantic Web and 2nd Asian Conference on Asian Semantic Web Conference, ISWC’07/ASWC’07, page 722 – 735, Berlin, Heidelberg, 2007. Springer-Verlag.
D. Bahdanau, K. Cho, and Y. Bengio. Neural machine translation by jointly learning to align and translate. CoRR, abs/1409.0473, 2014.
S. Banerjee and A. Lavie. METEOR: An automatic metric for MT evaluation with improved correlation with human judgments. In Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization, pages 65–72, Ann Arbor, Michigan, June 2005. Association for Computational Linguistics.
T. Berners-Lee, J. Hendler, and O. Lassila. The semantic web. Scientific American, 284(5):34–43, May 2001.
T. Castro Ferreira, C. Gardent, N. Ilinykh, C. van der Lee, S. Mille, D. Moussallem, and A. Shimorina. The 2020 bilingual, bi-directional WebNLG+ shared task: Overview and evaluation results (WebNLG+ 2020). In Proceedings of the 3rd International Workshop on Natural Language Generation from the Semantic Web (WebNLG+), pages 55–76, Dublin, Ireland (Virtual), 12 2020. Association for Computational Linguistics.
A. Celikyilmaz, E. Clark, and J. Gao. Evaluation of text generation: A survey. ArXiv, abs/2006.14799, 2020.
X. Chen, C. Liang, D. Huang, E. Real, K. Wang, Y. Liu, H. Pham, X. Dong, T. Luong, C.-J. Hsieh, Y. Lu, and Q. V. Le. Symbolic discovery of optimization algorithms. ArXiv, abs/2302.06675, 2023.
A. Colas, M. Alvandipour, and D. Z. Wang. Gap: A graph-aware language model framework for knowledge graph-to-text generation. In International Conference on Computational Linguistics, 2022.
A. Colas, A. Sadeghian, Y. Wang, and D. Z. Wang. Eventnarrative: A large-scale event-centric dataset for knowledge graph-to-text generation, 2022.
J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics.
P. Dognin, I. Padhi, I. Melnyk, and P. Das. ReGen: Reinforcement learning for text and knowledge base generation using pretrained language models. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 1084–1099, Online and Punta Cana, Dominican Republic, Nov. 2021. Association for Computational Linguistics.
P. Dufter, M. Schmitt, and H. Schütze. Position information in transformers: An overview. Computational Linguistics, 48(3):733–763, Sept. 2022.
V. P. Dwivedi and X. Bresson. A generalization of transformer networks to graphs. ArXiv, abs/2012.09699, 2020.
V. P. Dwivedi, C. K. Joshi, T. Laurent, Y. Bengio, and X. Bresson. Benchmarking graph neural networks. ArXiv, abs/2003.00982, 2020.
M. Freitag and Y. Al-Onaizan. Beam search strategies for neural machine translation. In Proceedings of the First Workshop on Neural Machine Translation, pages 56–60, Vancouver, Aug. 2017. Association for Computational Linguistics.
S. Gottschalk and E. Demidova. Eventkg: A multilingual event-centric temporal knowledge graph. In Extended Semantic Web Conference, 2018.
Q. Guo, Z. Jin, X. Qiu, W. Zhang, D. Wipf, and Z. Zhang. CycleGT: Unsupervised graph-to-text and text-to-graph generation via cycle training. In Proceedings of the 3rd International Workshop on Natural Language Generation from the Semantic Web (WebNLG+), pages 77–88, Dublin, Ireland (Virtual), 12 2020. Association for Computational Linguistics.
D. Hendrycks and K. Gimpel. Gaussian error linear units (gelus). arXiv: Learning, 2016.
J. Hewitt and C. D. Manning. A structural probe for finding syntax in word representations. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4129–4138, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics.
L. Hu, T. Yang, L. Zhang, W. Zhong, D. Tang, C. Shi, N. Duan, and M. Zhou. Compare to the knowledge: Graph neural fake news detection with external knowledge. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 754–763, Online, Aug. 2021. Association for Computational Linguistics.
F. Ilievski, P. Szekely, and B. Zhang. Cskg: The commonsense knowledge graph. Extended Semantic Web Conference (ESWC), 2021.
Z. Jin, Q. Guo, X. Qiu, and Z. Zhang. GenWiki: A dataset of 1.3 million content-sharing text and graphs for unsupervised graph-to-text generation. In Proceedings of the 28th International Conference on Computational Linguistics, pages 2398–2409, Barcelona, Spain (Online), Dec. 2020. International Committee on Computational Linguistics.
M. Johnson, M. Schuster, Q. V. Le, M. Krikun, Y. Wu, Z. Chen, N. Thorat, F. B. Viégas, M. Wattenberg, G. S. Corrado, M. Hughes, and J. Dean. Google' s multilingual neural machine translation system: Enabling zero-shot translation. Transactions of the Association for Computational Linguistics, 5:339–351, 2016.
J. Kim, D. Nguyen, S. Min, S. Cho, M. Lee, H. Lee, and S. Hong. Pure transformers are powerful graph learners. Advances in Neural Information Processing Systems, 35:14582–14595, 2022.
D. P. Kingma and J. Ba. Adam: A method for stochastic optimization. CoRR, abs/1412.6980, 2014.
T. Kipf and M. Welling. Semi-supervised classification with graph convolutional networks. ArXiv, abs/1609.02907, 2016.
T. Kudo. Subword regularization: Improving neural network translation models with multiple subword candidates. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 66–75, Melbourne, Australia, July 2018. Association for Computational Linguistics.
M. Lewis, Y. Liu, N. Goyal, M. Ghazvininejad, A. Mohamed, O. Levy, V. Stoyanov, and L. Zettlemoyer. BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7871–7880, Online, July 2020. Association for Computational Linguistics.
T. Lin, Y. Wang, X. Liu, and X. Qiu. A survey of transformers. AI Open, 3:111–132, 2021.
L. Liu, H. Jiang, P. He, W. Chen, X. Liu, J. Gao, and J. Han. On the variance of the adaptive learning rate and beyond. ArXiv, abs/1908.03265, 2019.
Y. Liu, J. Gu, N. Goyal, X. Li, S. Edunov, M. Ghazvininejad, M. Lewis, and L. Zettlemoyer. Multilingual denoising pre-training for neural machine translation. Transactions of the Association for Computational Linguistics, 8:726–742, 2020.
M. Mayank, S. Sharma, and R. Sharma. Deap-faked: Knowledge graph based approach for fake news detection. 2022 IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining (ASONAM), pages 47–51, 2021.
I. Melnyk, P. Dognin, and P. Das. Knowledge graph generation from text. In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 1610–1622, Abu Dhabi, United Arab Emirates, Dec. 2022. Association for Computational Linguistics.
E. Min, R. Chen, Y. Bian, T. Xu, K. Zhao, W. bing Huang, P. Zhao, J. Huang, S. Ananiadou, and Y. Rong. Transformer for graphs: An overview from architecture perspective. ArXiv, abs/2202.08455, 2022.
M. Mintz, S. Bills, R. Snow, and D. Jurafsky. Distant supervision for relation extraction without labeled data. In Proceedings of the Joint Conference of the 47th Annual Meeting of the ACL and the 4th International Joint Conference on Natural Language Processing of the AFNLP, pages 1003–1011, Suntec, Singapore, Aug. 2009. Association for Computational Linguistics.
S. Ouyang, Z. Zhang, and H. Zhao. Dialogue graph modeling for conversational machine reading. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 3158–3169, Online, Aug. 2021. Association for Computational Linguistics.
K. Papineni, S. Roukos, T. Ward, and W.-J. Zhu. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pages 311–318, Philadelphia, Pennsylvania, USA, July 2002. Association for Computational Linguistics.
A. Radford and K. Narasimhan. Improving language understanding by generative pre-training. 2018.
C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y. Zhou, W. Li, and P. J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21(140):1–67, 2020.
A. Rogers, O. Kovaleva, and A. Rumshisky. A primer in BERTology: What we know about how BERT works. Transactions of the Association for Computational Linguistics, 8:842–866, 2020.
A. Sinha, Z. Shen, Y. Song, H. Ma, D. Eide, B.-J. P. Hsu, and K. Wang. An overview of microsoft academic service (mas) and applications. In International World Wide Web Conferences. Microsoft, May 2015.
N. Srivastava, G. Hinton, A. Krizhevsky, I. Sutskever, and R. Salakhutdinov. Dropout: A simple way to prevent neural networks from overfitting. Journal of Machine Learning Research, 15(56):1929–1958, 2014.
I. Sutskever, O. Vinyals, and Q. V. Le. Sequence to sequence learning with neural networks. In Z. Ghahramani, M. Welling, C. Cortes, N. Lawrence, and K. Weinberger, editors, Advances in Neural Information Processing Systems, volume 27. Curran Associates, Inc., 2014.
A. Vaswani, N. M. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin. Attention is all you need. In NIPS, 2017.
D. Vrandečić and M. Krötzsch. Wikidata: A free collaborative knowledgebase. Commun. ACM, 57(10):78 – 85, sep 2014.
Y. Xu, L. Fu, Z. Lin, J. Qi, and X. Wang. Infinity: A simple yet effective unsupervised framework for graph-text mutual conversion. ArXiv, abs/2209.10754, 2022.
J. Yang, G. Xiao, Y. Shen, W. Jiang, X. Hu, Y. Zhang, and J. Peng. A survey of knowledge enhanced pre-trained models. ArXiv, abs/2110.00269, 2021.
C. Ying, T. Cai, S. Luo, S. Zheng, G. Ke, D. He, Y. Shen, and T.-Y. Liu. Do transformers really perform bad for graph representation? In Neural Information Processing Systems, 2021.
J. You, R. Ying, X. Ren, W. L. Hamilton, and J. Leskovec. Graphrnn: Generating realistic graphs with deep auto-regressive models. In International Conference on Machine Learning, 2018.
L. Zhang and R. Li. KE-GCL: Knowledge enhanced graph contrastive learning for commonsense question answering. In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 76–87, Abu Dhabi, United Arab Emirates, Dec. 2022. Association for Computational Linguistics.
T. Zhang, V. Kishore, F. Wu, K. Q. Weinberger, and Y. Artzi. Bertscore: Evaluating text generation with bert. ArXiv, abs/1904.09675, 2019.
Y. Zhang, H. Dai, K. Toraman, and L. Song. Kg^2: Learning to reason science exam questions with contextual knowledge graph embeddings, 2018.
L. Zhong, J. Wu, Q. Li, H. Peng, and X. Wu. A comprehensive survey on automatic knowledge graph construction, 2023.
Y. Zhu, Y. Du, Y. Wang, Y. Xu, J. Zhang, Q. Liu, and S. Wu. A survey on deep graph generation: Methods and applications. In LOG IN, 2022.