臺灣博碩士論文加值系統

高斯過程(Gaussian process)主要被用在解決迴歸(regression)與分類(classification)問題上，在眾多機器學習方法之中預測表現良好，且是非常彈性的非參數(nonparametric)模型。但其擁有一些明顯的缺點，如時間複雜度高，導致不容易應用在資料量較多的問題上。為了解決這個缺點，本次研究將結合高斯過程(Gaussian process)與梯度提升方法(gradient boosting)應用在分類問題上。根據實驗結果顯示，與原本的高斯過程相比，此結合後產生的演算法可大幅增加模型訓練速度，使模型可以在訓練中使用更多的資料，且可在某些情況下增加模型的表現，此外也會探討此方法與其他分類器在不同資料集與不同訓練資料量上之差別。

Gaussian process (GP) is mainly used to solve regression and classification problems in machine learning. GP is a nonparametric model with good prediction performance and wide applications. However, GP has an obvious drawback: high time complexity. This drawback makes it inappropriate for large data. In this work, we combine Gaussian process and gradient boosting to form Gradient Boosting Gaussian Process Classifier (GBGPC), then apply it to classification problems. The experiment results show that the proposed algorithm can largely improve training efficiency and performance in some situation compare to the standard Gaussian process. We also discuss the difference of performance between GBGPC and other well-known classifiers among different datasets and training data sizes.

誌謝 i
中文摘要 ii
ABSTRACT iii
CONTENTS iv
LIST OF FIGURES vi
LIST OF TABLES vii
Chapter 1 Introduction 1
Chapter 2 Literature Review 3
2.1 Gaussian Process 3
2.1.1 Gaussian Process Regression 4
2.1.2 Gaussian Process Classification 6
2.1.3 Latent Variable Inferences 8
2.1.4 Automatic Relevance Determination 15
2.1.5 Approximation Methods for Gaussian Process 16
2.2 Gradient Boosting Machine 18
2.2.1 Boosting 19
2.2.2 Gradient Boosting 19
2.2.3 Applications and Extensions of Gradient Boosting 21
Chapter 3 Design of Gradient Boosting Gaussian Process Classifier 24
3.1 Loss Function 24
3.2 Algorithm of Gradient Boosting Gaussian Process Classifier 27
3.2.1 Training and Test Algorithm of GBGPC 27
3.2.2 Base Learner Gaussian Process Regressor 30
3.2.3 Batch size 31
3.2.4 Early Stopping 32
Chapter 4 Experiment and Discussion 34
4.1 Datasets 34
4.2 Experiment Design and Baseline Models 35
4.3 Experiment Results 37
4.3.1 Performance Comparison 37
4.3.2 Early Stopping and Training time 48
4.3.3 Pseudo-Response Comparison 51
Chapter 5 Conclusion and Future Work 55
REFERENCE 57

[1]Abadi M. et al, TensorFlow: Large-scale Machine Learning on Heterogeneous Systems (https://www.tensorflow.org/), 2015.
[2]Baldi, P., Sadowski, P., Whiteson, D., Searching for Exotic Particles in High-energy Physics with Deep Learning, Nature Communications 5, July 2, 2014.
[3]Bishop, C. M., Pattern Recognition and Machine Learning (Information Science and Statistics), Springer-Verlag Berlin, Heidelberg, 2006.
[4]Breiman, L., Random Forest, Machine Learning, 45(1), 2001, pp. 5-32.
[5]Carcillo, F., Le Borgne, Y., Caelen, O., Bontempi, G., Streaming Active Learning Strategies for Real-life Credit Card Fraud Detection: Assessment and Visualization, International Journal of Data Science and Analytics, 5, 4, Springer International Publishing, 2018, pp. 285-300.
[6]Carcillo, F., Le Borgne, Y., Caelen, O., Oblé, F., Bontempi, G., Combining Unsupervised and Supervised Learning in Credit Card Fraud Detection, Information Sciences, 2019.
[7]Carcillo, F., Dal Pozzolo, A., Le Borgne, Y., Caelen, O., Mazzer, Y., Bontempi, G., Scarff: A Scalable Framework for Streaming Credit Card Fraud Detection with Spark, Information Fusion, 41, Elsevier, 2018, pp. 182-194
[8]Chen, T., Guestrin, C., XGBoost: A Scalable Tree Boosting System, Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 2016, pp. 785-794.
[9]Dal Pozzolo, A., Boracchi, G., Caelen, O., Alippi, C., Bontempi, G., Credit Card Fraud Detection: A Realistic Modeling and a Novel Learning Strategy, IEEE Transactions on Neural Networks and Learning Systems, 29, 8, 2018, pp. 3784-3797
[10]Dal Pozzolo, A., Caelen, O., Le Borgne, Y., Waterschoot, S., Bontempi, G., Learned Lessons in Credit Rard Fraud Detection from a Practitioner Perspective, Expert Systems with Applications, 41, 10, Pergamon, 2014, pp. 4915-4928
[11]Dal Pozzolo, A., Caelen, O., Johnson, R. A., Bontempi, G., Calibrating Probability with Undersampling for Unbalanced Classification, in Symposium on Computational Intelligence and Data Mining (CIDM), IEEE, 2015.
[12]Dal Pozzolo, A., Adaptive Machine Learning for Credit Card Fraud Detection, Université Libre de Bruxelles Machine Learning Group, PhD thesis (supervised by G. Bontempi), 2015.
[13]Dorogush, A. V., Ershov, V., Gulin, A., CatBoost: Gradient Boosting with Categorical Features Support, Proceedings of Neural Information Processing Systems 2018 (NIPS 2018), 2018.
[14]Dua, D., Graff, C., UCI Machine Learning Repository [http://archive.ics.uci.edu/ml]. Irvine, CA: University of California, School of Information and Computer Science, 2019.
[15]Friedman J. H., Greedy Function Approximation: A Gradient Boosting Machine, The Annals of Statistics, 2001, pp. 1189-1232
[16]Ke G. et al, LightGBM: A Highly Efficient Gradient Boosting Decision Tree, Proceedings of Neural Information Processing Systems 2017(NIPS 2017), 2017.
[17]Lawrence, N., Seeger, M., Herbrich, R., Fast Sparse Gaussian Process Methods: The Informative Vector Machine, In Becker, S., Thrun, S., Obermayer, K. editors, Advances in Neural Information Processing Systems 15, MIT Press, 2003, pp. 625-632.
[18]Lebichot, B., Le Borgne, Y., He, L. He, Oblé, F., Bontempi, G., Deep-Learning Domain Adaptation Techniques for Credit Cards Fraud Detection, INNSBDDL 2019: Recent Advances in Big Data and Deep Learning, 2019, pp. 78-88.
[19]Liao, W., A Gradient Boosting Algorithm Based on Gaussian Process Regression, National Taiwan University, Department of Information Management Master thesis (supervised by Hsin-Min Lu), 2018.
[20]Minka, T. P., Expectation Propagation for Approximate Bayesian Inference, UAI, Morgan Kaufmann, 2001a, pp 362-369.
[21]Nguyen-Tuong, D., Seeger, M., Peters, J., Model Learning with Local Gaussian Process Regression, Advanced Robotics, 2009.
[22]Nickisch, H., Approximations for Binary Gaussian Process Classification, Journal of Machine Learning Research 9, 2008, pp. 2035-2078.
[23]Opper, M., Archambeau, C., The Variational Gaussian Approximation Revisited. Neural Computation, 2008.
[24]Pedregosa et al, Scikit-learn: Machine Learning in Python, Journal of Machine Learning Research 12, pp. 2825-2830, 2011.
[25]Quiñonero-Candela, J., Ramussen, C. E., Williams, C. K. I., Approximation Methods for Gaussian Process Regression, 2007.
[26]Rasmussen, C. E., Williams, C. K. I., Gaussian Processes for Machine Learning, The MIT press, 2006
[27]Snelson, E., Ghahramani, Z., Sparse Gaussian Processes Using Pseudo-inputs, Advances in Neural Information Processing Systems 18, 2005.
[28]Tresp, V., A Bayesian Committee Machine, Neural Computation, Vol. 12, 2000, pp. 2719-2741
[29]Vapnik, V. N., The Nature of Statistical Learning Theory. Springer, 1995.
[30]Wahba, G., Spline Models for Observational Data. Society for Industrial and Applied Mathematics, 1990.
[31]Williams, C. K. I., Barber, D., Bayesian Classification with Gaussian Processes. IEEE Transactions on Pattern Analysis and Machine Intelligence, 20(12), 1998, pp. 1342-1351.
[32]Yeh, I. C., Lien, C. H., The Comparisons of Data Mining Techniques for the Predictive Accuracy of Probability of Default of Credit Card Clients. Expert Systems with Applications, 36(2), 2009, pp. 2473-2480.