跳到主要內容

臺灣博碩士論文加值系統

(44.200.194.255) 您好!臺灣時間:2024/07/19 04:52
字體大小: 字級放大   字級縮小   預設字形  
回查詢結果 :::

詳目顯示

: 
twitterline
研究生:何非凡
研究生(外文):He, Fei-Fan
論文名稱:以多智能體虛擬市場模型提升強化學習交易策略之泛化性
論文名稱(外文):A Multi-Agent Virtual Market Model for Generalization in Reinforcement Learning Based Trading Strategies
指導教授:黃思皓黃思皓引用關係
指導教授(外文):Huang, Szu-Hao
口試委員:蔡垂雄林瑞嘉陳安斌
口試委員(外文):Tsai, Chwei-ShyongLin, Jui-ChiaChen, An-Pin
口試日期:2021-07-30
學位類別:碩士
校院名稱:國立陽明交通大學
系所名稱:資訊管理研究所
學門:電算機學門
學類:電算機一般學類
論文種類:學術論文
論文出版年:2021
畢業學年度:109
語文別:英文
論文頁數:51
中文關鍵詞:深度學習交易策略金融科技強化學習
外文關鍵詞:Deep learningTrading strategyNeural network for financeReinforcement learning
相關次數:
  • 被引用被引用:0
  • 點閱點閱:203
  • 評分評分:
  • 下載下載:30
  • 收藏至我的研究室書目清單書目收藏:0
許多研究成功使用強化學習的方法,以金融市場資料訓練出有獲利能力的交易模型。
大多數的方法都簡化了交易行為對市場狀態的影響,也就是假設買賣行為不會影響交易環境,如此得以在強化學習的框架中直接重演歷史價格資料作為交易環境。
然而,以固定的歷史資料尋找最大化長期獲利的交易策略,容易有泛化性差的問題,在使用未來資料測試時的表現往往和訓練時有很大的落差。

為了改善交易策略的泛化性,我們提出了多智能體虛擬市場模型,使用多個生成網路模擬出市場未來可能的變化。
由於生成對抗網路擅長產生出擬真的資料,我們的虛擬市場模型學習了真實市場的價格和交易量分佈,可以生成無數個模擬的市場走勢。
此外,交易動作可以和生成的市場狀態疊加,作為虛擬市場模型的輸入,使後續生成的市場狀態能夠和過去交易行為相關。
我們認為結合多個生成網路可以學習到市場上多種價格走勢,比起使用單一網路更有機會精細的模擬真實市場的變化。
我們的研究以多智能體虛擬市場模型所生成的市場資料取代原始的歷史資料,建構強化學習的交易環境。

我們使用中國股票期貨以及美國股票市場的資料來建模並測試使用多智能體虛擬市場模型對強化學習交易策略的改善。
採用多智能體虛擬市場訓練出的交易策略比起直接使用歷史資料,交易滬深300股票期貨一年的總收益增加了至少12%;而交易蘋果公司股票,半年就能夠有至少34%的收益成長。
如此證明了使用多智能體虛擬市場模型訓練出的交易策略有更好的泛化性,能夠在測試資料集上得到更高的報酬且更低的損失風險。

多智能體虛擬市場模型不僅有模擬真實市場變化的能力,生成的價格資料更可以用來訓練基於強化學習的交易策略,並且能夠使交易模型表現出更好的獲利能力。
Many studies have successfully used reinforcement learning (RL) to train an intelligent agent that learns profitable trading strategies from financial market data. However, most of these studies have simplified the effect of the actions of the trading agent on the market state. Trading actions do not affect the market environment reproduced in RL. The trading agent is trained to maximize long-term profit by optimizing fixed historical data. Such approach frequently results in the trading performance during out-of-sample validation being considerably different from that during training. In this paper, we propose a multi-agent virtual market model (MVMM) comprised of multiple generative adversarial networks (GANs) which cooperate with each other to reproduce market price changes. Due to the advantages of GANs, the MVMM can learn the distribution of real market prices and generate numerous simulated market trends. In addition, the action of the trading agent can be superimposed on the current market state as the input of the MVMM to generate an action-dependent next state. In this research, real historical data were replaced with the simulated market data generated by the MGVMM. The experimental results indicated that the RL agent is appropriately generalized by the aforementioned methods. The trading strategy of the trained RL agent achieved high profits and exhibited low risk of loss during the testing phase.
中文摘要 i
英文摘要 ii
Acknowledgement iii
Table of Contents iv
List of Figures vi
List of Tables vii
1 Introduction 1
1.1 Background 1
1.2 Motivation 2
1.3 Research Goal and Hypothesis 4
1.4 Contributions and Value 5
1.5 Organization 5
2 Related Works 7
2.1 Deep learning in Finance 7
2.2 Reinforcement Learning in Finance 8
2.3 Generative Adversarial Networks in Finance 11
3 Proposed Method 13
3.1 System Overview 13
3.2 RL in Financial Trading 14
3.3 Local GANs 16
3.4 Global Controller and Global Discriminator 19
3.5 DRNN Agent 22
3.6 Proposed RL Trading Framework 24
4 Experiments 28
4.1 Experimental Settings 28
4.2 Evaluation of VM Simulation 31
4.3 Profitability Evaluation for the IF Dataset 36
4.4 Evaluation of Profitability for the AAPL Dataset 39
4.5 Comparison of the VM Models with Different Numbers of Local GANs 40
4.6 Evaluation of Profitability for Other US Stocks 43
5 Conclusion 46
References 48
[1] D. Shah, H. Isah, and F. Zulkernine, “Stock market analysis: A review and taxonomy of prediction techniques,” International Journal of Financial Studies, vol. 7, no. 2, p. 26, 2019.
[2] L. Zhang, C. Aggarwal, and G.-J. Qi, “Stock price prediction via discovering multi-frequency trading patterns,” in Proceedings of the 23rd ACM SIGKDD international conference on knowledge discovery and data mining, 2017, pp. 2141–2149.
[3] Y. Deng, F. Bao, Y. Kong, Z. Ren, and Q. Dai, “Deep direct reinforcement learning for
financial signal representation and trading,” IEEE transactions on neural networks and learning systems, vol. 28, no. 3, pp. 653–664, 2016.
[4] A. Tsantekidis, N. Passalis, A.-S. Toufa, K. Saitas-Zarkias, S. Chairistanidis, and A. Tefas, “Price trailing for financial trading using deep reinforcement learning,” IEEE Transactions on Neural Networks and Learning Systems, 2020.
[5] Z. Jiang, D. Xu, and J. Liang, “A deep reinforcement learning framework for the financial portfolio management problem,” arXiv preprint arXiv:1706.10059, 2017.
[6] T. Spooner, J. Fearnley, R. Savani, and A. Koukorinis, “Market making via reinforcement learning,” arXiv preprint arXiv:1804.04216, 2018.
[7] A. Charpentier, R. Elie, and C. Remlinger, “Reinforcement learning in economics and finance,” Computational Economics, pp. 1–38, 2021.
[8] Y. Ye, H. Pei, B. Wang, P.-Y. Chen, Y. Zhu, J. Xiao, and B. Li, “Reinforcement-learning based portfolio management with augmented asset movement prediction states,” in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 34, no. 01, 2020, pp. 1112–1119.
[9] Z. Liang, H. Chen, J. Zhu, K. Jiang, and Y. Li, “Adversarial deep reinforcement learning in portfolio management,” arXiv preprint arXiv:1808.09940, 2018.
[10] E. Chong, C. Han, and F. C. Park, “Deep learning networks for stock market analysis and prediction: Methodology, data representations, and case studies,” Expert Systems with Applications, vol. 83, pp. 187–205, 2017.
[11] F. Feng, H. Chen, X. He, J. Ding, M. Sun, and T.-S. Chua, “Enhancing stock movement prediction with adversarial training,” arXiv preprint arXiv:1810.09936, 2018.
[12] W. Li, R. Bao, K. Harimoto, D. Chen, J. Xu, and Q. Su, “Modeling the stock relation with graph network for overnight stock movement prediction,” in no. CONF, 2020, pp. 4541–4547.
[13] R. S. Sutton and A. G. Barto, Reinforcement learning: An introduction. MIT press, 2018.
[14] A. Tsantekidis, N. Passalis, and A. Tefas, “Improving deep reinforcement learning for financial trading using neural network distillation,” in 2020 IEEE 30th International Workshop on Machine Learning for Signal Processing (MLSP). IEEE, 2020, pp. 1–6.
[15] K. Lei, B. Zhang, Y. Li, M. Yang, and Y. Shen, “Time-driven feature-aware jointly deep reinforcement learning for financial signal representation and algorithmic trading,” Expert Systems with Applications, vol. 140, p. 112872, 2020.
[16] Y. Liu, Q. Liu, H. Zhao, Z. Pan, and C. Liu, “Adaptive quantitative trading: an imitative deep reinforcement learning approach,” in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 34, no. 02, 2020, pp. 2128–2135.
[17] J. Wang, Y. Zhang, K. Tang, J. Wu, and Z. Xiong, “Alphastock: A buying-winners-and-selling-losers investment strategy using interpretable deep reinforcement attention networks,” in Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, 2019, pp. 1900–1908.
[18] J. Moody and M. Saffell, “Learning to trade via direct reinforcement,” IEEE transactions on neural Networks, vol. 12, no. 4, pp. 875–889, 2001.
[19] R. S. Sutton, “Learning to predict by the methods of temporal differences,” Machine learning, vol. 3, no. 1, pp. 9–44, 1988.
[20] C. J. Watkins and P. Dayan, “Q-learning,” Machine learning, vol. 8, no. 3-4, pp. 279–292, 1992.
[21] M. A. Dempster and V. Leemans, “An automated fx trading system using adaptive reinforcement learning,” Expert Systems with Applications, vol. 30, no. 3, pp. 543–552, 2006.
[22] C. Zhang, O. Vinyals, R. Munos, and S. Bengio, “A study on overfitting in deep reinforcement learning,” arXiv preprint arXiv:1804.06893, 2018.
[23] S.-H. Huang, Y.-H. Miao, and Y.-T. Hsiao, “Novel deep reinforcement algorithm with adaptive sampling strategy for continuous portfolio optimization,” IEEE Access, 2021.
[24] I. J. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio, “Generative adversarial networks,” arXiv preprint arXiv:1406.2661, 2014.
[25] A. Kumar, A. Biswas, and S. Sanyal, “ecommercegan: A generative adversarial network for e-commerce,” arXiv preprint arXiv:1801.03244, 2018.
[26] J.-C. Shi, Y. Yu, Q. Da, S.-Y. Chen, and A.-X. Zeng, “Virtual-taobao: Virtualizing real-world online retail environment for reinforcement learning,” in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 33, no. 01, 2019, pp. 4902–4909.
[27] K. Zhang, G. Zhong, J. Dong, S. Wang, and Y. Wang, “Stock market prediction based on generative adversarial network,” Procedia computer science, vol. 147, pp. 400–406, 2019.
[28] F. De Meer Pardo, “Enriching financial datasets with generative adversarial networks,” 2019.
[29] J. Li, X. Wang, Y. Lin, A. Sinha, and M. Wellman, “Generating realistic stock market order streams,” in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 34, no. 01, 2020, pp. 727–734.
[30] A. Koshiyama, N. Firoozye, and P. Treleaven, “Generative adversarial networks for financial trading strategies fine-tuning and combination,” Quantitative Finance, vol. 21, no. 5, pp. 797–813, 2021.
[31] M. Mirza and S. Osindero, “Conditional generative adversarial nets,” arXiv preprint arXiv:1411.1784, 2014.
[32] C.-H. Kuo, C.-T. Chen, S.-J. Lin, and S.-H. Huang, “Improving generalization in reinforcement learning–based trading by using a generative adversarial market model,” IEEE Access, vol. 9, pp. 50 738–50 754, 2021.
[33] M. Arjovsky, S. Chintala, and L. Bottou, “Wasserstein generative adversarial networks,” in International conference on machine learning. PMLR, 2017, pp. 214–223.
連結至畢業學校之論文網頁點我開啟連結
註: 此連結為研究生畢業學校所提供,不一定有電子全文可供下載,若連結有誤,請點選上方之〝勘誤回報〞功能,我們會盡快修正,謝謝!
QRCODE
 
 
 
 
 
                                                                                                                                                                                                                                                                                                                                                                                                               
第一頁 上一頁 下一頁 最後一頁 top
無相關期刊