( 您好!臺灣時間:2024/07/19 04:52
字體大小: 字級放大   字級縮小   預設字形  
回查詢結果 :::


研究生(外文):He, Fei-Fan
論文名稱(外文):A Multi-Agent Virtual Market Model for Generalization in Reinforcement Learning Based Trading Strategies
指導教授(外文):Huang, Szu-Hao
口試委員(外文):Tsai, Chwei-ShyongLin, Jui-ChiaChen, An-Pin
外文關鍵詞:Deep learningTrading strategyNeural network for financeReinforcement learning
  • 被引用被引用:0
  • 點閱點閱:203
  • 評分評分:
  • 下載下載:30
  • 收藏至我的研究室書目清單書目收藏:0



Many studies have successfully used reinforcement learning (RL) to train an intelligent agent that learns profitable trading strategies from financial market data. However, most of these studies have simplified the effect of the actions of the trading agent on the market state. Trading actions do not affect the market environment reproduced in RL. The trading agent is trained to maximize long-term profit by optimizing fixed historical data. Such approach frequently results in the trading performance during out-of-sample validation being considerably different from that during training. In this paper, we propose a multi-agent virtual market model (MVMM) comprised of multiple generative adversarial networks (GANs) which cooperate with each other to reproduce market price changes. Due to the advantages of GANs, the MVMM can learn the distribution of real market prices and generate numerous simulated market trends. In addition, the action of the trading agent can be superimposed on the current market state as the input of the MVMM to generate an action-dependent next state. In this research, real historical data were replaced with the simulated market data generated by the MGVMM. The experimental results indicated that the RL agent is appropriately generalized by the aforementioned methods. The trading strategy of the trained RL agent achieved high profits and exhibited low risk of loss during the testing phase.
中文摘要 i
英文摘要 ii
Acknowledgement iii
Table of Contents iv
List of Figures vi
List of Tables vii
1 Introduction 1
1.1 Background 1
1.2 Motivation 2
1.3 Research Goal and Hypothesis 4
1.4 Contributions and Value 5
1.5 Organization 5
2 Related Works 7
2.1 Deep learning in Finance 7
2.2 Reinforcement Learning in Finance 8
2.3 Generative Adversarial Networks in Finance 11
3 Proposed Method 13
3.1 System Overview 13
3.2 RL in Financial Trading 14
3.3 Local GANs 16
3.4 Global Controller and Global Discriminator 19
3.5 DRNN Agent 22
3.6 Proposed RL Trading Framework 24
4 Experiments 28
4.1 Experimental Settings 28
4.2 Evaluation of VM Simulation 31
4.3 Profitability Evaluation for the IF Dataset 36
4.4 Evaluation of Profitability for the AAPL Dataset 39
4.5 Comparison of the VM Models with Different Numbers of Local GANs 40
4.6 Evaluation of Profitability for Other US Stocks 43
5 Conclusion 46
References 48
[1] D. Shah, H. Isah, and F. Zulkernine, “Stock market analysis: A review and taxonomy of prediction techniques,” International Journal of Financial Studies, vol. 7, no. 2, p. 26, 2019.
[2] L. Zhang, C. Aggarwal, and G.-J. Qi, “Stock price prediction via discovering multi-frequency trading patterns,” in Proceedings of the 23rd ACM SIGKDD international conference on knowledge discovery and data mining, 2017, pp. 2141–2149.
[3] Y. Deng, F. Bao, Y. Kong, Z. Ren, and Q. Dai, “Deep direct reinforcement learning for
financial signal representation and trading,” IEEE transactions on neural networks and learning systems, vol. 28, no. 3, pp. 653–664, 2016.
[4] A. Tsantekidis, N. Passalis, A.-S. Toufa, K. Saitas-Zarkias, S. Chairistanidis, and A. Tefas, “Price trailing for financial trading using deep reinforcement learning,” IEEE Transactions on Neural Networks and Learning Systems, 2020.
[5] Z. Jiang, D. Xu, and J. Liang, “A deep reinforcement learning framework for the financial portfolio management problem,” arXiv preprint arXiv:1706.10059, 2017.
[6] T. Spooner, J. Fearnley, R. Savani, and A. Koukorinis, “Market making via reinforcement learning,” arXiv preprint arXiv:1804.04216, 2018.
[7] A. Charpentier, R. Elie, and C. Remlinger, “Reinforcement learning in economics and finance,” Computational Economics, pp. 1–38, 2021.
[8] Y. Ye, H. Pei, B. Wang, P.-Y. Chen, Y. Zhu, J. Xiao, and B. Li, “Reinforcement-learning based portfolio management with augmented asset movement prediction states,” in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 34, no. 01, 2020, pp. 1112–1119.
[9] Z. Liang, H. Chen, J. Zhu, K. Jiang, and Y. Li, “Adversarial deep reinforcement learning in portfolio management,” arXiv preprint arXiv:1808.09940, 2018.
[10] E. Chong, C. Han, and F. C. Park, “Deep learning networks for stock market analysis and prediction: Methodology, data representations, and case studies,” Expert Systems with Applications, vol. 83, pp. 187–205, 2017.
[11] F. Feng, H. Chen, X. He, J. Ding, M. Sun, and T.-S. Chua, “Enhancing stock movement prediction with adversarial training,” arXiv preprint arXiv:1810.09936, 2018.
[12] W. Li, R. Bao, K. Harimoto, D. Chen, J. Xu, and Q. Su, “Modeling the stock relation with graph network for overnight stock movement prediction,” in no. CONF, 2020, pp. 4541–4547.
[13] R. S. Sutton and A. G. Barto, Reinforcement learning: An introduction. MIT press, 2018.
[14] A. Tsantekidis, N. Passalis, and A. Tefas, “Improving deep reinforcement learning for financial trading using neural network distillation,” in 2020 IEEE 30th International Workshop on Machine Learning for Signal Processing (MLSP). IEEE, 2020, pp. 1–6.
[15] K. Lei, B. Zhang, Y. Li, M. Yang, and Y. Shen, “Time-driven feature-aware jointly deep reinforcement learning for financial signal representation and algorithmic trading,” Expert Systems with Applications, vol. 140, p. 112872, 2020.
[16] Y. Liu, Q. Liu, H. Zhao, Z. Pan, and C. Liu, “Adaptive quantitative trading: an imitative deep reinforcement learning approach,” in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 34, no. 02, 2020, pp. 2128–2135.
[17] J. Wang, Y. Zhang, K. Tang, J. Wu, and Z. Xiong, “Alphastock: A buying-winners-and-selling-losers investment strategy using interpretable deep reinforcement attention networks,” in Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, 2019, pp. 1900–1908.
[18] J. Moody and M. Saffell, “Learning to trade via direct reinforcement,” IEEE transactions on neural Networks, vol. 12, no. 4, pp. 875–889, 2001.
[19] R. S. Sutton, “Learning to predict by the methods of temporal differences,” Machine learning, vol. 3, no. 1, pp. 9–44, 1988.
[20] C. J. Watkins and P. Dayan, “Q-learning,” Machine learning, vol. 8, no. 3-4, pp. 279–292, 1992.
[21] M. A. Dempster and V. Leemans, “An automated fx trading system using adaptive reinforcement learning,” Expert Systems with Applications, vol. 30, no. 3, pp. 543–552, 2006.
[22] C. Zhang, O. Vinyals, R. Munos, and S. Bengio, “A study on overfitting in deep reinforcement learning,” arXiv preprint arXiv:1804.06893, 2018.
[23] S.-H. Huang, Y.-H. Miao, and Y.-T. Hsiao, “Novel deep reinforcement algorithm with adaptive sampling strategy for continuous portfolio optimization,” IEEE Access, 2021.
[24] I. J. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio, “Generative adversarial networks,” arXiv preprint arXiv:1406.2661, 2014.
[25] A. Kumar, A. Biswas, and S. Sanyal, “ecommercegan: A generative adversarial network for e-commerce,” arXiv preprint arXiv:1801.03244, 2018.
[26] J.-C. Shi, Y. Yu, Q. Da, S.-Y. Chen, and A.-X. Zeng, “Virtual-taobao: Virtualizing real-world online retail environment for reinforcement learning,” in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 33, no. 01, 2019, pp. 4902–4909.
[27] K. Zhang, G. Zhong, J. Dong, S. Wang, and Y. Wang, “Stock market prediction based on generative adversarial network,” Procedia computer science, vol. 147, pp. 400–406, 2019.
[28] F. De Meer Pardo, “Enriching financial datasets with generative adversarial networks,” 2019.
[29] J. Li, X. Wang, Y. Lin, A. Sinha, and M. Wellman, “Generating realistic stock market order streams,” in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 34, no. 01, 2020, pp. 727–734.
[30] A. Koshiyama, N. Firoozye, and P. Treleaven, “Generative adversarial networks for financial trading strategies fine-tuning and combination,” Quantitative Finance, vol. 21, no. 5, pp. 797–813, 2021.
[31] M. Mirza and S. Osindero, “Conditional generative adversarial nets,” arXiv preprint arXiv:1411.1784, 2014.
[32] C.-H. Kuo, C.-T. Chen, S.-J. Lin, and S.-H. Huang, “Improving generalization in reinforcement learning–based trading by using a generative adversarial market model,” IEEE Access, vol. 9, pp. 50 738–50 754, 2021.
[33] M. Arjovsky, S. Chintala, and L. Bottou, “Wasserstein generative adversarial networks,” in International conference on machine learning. PMLR, 2017, pp. 214–223.
註: 此連結為研究生畢業學校所提供,不一定有電子全文可供下載,若連結有誤,請點選上方之〝勘誤回報〞功能,我們會盡快修正,謝謝!
第一頁 上一頁 下一頁 最後一頁 top