本篇文章使用了增強學習與捲積深度學習結合的DQCN模型製作交易系統，希望藉由此交易系統能自行判斷是否買賣ETF，由於ETF屬於穩定性高且手續費高的衍生性金融商品，所以該系統不即時性的做買賣，採用每二十個開盤日進行一次買賣，並由這20個開盤日進行買賣的預測，希望該系統能最大化我們未來的報酬。
DQN是一種增強學習的模型，並在其中使用深度學習進行動作價值的預測，利用增強學習的自我更新動作價值的機制，再用深度學習強大的學習能力成就了人工智慧，並在其取得良好的成效。

In this paper, we used DCQN model, which is combined with reinforcement learning and CNN to train a trading system and hope the trading system could judge whether buy or sell ETFs. Since ETFs is a derivative financial good with high stability and related fee, the system does not perform real-time trading and it performs every 20 trading day. The system predicts value of action based on data in the last 20 opening days to maximize our future rewards.
DQN is a reinforcement learning model, using deep learning to predict value of actions in model. Combined with the RL's mechanism, which updates value of actions, and deep learning, which has a strong ability of learning, to finish an artificial intelligence. We got a perfect effect.

1 Introduction 1
2 Deep Learning 3
2.1 NeuralNetworkandNeuron.......................... 5
2.1.1 ActivationFunction .......................... 7
2.1.2 LossFunction.............................. 9
2.1.3 GradientDescentMethod ....................... 10
2.2 ConvolutionalNeuralNetwork...................... 11
3 Reinforcement Learning 14
3.1 Introduction................................... 15
3.2 Temporal-DifferencePrediction .................. 18
3.3 Q-Learning ................................... 20
3.4 DeepQ-Learning ................................ 21
3.5 Policy-BasedMethod............................. 24
3.6 Actor-Critic................................... 26
3.7 AsynchronousAdvantageActor-Critic(A3C). . . . . . . .. . . . . 28
4 Exchange-Traded Fund 30
4.1 ExchangeTradedFunds ........................... 31
4.2 AdvantageofETF ............................... 32
4.3 ExampleofETF ................................ 33
5 Automated Trading System 35
5.1 ETFdata.................................... 35
5.2 AutomatedTradingSystem ....................... 37
5.2.1 Introduction .............................. 37
5.2.2 Definition................................ 38
5.2.3 InitialParameterSettlement................ 40
5.2.4 NeuralNetwork............................. 41
5.2.5 DCQN.................................. 42
5.3 Result..................................... 43
6 Conclusion.................................. 45
Bibliography.................................. 47

[1] Anastasia Borovykh, Sander Bohte, and Cornelis W Oosterlee. Conditional time se- ries forecasting with convolutional neural networks. arXiv preprint arXiv:1703.04691, 2017.
[2] Guglielmo Maria Caporale, Juncal Cuñado, and Luis A Gil-Alana. Modelling long- run trends and cycles in financial time series data. Journal of Time Series Analysis, 34(3):405–421, 2013.
[3] Thira Chavarnakul and David Enke. Intelligent technical analysis based equivolume charting for stock trading using neural networks. Expert Systems with Applications, 34(2):1004–1017, 2008.
[4] Tim de Bruin, Jens Kober, Karl Tuyls, and Robert Babuška. The importance of experience replay database composition in deep reinforcement learning. In Deep Reinforcement Learning Workshop, NIPS, 2015.
[5] John Cristian Borges Gamboa. Deep learning for time-series analysis. arXiv preprint arXiv:1701.01887, 2017.
[6] Yoon Kim. Convolutional neural networks for sentence classification. arXiv preprint arXiv:1408.5882, 2014.
[7] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. In Advances in neural information processing systems, pages 1097–1105, 2012.

[8] Ramon Lawrence. Using neural networks to forecast stock market prices. University of Manitoba, 1997.
[9] Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. Deep learning. Nature, 521(7553): 436–444, 2015.
[10] Timothy P Lillicrap, Jonathan J Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep rein- forcement learning. arXiv preprint arXiv:1509.02971, 2015.
[11] Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforcement learning. In International Conference on Machine Learning, pages 1928–1937, 2016.
[12] Arun Nair, Praveen Srinivasan, Sam Blackwell, Cagdas Alcicek, Rory Fearon, Alessandro De Maria, Vedavyas Panneershelvam, Mustafa Suleyman, Charles Beat- tie, Stig Petersen, et al. Massively parallel methods for deep reinforcement learning. arXiv preprint arXiv:1507.04296, 2015.
[13] Pierre Sermanet, David Eigen, Xiang Zhang, Michaël Mathieu, Rob Fergus, and Yann LeCun. Overfeat: Integrated recognition, localization and detection using convolutional networks. arXiv preprint arXiv:1312.6229, 2013.
[14] Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction, volume 1. MIT press Cambridge, 1998.
[15] Ziyu Wang, Tom Schaul, Matteo Hessel, Hado Van Hasselt, Marc Lanctot, and Nando De Freitas. Dueling network architectures for deep reinforcement learning. arXiv preprint arXiv:1511.06581, 2015.
[16] Yudong Zhang and Lenan Wu. Stock market prediction of s&p 500 via combination of improved bco approach and bp neural network. Expert systems with applications, 36(5):8849–8854, 2009.