臺灣博碩士論文加值系統

近年來，隨著雲端技術的快速發展，數據中心一向是評估雲端服務的重要考量之一，其可靠度與妥善率一直是大家關注的焦點。然而，服務中斷是每個數據中心最需要考量與避免的災害事件，輕則影響用戶體驗；重則造成企業巨大損失。因此，若能對資料數據中心進行自動化的故障預防性監測，將可大大提升整體雲端服務的可靠度。預知維修保養有別於傳統例行維護、事後修復的維修方式，在設備運行時，依據其所處狀態與未來的發展趨勢，探知有無狀態異常或故障預兆。
本研究係以預測硬碟故障為例，發展出一「預警監測系統」(PAF-HD)。結合大數據分析與機器學習技術，利用機台故障預診斷技術，從故障預兆的角度為出發點，並結合自我監測、分析及報告技術(SMART) ，用以識別出故障前的早期異常徵兆。最後，藉由隨機森林演算法建構預測模型。本研究旨在發展一套預警監測流程，提供設備之狀態監測與故障診斷，掌握異常徵兆，藉此及早發覺及解決設備異常，讓設備維持在最佳狀態。

In recent years, with the rapid advancement of cloud technology, data centers have always been considered one of cloud services most important aspect to evaluate, and its reliability and availability have been the focus of every IT engineer’s attention. However, service interruption is the most important factor to consider for every data center, affecting the user experience, or causing loss in a business. Therefore, automated fault prevention and monitoring of data center will effectively improve the reliability of cloud services. Predictive maintenance differs from traditional maintenance process (i.e. routine maintenance and corrective maintenance), it evaluates the state by performing device condition monitoring, and according to the state, it predicts when maintenance should be performed.
This research focuses on hard drive failure prediction, with big data analysis and machine learning technology, we have developed a Preventive Monitoring System (PMS). Utilizing Prognostics and Health Management (PHM) to identify the failure mechanism, and combining Self-Monitoring, Analysis and Reporting Technology (SMART) to identify early signs of abnormalities before the device fail. Finally, we use random forest algorithm to construct the predictive model. This research aims to develop a predictive monitoring system to provide device condition monitoring and fault diagnosis, thereby identifying the device malfunction and resolving it as soon as possible, keeping the system maintained at optimal condition.

摘 要 iii
ABSTRACT iv
Contents v
List of Figures 7
List of Tables viii
Chapter 1 Introduction 1
1.1 Background and Motivation 1
1.1.1 Preventive Maintenance in Big Data 2
1.1.2 Cloud Prediction Tool 2
1.1.3 Failure Prediction in Computer Science 3
1.2 Research Objective 5
1.3 Research Overview 5
Chapter 2 Literature Review 6
2.1 Big Data analysis 6
2.1.1 Failure Precursor 6
2.1.2 Preventive Maintenance in Big Data 7
2.2 Self-Monitoring and Reporting Technology (SMART) 8
2.3 Machine Learning 9
2.3.1 Random Forest 11
Chapter 3 Research Methodology 16
3.1 The overview of PAF-HD 16
3.2 Backblaze SMART dataset 18
3.3 The Module 1: ETL 19
3.3.1 Failure Mode (FM) 20
3.3.2 Extraction and Storage (ES) 20
3.4 The Module 2: ML 22
3.4.1 Feature Engineering (FE) 22
3.4.2 Generate Model (GM) 23
3.5 The Module 3: PHM 24
Value Prediction (VP) 26
Warning System (WS) 27
Chapter 4 Implementation 29
4.1 Hardware and Software Configuration 29
4.2 Initiate PAF-HD system 30
4.3 Demonstration of Scenario 32
4.3.1 Scenario 1 - Batch Training 33
4.3.2 Scenario 2 - Real-time Predicting 39
Chapter 5 Concluding remarks and future works 43
5.1 Conclusions 43
5.2 Limitations 44
5.3 Future Works 44

Gu, J., Vichare, N., Tracy, T., & Pocht, M. (2007). Prognostics Implementation Methods for Electronics. Paper presented at the Annual Reliability and Maintainability Symposium.
Covic, V. (2015). Backblaze Reveals Hard Drive Reliability Stats for Q3 2015. Retrieved from http://www.cloudwards.net/news/backblaze-reveals-hard-drive-reliability-stats-for-q3-2015-10905/
Mayer-Schönberger, V., & Cukier, K. (2013). Big data: A revolution that will transform how we live, work, and think: Houghton Mifflin Harcourt.
Edge, M. E., Sampaio, P. R. F., & Choudhary, M. (2007). Towards a proactive fraud management framework for financial data streams. Paper presented at the Dependable, Autonomic and Secure Computing, 2007. DASC 2007. Third IEEE International Symposium on.
Lian, X., Chen, L., Yu, J. X., Han, J., & Ma, J. (2009). Multiscale representations for fast pattern matching in stream time series. Knowledge and Data Engineering, IEEE Transactions on.
Ko, A. H. R., Jousselme, A.-L., & Maupin, P. (2011). A novel measure for data stream anomaly detection in a bio-surveillance system. Paper presented at the Information Fusion (FUSION), 2011 Proceedings of the 14th International Conference on.
Yang, S., Kalpakis, K., & Biem, A. (2014). Detecting road traffic events by coupling multiple timeseries with a nonparametric bayesian method. Intelligent Transportation Systems, IEEE Transactions on.
Hamerly, G., & Elkan, C. (2001). Bayesian approaches to failure prediction for disk drives. Paper presented at the ICML.
Hughes, G. F., Murray, J. F., Kreutz-Delgado, K., & Elkan, C. (2002). Improved disk-drive failure warnings. Reliability, IEEE Transactions on.
Murray, J. F., Hughes, G. F., & Kreutz-Delgado, K. (2003). Hard drive failure prediction using non-parametric statistical methods. Paper presented at the Proceedings of ICANN/ICONIP.
Pinheiro, E., Weber, W.-D., & Barroso, L. A. (2007). Failure Trends in a Large Disk Drive Population.
Agarwal, V., Bhattacharyya, C., Niranjan, T., & Susarla, S. (2009). Discovering rules from disk events for predicting hard drive failures. Paper presented at the Machine Learning and Applications, 2009. ICMLA'09. International Conference on.
Hosch, W. L. (2016). Machine learning. Retrieved from http://global.britannica.com/technology/machine-learning
Samuel, A. L. (1959). Some studies in machine learning using the game of checkers. IBM Journal of research and development, 3(3), 210-229.
Mitchell, T. M. (1997). Machine Learning: McGraw.
Kotsiantis, S. B., Zaharakis, I., & Pintelas, P. (2007). Supervised machine learning: A review of classification techniques.
Breiman, L., & Cutler, A. Random Forests Leo Breiman and Adele Cutler. Retrieved from https://www.stat.berkeley.edu/~breiman/RandomForests/cc_home.htm
Breiman, L. (2001). Random forests. Machine Learning.
Ho, T. K. (1995). Random decision forests. Paper presented at the Document Analysis and Recognition, 1995., Proceedings of the Third International Conference on.
Ho, T. K. (1998). The random subspace method for constructing decision forests. Pattern Analysis and Machine Intelligence, IEEE Transactions on.
Breiman, L. (1996). Bagging predictors. Machine Learning.
Utgoff, P. E. (1989). Incremental Induction of Decision Trees. Machine Learning.
Quinlan, J. R. (1999). Simplifying decision trees. International Journal of Human-Computer Studies.
Yan, W. (2006, 4-6 Oct. 2006). Application of Random Forest to Aircraft Engine Fault Diagnosis. Paper presented at the Computational Engineering in Systems Applications, IMACS Multiconference on.
Vincenzi, S., Zucchetta, M., Franzoi, P., Pellizzato, M., Pranovi, F., De Leo, G. A., & Torricelli, P. (2011). Application of a Random Forest algorithm to predict spatial distribution of the potential yield of Ruditapes philippinarum in the Venice lagoon, Italy. Ecological Modelling.
Li, Z., & He, Q. (2015). Prediction of Railcar Remaining Useful Life by Multiple Data Source Fusion. IEEE Transactions on Intelligent Transportation Systems.
Gambill, D. R., Wall, W. A., Fulton, A. J., & Howard, H. R. (2016). Predicting USCS soil classification from soil property variables using Random Forest. Journal of Terramechanics.
Ng, A. (2015). [Machine Learning and AI via Brain simulations].
Miel, S. (2015). [What is the intuitive explanation of feature engineering in machine learning? - Quora].