(3.235.108.188) 您好!臺灣時間:2021/02/26 18:59
字體大小: 字級放大   字級縮小   預設字形  
回查詢結果

詳目顯示:::

我願授權國圖
: 
twitterline
研究生:陳紀甌
研究生(外文):Chi-Ou Chen
論文名稱:以機器學習改善Hadoop系統優化
論文名稱(外文):Configuration Tuning on Hadoop System Based on Machine Learning
指導教授:蘇雅韻
指導教授(外文):Ya-Yunn Su
口試委員:廖世偉林守德
口試委員(外文):Shih-Wei LiaoShou-De Lin
口試日期:2014-07-17
學位類別:碩士
校院名稱:國立臺灣大學
系所名稱:資訊工程學研究所
學門:工程學門
學類:電資工程學類
論文種類:學術論文
論文出版年:2014
畢業學年度:102
語文別:英文
論文頁數:28
中文關鍵詞:巨量資料分散式系統機器學習全局優化隨機抽樣
外文關鍵詞:big datadistributed systemmachine learningglobal optimizationrandom sampling
相關次數:
  • 被引用被引用:0
  • 點閱點閱:251
  • 評分評分:系統版面圖檔系統版面圖檔系統版面圖檔系統版面圖檔系統版面圖檔
  • 下載下載:0
  • 收藏至我的研究室書目清單書目收藏:2
隨著巨量資料分析的興起, 支持此類大規模資料處理的系統, 如分散
式系統也越受到關注, 在管理建立在日益龐大機器叢集的系統, 系統管
理者必須花更多心力管理。除了使系統能夠穩定地支援各式各樣的資
料分析應用, 也需要對系統作優化, 讓效能夠有效的提昇, 提高系統的使
率及降低運行這些資料分析應用的時間。然而, 對大規模機器叢集而
言, 系統參數調校是複雜的, 管理者除了要處理各個機器之間互動的問
題, 也必須針對不同應用, 了解其運算特性, 進而調校系統參數。而現行
系統參數調校的方法有可用性不高, 以及可調校的參數受到限制等缺
點。本研究基於這些現行的的方法, 以機器學習來改善上述的這些問
題, 打破這些限制使系統效能更進一步提昇


Big Data has emerged in recent year. Systems which is able to support such large-scale data analysis are received more attentions. The distributed system like Hadoop is most used for the analysis. However, it will be increasingly difficult for system administrators to manage the whole system when the cluster of the system scales out. System administrator should maintain the system to execute applications stably. Besides, they need to optimize the system to improve the performance, increase the system utilization and reduce the latency of application executing. And the configuration problem is the most important issue of system optimization. Configuration parameter tuning is related lots of complicated issues. It needs to understand the interaction between physical machines and the behavior of each applications. The current method, rule-based and cost-based optimization, have drawbacks like unfeasibility and limitation of configuration parameter space. Our work exploit machine learning to solve the problem to improve the performance.


摘要 i
Abstract ii
1 Introduction 1
1.1 Misconfiguration in Hadoop . . . . . . . . . . . . . . . . . . . . . . . . 3
1.2 Contribution . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
2 Configuration Tuning 5
2.1 Rule-based Optimization in Hadoop:Vaidya . . . . . . . . . . . . . . . . 5
2.2 Cost-based Optimization . . . . . . . . . . . . . . . . . . . . . . . . . . 6
2.2.1 Limitation of Configuration Space . . . . . . . . . . . . . . . . . 7
2.2.2 Limitation of Portability . . . . . . . . . . . . . . . . . . . . . . 10
3 Design Concept 11
3.1 Configuration Parameters Space . . . . . . . . . . . . . . . . . . . . . . 11
3.2 Machine Learning-Based Predictor . . . . . . . . . . . . . . . . . . . . . 12
3.3 RRS Optimizer . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14
4 Implementation 18
4.1 ML Predictor and RRS Optimizer . . . . . . . . . . . . . . . . . . . . . 18
5 Evaluation 20
5.1 Importance of Configuration Parameters . . . . . . . . . . . . . . . . . . 21
5.2 Accuracy of ML Predictor . . . . . . . . . . . . . . . . . . . . . . . . . 23
5.3 Improvement from Machine Learning-based Optimization . . . . . . . . 23
6 Conclusion and Future Work 26
Bibliography 27


[1] Apache hadoop. http://hadoop.apache.org/.
[2] Apache hadoop rumen. http://hadoop.apache.org/docs/r1.2.1/rumen.html.
[3] scikit-learn: machine learning in python. http://scikit-learn.org/stable/.
[4] Planning guide:getting started with big data. Intel IT Center, January 2013.
[5] S. Babu. Towards automatic optimization of mapreduce programs. In SoCC, 2010.
[6] J. Dean and S. Ghemawat. Mapreduce: simplified data processing on large clusters.
In Communications of the ACM, 2008.
[7] H. Herodotos and S. Babu. Profiling, what-if analysis, and cost-based optimization
of mapreduce programs. In Proc. of the VLDB Endowment, 2011.
[8] H. Herodotou. Hadoop performance models. In Technical Report CS-2011-05, Duke
University, 2011.
[9] H. Herodotou, H. Lim, G. Luo, N. Borisov, L. Dong, F. B. Cetin, and S. Babu.Starfish: A self-tuning system for big data analytics. In 5th Conference on Innovative Data Systems Research, 2011
[10] S. Huang. The hibench benchmark suite: Characterization of the mapreduce-based
data analysis. In Data Engineering Workshops (ICDEW), 2010 IEEE 26th International Conference on, 2010.
[11] L. Jimmy and C. Dyer. Data-intensive text processing with mapreduce. In Synthesis
Lectures on Human Language Technologies, 2010.
[12] O. O’Malley. Terabyte sort on apache hadoop. In Yahoo, available online at:
http://sortbenchmark. org/Yahoo-Hadoop. pdf, 2008.
[13] A. Rabkin and R. Katz. How hadoop clusters break. In Software, IEEE 30.4, 2013.
[14] C. Shalizi. Lecture 10: Regression trees. http://www.stat.cmu.edu/ cshalizi/350-
2006/lecture-10.pdf, October 2006.
[15] T. Ye, H. T. Kaur, and S. Kalyanaraman. A recursive random search algorithm for
large-scale network parameter con&;#64257;guration. In ACM SIGMETRICS, 2003.

QRCODE
 
 
 
 
 
                                                                                                                                                                                                                                                                                                                                                                                                               
第一頁 上一頁 下一頁 最後一頁 top
無相關期刊
 
系統版面圖檔 系統版面圖檔