臺灣博碩士論文加值系統

本論文提出了Green MapReduce System(GMS)的系統架構，主要是在解決Load Balance負載平衡以及Power Saving節能的部分。下列提出三點改進方法：一、提出Green Master架構。二、為群組內的伺服器加上Benchmark Score評分值。三、提出演算法關於如何區別高分數的伺服器以及低分數的伺服器，以及如何最有效率的運用運算資源而不造成浪費。
本論文是基於Hadoop的MapReduce系統做出改進，Hadoop是一套由Google MapReduce系統演化出來的的開放性軟體，可以讓使用者利用安裝此軟體而互相建立連線，獲得大量運算資源，之後再透過撰寫Map Function以及Reduce Function決定運算的目的以及執行方法。但是通常為了獲得群組內最大的運算資源，往往會使群組內的所有伺服器始終處於開機狀態或是保持在高速運轉的狀態，這在無形中會造成不必要的浪費。例如：電腦性能的好壞不一，運算速度自然不同，若是分配相同的工作量給所有的運算伺服器勢必會造成某些伺服器的工作提早結束，但是還要等待其他運算速度較差的伺服器完成工作，這段時間就會空轉造成資源上的浪費；又例如：群組伺服器性能相近，但是隨著時間的推移，速度較慢的伺服器的工作會慢慢累積，又會造成上述例子的情況再一次發生。

MapReduce is a kind of distributed computing system, and also many people use it nowadays. In this paper, the Green Master based on MapReduce is proposed to solve the problem between load balance and power saving. There are three mechanism proposed by this paper to improve the MapReduce system efficiency. First, a brand new architecture called Green Master is designed in the system. Second, Benchmark Score is added to each service in the cluster. In the last, an algorithm about how to distinguish the high score service and the low score service, and explain how to use them effectively.
The algorithm in this paper will be used to improve the system efficiency based on MapReduce of Hadoop. Hadoop is a kind of open source software that develop from Google MapReduce, and it can will create a cluster that connects each services. The cluster is used to make more computing resources called computing pool, and it can be expanded more and more. In the end, we can decide what we want to get or how to execute the program through coding the Map Function and Reduce Function. As usual, in order to make the maximum computing resources, the services must keep the high-speed state, but it also has a lot of unnecessary waste. For example, service performance are not the same, some of them are very high, but some of them are very low. if we allocate the same amount of work to all service, it must cause a part of service will complete the work early, but it still have to wait other service that performance is poor, and the waiting time means resources wastes. We will talk about how to make the service off if the performance is too low that seriously affects the system performance.

目錄
第一章 緒論 1
1.1 研究動機 2
1.2 論文架構 4
第二章 相關研究 6
2.1 Virtual Machine 6
2.2 Google File System 7
2.3 MapReduce 9
2.4 Hadoop 11
2.5 Hadoop MapReduce 13
2.5.1 HDFS 15
2.5.2 NameNode 16
2.5.3 DataNode 16
2.5.4 JobNode [JobTracker] 17
2.5.5 TaskNodep [TaskTracker] 17
2.6 Benchmark 18
2.7 相關論文研究 19
第三章 Green Master of MapReduce 24
3.1 Green Master System 25
3.1.1 Input File Index 27
3.1.2 Queue 27
3.1.3 Server Information & Benchmark 28
3.1.4 Record 30
3.1.5 Load Balance Optimization 30
3.1.6 Power Saving Algorithm 31
3.1.7 Decision 33
3.2 節能計算公式 34
3.3 系統流程圖 35
第四章 模擬結果以及效能分析 37
4.1. 實驗環境 37
4.2. Green Master Simulation Result 39
4.3. 系統效能 42
4.4. 系統泛用性 45
第五章 結論以及未來展望 51
參考文獻 (References) 52

圖目錄
圖2.1、虛擬化實現方式 7
圖2.2、MapReduce簡易結構圖 10
圖 2.3、Key/Value示意圖 11
圖2.4、MapReduce比較圖 12
圖2.5、Hadoop環境建置 14
圖2.6、主從式關係圖 17
圖2.7、主從式關係圖 18
圖3.1、雲端管理示意圖 24
圖3.2、Green Master Architecture 25
圖3.3、Record Table 29
圖3.4、系統流程圖 35
圖4.1、Virtual Machine數目以及BS初始設定值 37
圖4.2、VM1的環境設定值 38
圖4.3、系統消耗以及系統時間比較圖 39
圖4.4、System Efficiency Consumption 40
圖4.5、差值比比較圖 41
圖4.6、系統運算時間 42
圖4.7、系統耗能比較圖 43
圖4.8、系統節能比率 43
圖4.9、伺服器性能分布為常態分布(高斯分布) 45
圖4.12、伺服器性能分布於極值低者多 47
圖4.13、常態分布(高斯分布)使用率 48
圖4.14、分布於極值高者多使用率 48
圖4.15、分布於極值低者多使用率 49
圖4.16、分布於兩極者使用率 49
圖4.17、節能比率 50

[1]URL: http://trac.nchc.org.tw/cloud/wiki/NCHCCloudCourse110718
[2]URL: http://www.microsoft.com/zh-tw/default.aspx
[3]URL: http://www.google.com/intl/zh-TW/drive/features.html
[4]URL: http://www.amazon.com/
[5]Jeffrey Dean and Sanjay Ghemawat, “MapReduce: Simpli_ed Data Processing on Large Clusters,” OSDO 2004.
[6]Ling-Shang Kuo, “MapReduce-based Image Processing System with Priority-based DSRF Algorithm,”
[7]陳信宇, “雲端計算中心力用DVFS技術之工作排程節能演算法,2011.
[8]Yifeng Sun, “Fast Live Cloning of Virtual Machine based on Xen,” 2009 11 th IEEE International Conference on High Performance Computing and Communication.
[9]Ling-Shang Kuo, “MapReduce-based Image Processing System with Priority-based DSRF Algorithm,” 淡江大學, 2012.
[10]Xia Xie, Qingcha Chen, Wenzhi Cao, Pingpeng Yuan, Hai Jin, “Benchmark Object for Virtual Machines,” 2010 Second International Workshop on Education Technology and Computer Science.
[11]Yury Audzevich, “GreenCloud: A Packet-level Simulator of Energy-aware Cloud Computing Data Centers,” This full text paper was peer reviewed at the direction of IEEE Communications Society subject matter experts for publication in the IEEE Globecom 2010 proceedings.
[12]Jiong Xie, Shu Yin, Xiaojun Ruan, Zhiyang Ding, Yun Tian,, “Improving MapReduce Performance through Data Placement in Heterogeneous Hadoop Cluster,” 978-1-4244-6534-7/10/$26.00 c2010 IEEE.
[13]Sandhya S V, Sanjay H A*, Netravathi S J, Sowmyashree M V, Yogeshwari R N, “Fault–Tolerant Master-Workers framework for MapReduce Application,” 2009 International Conference on Advances in Recent Technologies in Communication and Computing.
[14]王亭穎, “基於統計方法之雲端運算虛擬機器節能方法,”國立高雄應用科技大學,2012
[15]鄭瑞明, “在有限資源的雲端下依機器性能優化其虛擬機器的配置,”國立東華大學資訊工程系,2012
[16]王柏翔, “雲端運算下的節能負載平衡,”國立交通大學,2011
[17]Tao Zhu, “Green Scheduling: A Scheduling Policy for Improving the Energy Efficiency of Fair Scheduler,” 2011 12th International Conference on Parallel and Distributed Computing, Applications and Technologies.
[18]Lin-Shang Kuo, “MapReduce-based Image Processing System with Priority-based DSRF Algorithm,”
[19]Yi Zhao, Wenlong Huang, “Adaptive Distributed Load Balancing Algorithm based on Live Migration of Virtual Machines in Cloud,” 2009 Fifth International Joint Conference on INC, IMS and IDC.