臺灣博碩士論文加值系統

Hadoop 是工業界和學術界最流行的雲端系統之一。其預設的檔案系統，
Hadoop Distributed File System(HDFS)，透過將資料儲存多個備份在系統中以維持
資料的可用性。但是，萬一失去整個 Hadoop 叢集時，保留多個資料的備份仍然
不足以確保資料的可用性。因此，為了應對 Hadoop 叢集在極端情況下完全毀損，
Hadoop 提供了一個工具，distcp，用於在 Hadoop 叢集之間複製檔案。然而，在任
何 Hadoop 叢集上修改重複資料時，從兩個 Hadoop 叢集之間複製的重複資料可能
會產生不一致，這可能會顯著降低 Hadoop 叢集之間重複資料的可靠性。然而，
Hadoop 並沒有提供任何有效的工具來解決資料可靠性的問題。我們設計並實作了
一種新的方案，幫助使用者輕鬆辨識 Hadoop 叢集之間重複資料的不一致。因此
可以維持 Hadoop 叢集之間備份資料的可靠性。
關鍵字: Hadoop、HDFS、Replica、Snapshot、Distcp、雜湊值

The Apache Hadoop is one of the most popular cloud systems in both industry and
academia. Its default file system, Hadoop Distributed File System (HDFS) maintains the availability of data through keeping multiple replicas of data in the system. However, keeping multiple copies of data still cannot assure the data availability in the case
of a total loss of an Hadoop cluster. As a result, Hadoop provides a tool, distcp, to copy
files between Hadoop clusters to deal with the extreme situation when an Hadoop cluster is completely destroyed. However, the discrepancy between data copied between two
Hadoop clusters could occur when duplicated data is modified on any of the Hadoop clusters, which could significantly reduce the reliability of duplicated data between Hadoop
clusters. Unfortunately, Hadoop does not provide any efficient tool to address the issue
of the data reliability. We designed and implemented a new scheme to help users easily
identify the discrepancy of duplicated data between Hadoop clusters. Consequently, the
reliability of data duplicated between Hadoop clusters could be maintained.
keywords: Hadoop、HDFS、Replica、Snapshot、Distcp、Checksum

1 導論 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
1.1 研究動機 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
1.2 研究目的 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
1.3 論文結構 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
2 背景知識與相關研究 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
2.1 Hadoop 基本架構 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
2.1.1 Hadoop 讀取流程 . . . . . . . . . . . . . . . . . . . . . . . . . 5
2.1.2 Hadoop 寫入流程 . . . . . . . . . . . . . . . . . . . . . . . . . 6
2.2 HDFS Shell 指令 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
2.2.1 Ls 指令運作 . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
2.2.2 Checksum 指令運作 . . . . . . . . . . . . . . . . . . . . . . . . 7
2.3 Snapshot . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
2.4 Distcp 介紹 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
2.5 雜湊函數與雜湊值介紹 . . . . . . . . . . . . . . . . . . . . . . . . . . 9
3 設計與方法實作 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10
3.1 新增 diff 指令 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
3.2 修改 FSCommand 類別 . . . . . . . . . . . . . . . . . . . . . . . . . . 12
4 實驗設計與結果分析 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13
4.1 實驗設計 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13
4.2 實驗環境 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
4.3 實驗一: 一個目標檔案內容不一致 . . . . . . . . . . . . . . . . . . . . 18
I
目錄 目錄
4.4 實驗二: 七個最大目標檔案內容不一致 . . . . . . . . . . . . . . . . . 34
4.5 實驗三: 七個最小目標檔案內容不一致 . . . . . . . . . . . . . . . . . 35
4.6 實驗四: 七個最大目標檔案和七個最小目標檔案內容不一致 . . . . . 36
4.7 實驗五: 直接使用 diff 與 checksum 計算雜湊值 . . . . . . . . . . . . . 37
5 結論與未來展望 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55

[1]“Apache hadoop. ＂ https://hadoop.apache.org/.
[2]“Spark. ＂ https://spark.apache.org/.
[3]“OSCL. ＂ http://140.136.149.249/lxr/source/.
[4] S.Agarwal, D,Borthakur, I.Stoica,“Snapshots in hadoop distributed file system. ＂
Technical report (2011).
[5] K.Qu, L.Meng, Y.Yang,“A dynamic replica strategy based on Markov model for
hadoop distributed file system (HDFS). ＂ 2016 4th International Conference on
Cloud Computing and Intelligence Systems (CCIS). IEEE, 2016.
[6] J.Dean, S.Ghemawat,“MapReduce: Simplified data processing on large clusters. ＂
(2004).
[7] S.Ghemawat, H.Gobioff, S.Leung,“The Google file system. ＂ Proceedings of the
nineteenth ACM symposium on Operating systems principles. 2003.
[8] P.Hunt, M.Konar, F.P.Junqueira and B.Reed,“ZooKeeper: Wait-free Coordination for Internet-scale Systems. ＂ 2010 USENIX Annual Technical Conference
(USENIX ATC 10). 2010.
57
參考文獻 參考文獻
[9] A.Oriani, I.C.Garcia,“From backup to hot standby: High availability for hdfs. ＂
2012 IEEE 31st Symposium on Reliable Distributed Systems. IEEE, 2012.
[10] R.Rivest, S.Dusse,“The MD5 message-digest algorithm. ＂ (1992): 330-344.
[11] K.Shvachko, H.Kuang, S.Radia, R.Chansler,“The hadoop distributed file system.
＂ 2010 IEEE 26th symposium on mass storage systems and technologies (MSST).
Ieee, 2010.
[12] F.Wang, J.Qiu, J.Yang, B.Dong, X.Li, Y.Li,“Hadoop high availability through metadata replication. ＂ Proceedings of the first international workshop on Cloud data
management. 2009.
[13] T.White,“Hadoop: The definitive guide. ＂ O＇Reilly Media, Inc.＂, 2012.
[14] T.Yeh, H.Lee,“Enhancing availability and reliability of cloud data through syncopy.
＂ 2014 IEEE International Conference on Internet of Things (iThings), and IEEE
Green Computing and Communications (GreenCom) and IEEE Cyber, Physical and
Social Computing (CPSCom). IEEE, 2014.
[15] T.Yeh, Y.Wang,“Enhancing Hadoop System Dependability Through Autonomous
Snapshot. ＂ 2018 IEEE 16th Intl Conf on Dependable, Autonomic and Secure Computing, 16th Intl Conf on Pervasive Intelligence and Computing, 4th Intl Conf on
Big Data Intelligence and Computing and Cyber Science and Technology Congress
(DASC/PiCom/DataCom/CyberSciTech). IEEE, 2018.