臺灣博碩士論文加值系統

在IoT(Internet of Things)時代無線通訊系統將涵蓋 5G 與低功耗廣域網路，5G 具有高行動速率、低遲延性、超高速通訊速率、高佈建密度、大量裝置連結、防災等，隨著時代的變遷，資訊領域與網路的蓬勃發展，在行動裝置的快速發展與社交網路的崛起，網路品質在生活上亦發重要，以往電信網路的維運狀況，都是要等設備出現問題後，再聯絡工程師到現場處理，增加客戶抱怨也會增加工程師的工作量，現在可以透過大量數據資料的統計預測發生問題與時間，進而改善預防網路災害的發生。

本研究使用HDFS(Hadoop Distributed File System)架構設計，採用分散式檔案系統，對資料進行管理與控制，減少因設備發生故障時造成的資料遺失問題，以及使用更加高級別的密碼學方法促進數據安全，使用隨機森林演算法 (Random Forest Algorithm)判斷設備給予的資料，預測設備的平均故障間隔(Mean Time Between Failures) ，提前做好預防，測試結果準確率達75%，可以提升網路的品質，減少客戶抱怨及工程師的工作量。

關鍵詞：HDFS、隨機森林、MTBF、物聯網。

In the era of IoT (Internet of Things) wireless communication system will cover 5G and low-power wide-area network, 5G has high action rate, low latency, ultra-high-speed communication speed, high built-in density, a large number of device connections, disaster prevention, etc. The changes, the information field and the rapid development of the Internet, the rapid development of mobile devices and the rise of social networks, the quality of the Internet is also important in life. In the past, the maintenance of telecommunications networks was waiting for devices to appear. After the problem, contact the engineer to the site to deal with it. Increasing customer complaints will also increase the workload of the engineers. Now it is possible to predict the occurrence of problems and time through statistical data of a large number of data, thereby improving the prevention of network disasters.

This study uses HDFS (Hadoop Distributed File System) architecture design, uses a distributed file system to manage and control data, reduces data loss caused by equipment failure, and promotes data using a higher level of cryptography. Security, use the Random Forest Algorithm to judge the data given by the equipment, predict the average fault interval of the equipment (mean time between failures), prevent it in advance, and the accuracy of the test results is 75%, which can improve the quality of the network. Reduce customer complaints and the workload of engineers.

Keywords：HDFS、Random Forest Algorithm、Mean Time Between. Failures、Internet of Things.

中文摘要 ----------------------------------------------------------------- i
英文摘要 ----------------------------------------------------------------- ii
誌謝 ----------------------------------------------------------------- iv
目錄 ----------------------------------------------------------------- v
表目錄 ----------------------------------------------------------------- ix
圖目錄 ----------------------------------------------------------------- x
第一章 緒論 ---------------------------------------------------------- 1
1.1 引言 ---------------------------------------------------------- 1
1.2 研究動機 ---------------------------------------------------- 2
1.3 章節說明 ---------------------------------------------------- 6
第二章 文獻探討 ---------------------------------------------------- 7
2.1 Alcatel-Lucent 1660 SM緣起與發起--------------------
----------------------------- 7
2.1.1 Alcatel-Lucent 1660 SM 運作原理 ------------------ 6
2.2 Alcatel-Lucent 1660 SM 硬體介紹 ----------------- 8
2.2.1 Alcatel-Lucent P63E1與A21E1硬體 ---------------- 10
2.2.2 Alcatel-Lucent P3T3與A3T3硬體 --------------------- 13
2.2.3 Alcatel-Lucent P4S1N與A2S1硬體------------------- 16
2.2.4 Alcatel-Lucent S-4.1N硬體------------------------------- 19
2.2.5 Alcatel-Lucent L-16.1N硬體------------------------------ 20
2.2.6 Alcatel-Lucent EQUICO硬體----------------------------- 21
2.2.7 Alcatel-Lucent MATRIX硬體 --------------------------- 22
2.2.8 Alcatel-Lucent CONGI硬體------------------------------- 23
2.2.9 Alcatel-Lucent SERVIC硬體----------------------------- 24
2.2.10 Alcatel-Lucent Fan Unit 硬體---------------------------- 25
2.2.11 Alcatel-Lucent EPS保護方式---------------------------- 26
2.3 Apache Hadoop 作業系統-------------------------------- 27
2.4 Apache Hadoop 緣起與發展----------------------------- 27
2.5 Apache Hadoop 運作原理-------------------------------- 28
2.6 Apache Hadoop 操作指令-------------------------------- 30
2.7 Apache Hadoop 相關組件介紹-------------------------- 34
2.8 Apache Hadoop HDFS------------------------------------- 36
2.9 Apache Hadoop Yarn -------------------------------------- 38
2.10 Apache Hadoop Hive -------------------------------------- 40
2.11 Apache Hadoop HBase ----------------------------------- 42
2.11.1 Apache Hadoop HBase系統架構------------------------ 42
2.12 Apache Hadoop MapReduce ------------------------------ 44
2.13 Apache Hadoop Spark-------------------------------------- 50
2.14 Apache Hadoop Spark與Apache Hadoop差異------- 53
2.15 Random Forest 隨機森林概念--------------------------- 55
2.15.1 訊息、熵以及訊息增益的概念-------------------------- 59
2.15.2 Random Forest 隨機森林演算法特徵------------------ 59
2.16 開發軟體系統說明----------------------------------------- 61
2.16.1 VirtualBox軟體介紹--------------------------------------- 61
2.17 Ubuntu作業系統介紹------------------------------------- 61
第三章 Apache Hadoop系統規劃設計--------------------------- 63
3.1 系統架構----------------------------------------------------- 63
3.2 系統硬體----------------------------------------------------- 63
3.3 軟體執行程序-----------------------------------------------
----------------------------------- 65
第四章 系統整合與測試------------------------------------------- 78
4.1 系統整合-------------------------------------------------- 78
4.2 軟硬體架設------------------------------------------ 78
4.3 系統測試--------------------------------------------- 79
第五章 結果與討論-------------------------------------------------- 83
參考文獻 ----------------------------------------------------------------- 87
公開發表 ----------------------------------------------------------------- 91

表2-1 P63E1與A21E1卡板對應關係-------------------------------- 12
表2-2 P3E3與A3E3卡板對應關係---------------------------------- 15
表2-3 P4S1N與A2S1卡板對應關係-------------------------------- 18
表2-4 HDFS指令--------------------------------------------------------- 30
表2-4-1 HDFS指令--------------------------------------------------------- 31
表2-4-2 HDFS指令--------------------------------------------------------- 32
表2-4-3 HDFS指令--------------------------------------------------------- 33
表2-4-4 HDFS指令--------------------------------------------------------- 33
表2-5 Apache Hadoop 組件服務-------------------------------------- 34
表2-6 Hive與傳統的關係數據庫主要區別------------------------- 40
表2-7 HBase中的表特徵---------------------------------------------- 42
表2-8 Hadoop生態架構表---------------------------------------------- 54
表2-9 Spark生態架構表------------------------------------------------ 55
表2-10 隨機森林演算法優點-------------------------------------------- 60
表2-11 隨機森林演算法特徵-------------------------------------------- 60
表4-1 系統模擬軟硬體需求-------------------------------------------- 78

圖1-1 中華電信與多個廠商推動5G-------------------------------- 1
圖1-2 全球人才大數據------------------------------------------------- 2
圖1-3 PC1數據集上分類器使用不同特徵選擇的準確性------- 4
圖1-4 PC2數據集上分類器使用不同特徵選擇的準確性------- 5
圖2-1 Alcatel-Lucent 1660SM設備---------------------------------- 9
圖2-2 Alcatel-Lucent A21E1卡板------------------------------------ 10
圖2-3 Alcatel-Lucent P63E1卡板------------------------------------ 11
圖2-4 Alcatel-Lucent P3T3卡板-------------------------------------- 13
圖2-5 Alcatel-Lucent A3T3卡板------------------------------------- 14
圖2-6 Alcatel-Lucent A2S1卡板------------------------------------- 16
圖2-7 Alcatel-Lucent P4S1N卡板------------------------------------ 17
圖2-8 Alcatel-Lucent S-4.1N 卡板----------------------------------- 19
圖2-9 Alcatel-Lucent L-16.1N卡板---------------------------------- 20
圖2-10 Alcatel-Lucent EQUCIO卡板--------------------------------- 21
圖2-11 Alcatel-Lucent MATRIX卡板-------------------------------- 22
圖2-12 Alcatel-Lucent CONGI卡板----------------------------------- 23
圖2-13 Alcatel-Lucent SERVIC卡板--------------------------------- 24
圖2-14 Alcatel-Lucent Fan Unit 模組--------------------------------- 25
圖2-15 EPS設備保護卡板---------------------------------------------- 26
圖2-16 Hadoop 基本架構----------------------------------------------- 29
圖2-17 Hadoop 生態系統的各種組件-------------------------------- 35
圖2-18 Hadoop Distributed File System架構------------------------ 37
圖2-19 YARN (Yet Another Resource Negotiator)架構------------ 39
圖2-20 Hadoop Hive 組件架構---------------------------------------- 41
圖2-21 Hadoop HBase系統架構--------------------------------------- 43
圖2-22 Hadoop MapReduce執行任務流程-------------------------- 46
圖2-23 Hadoop MapReduce 架構------------------------------------- 47
圖2-24 Hadoop MapReduce邏輯資料流程-------------------------- 48
圖2-25 MapReduce word count Input---------------------------------- 48
圖2-26 MapReduce word count 流程--------------------------------- 49
圖2-27 MapReduce word count output-------------------------------- 50
圖2-28 Hadoop MapReduce執行運算-------------------------------- 50
圖2-29 Spark MapReduce 執行運算---------------------------------- 51
圖2-30 Apache Spark搭配叢集管理分散式儲存系統架構------- 52
圖2-31 隨機森林演算法運算流程------------------------------------- 57
圖2-32 隨機森林結構---------------------------------------------------- 58
圖2-33 Oracle VirtualBox ----------------------------------------------- 61
圖2-34 Ubuntu作業系統畫面------------------------------------------ 62
圖3-1 系統架構圖------------------------------------------------------- 63
圖3-2 系統流程圖------------------------------------------------------- 64
圖3-3 Ubuntu Operating System -------------------------------------- 65
圖3-4 Virtual Box ------------------------------------------------------- 65
圖3-5 Java_1.7.0_115版本-------------------------------------------- 66
圖3-6 SSH Key放置授權檔案---------------------------------------- 66
圖3-7 下載Apache Hadoop-2-6.5.tar.gz ---------------------------- 67
圖3-8 解壓縮Hadoop-2-6.5.tar.gz ----------------------------------- 67
圖3-9 hadoop-2.6.5目錄至/usr/local/hadoop目錄中------------- 67
圖3-10 顯示解壓縮後的檔案是否儲存在指定路徑---------------- 68
圖3-11 編輯～/.bashrc 指令-------------------------------------------- 68
圖3-12 設定～/.bashrc 檔案環境變數-------------------------------- 69
圖3-13 編輯Hadoop core-site.xml指令------------------------------ 69
圖3-14 設定Hadoop core-site.xml環境變數------------------------ 70
圖3-15 編輯Hadoop hdfs-site.xml指令------------------------------ 70
圖3-16 設定Hadoop hdfs-site.xml環境變數------------------------ 71
圖3-17 編輯Hadoop yarn-site.xml指令------------------------------ 71
圖3-18 設定yarn-site.xml環境變數---------------------------------- 72
圖3-19 編輯Hadoop mapred-site.xml指令-------------------------- 72
圖3-20 設定mapred-site.xml環境變數------------------------------- 73
圖3-21 設定hduser在local端存放的檔案位置-------------------- 73
圖3-22 start-all.sh 指令-------------------------------------------------- 74
圖3-23 JPS指令---------------------------------------------------------- 74
圖3-24 Hadoop ResourceManager Web查看已執行的Nodes---- 75
圖3-25 Hadoop Resource-Manager HDFS Web UI查看啟動的Nodes---------------------------------------------------------------
76
圖3-26 Hadoop Resource-Manager HDFS Web UI查看啟動DataNodes資訊--------------------------------------------------
76
圖3-27 Hadoop Spark 互動畫面--------------------------------------- 77
圖4-1 Virtual Box 執行Hadoop Nodes----------------------------- 79
圖4-2 ANACONDA NAVIGATOR執行畫面--------------------- 80
圖4-6 Hadoop HDFS & Spark讀取架構圖------------------------- 80
圖4-7 設備Raw Data.csv檔------------------------------------------ 81
圖4-8 隨機森林演算法運算程式碼------------------------------ 81
圖4-9 隨機森林演算法預測結果與準確度----------------------- 81
圖5-1 Alcatel Lucent Replaceable Unit Problem alarm------------ 83
圖5-2 Alcatel Lucent Degraded Signal alarm---------------------- 83
圖5-3 Alcatel Lucent Inside Failure alarm度--------------------- 84
圖5-4 Alcatel Lucent BBE error performance data趨勢圖------- 84

[1]數位時代(2018年04月02日)。為未來設計的網路-5G掀起全新戰場【科技新報】。取自
http://technews.tw/2018/04/02/future-is-5g/
[2]拓墣產研(2017年03月11日)。拓墣觀點」不只速度快，5G帶來4G無法取代的新價值【科技新報】。取自
http://technews.tw/2017/03/11/5g-value/
[3]曾靉(2018年01月29日)。台灣首支「5G國家隊」正式成軍，中華電：2020年轉商【數位時代】。取自
http://technews.tw/2017/03/11/5g-value/
[4]新智元(2017年07月06日)。「首發」LinkedIn《全球AI領域人才報告》階示全球AI人才圖蹼【每日頭條】。取自
https://kknews.cc/zh-tw/tech/gqvbe8m.html
[5]Dyana Rashid Ibrahim, Rawan Ghnemat, Amjad Hudaib. (2017, Octber)."Software Defect Prediction using Feature Selection and Random Forest Algorithm", 2017 International Conference on New Trends in Computing Sciences (ICTCS), Amman,Jordan.
[6]Manuals Alcatel 1660 SM Technical Handbook【Alcatel 1660 SM Manuals】。取自
https://www.manualslib.com/products/Alcatel-1660sm-8743216.html
[7]陳振群，2016，ALBERT:利用自動學習方式優化Hadoop執行與資源使用之計算管理系統，國立清華大學，碩士論文。
[8]其他(2018年12月10日)。Hadoop的介紹以及發展歷史【ITREAD01】。取自
https://www.itread01.com/content/1544423064.html
[9]Renata Machova, Jitka Komarkova, Martin Lenicka. (2016, Octber)."Processing of big educational data in the cloud using Apache Hadoop", 2016 International Conference on Information Society (i-Society) ,Dublin, lreland.
[10]IT技術(2016年10月28日)。Hadoop學習筆記（1）：概念和整體架構【ITREAD01】。取自
https://www.itread01.com/articles/1475877910.html
[11]Apache Hadoop(2018年09月15日)。FS Shell 使用指南【Hadoop Core】。取自
https://hadoop.apache.org/docs/r1.0.4/cn/hdfs_shell.html
[12]極客書(2015)。Hadoop介紹快速入門【GITBOOK.NET】。取自http://tw.gitbook.net/hadoop/hadoop_10mins_start.html
[13]Xiaoyu Jin , Xilin Ji , Yang Yang , Zhipeng Gao , Xuesong Qiu. (2018, January), "An Evaluation Mechanism Based on HDFS in Unstable Network Environment", 2018 IEEE International Conference on Big Data and Smart Computing (BigComp), Shanghai, China.
[14]Stana(2017年12月05日)。Hadoop Ecosystem之Hadoop介紹【IT邦幫忙】。取自
https://ithelp.ithome.com.tw/articles/10190756
[15]洪羿漣(2009年11月)。初探Hadoop開放原始碼平台環境【RUN!PC雜誌】。取自
http://www.runpc.com.tw/content/content.aspx?id=105318
[16]網路編輯愛好者(2018年03月04日)。Hadoop YARN簡介【每日頭條】。取自
https://kknews.cc/zh-tw/other/gya93m9.html
[17]Janardhanan PS, Philip Samuel. (2017, November), "Analysis and modeling of Resource Management Overhead in Hadoop YARN Clusters", 2017 IEEE 15th Intl Conf on Dependable, Autonomic and Secure Computing, 15th Intl Conf on Pervasive Intelligence and Computing, 3rd Intl Conf on Big Data Intelligence and Computing and Cyber Science and Technology Congress, Orlando, FL, USA .
[18]DATAFLAIR TEAM(2018年11月07日)。Apache Hive Architecture & Components【Data Flair】。取自
https://data-flair.training/blogs/apache-hive-architecture/
[19]Cnblogs(105年02月29日)。HBase介紹和工作原理【壹讀】。取自
https://read01.com/zh-tw/6GP7AG.html#.XMwhr5MzY_U
[20]Xiuqin LIN, Peng WANG, Bin WU. (2013, November), "LOG ANALYSIS IN CLOUD COMPUTING ENVIRONMENT WITH HADOOP AND SPARK", 2013 5th IEEE International Conference on Broadband Network & Multimedia Technology, Guilin, China.
[21]Khadija Aziz, Dounia Zaidouni, Mostafa Bellafkih. (2018, April), "Real-Time Data Analysis Using Spark and Hadoop", 2018 4th International Conference on Optimization and Applications (ICOA), Mohammedia, Morocco.
[22]Zhu Mingyu(2019年03月03日)。Hadoop組件介紹【Eternal Center】。取自
https://eternalcenter.com/?p=3621
[23]Taneyahomilk(2017年12月30日)。Apache Spark簡介【IT邦幫忙】。取自
https://ithelp.ithome.com.tw/articles/10194895
[24]W Huang, L. Meng, D. Zhang , W. Zhang. (2016), "In-memory Parallel Processing of Massive Remotely Sensed Data Using an Apache Spark on Hadoop YARN Model", IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, Volume.10, pp. 3-19.
[25]Shyam R, Bharathi Ganesh H.B, Sachin Kumar S, Prabaharan Poornachandran. (2015), "Apache Spark a Big Data Analytics Platform for Smart Grid", Procedia Technology, Volume.21, pp. 171-
178.
[26]I.Chebbi, W.Boulila, M.Lamolle, I.R.Farah, N.Mellouli. (2018, March), "A Comparison of Big Remote Sensing Data Processing with Hadoop MapReduce and Spark", 2018 4th International Conference on Advanced Technologies for Signal and Image Processing (ATSIP), Sousse, Tunisia.
[27]深思海數(2016年10月17日)。機器學習算法之隨機森林 (Random Forest) 【每日頭條】。取自
https://kknews.cc/zh-tw/news/brzmvj.html
[28]Site of Leo Breiman (2004年06月02日)。Random Forest classifier description【Internet Archive WayBackmachine】。取自
https://web.archive.org/web/20080622230434/http://stat-www.berkeley.edu/users/breiman/RandomForests/cc_home.htm
[29]May Phu Paing, Somsak Choomchuay. (2018, July), "Improved Random Forest (RF) Classifier for Imbalanced Classification of Lung Nodules", 2018 International Conference on Engineering, Applied Sciences, and Technology (ICEAST), Phuket, Thailand.
[30]Breiman,Leo. (2001), "Random Forests Machine Learning", Kluwer Academic,Volume.45, pp. 5-32.
[31]YouLi Feng, ShanShan Wang. (2017, December), "A forecast for bicycle rental demand based on random forest and multiple linear regression", 2017 IEEE/ACIS 16th International Conference on Computer and Information Science (ICIS), Whuan, China.
[32]JianKai Wang(2018年03月26日)。隨機森林(Random Forest)【sophia】。取自
https://rstudio-pubs-static.s3.amazonaws.com/378052_30d987a09ea54b6db5aa1e82f5dce6bf.html
[33]Xiaolong Xu, Wen Chen. (2017, October), "Implementation and Performance Optimization of Dynamic Random Forest", 2017 International Conference on Cyber-Enabled Distributed Computing and Knowledge Discovery (CyberC), Nanjing,China.
[34]Yeh James(2017年11月5日)。資料分析&機器學習：決策樹(Decision Tree)以及隨機森林(Random Forest)介紹【medium】取自
https://medium.com/jameslearningnote
[35]Shiori Toyoshima, Saneyasu Yamaguchi, Masato Oguch. (2010, April) ,"Storage Access Optimization with Virtual Machine Migration During Execution of Parallel Data Processing on a Virtual Machine PC Cluster", 2010 IEEE 24th International Conference on Advanced Information Networking and Applications Workshops, Perth, WA, Australia.
[36]熊嬡(2015年12月24日)。柯克蘭:Ubuntu操作系統全球用戶已超10億【天極網IT新聞頻道】。取自
http://wap.yesky.com/477/99506977.shtml