臺灣博碩士論文加值系統

在邁向高速網路紀元，對於骨幹100G/區域10G的網路所湧出的巨量資料（Big Data），不僅資料量達PB等級且有許多非結構化資料，其高速網路時代的資安事件，如近年400Gbps的DDOS攻擊其龐大訊息也造成資安分析的重大挑戰。在高速骨幹網路上進行流量偵測及資安分析是一個新興挑戰性課題，不論是資料量或資料類型已無法透過既有技術完善處理偵測，需要藉由新興科技如雲端運算與其它IT技術來加以解決。本論文提出利用巨量資料分析技術進行高速網路下巨量資料資安分析平台之研發，本文提出的架構具備巨量資料儲存、可動態擴充、高效率分散式運算等特點，可用來處理高速網路的流量分析與監控等應用。本平台整合架構涵蓋第三方相關開放原始碼套件Logstash，Elasticsearch及Redis NoSQL資料庫技術，在透過Logstash採集資料源並將各種不同結構資料經篩選後轉化為結構資料，並借助Elasticsearch快速搜尋資料及自動索引功能達到即時之效果，Redis則是以RAM為儲存媒介的資料庫，並且以key-value為儲存型態，透過In-memory架構快速且穩定儲存資料，本文並整合Kibana進行資料視覺化的呈現。本文所研發的系統已布署至台灣學術網路TANet的MiniSoc系統環境內，初步的資料顯示其對於邁向骨幹100G/區域10G的台灣學術網路之資安分析有著莫大助益，透過偵測系統所產生的龐大警報所呈現的資安分析都顯示本平台適合提供給大型企業及政府環境進行高速網路環境下的資安分析。

The threat of the endless information security incidents continuously expand due to the development of internet. All kinds of information security incidents ranging from Openssl heartbleed to Bash Shellshock seriously impact the computer operations of the enterprises and governments. Under the circumstances of big data times incurred by the popularity of internet, this paper proposes the BigSA massive analysis of information security framework by integrating various massive data analysis technologies to meet the requirement for the high speed network. The proposed BigSA framework is implemented by integrating several related open sources of information security, such as logstach, Elasticsearch and Redis NoSQL database technologies. A prototype has been deployed in TANet, Taiwan Academic Network. For TANet with the orientation of 100Gbps backbone and 10Gbps regional network, the preliminary data shows an enormous improvement in the information security analysis. Both of the alarm resulting from the detection system and the information from Netflow indicate that the BigSA framework meet the requirements of the information security analysis in high speed network, regardless of huge enterprises and governments. The ‎robustness and flexibility are also discussed in the end of this paper.

目錄
摘 要 ................................ ................................ ................................ ............................. i
ABSTRACT ................................ ................................ ................................ .................. ii
誌 謝 ................................ ................................ ................................ ........................... iii
目錄 ................................ ................................ ................................ ............................... iv
表目錄 ................................ ................................ ................................ ........................... vi
圖目錄 ................................ ................................ ................................ .......................... vii
第一章、緒論 ................................ ................................ ................................ ................ 1
1.1 研究背景 ................................ ................................ ................................ .......... 1
1.2 研究動機 ................................ ................................ ................................ .......... 3
1.3 論文架構 ................................ ................................ ................................ .......... 4
第二章、文獻探討................................ ................................ ................................ ........ 5
2.1 雲端運算 ................................ ................................ ................................ .......... 5
2.2 巨量資料分析 ................................ ................................ ................................ .. 9
2.3 分散式搜尋系統 ................................ ................................ ............................ 22
2.4 NoSQL 資料庫 ................................ ................................ ............................... 24
第三章、 BigSA系統架構設計 ................................ ................................ .................. 27
3.1 系統架構 ................................ ................................ ................................ ........ 27
3.2 元件模組 ................................ ................................ ................................ ........ 29
第四章、系統開發及設計 ................................ ................................ .......................... 34
4.1 iDocker┼ Security Lab ................................ ................................ ................... 34
4.2BigSA在 MiniSOC基本架構設計及開發 ................................ .................... 37
4.3 資料視覺化 Data Visualization 之 My Security Dashboar ......................... 44
第五章、測試情境 (scenario testing) ................................ ................................ ......... 48
5.1 資安分析之 TOP_10 攻擊 IP 分析 ................................ ............................... 48
5.2 資安 分析之 TOP_10 受攻擊 IP 分析 ................................ ........................... 50
5.3 資安分析之 TOP_10 觸發規則分析 ................................ ............................. 52
第六章、結論與未來工作 ................................ ................................ .......................... 56
參考文獻 ................................ ................................ ................................ ...................... 57
附件一、 Snort Log 格式分析 ................................ ................................ ................... 60

參考文獻
[1] Wikipedia,”Heartbleed.”,https://en.wikipedia.org/wiki/Heartbleed
[2] iThome, “OpenSSL Heartbleed漏洞危機特別報導”, http://www.ithome.com.tw/special_report/heartbleed
[3] TIME, “Report: Devastating Heartbleed Flaw Was Used in Hospital Hack”,http://time.com/3148773/report-devastating-heartbleed-flaw-was-used-in-hospital-hack/
[4] Wikipedia,”Shellshock_(software_bug)”, https://en.wikipedia.org/wiki/Shellshock_(software_bug)
[5] iThome,”羅馬尼亞駭客利用Shellshock漏洞入侵雅虎，不小心打中Web log 漏洞”,http://www.ithome.com.tw/news/91365
[6] iThome, “IT產品Shellshock災情大清查”, http://www.ithome.com.tw/news/91433
[7] Microsoft,”Microsoft 資訊安全公告 MS15-034 - 重大”, https://technet.microsoft.com/zh-tw/library/security/ms15-034.aspx
[8] Cloudflare,”Technical Details Behind a 400Gbps NTP Amplification DDoS Attack”,https://blog.cloudflare.com/technical-details-behind-a-400gbps-ntp-amplification-ddos-attack/
[9] computerworld,”Applying big data approaches to information security a challenge”,http://www.computerworld.com/article/2495671/big-data/applying-big-data-approaches-to-information-security-a-challenge.html
[10] Wikipedia,”Cloud computing. “, https://en.wikipedia.org/wiki/Cloud_computing
[11] 大紀元,” 移動裝置更安全 2015年六大科技趨勢”http://www.epochtimes.com/b5/14/12/7/n4312959.htm%E7%A7%BB%E5%8B%95%E8%A3%9D%E7%BD%AE%E6%9B%B4%E5%AE%89%E5%85%A8-2015%E5%B9%B4%E5%85%AD%E5%A4%A7%E7%A7%91%E6%8A%80%E8%B6%A8%E5%8B%A2.html
[12] readwrite ,”Apps.gov Gives Cloud Computing a Slice of the $75 Billion IT Pie” ,http://readwrite.com/2009/09/15/appsgov-gives-cloud-computing
[13] 美國國家標準與技術研究院 , “The NIST Definition of Cloud Computing” , NIST SP - 800-145 ,
[14] IDC,”THE DIGITAL UNIVERSE IN 2020: Big Data” ,http://www.emc.com/collateral/analyst-reports/idc-the-digital-universe-in-2020.pdf ,pp.3-3
[15] McKinsey,”Big data: The next frontier for innovation, competition, and productivity” ,http://www.mckinsey.com/insights/business_technology/big_data_the_next_frontier_for_innovation
[16] iThome ,“IDC：Big Data市場規模2015年將達169億美元”, http://www.ithome.com.tw/node/72571
[17] TIME,”Inside the Secret World of the Data Crunchers Who Helped Obama Win” , http://swampland.time.com/2012/11/07/inside-the-secret-world-of-quants-and-data-crunchers-who-helped-obama-win/2/
[18] Nature,”Big data: science in the petabyte era”, http://www.nature.com/nature/journal/v455/n7209/edsumm/e080904-01.html
[19] Wikipedia，”Big data.”, https://en.wikipedia.org/wiki/Big_data
[20] Gartner,”3D Data Management: Controlling Data Volume, Velocity, and Variety.”,http://blogs.gartner.com/doug-laney/files/2012/01/ad949-3D-Data-Management-Controlling-Data-Volume-Velocity-and-Variety.pdf
[21] Ibm,”Analytics: The real-world use of big data”, http://www-935.ibm.com/services/us/gbs/thoughtleadership/ibv-big-data-at-work.html
[22] Wikipedia,”Hadoop” ,https://en.wikipedia.org/wiki/Apache_Hadoop
[23] S. Ghemawat, H. Gobioff, and S.-T. Leung, “The Google file system”, in SIGOPS Oper. Syst. Rev. 2003. p. 29-43
[24] DEAN J, GHEMAWAT S. ,“MapReduce: Simplified data processing on large clusters” ,Proceedings of the 6th USENIX Symposium on Operating System Design and Implementation (OSDI’04), Dec 6-8,2004,San Francisco, CA,USA. Berkeley, CA, USA: USENIX Association, 2004:137-150.
[25] Vavilapalli V K, Murthy A C, Douglas C, “Apache hadoop yarn: Yet another resource negotiator.” Proceedings of the 4th annual Symposium on Cloud Computing. ACM, 2013: 5.
[26] Wikipedia, “HBase”, https://en.wikipedia.org/wiki/Apache_HBase
[27] Wikipedia, “Hive” , https://en.wikipedia.org/wiki/Apache_Hive
[28] P. Hunt, M. Konar, F.P. Junqueira, and B. Reed, “ZooKeeper: wait-free coordination for internet-scale systems”, in Proceedings of the 2010 USENIX conference on USENIX annual technical conference. 2010, USENIX Association: Boston, MA. p. 11-11
[29] Wikipedia,”Pig”, https://en.wikipedia.org/wiki/Pig_(programming_tool)
[30] Wikipedia, “Apache_Mahout” , https://en.wikipedia.org/wiki/Apache_Mahout
[31] Wikipedia, “Sqoop” , https://en.wikipedia.org/wiki/Sqoop
[32] Wikipedia,”R_(programming_language)”, https://en.wikipedia.org/wiki/R_(programming_language)
[33] Wikipedia,”Storm_(event_processor)”, https://en.wikipedia.org/wiki/Storm_(event_processor)
[34] Wikipedia, “Apache_Spark” , https://en.wikipedia.org/wiki/Apache_Spark
[35] mmdays , “Big Data 大數據 大商機 大未來”, http://mmdays.com/2012/12/22/big-data-%E5%A4%A7%E8%B3%87%E6%96%99-%E5%A4%A7%E5%95%86%E6%A9%9F/
[36] Tsung's Blog,”淘寶 1111(雙11) 光棍節 2014年的紀錄”, http://blog.longwin.com.tw/2014/11/taobao-alibaba-1111-record-2014/
[37] Wikipedia,”Lucene”, https://en.wikipedia.org/wiki/Lucene
[38] Wikipedia,”Apache_Solr”, https://en.wikipedia.org/wiki/Apache_Solr
[39] Wikipedia,”Elasticsearch”, https://en.wikipedia.org/wiki/Elasticsearch
[40] Borthakur, D., Gray, J., Sarma, J. S., Muthukkaruppan, K., Spiegelberg, N., Kuang, H.,Ranganathan, K., Molkov, D., Menon, A., Rash, S., Schmidt R. and Aiyer, A. “Apachehadoop goes realtime at Facebook,” Proceedings of the 2011 ACM SIGMODInternational Conference on Management of data, June 12-16, 2011, Athens, Greece.
[41] Pramod J. Sadalage、Martin Fowler,” NoSQL Distilled”, 2013年
[42] Phimedia,”STORAGE AND BIG DATA 大資料時代的儲存方式”, http://blog.phimedia.tv/2012/11/storage-and-big-data/
[43] F. Chang, J. Dean, S. Ghemawat, W. Hsieh, D. Wallach, M. Burrows, T. Chandra, A. Fikes and R. Gruber, “Bigtable: A Distributed Storage System for Structured Data,” Proceedings of the 7th Symposium on Operating System Design and Implementation, 2006, pages 1-26.
[44] Wikipedia,”Redis” ., https://en.wikipedia.org/wiki/Redis
[45] 曾龍、周志學、王英傑、陳柔伊、蔡龍佑，“BigSA:以巨量資料分析為基礎的資安分析平台開發與設計”， UHC2014第八屆優質家庭生活研討會，2014
[46] Wikitech,”Logstash .”, https://wikitech.wikimedia.org/wiki/Logstash
[47] “ELKstack中文指南”,http://kibana.logstash.es/content/
[48] Wikipedia, “Docker”, https://en.wikipedia.org/wiki/Docker_(software)
[49] Wikipedia,:” CoreOS”, https://en.wikipedia.org/wiki/CoreOS
[50] “Open Container Project” ,https://www.opencontainers.org/
[51] Symantec,”The Evolution of Malicious IRC Bots” , http://www.symantec.com/avcenter/reference/the.evolution.of.malicious.irc.bots.pdf