臺灣博碩士論文加值系統

在伺服器規模日漸擴張的單位中，日誌管理與維護重要性越來越高，除了最基本的系統日誌 ( syslog ) 外，應用層面的日誌更能反映伺服器中運行的服務狀態，同時在問題發生時也更能夠追蹤事件來源。
本論文嘗試佈建伺服器集群運作日誌蒐集系統，符合學術單位中電子計算中心架構。以 Elastic Stack 為基礎，提供交通大學資工系計算機中心一穩健日誌蒐集平台，並於此之上開放資工系計算機中心服務日誌相關 Opendata ，以利於其他學術研究或第三方分析使用。

Today, more and more servers installed into data/compute cventer, it’s become more important for IT maintainer to collect and analysis log. Not only the basic system running log (syslog), but also logs generated in application level. Those logs can help we address the real problem cause the event.
In this paper, we are trying to build a streamlined bug report mechanism for large-scale IT infrastructure based on multi-source log aggregation, and provide open data base on it. We deploy Elastic Stack on Computer Center of CS of NCTU to implement a stable log collecting, analyzing platform. And finally build an Open Data system for third party study.

1. 動機目標 p1
2. 研究背景與資源 p3
3. 相關研究 p4
Splunk [1] p4
Detecting Large-Scale System Problems by Mining Console Logs [2] p4
Analysis farm: A cloud-based scalable aggregation and query platform for network log analysis [3] p5
Statistic and analysis for Host-based syslog [4] p5
Elastic Stack [5] p5
4. 研究設計 p8
4.1規劃架構 p8
4.2 架設系統 p11
4.3 整合資料來源 p11
4.4 處理資料 p15
4.5 呈現資料 p15
4.6 偵測事件 p16
4.7 開放資料 p17
5. 研究實踐 p18
5.1 實驗環境 p18
5.2 平台環境 p20
5.2.1 架設系統 p20
5.2.2 管理平台 p23
5.3. Open Data 系統實作 p25
5.4 運行成果 p28
6. 實驗評測 p28
6.1系統負載 p28
6.2即時分析 p29
6.3 視覺化分析 p30
6.4 事件監控通知 p32
6.5開放資料 p33
7. 總結 p36
8. Future work p37
Reference p38

[1] Splunk, https://www.splunk.com
[2] Wei Xu, Ling Huang, Armando Fox, David Patterson, and Michael I. Jordan. 2009. Detecting large-scale system problems by mining console logs. In Proceedings of the ACM SIGOPS 22nd symposium on Operating systems principles (SOSP '09). ACM, New York, NY, USA, 117-132. DOI=http://dx.doi.org/10.1145/1629575.1629587
[3] Jianwen WEI, Yusu ZHAO, Kaida JIANG, Rui XIE and Yaohui JIN,"Analysis Farm: A Cloud-based Scalable Aggregation and Query Platform for Network Log Analysis", International Conference on Cloud and Service Computing, 2011.
[4] Z. Gu and C. Wang, "Statistic and Analysis for Host-Based Syslog," in Education Technology and Computer Science (ETCS), 2010 Second International Workshop on, pp. 277-280.
[5] Elastic Stack, https://www.elastic.co/
[6] Apache Lucene, https://lucene.apache.org/core/
[7] GeoLite2, https://dev.maxmind.com/geoip/geoip2/geolite2/
[8] R. Vaarandi, A Data Clustering Algorithm for Mining Patterns from Event Logs, In Proc. IPOM, 2003.
[9] Curator, https://github.com/elastic/curator
[10] Elastalert, https://github.com/Yelp/elastalert