臺灣博碩士論文加值系統

物聯網管理平台整合設備資源並提供進階的雲端服務，如何從即時串流資料中分離出有用的資訊是服務應用的核心問題，因此我們透過Spark的串流處理模組實作異常偵測方法，並利用叢集系統的運算效能優勢，提供即時的服務內容。
本論文提出一套完整的異常偵測和事件警報系統，使用Spark Streaming RDD資料處理概念實作不同功能的資料分析模組，並與其他第三方服務連接提供不同的訊息推送方式、設計偵測系統與既有平台整合的資料交換格式。延伸常態分佈與局部異常因子（LOF）的想法，於串流資料中實作五種異常偵測演算法，再藉由模擬數據的實驗證明方法在不同資料變化特徵下的使用價值。

An IoT（Internet of Things）management platform provides advance cloud applications by integrating a variety of device resources. The core of service support is to separate valuable information from original data. In order to construct real-time services, we implement an anomaly detection module by the support of streaming process and cluster computing from Spark.
This thesis aims to propose a fully developed anomaly detection and event alert system. With the RDD data process concept in Spark Streaming, we implement a data analysis module that supports various functions. This module can receive data from an integrated platform and push messages out by using third-party services. We extend many concepts such as normal distribution and local outlier factor to implement five streaming-based detection algorithms. Simulation results provides reference values to these algorithms with different data features.

致謝----------------------------------------------------------------i
中文摘要------------------------------------------------------------ii
ABSTRACT------------------------------------------------------------iii
目錄----------------------------------------------------------------iv
圖目錄--------------------------------------------------------------vii
表目錄--------------------------------------------------------------ix
Chapter 1 前言------------------------------------------------------1
1.1 研究背景------------------------------------------------------1
1.2 研究動機與目的------------------------------------------------2
1.3 論文架構------------------------------------------------------4
Chapter 2 文獻探討--------------------------------------------------5
2.1 都市環境品質監控----------------------------------------------5
2.2 感測數據分析--------------------------------------------------6
2.3 異常偵測------------------------------------------------------8
2.3.1 異常的種類------------------------------------------------8
2.3.2 異常偵測的方法--------------------------------------------9
2.4 時序資料的異常偵測（Anomaly Detection of Time Series）--------10
2.5 應用技術介紹--------------------------------------------------11
2.5.1 Apache Spark----------------------------------------------12
2.5.2 Node.js---------------------------------------------------13
2.5.3 MQTT------------------------------------------------------15
2.5.4 REST （Representational State Transfer）------------------17
2.5.5 MongoDB---------------------------------------------------18
2.5.6 Redis-----------------------------------------------------18
2.5.7 Firebase Cloud Messaging----------------------------------19
Chapter 3 異常檢測和事件警報系統------------------------------------21
3.1 裝置管理平台--------------------------------------------------21
3.2 系統設計------------------------------------------------------22
3.3 系統資料傳遞--------------------------------------------------24
3.3.1 Facebook Token--------------------------------------------24
3.3.2 FCM Token-------------------------------------------------27
3.3.3 Spark Cluster---------------------------------------------28
3.4 異常偵測方法設計----------------------------------------------32
3.4.1 Threshold-------------------------------------------------32
3.4.2 Normal Distribution---------------------------------------34
3.4.3 CUSUM（Cumulative Sum）-----------------------------------37
3.4.4 LOF（Local Outlier Factor）-------------------------------40
3.5 系統功能實作--------------------------------------------------46
3.5.1 即時偵測策略----------------------------------------------46
3.5.2 異常偵測方法實作------------------------------------------50
3.5.3 異常事件通知----------------------------------------------61
Chapter 4 系統實驗--------------------------------------------------65
4.1 實驗方法------------------------------------------------------65
4.1.1 實驗流程--------------------------------------------------66
4.2 實驗數據設計--------------------------------------------------66
4.2.1 訓練數據--------------------------------------------------66
4.2.2 測試數據--------------------------------------------------68
4.3 實驗結果------------------------------------------------------73
4.3.1 實驗一----------------------------------------------------73
4.3.2 實驗二–1--------------------------------------------------85
4.3.3 實驗二–2--------------------------------------------------90
4.4 實驗結論------------------------------------------------------93
Chapter 5 結論與未來工作--------------------------------------------95
參考文獻------------------------------------------------------------97
附錄–開發與執行環境建置---------------------------------------------101

[1]WHO releases country estimates on air pollution exposure and health impact. World Health Organization. from:http://www.who.int/mediacentre/news/releases/2016/air-pollution-estimates/en/
[2]Air Pollution Deaths Cost Global Economy US$225 Billion. World Bank. from:http://www.worldbank.org/en/news/press-release/2016/09/08/air-pollution-deaths-cost-global-economy-225-billion
[3]NEC智慧城市 (Smart City) | NEC. Tw.nec.com. from:http://tw.nec.com/zh_TW/solutions/smartcity/smartcity1.html
[4]Baralis, E., Cerquitelli, T., Chiusano, S., Garza, P., & Kavoosifar, M. (2016). Analyzing air pollution on the urban environment. In MIPRO (pp. 1464-1469). Opatija, Croatia.
[5]Rishee K.Jain, Jose M.F. Moura, & Constaintine E. Kontokosta (2014). Big Data + Big Cities: Graph Signals of Urban Air Pollution. In IEEE SIGNAL PROCESSING MAGAZINE(pp. 130-136).
[6]Zheng, Y., Liu, F., & Hsieh, H. (2013). U-Air: when urban air quality inference meets big data. In KDD '13 Proceedings of the 19th ACM SIGKDD international conference on Knowledge discovery and data mining (pp. 1436-1444). Chicago, Illinois, USA: KDD 2013.
[7]Charu C. Aggarwal.(2013). An Introduction to Sensor Data Analytics. In Managing and Mining Sensor Data(pp.1-8).
[8]Han J, Kamber M, Pei J. Data Mining: Concepts and Techniques, 3rd ed. Burlington: Morgan Kaufmann Publishers Inc; 2011.
[9]Zaki MJ, Meira W. Data Mining and Analysis: Fundamental Concepts and Algorithms. New York: Cambridge University Press; 2014.
[10]García, S., Ramírez-Gallego, S., Luengo, J., Benítez, J., & Herrera, F. (2016). Big data preprocessing: methods and prospects. Big Data Analytics, 1(1). http://dx.doi.org/10.1186/s41044-016-0014-0
[11]Chandola, V., Banerjee, A., & Kumar, V. (2009). Anomaly detection: A survey. ACM Computing Surveys (CSUR), Volume 41 Issue 3, July 2009, Article No. 15.
[12]Song, X., Wu, M., Jermaine, C. and Ranka, S. (2007). Conditional Anomaly Detection. IEEE Transactions on Knowledge and Data Engineering, 19(5), pp.631-645.
[13]U.A.B.U.A Bakar, H. Ghayvat, S.F. Hasanm, S.C. Mukhopadhyay (2016). Activity and Anomaly Detection in Smart Home: A Survey. Springer International Publishing Switzerland(pp. 191-220).
[14]Friedman, N., Geiger, D. and Goldszmidt, M. (1997). Machine Learning, 29(2/3), pp.131-163.
[15]Suykens, J. and Vandewalle, J. (1999). Neural Processing Letters, 9(3), pp.293-300.
[16]Pham, D., Dimov, S., & Nguyen, C. (2005). Selection of K in K-means clustering. Proceedings Of The Institution Of Mechanical Engineers, Part C: Journal Of Mechanical Engineering Science, 219(1), 103-119. http://dx.doi.org/10.1243/095440605x8298
[17]Peterson, L. (2009). K-nearest neighbor. Scholarpedia, 4(2), p.1883.
[18]Deepthi, C. (2010). Anomaly detection of time series.. Retrieved from the University of Minnesota Digital Conservancy, http://hdl.handle.net/11299/92985.
[19]Stats and Bots. (2017). Time Series Anomaly Detection Algorithms – Stats and Bots. from:https://blog.statsbot.co/time-series-anomaly-detection-algorithms-1cef5519aef2
[20]Apache Spark Introduction. www.tutorialspoint.com. from:https://www.tutorialspoint.com/apache_spark/apache_spark_introduction.htm
[21]王家林(2016). 大數據分析處理Spark技術應用與性能優質化 (1st ed.). 上奇資訊股份有限公司.
[22]錢逢祥, 蔡政崇, 林政毅(2015). 不一樣的Node.js_用JavaScript打造高效能的前後台網頁程式 (2nd ed.). 松崗資產管理股份有限公司.
[23]Laurens, T. How the V8 engine works?. Thibaultlaurens.github.io. http://thibaultlaurens.github.io/javascript/2013/04/29/how-the-v8-engine-works/
[24]MQTT Essentials. HiveMQ. from:http://www.hivemq.com/mqtt-essentials/
[25]MQTT Version 3.1.1. Docs.oasis-open.org. from:http://docs.oasis-open.org/mqtt/mqtt/v3.1.1
[26]Roy Thomas Fielding(2000). Architectural styles and the design of network-based software architectures. Doctoral Dissertation.
[27]淺談 REST 軟體架構風格 - 從了解 REST 到設計 RESTful！ - Soul & Shell Blog. from:https://blog.toright.com/
[28]MongoDB Tutorial. www.tutorialspoint.com. from:https://www.tutorialspoint.com/mongodb/index.htm
[29]Redis Tutorial. www.tutorialspoint.com. from:https://www.tutorialspoint.com/redis/index.htm
[30]Firebase Documentation. Firebase. from:https://firebase.google.com/docs/
[31]觀測資料查詢系統. E-service.cwb.gov.tw. from:http://e-service.cwb.gov.tw/HistoryDataQuery/index.jsp
[32]Cluster Mode Overview - Spark 2.1.1 Documentation. Spark.apache.org. from:https://spark.apache.org/docs/latest/cluster-overview.html
[33]Apache Spark Hidden REST API. Gist, from:https://gist.github.com/arturmkrtchyan/5d8559b2911ac951d34a
[34]行政院環境保護署－空氣品質監測網. Taqm.epa.gov.tw. from:http://taqm.epa.gov.tw/taqm/tw/b0201.aspx
[35]常態分布. Zh.wikipedia.org. from:https://zh.wikipedia.org/zh-tw/%E6%AD%A3%E6%80%81%E5%88%86%E5%B8%83
[36]Knuth D, Art of Computer Programming, Vol 2, page 232, 3rd edition
[37]Accurately computing sample variance online. Johndcook.com. https://www.johndcook.com/blog/standard_deviation/
[38]Trilles, S., Schade, S., Belmonte, Ó., & Huerta, J. (2015). Real-Time Anomaly Detection from Environmental Data Streams. Lecture Notes In Geoinformation And Cartography, 125-144. http://dx.doi.org/10.1007/978-3-319-16787-9_8
[39]Keeping the Process on Target: CUSUM Charts | BPI Consulting. (2017). Spcforexcel.com. from:https://www.spcforexcel.com/knowledge/variable-control-charts/keeping-process-target-cusum-charts
[40]Breunig, M., Kriegel, H., Ng, R., & Sander, J. (2000). LOF. Proceedings Of The 2000 ACM SIGMOD International Conference On Management Of Data - SIGMOD '00. http://dx.doi.org/10.1145/342009.335388
[41]Das, T., & Zhu, S. (2017). Faster Stateful Stream Processing in Apache Spark Streaming. Databricks. from:https://databricks.com/blog/2016/02/01/faster-stateful-stream-processing-in-apache-spark-streaming.html
[42]Hayes, M., & Capretz, M. (2015). Contextual anomaly detection framework for big sensor data. Journal Of Big Data, 2(1). http://dx.doi.org/10.1186/s40537-014-0011-y