臺灣博碩士論文加值系統

在ETSI的標準裡，每筆從感測器收集到的資料，在它們被處理之前都會先存於平台的資料庫裡頭。換言之，那些收集到的資訊不只會佔用網路的頻寬，也同時會持有部分的平台儲存資源。此外，在ETSI裡，那些感測數據並不只是像傳統的SQL架構的方式做儲存，而是以更複雜的資源樹的架構做儲存，用來描述層級式的屬性以達成更好的資料管理；也因此需要夠多的資源來管理這樣的資料結構。然而，那些收集到的資料，對於應用來說，並不是所有的資訊都是有用的；事實上，有些資訊其實是多餘且累贅的，而不應分配過多的資源予它們。
在此論文研究裡，我們提出先行處理這些串流資訊，接著決定資料的保存與否。主要的理念是根據資料間不同的本質去進行全然不同地處理。像是，在最一開始，我們可以試著過濾出多餘、無用或是錯誤的資訊；而對於那些不全然有用的資訊，在它們被送入伺服器之前，我們應小心地精簡化，並取出主要的數據加以儲存和更進一步地處理。至於那些十分重要的資訊，我們應馬上處理它們且盡快地採取相對應的行動措施，然後如果有需要則必須完整地將它們儲存在資料庫裡。
藉由過濾及先行處理那些資訊，可節省大量的資源。透過減少物聯網感測器所收集到的資訊其資料傳輸、資料儲存、資料管理及處理成本，進而達到增進物聯網平台的效率及高度使用率的目標，而不浪費資源成本在無用的資料上。
此研究將會著重於如何使得在ETSI M2M標準兼容的物聯網平台 ─ OpenMTC裡進行高速且大量的資料處理。為了比較我們與傳統在處理資料的方法上之差異，將會使用工廠管理作為展示在成本和效率方面的結果範例。在成本的分析上，我們將會測量各方法所需的儲存空間與資料傳輸量。而對於效能分析，我們將會觀察兩種方法在中央處理器與記憶體上使用量的不同之處。我們將會展示我們處理資料的方法可以大幅地改善在ETSI M2M架構裡對於高速且大量的資料處理。

In the ETSI standard, each data collected by sensors would be stored in database before they are processed. In other words, those collecting data will not only occupy bandwidth of the internet, but also hold some storage resources in the platform. Furthermore, in ETSI the sensing data would not be stored like traditional format as SQL-like structure, but become more complicate one - resource tree which describes hierarchical attributes for better data management. Consequently, it takes more efforts to manage IoT data stored in the IoT platform. However, for those collected data, not all of them are really useful for the applications. Some of them, actually, are redundant and doesn’t need to be allocated too much resource.
In this thesis research, we propose to process streaming data first then determine whether the data should be kept or not. The idea is to treat data differently based on their different nature. For example, we can filter redundant, useless and fallacious data out in the very beginning. Then for those not completely useful data, we can refine them carefully and take key values out for storage and further processing before they are sent to the server. As for important data, we can process them and take immediate action as soon as possible, then fully store them in the database if needed.
By filtering and pre-processing those data, it can save lots of resources by reducing data transmission, data storage and data management and processing overhead for those data collected by IoT sensors, the goal is to make an IoT platform more efficient and highly utilized without wasting the resources on useless data.
This research will focus on how to enable big velocity and large volume data processing in an ETSI M2M standard compliant IoT platform - OpenMTC. To compare the differences between our approach and traditional approach of handling data, a use case from factory management will be used to demonstrate the results in terms of cost and efficiency. For the cost analysis, we will measure the storage space and the data transmission volume required for each approach. For the efficiency analysis, we will observe the difference between these two approaches in terms of their cpu and memory usage. We are going to demonstrate our approach of handling data can largely improve big velocity and large volume data processing in an ETSI M2M architecture.

Chapter 1. Introduction 1
1.1 Motivation 1
1.2 Contribution 1
1.3 Thesis Organization 1
Chapter 2. Background 3
2.1 Streaming Data Processing 3
2.1.1 Characteristics of Big Data in IoT/M2M Environment 3
2.1.2 Stream Processing 4
2.2 Technologies 6
2.2.1 Representational State Transfer 6
2.2.2 Machine to Machine Communications 7
2.2.3 M2M Horizontal common service layer 8
2.3 Summary 10
Chapter 3. Related Work 11
3.1 Data preprocessing method based on user characteristic of interests for Web log mining 11
3.2 Research on Data Preprocessing in Supermarket Customers Data Mining 11
3.3 Preprocessing and Symbolic Representation of Stock Data 11
3.4 Web Log Data Preprocessing Based on Collaborative Filtering 11
3.5 Research on Data Preprocessing Technology in Safety Equipment Linkage System 11
3.6 A New Data Filtering Scheme Based on Statistical Data Analysis for Monitoring Systems in Wireless Sensor Networks 12
3.7 A Sampling-based Data Filtering Scheme for Reducing Energy Consumption in Wireless Sensor Networks 12
3.8 Data Filtering for Wireless Sensor Networks Using Forecasting and Value of Information 12
3.9 Summary 13
Chapter 4. M2M Fast Velocity and Large Volume Data processing 14
4.1 Proposed Method 14
4.1.1 Streaming Data Processor 15
4.1.2 Data Specification 16
4.1.3 System Feature 16
4.1.4 Parser 18
4.1.5 Processing Engine 19
4.1.6 Filter 19
4.2 Use Case Description 19
4.2.1 Factory Management Assumptions 19
4.2.2 System Features of Factory Management 21
4.2.3 System Feature Specifications 22
Chapter 5. Experimental Results 26
5.1 Experimental Environment 26
5.1.1 OpenMTC Release 3 26
5.1.2 IBM InfoSphere 26
5.2 Experimental Results 27
Chapter 6. Conclusion and Future work 30
References 31
Appendix A: Streaming Data Processor Implementation 33
A.1 Lex and Patterns 33
A.2 Yacc and Grammars 34
A.3 Process Engine Building Procedure 36
A.4 Processing Handler 36
Appendix B: Factory Management Data Simulator 38
B.1 Factory Management Simulation Assumptions 38
B.2 Factory Management Simulation 38
B.3 Data Generator Design and Architecture Diagram 38
B.4 Implementation Tool – IBM InfoSphere 41
B.5 Entering Process 42
B.5 Leaving Process 44
B.6 Abnormal Case Simulation 46
B.7 Interaction between IoT/M2M platform and Data Generator 48

[1] IBM The Four V’s of Big Data, Available: http://www.ibmbigdatahub.com/infographic/four-vs-big-data.
[2] IBM Stream Processing, Available: http://www-03.ibm.com/systems/infrastructure/us/en/technical-breakthroughs/stream-processing.html.
[3] Apache Storm, Available: https://storm.apache.org/
[4] Leonard Richardson and Sam Ruby, RESTful Web Services, O'Reilly Media, May 2007.
[5] European Telecommunications Standard Institute TS 102 690, "Machine-to-Machine Communications (M2M); Functional architecture”, October 2014 [Online]. Available: http://www.etsi.org/deliver/etsi_ts/102600_102699/102690/02.01.01_60/ts_102690v020101p.pdf
[6] Ying Han, Kejian Xia, “Data preprocessing method based on user characteristic of interests for Web log mining”, Instrumentation and Measurement, Computer, Communication and Control (IMCCC), 2014 Fourth International Conference on, Harbin, Sept. 2014, pp. 867-872.
[7] Wei Jianping, “Research on Data Preprocessing in Supermarket Customers Data Mining”, Information Engineering and Computer Science (ICIECS), 2010 2nd International Conference on, Wuhan, Dec. 2010, pp. 1-4.
[8] Mukesh Kumar, Arvind Kalia, “Preprocessing and Symbolic Representation of Stock Data”, Advanced Computing &; Communication Technologies (ACCT), 2012 Second International Conference on, Rohtak, Haryana, Jan. 2012, pp. 83-88.
[9] JIANG Chang-bin, Chen Li, “Web Log Data Preprocessing Based on Collaborative Filtering”, Education Technology and Computer Science (ETCS), 2010 Second International Workshop on, Wuhan, March 2010, pp. 118-121.
[10] Xiaorong Cheng, Hui Liu, “Research on Data Preprocessing Technology in Safety Equipment Linkage System”, Computational and Information Sciences (ICCIS), 2013 Fifth International Conference on, Shiyang, June 2013, pp. 1713-1716.
[11] Seung Tae Hong, Jae Woo Chang, “A New Data Filtering Scheme Based on Statistical Data Analysis for Monitoring Systems in Wireless Sensor Networks”, High Performance Computing and Communications (HPCC), 2011 IEEE 13th International Conference on, Banff, AB, Sept. 2011, pp. 635-640.
[12] Seung Tae Hong, Byeong-Seok Oh, Jae Woo Chang, “A Sampling-based Data Filtering Scheme for Reducing Energy Consumption in Wireless Sensor Networks”, Services Computing Conference (APSCC), 2011 IEEE Asia-Pacific, Jeju Island, Dec. 2011, pp. 353-359.
[13] Sebastian Z¨oller, Christian Vollmer, Markus Wachtel, Ralf Steinmetz, Andreas Reinhardt, “Data filtering for wireless sensor networks using forecasting and value of information”, Local Computer Networks (LCN), 2013 IEEE 38th Conference on, Sydney, NSW, Oct. 2013, pp. 441-449.
[14] PLY (Python Lex-Yacc), Available: http://www.dabeaz.com/ply/
[15] Fraunhofer FOKUS, (June 2014) "The OpenMTC Platform", [Online]. Available: http://www.open-mtc.org.
[16] IBM, (July 2014), “InfoSphere Platform”, [Online]. Available: http://www-01.ibm.com/software/data/infosphere
[17] IBM White Paper, “IBM Infosphere Streams: Redefining Real Time Analytics”, Apr 2014, Available: https://www-01.ibm.com/marketing/iwm/iwm/web/signup.do?source=sw-infomgt&;S_PKG=ov4110&;S_TACT=109HF53W&;S_CMP=is_wp67