臺灣博碩士論文加值系統

2015年全球最具權威的IT研究與顧問咨詢公司Gartner在奧蘭多(Orlando)召開年度科技研討會(Gartner Symposium/ITxpo)，Gartner介紹關於未來的十大戰略性技術趨勢預測，機器學習被評選為未來科技潮流之一。
而ICT(Information and Communication Technology)產業將物聯網、機器學習視為次世代重點的科技，這些科技都是以大數據為基礎，誰先佔有大數據，便能領先同業，如此可知大數據對於未來趨勢科技的重要性，發展潛力也讓各大國際企業加快腳步搶進市場。
Apache Spark提供快速的邏輯處理與In Memory技術跟叢集運算框架，並提供各種的機器學習演算法，因此研究Apache Spark的決策樹二元分類演算法、支援向量機器SVM二元分類演算法經由作業系統虛擬化技術，將本機電腦增加多執行緒運算方式跟增加多台電腦叢集運算架構，在相同數據資料筆數增加情況下進行速度與準確度的差異分析效能比較。

In 2015, machine learning has been selected by Gartner as one of the top ten strategic technologies of the future in the Technology of the Year Seminar (Gartner Symposium / ITxpo) held in Orlando. In addition, IoT (Internet of Things) and machine learning are also considered the essential next-generation technologies by the ICT (Information and Communication Technology) industries, which both based on the big data analysis techniques. It is believed that whoever holds the edge in big data may lead in the industry. Accordingly, the potential of big data has attracted major international companies into this market.
In this thesis, we investigate the Apache Spark, which provides fast processing of various machine learning algorithms through its In Memory technology and cluster computing framework. In particular, performance of two algorithms, decision tree and support vector machine, will be evaluated in the contexts of multi-threading and multi-host environments. Big data re-generated through date set collected for the diagnosis of students with learning disabilities will be used as the test samples for the evaluation.
Keywords: Machine learning, Big Data, Apache Spark, decision tree, Support Vector Machine.

中文摘要 I
ABSTRACT II
誌謝 IV
目錄 V
圖索引 VI
表索引 VIII
第一章 緒論 1
第一節 研究背景 1
第二節 研究動機與目的 3
第三節 論文架構 6
第二章 文獻探討 7
第一節 大數據(BIG DATA) 7
第二節 APACHE SPARK 9
第三節 機器學習 17
第四節 SCALA 20
第三章 研究方法 21
第一節 機器硬體規格表 21
第二節 軟體介紹 23
第三節 系統實作說明 24
第四章 測試效能分析 29
第一節 本機增加執行緒 29
第二節 叢集架構 37
第五章 結論與未來展望 45
第一節 結論 45
第二節 未來展望 46
第六章 參考文獻 47

中文部分：
施伯燁. (2014). 社群媒體-使用者研究之概念, 方法與方法論初探. 傳播研究與實踐, 4(2), 207-227.
劉志強, 顧榮, 袁春風, & 黃宜華. (2015). 基於 Spark 的分類算法並行化研究. Journal of Frontiers of Computer Science and Technology, 9(11), 1281-1294
陳淑芬. (2014). 大數據時代來臨!. 禪天下, (117), 62-65.
蔡維哲, & 虞孝成. (2010). 物聯網發展趨勢與政府政策研究(Doctoral dissertation).
Tai, T. L. (2016). 可執行於多個 Spark 上之彈性分散式資料集. 成功大學電腦與通信工程研究所學位論文, 1-37.
詹義方 媛. (2016). 基於 Spark 技術的網絡大數據分析平台搭建與應用. 互聯網天地, 13(2), 75-78.
支援向量機於信用評等之應用. 計量管理期刊, 2004, 1.第 2: 155-172.
林大貴.(2015).Hadop+Spark大數據巨量分析與機器學習整合開發實戰


英文部分：
Zaharia, M. A. (2013). An Architecture for and Fast and General Data Processing on Large Clusters.
Meng, X., Bradley, J., Yavuz, B., Sparks, E., Venkataraman, S., Liu, D., ... & Xin, D. (2015). Mllib: Machine learning in apache spark. arXiv preprint arXiv:1505.06807.
Xin, R., Deyhim, P., Ghodsi, A., Meng, X., & Zaharia, M. (2014). GraySort on Apache Spark by Databricks. GraySort Competition.
Wiewiórka, M. S., Messina, A., Pacholewska, A., Maffioletti, S., Gawrysiak, P., & Okoniewski, M. J. (2014). SparkSeq: fast, scalable, cloud-ready tool for the interactive genomic data analysis with nucleotide precision. Bioinformatics, btu343.
Gopalani, S., & Arora, R. (2015). Comparing Apache Spark and Map Reduce with Performance Analysis using K-Means.International Journal of Computer Applications, 113(1).
Zheng, J., & Dagnino, A. (2014, October). An initial study of predictive machine learning analytics on large volumes of historical data for power system applications. In Big Data (Big Data), 2014 IEEE International Conference on (pp. 952-959). IEEE.
Solaimani, M., Iftekhar, M., Khan, L., & Thuraisingham, B. (2014, October). Statistical technique for online anomaly detection using spark over heterogeneous data from multi-source VMware performance data. In Big Data (Big Data), 2014 IEEE International Conference on (pp. 1086-1094). IEEE.
Lin, C. Y., Tsai, C. H., Lee, C. P., & Lin, C. J. (2014, October). Large-scale logistic regression and linear support vector machines using Spark. In Big Data (Big Data), 2014 IEEE International Conference on (pp. 519-528). IEEE.
Odersky, M., Spoon, L., & Venners, B. (2008). Programming in scala. Artima Inc.
Chiusano, P., & Bjarnason, R. (2014). Functional programming in Scala. Manning Publications Co..
Zaharia, M., Chowdhury, M., Franklin, M. J., Shenker, S., & Stoica, I. (2010). Spark: Cluster Computing with Working Sets.HotCloud, 10, 10-10.
Pfeffer, A. (2009). Figaro: An object-oriented probabilistic programming language. Charles River Analytics Technical Report, 137.
Kohavi, R. (1996, August). Scaling Up the Accuracy of Naive-Bayes Classifiers: A Decision-Tree Hybrid. In KDD (Vol. 96, pp. 202-207).
Safavian, S. R., & Landgrebe, D. (1990). A survey of decision tree classifier methodology.
Bishop, C. M. (2006). Pattern Recognition. Machine Learning.
Cortes, C., & Vapnik, V. (1995). Support-vector networks.Machine learning, 20(3), 273-297.


網頁部分：
Apache.(2014).The Apache Software Foundation Blog.，網址：
https://blogs.apache.org/foundation/entry/the_apache_software_foundation_announces50
Spark 編程指南繁體中文版. (2015). – GitBook.，網址：
https://taiwansparkusergroup.gitbooks.io/spark-programming-guide-zh-tw/content/mllib/index.html
Apache Spark Blog.(2014).http://spark.apache.org/
石山園.(2015).Spark 入門實戰系列.，網址：
http://www.cnblogs.com/shishanyuan/p/4747761.html
李欣宜.(2015).[大數據怎麼做？] 美國7-Eleven主動出擊送優惠，唯快不破.，網址：http://www.bnext.com.tw/article/view/id/35848
工研院IEK.(2016).2016十大ICT產業關鍵議題.，網址：
https://www.itri.org.tw/chi/Content/NewsLetter/contents.aspx?&SiteID=1&MmmID=620605426331276153&MSID=707246230323432252&PageID=20
科技產業資訊室–Hana.(2016).大數據是次世代科技之根本技術.，網址：
http://iknow.stpi.narl.org.tw/post/Read.aspx?PostID=12007
科技產業資訊室–Hana.(2015).大數據帶動機器學習形成IT新戰場.，網址：
http://iknow.stpi.narl.org.tw/post/Read.aspx?PostID=10954
余至浩.(2014).Spark擊敗Hadoop刷新資料排序世界記錄.，網址：
http://www.ithome.com.tw/news/92449
Sort Benchmark.(2014). ，網址：http://sortbenchmark.org/
林妍溱.(2015).IBM加入Apache Spark社群，打算培育百萬名資料科學家：
可能是未來十年最重要的開放源碼計畫.，網址：
http://www.ithome.com.tw/news/96777
馬岳琳.(2015).小米雷軍：不被恥笑的夢想就不是夢想.，網址：
http://www.cw.com.tw/article/article.action?id=5063785
彭凡.(2014). 解析Spark在騰訊、雅虎、優酷的成功應用.，網址：
http://database.51cto.com/art/201406/442055.htm
Stack Overflow.(2015).2015 Developer Survey.，網址：
http://stackoverflow.com/research/developer-survey-2015