臺灣博碩士論文加值系統

根據TWNIC 2005年1月中旬之調查報告指出，臺灣地區上網人口成長約1,380萬人，上網的家庭已達463萬戶，普及率高達65.02%，故Internet不僅已成為一個強大的媒體平臺，它也變成每個企業不可或缺的重要通路。企業都思索如何應用此通路來收集客戶瀏灠行為資料及維護客戶關係，找出顧客的真正需求，提昇服務品質和滿意度，以鞏固顧客忠誠度，使企業能長期從客戶價值中獲利。
但是要從網站的記錄來分析客戶瀏灠行為，在實務上還是有許多難題待解: (1) 網站的記錄，資訊是片斷分散的資訊，先天上資料並不夠完整。(2)網站的記錄的量通常都非常大，如何萃取、轉換成有用的資訊都是挑戰。(3)如何挖掘出對企業有用的知識也是難題。
本研究的貢獻是: (1)提出一套實用又簡單的分析架構可以有效的分析網站的記錄，對上述難題提供解決方案，(2)同時還提出一項演算法，它可算出網頁被點閱的熱門程度，進一步繪製出網站二維的瀏灠地圖，從視覺化呈現探勘結果，讓解釋上更明瞭易懂。應用本研究之分析架構和準則，就能從分析網站的記錄中萃取出網友的瀏灠行為，可協助企業多瞭解客戶，獲知客戶喜好有關的知識。
其結果可用於改善: (1)網站結構設計、(2)網頁瀏灠動線設計、(3)也可以用來分析單一客戶、一群特定目標客戶、或是全站客戶點閱記錄。從分析結果中，就能得到熱門網頁及客戶喜好的瀏灠動線，有了這些資訊後，只要再加入網站較高層次的語意資訊，例如:那些網頁是與購物車相關網頁、那些是檢索服務入口網頁等資訊，就可獲得許多網站經營上的實務知識。
本研究並以一個實際在線上服務的內容網站，從2005年3月至4月，一個月期間約六千六百萬筆記錄，共1.26GB大小的原始網站日誌資料為分析對象，經處理後選取其中三個最具代表性的典型資料為範例，來實證本研究所提的分析架構和演算法的適用性。

According to the survey report, issued by TWNIC Jan. 2005, Internet popularity had grown to 13,800,000 users, about 4,630,000 home families, approaching 65% of whole families in Taiwan. Therefore, the Internet not only is a powerful media, but also become an important channel to enterprises. All enterprises are eager to find out a useful way to synergize such a powerful channel. They have been trying to analyze the visiting log of the web, and mine the behavior of customers who had contacted the enterprise through the Internet, willing to collect more customer information and provide more personalized services to customers. However, in practicality, there are some difficulties encountered. The First is the web logs are distributed information, which are separated on several servers, and need to be integrated and do lots of processing. Secondary, one of the difficulties is how to extract the key features from the huge logs, and how to solve the scalability issues. The third problem is how to find the suitable mining tools to discover the implicit knowledge from bunch of irrelevant raw data.
Our research proposes a novel framework, which integrates most useful public domain resources and some self-developed tools, provides powerful analyzing tools to overcome such difficulties. This thesis also illustrates a novel algorithm to visualize click-stream mining result, named “Click-map”. This presentation is able to assist the web master to discover users’ navigation behaviors from the click path analysis more easily.
For examining the availability of the framework and analysis methods, we use online web logs for the period of one month as examples. The logs came from an online content search services site, with 1.26GB data size and over 66 million records, recorded from March to April in 2005. The results proofed our framework to be useful and effective.

目錄


謝詞 一
中文摘要 二
英文摘要 三

第一章 緒論 1
第一節 研究背景 1
第二節 研究動機 4
第三節 研究目的 7
第四節 研究範圍 10
第五節 章節結構 11


第二章 文獻探討 12
第一節資料探勘與網路探勘的異同 13
第二節網路探勘的分類與應用 16
第三節網頁使用探勘的執行步驟 18
第四節網頁使用探勘技術分析 20
第五節網路探勘文獻評述 23


第三章 研究方法 25
第一節 研究架構 25
第二節 探勘資料源網站日誌檔格式 27
第三節 資料前處理過程及工具 31
3.3.1資料清除程序 32
3.3.2網址代碼化處理程序 33
3.3.3區段化處理程序 33
3.3.4將資料轉換成入庫格式之處理程序 36
3.3.5從資料庫將探勘資料匯出轉換程序 40
第四節 網路探勘工具和探勘資料準備 42

3.4.1 WEKA的探勘資料格式-ARFF 43
3.4.2 點閱路徑分析 43
3.4.3以 Click-map視覺化呈現點閱路徑 44


第四章 實證分析 50
第一節 本研究對象說明 50
第二節 以關聯法則探勘器挖掘隱性知識 53
第三節 以路徑縱灠探勘器挖掘隱性知識 56
4.3.1 分析所有身份可識別者的集體點閱行為 56
4.3.2 分析個人的點閱行為 58
4.3.3 分析單週全站集體點閱記錄以挖掘熱門路徑 60


第五章 結論及未來工作 65
第一節 結論及貢獻 65
第二節 研究限制 69
第二節 未來發展建議 70


參考文獻 71
相關網站 75
附錄一 把點閱資料區段化(Sessionize)的三種經驗計算 76
附錄二 本研究所提的網站探勘資料庫設計架構 79
附錄三 資料庫處理Store Procedure spWUMT00程式碼 82
附錄四 Chen(1998) MFR演算法範例 86
附錄五 以PERL程式語言實作MFR演算法程式碼 90
附錄六 本研究網站vip.tol.com.tw網址與網頁代碼對應表 93
附錄七 把網站探勘資料轉換成 ARFF格式之範例 94
附錄八 以PERL程式語言實作Click-map演算法程式碼 96

參考文獻

一、中文部分
丁一賢(2001),"運用網頁探勘為基礎的個人化技術於網路廣告之探討",彰化師範大學,資訊管理學系,碩士論文,民國90年。

工商時報(2005), 2005年2月22日十三版 科技產銷報導, http://www.tol.com.tw/

李宜懃譯(2000)「位元風暴」，天下文化公司，原著 P. Evans and T. S. Wurster, Blown to Bits, Boston Consulting Group, Inc., 2000

季延平譯(2000)「網際網路行銷」，華泰文化公司，原著 W. Hanson, Internet Marketing, International Thomson Publishing Co., 2000。

邱如美(2003), 「下一個經濟盛世 (The Next Economy) 」原著 Elliott Ettenberg, 天下雜誌, 2003年6月初版

林義淵(1997) ”一個應用於全球資訊網中資訊勘測之資料收集機制的設計與實作”, 國立台灣大學,電機工程學系研究所,碩士論文,民國86年。

陳世訓(2001),"網站日誌探勘的研究與實作",國立臺灣大學,資訊工程學研究所,碩士論文,民國90年

黃照貴(2000),"導入網際探勘技術以支援電子商務經營-競爭策略與效益分析",國立成功大學,企業管理學系,博士論文,民國89年。

黃雅慧(2002),"應用網站探勘於網站瀏覽之個人化－以健康檢查業為例"東吳大學,資訊科學系,碩士論文,民國91年

趙景明(2003)，趙景明、黃雅慧 “應用網頁探勘於網站瀏灠之個人化-以醫療產業為例”, 中原學報，第31卷，第3期，pp. 271-282，2003年

盧木賢(2002),"資料採掘應用於Web Marketing",淡江大學,資訊工程學系,碩士論文,民國91年

嚴久欽(2001),"實作序列型樣探勘系統並應用到網站日誌",國立臺灣大學,資訊工程學研究所,碩士論文,民國90年


二、英文部份
[Agrawal 1994] R. Agrawal, and R. Srikant, “Fast Algorithms for mining association rules.“, VLDB-94, 1994.
[Araya 2004] S. Araya, M. Silva, and R. Weber, “A methodology for web usage mining and its application to target group identification.“, Fuzzy Sets and System, Vol., 148, pp.139-152, 2004.
[Asia WSJ 2005] K. J. Delaney, “In Hunt for Online Ads, Yahoo Makes a Big Bet On Media-Based Tack.“, Asia Wall Street Journal, March 2, p. A1, 2005.
[Baldi 2003] P. Baldi, P. Frasconi, and P. Smyth, Modeling the Internet and the Web, John Wiely & Sons Ltd., 2003.
[Berendt 2000] B. Berendt, and M. Spiliopoulou,"Analysis of navigation behaviour in web sites integrating multiple information systems",The VLDB Journal (2000) 9: 56–75.
[Berendt 2002a] B. Berendt, B. Mobasher, M. Nakagawa and M. Spiliopoulou. "The Impact of Site Structure and User Environment on Session Reconstruction in Web Usage Mining", WEBKDD 2002, pp 159-179, 2002.
[Berendt 2002b] B. Berendt, B. Mobasher, and M. Spiliopoulou. "Web Usage Mining for E-Business Application", ECML/PKDD Helsinki, Finland, 19, Aug., 2002.
[Brin 1998] S. Brin, and L. Page. "The Anatomy of a Large-Scale Hypertextual Web Search Engine", in Proceedings of World-Wide Web ''98 (WWW7), [Online: http://www-db.stanford.edu/pub/papers/google.pdf ], April 1998.
[BusinessWeek 2005] BusinessWeek report, "Keywords for AD, Buyers: Pay Up", BusinessWeek, Feb. 21, p33 , 2005
[Chen 1996] M-S. Chen, J .S. Park, and P.S. Yu. "Data Mining for Path Traversal Patterns in a Web Environment." In Proceedings of the 16th International Conference on Distributed Computing Systems,pp. 385–392, 1996.
[Chen 1998] M-S Chen, J. S. Park, and P. S. Yu, “Efficient Data Mining for Path Traversal Patterns”, IEEE Trans. Knowledge Data Eng. 10 (2), pp. 209-221, 1998.
[Cooley 1999] R. Cooley, B. Mobasher, and J. Srivastava,”Data Preparation for Mining World Wide Web Browsing Patterns”, Knowledge and Information System, 1(1), pp 5-32, 1999.
[Cooley 2000] R. Cooley, “Web usage mining: discovery and application of interesting patterns from web data”, PhD. dissertation, Univ. Minnesota, 2000.
[Cooley 2003] R.Cooley, “The Use of Web Structure and Content to Identify Subjectively Interesting Web Usage Patterns.”, ACM Transactions on Internet Technology, 3(2), pp. 93-116, 2003.
[Eirinaki 2003] M. Eirinaki, and M. Vazirgiannis, “Web Mining for Web Personalization.”, ACM Transaction on Internet Technology, Vol. 3, No. 1, pp. 1-27, 2003.
[Gaul 2000] W. Gaul, and L. Schmidt-Thieme, “Mining web navigation path fragments”, In Workshop on Web Mining for E-Commerce - Challenges and Opportunities Working Notes (KDD2000), pp. 105-110, Boston, MA, August, 2000.
[Han 2001] J Han, and M. Kamber, Data Mining: concepts and Techniques., Morgan Kaufmann Publishers, Academic Press 2001.
[Heller 2001] R. Heller, Andrew Grove., Dorling Kindersley Limited, London 2001.
[Hu 2004] X. Hu, and N. Cercone “A Data Warehouse/OLAP Framework for Web Usage Mining and Business Intelligence Reporting”, International Journal of Intelligence System, 19(7), pp. 567-584, 2004.
[Huang 2001] Z. Huang, J. Ng, D. W. Cheung, M. Ng, and W. K. Ching “A Cube Model for Web Access Sessions and Cluster Analysis”, WEBKDD 2001, San Francisco, Aug. 2001.
[Kamdar 2000] T. Kamdar, and A. Joshi, ”On Creating Adaptive Web Servers Using Weblog Mining”, Technical Report TR-CS-00-05, 20, Nov. 2000.
[Kantardzic 2003] M. Kantardzic, Data Mining: Concpets, Models, Methods, and Algorithms, IEEE Press. Wiley-Interscience 2003.
[Kohavi 2004] R. Kohavi, L. Mason, and Z. Zheng ”Lessons and Challenges from Mining Retail E-Commerce Data”, Machine Learning Journal, Special Issue onn Data Mining Lessons Learned, 2004
[Kohavi 2001] R. Kohavi ”Mining E-Commerce Data: The Good, the Bad, and the Ugly”, ACM KDD’01, San Francisco, CA, 2001
[Kosala 2000] R. Kosala ”Web Mining Research: A Survey”, ACM SIGKDD, Vol. 2, (1), pp. 1-15, 2000
[Liu 1999] B. Liu, W. Hsu, and Y. Ma, ”Mining Association Rules with Multiple Minimum Supports”, ACM KDD-99, August 15-18, 1999, San Diego, CA, USA.
[Li 2004] H-F. Li, S-Y. Lee, and M-K. Shan, ”On Mining Webclick Streams for Path Traversal Patterns”, ACM WWW 2004, May 17-22, 2004, New York, USA.
[Mobasher 1999] B. Mobasher, R. Cooley, and J. Srivastava, “Creating adaptive web sites through usage-based clustering of urls”, In IEEE Knowledge and Data Engineering Workshop (KDEX’99), November 1999.
[Mobasher 2000] B. Mobasher, R. Cooley, and J. Srivastava.”Automatic personalization based on web usage mining”. In Communications of the ACM, (43) 8, August 2000.
[Mobasher 2000a] B. Mobasher, H. Dai, T. Luo, M. Nakagawa, Y. Sun, and J. Wiltshire, ”Discovery of aggregate usage profiles for Web personalization”, In Proceedings of the WebKDD 2000 Workshop at the ACM SIGKKD 2000, Boston, August 2000.
[Mobasher 2001a] B. Mobasher, H. Dai, T. Luo, and M. Nakagawa, “Effective Personalization Based on Association Rule Discovery from Web Usage Data ”, WIDM01 , 3rd ACM Workshop on Web Information and Data Management, November 9, 2001, Atlanta, Georgia, USA.
[Mobasher 2001b] Bamshad Mobasher, Bettina Berendt and Myra Spiliopoulou, “KDD for Personalization”, PKDD 2001 Tutorial September 6, 2001
[Mobasher 2004] Bamshad Mobasher, “Web Usage Mining and Personalization”, Draft Chapter in Practical Handbook of Internet Computing, Munindar P. Singh ed., CRC Press, 2004
[Resnick 1994] Resnick, P., Iacovou, N., Suchak, M., Bergstrom, P., and Riedl, J. Grouplens: An open architecture for Web Usage Mining of netnews. In Proceedings of ACM CSCW''94 Conference on Computer-Supported Cooperative Work, Sharing Information and Creating Meaning, pp. 175-186., 1994
[Rantzau 1997] R. Rantzau, “Extended Concepts for Association Rule Discovery.”, University of Stuttgart, Faculty of Computer Science, Diploma Thesis No. 1554 (1997).61 pages, English.
ftp://ftp.informatik.uni-stuttgart.de/pub/library/medoc.ustuttgart_fi/DIP-1554/DIP-1554.pdf
[Smith 2002] K. Smith, and Alan. Ng, “Web page clustering using a self-organizing map of user navigation patterns”, Decision Support System 35, pp. 245-256, 2002.
[Shahabi 1997] C. Shahabi, A. M. Zarkesh, J. Adibi, and V. Shah, “Knowledge Discovery from Users Web-Page Navigation”, Proceedings of the 7th International Workshop on Research Issues in Data Engineering (RIDE ''97) High Performance Database Management for Large-Scale Applications, 1997.
[Shahabi 2001] C. Shahabi and F. Banaei-Kashani, “A Framework for Efficient and Anonymous Web Usage Mining Based on Client-Side Tracking”, Proceedings of WEBKDD 3rd International Workshop, San Francisco,CA, USA, Aug. 26, 2001.
[Spiliopoulou 1999] M. Spiliopoulou, L. C. Faulstich, and K. Winkler “A Data Miner analyzing the Navigational Behavior of Web”, In Proc. of the Workshop on Machine Learning in User Modelling of the ACAI99, Greece, 1999.
[Spiliopoulou 2000] M. Spiliopoulou,"Web Usage Mining forWeb Site Evaluation", ACM Communications, 43(8), 2000
[Srivastava 2000] J. Srivastava, R. Cooley, M. Deshpande, and P. Tan “Web Usage Mining: Discovery and Applications of Usage Patterns from Web Data”, ACM SIGKDD, Vol. 1, No.2, pp 12-23, 2000.
[Srivastava 2003] J. Srivastava, P. Desikan, and V. Kumar, “Web Mining – Accomplishments and Future Directions”, Proceedings US Int’l Science Foundation Workshop on , 2003. http://www.ieee.org.ar/downloads/Srivastava-tut-paper.pdf
[Sweiger 2002] M. Sweiger, M. R. Madsen, J. Langston, and H. Lombard, Clickstream Data Warehousing, John Wiely & Sons, Inc., 2002
[Wang 2004] L. Wang, and C. Meinel “Behavior Recovery and Complicated Pattern Definition in Web Usage Mining”, ICWE 2004, LNCS 3140, pp. 531-543, 2004. (N. Koch, P. Fraternali and M. Wirsing (Eds.), Springer-Verlag Berlin Heidelberg 2004.)
[Witten 1999] I. H. Witten, and E. Frank, Data Mining: Practical Machine Learning Tools and Techniques with Java Implementations, Morgan Kaufmann, 1999, 416 pages, ISBN 1-55860-552-5.
[Xiao 2001] Y. Xiao, and M. H. Dunham, “Efficient mining of traversal patterns”, Data and Knowledge Eng. 39, pp. 191-214, 2001.