臺灣博碩士論文加值系統

目前的垃圾郵件過濾器技術常面臨一個瓶頸，就是在追求提高垃圾郵件辨識率的同時，也使得正常郵件被誤判為垃圾郵件的可能性上升。這是因為每位使用者所認定的正常郵件皆不相同，並沒有一套泛用的規則能夠適用於所有的使用者。為此，本論文提出一個雙層的垃圾郵件過濾方法，在傳統的單層垃圾郵件過濾架構中植入一個正常郵件過濾器。這個過濾器能夠依據使用者收件匣的內容迅速、自動地建立與更新正常郵件規則。對於與使用者的正常郵件相似度高的來信，正常郵件過濾器會直接送到使用者的信箱，其他的郵件則交給垃圾郵件過濾器進一步判定，以達到降低垃圾郵件誤判率之效果。在建立郵件規則部分，我們使用常見於資料探勘與資料檢索的TF-IDF加權方法來評估字詞在文件中的重要性，並整合MMSEG斷詞程式來負責中文斷詞。我們將此雙層過濾方法實作在郵件伺服器端，並另外設置一個裝載單層垃圾郵件過濾之郵件伺服器作為對照組，比較兩者的過濾情形。實驗結果顯示雙層式郵件過濾方法幾乎消弭正常郵件被誤判的機率，同時僅提高極少量的false negative rate。

In order to increase the filtering rate, current spam mail filters usually suffer the side effect of increasing false positive rate. One reason to this bottleneck is that, comparing to spam mail, the definition of white mail depends on the receiver, and thus the characteristics of white mail vary. In this thesis, a two-tier architecture is proposed to solve this bottleneck. A rule-based white mail filter is inserted in front of spam mail filter, and the rules it uses to filter are automatically built from existing white mails in personal mail folders. For incoming mails that show high relativity with existing white mail, the white mail filter directs them to the mail folders. The other mails are forwarded to spam mail filter for further check. To construct the white mail rules for the experimental system, TF-IDF method is used to evaluate the importance of words in documents, and the system also integrates MMSEG module to segment Chinese sentences and to identify words. Experiments are performed on two mail servers with one equipped ordinary spam mail filter, and another equipped both white mail filter and spam mail filter. The results of the experiment show that the two-stage mail filtering method almost eliminates false positive cases with very small increase to false negative rate.

摘要........................................................I
Abstract....................................................II
誌謝........................................................III
目錄........................................................IV
圖目錄......................................................VI
表目錄......................................................VII
第一章 緒論.................................................1
1.1 前言...................................................1
1.2 研究背景與動機.........................................1
1.3 研究目的...............................................2
1.4 論文架構...............................................2
第二章 文獻探討.............................................3
2.1 郵件通訊協定...........................................3
2.1.1 簡單郵件通訊協定.....................................3
2.1.2 郵局通訊協定與網際網路訊息存取通訊協定...............4
2.1.3 多用途網際網路研究延伸(MIME).........................5
2.2 郵件過濾方法...........................................6
2.2.1 名單比對.............................................6
2.2.2 內容過濾.............................................7
2.3 中文斷詞...............................................9
2.4 字詞篩選...............................................11
2.5 相似度比對.............................................12
第三章 研究方法.............................................13
3.1 過濾架構...............................................13
3.2 正常郵件規則...........................................15
3.3 正常郵件過濾器運作流程.................................15
3.3.1 郵件解碼.............................................16
3.3.2 中文斷詞與文章前處理.................................17
3.3.3 建立斷詞詞庫.........................................18
3.3.4 TF-IDF詞彙加權.......................................18
3.3.5 文章向量化與相似度比對...............................20
第四章 實驗系統與數據結果...................................22
4.1 實驗平台與系統設定.....................................22
4.2 門檻值選定.............................................24
4.3 實驗結果分析...........................................26
第五章 結論與未來工作.......................................28
5.1 結論...................................................28
5.2 未來工作...............................................28
參考文獻....................................................30

[1]葉生正, 蘇民揚, 張僩鈞,”兩階層式垃圾郵件過濾機制之研究”, 銘傳大學, 資訊傳播工程研究所, 資訊工程研究所, 2005.
[2]王文政, “垃圾郵件過濾系統之分析研究”, 國立台灣科技大學, 1995.
[3]RFC 2821, Simple Mail Transfer Protocol.
[4]Wietse Zweitze Venema,”Postfix”, http://www.postfix.org/
[5]RFC 2045, MIME Part One: Format of Internet Message Bodies.
[6]RFC 2046, MIME Part Two: Media Types. N. Freed, Nathaniel Borenstein. November 1996.
[7]趨勢科技, “网管员安全周记”, http://www.trendmicro.com/cn/security/general/article-bank/archive/2003/2003-12-10.htm
[8]張琮翔, 貝氏過濾法簡介, 2004. http://wiki.vgod.tw/doku.php?id=doc:bayesianfiltering
[9]黃純敏, 楊存一, 邱立豐,”TFIDF觀念於自動摘要實作評估”, 國立雲林科技大學, 資訊管理研究所
[10]黃純敏, 吳郁瑩, “網路中文文件自動摘要”, 國立雲林科技大學, 資訊管理研究所, 1999.
[11]陳克健, 陳正佳, 林隆基, “中文語句分析的研究－斷詞與構詞”, 中央研究院, 資訊所技術報告, TR86-004, 1986.
[12] ZBNO中文斷詞引擎, http://www.zbno.com/seg.jsp
[13] 張華平, “ICTCLAS漢語詞法分析系統”, 中國科學院計算技術研究所, http://sewm.pku.edu.cn/QA/reference/ICTCLAS/FreeICTCLAS/
[14] 詞庫小組, “中文斷詞系統”,中央研究院,http://ckipsvr.iis.sinica.edu.tw/
[15] 蔡志浩, “MMSEG: A Word Identification System for Mandarin Chinese Text Based on Two Variants of the Maximum Matching Algorithm”, http://technology.chtsai.org/mmseg/
[16] 魏忠志, “SCI/SSCI文章比對方法之研究”, 國立中央大學資訊管理研究所, 1995.
[17] Minoru Sasaki, Hiroyuki Shinnou, “Spam Detection Using Text Clustering”, IEEE International Conference on Cyberworlds, 2005.
[18] http://search.cpan.org/dist/MIME-tools/lib/MIME/Parser.pm
[19] http://spamassassin.apache.org/
[20] Type I and type II errors, http://en.wikipedia.org/wiki/False_positive