臺灣博碩士論文加值系統

文字採礦中經常使用TF-IDF(term frequency - inverse document frequency)的方法來計算權重，理由是：「如果一個詞在某一份文件中出現的頻率TF高，且沒有或比較少出現在其他文件中，則這個詞具有良好的區別能力，適合拿來進行分類。如此可以濾掉常見的詞語，留下重要的詞語。」因此，TF這個方法就被捨棄不用了。
但是，當時的文章來源大多為部落格，文章長度較長。某個詞在其他文章很少出現的話，真的可以認為這個詞擁有良好的區別能力。
目前各類評論文章的來源大多為推特（Twitter）、微博(microblog)、璞浪(plurk)或臉書(Facebook)，這些文章大多都非常簡短，推特跟微博甚至限制在140個字以內。如此簡短的內容造成許多詞語有可能只出現在少數文章中，使得許多詞語的IDF權重分數都會增加。在這種情況下，強調TF-IDF可以「找出具良好區別能力的詞語，進而導致分類正確率比TF高。」這個特性，還能達成嗎？
所以本研究想要比較一下TF與TF-IDF在導入SentiWordNet之後，其分類結果是否仍然是TF-IDF比較優良？

Text mining often uses the TF-IDF (term frequence-inverse document frequency) technique to preprocess documents because of a common knowledge that “if a word has a high frequency in a document and does’nt appear in too many other documents, then the word has a good discriminatory power for text classification”. Therefore, common words can be filtered out in the process of text mining jobs.
Blog articles come with text of a substantial length. When a word rarely appears in other articles, this word does have a good capability to distinguish text of different categories. Today, many review articles are created in microblog sites such as Twitter or plurk, which often restricts an article to a maximum of 140 characters. With such a short article, most words naturally appear in only a few articles creating the phenomenon of a high IDF. Is it still resonable to emphasize the IDF factor when we mine corpus with short text?
In order to answer the above question, we set up experiments to detect the sentiment of movie reviews using SentiWordNet. Two classification algorithms (Naïve Bayesian and decision tree) have been applied to learn and predict the polarity of a movie review. It is found that TF-IDF features performed no better than TF features.

目錄
中文摘要 i
英文摘要 ii
誌謝 iii
目錄 iv
表目錄 vi
一、緒論 1
1.1 研究背景與動機 1
1.2 研究目的 3
二、 文獻探討 5
2.1自然語言處理 6
2.1.1斷詞(tokenization) 6
2.1.2去除停用字（remove stop words） 6
2.1.3詞性標註(Part-of-speech,POS) 7
2.1.4詞形還原(Lemmatisation) 7
2.2文字探勘 8
2.2.1詞頻（TF） 8
2.2.2反向文件頻率（IDF） 8
2.2.3TF-IDF 9
2.3 SentiWordNet 9
2.4Naive Bayesian 10
2.5決策樹（Decision tree） 12
2.5.1 J48即決策樹C4.5演算法 14
2.5電影評論 14
三、 研究設計 16
3.1蒐集電影文件集 16
3.2文字前處理。 16
3.3導入情緒辭典、分群 18
四、 研究結果 22
4.1 Naïve Bayes分類器（未設定min_df，特徵詞數量為8271個） 23
4.2決策樹(J48)分類器（未設定min_df，特徵詞數量為8271個） 24
4.3 Naïve Bayes分類器（min_df=0.01，特徵詞數量為2278個） 25
4.4決策樹(J48)分類器（min_df=0.01，特徵詞數量為2278個） 26
4.5 Naïve Bayes分類器（min_df=0.03，特徵詞數量為999個） 27
4.6決策樹(J48)分類器（min_df=0.03，特徵詞數量為999個） 28
4.7 Naïve Bayes分類器（min_df=0.04，特徵詞數量為774個） 29
4.8決策樹(J48)分類器（min_df=0.04，特徵詞數量為774個） 30
4.9 Naïve Bayes分類器（min_df=0.05，特徵詞數量為624個） 31
4.10決策樹(J48)分類器（min_df=0.05，特徵詞數量為624個） 32
4.11 Naïve Bayes分類器（min_df=0.1，特徵詞數量為286個） 33
4.12決策樹(J48)分類器（min_df=0.1，特徵詞數量為286個） 34
五、 結論 35
參考文獻 39

[1]馬芳資, &; 林我聰. (2006). 決策樹形式知識的合併修剪之研究. Electronic Commerce Studies, 4(2), 123-156.
[2]李旭昇, &;郭耀煌. (2007).靈活的增強樸素貝葉斯分類器. 信息與控制 , 36 (6), 690-695.
[3]Baccianella, S., Esuli, A., &; Sebastiani, F. (2010, May). SentiWordNet 3.0: An Enhanced Lexical Resource for Sentiment Analysis and Opinion Mining. In LREC (Vol. 10, pp. 2200-2204).
[4]Balakrishnan, V., &; Lloyd-Yemoh, E. (2014). Stemming and Lemmatization: A Comparison of Retrieval Performances. Lecture Notes on Software Engineering , 2 (3).
[5]Bhattasali, S. (2014). Correlations Between Sentiment Analysis of Movie Tweets, Film Critics Reviews, and Box Office Earnings.
[6]Bird, S., Klein, E., &; Loper, E. (2009). Natural language processing with Python . " O''Reilly Media, Inc.".
[7]Bouckaert, R. R., Frank, E., Hall, M., Kirkby, R., Reutemann, P., Seewald, A., &; Scuse, D. (2013). WEKA Manual for Version 3-7-8.
[8]Carlin, B. P., &; Louis, T. A. (2011). Bayesian methods for data analysis. CRC Press.
[9]Chiang, D. A., Chen, W., Wang, Y. F., &; Hwang, L. J. (2001). Rules generation from the decision tree. Journal of Information Science and Engineering, 17(2), 325-339.
[10]Esuli, A., &; Sebastiani, F. (2006, May). Sentiwordnet: A publicly available lexical resource for opinion mining. In Proceedings of LREC (Vol. 6, pp. 417-422).
[11]Dolamic, L., &; Savoy, J. (2010). When stopword lists make the difference. Journal of the American Society for Information Science and Technology, 61(1), 200-203.
[12]Hamouda, A., &; Rohaim, M. (2011, January). Reviews classification using sentiwordnet lexicon. In World Congress on Computer Science and Information Technology .
[13]Hung, C., &; Lin, H. K. (2013). Using objective words in SentiWordNet to improve word-of-mouth sentiment classification. IEEE Intelligent Systems,28(2), 0047-54.
[14]Ku, L. W., &; Chen, H. H. (2007). Mining opinions from the Web: Beyond relevance retrieval. Journal of the American Society for Information Science and Technology, 58(12), 1838-1850.
[15]Li, T., Zhang, Y., &; Sindhwani, V. (2009, August). A non-negative matrix tri-factorization approach to sentiment classification with lexical prior knowledge. In Proceedings of the Joint Conference of the 47th Annual Meeting of the ACL and the 4th International Joint Conference on Natural Language Processing of the AFNLP: Volume 1-Volume 1 (pp. 244-252). Association for Computational Linguistics.
[16]Liu, B. (2012). Sentiment analysis and opinion mining. Synthesis Lectures on Human Language Technologies, 5(1), 1-167.
[17]Maas, AL, Daly, RE, Pham, PT, Huang, D., Ng, AY, &; Potts, C. (2011, June). Learning word vectors for sentiment analysis. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies-Volume 1 (pp. 142-150). Association for Computational Linguistics.
[18]Martineau, J., &; Finin, T. (2009, May). Delta TFIDF: An Improved Feature Space for Sentiment Analysis. In ICWSM .
[19]Miller, GA, Beckwith, R., Fellbaum, C., Gross, D., &; Miller, KJ (1990). Introduction to wordnet: An on-line lexical database*. International journal of lexicography , 3 (4), 235 -244.
[20]Mooney, R. J., &; Roy, L. (2000, June). Content-based book recommending using learning for text categorization. In Proceedings of the fifth ACM conference on Digital libraries (pp. 195-204). ACM.
[21]Nasukawa, T., &; Yi, J. (2003, October). Sentiment analysis: Capturing favorability using natural language processing. In Proceedings of the 2nd international conference on Knowledge capture (pp. 70-77). ACM.
[22]Ohana B., 2014, A Lexicon Based Sentiment Classifier in Python. Retrieved from http://kmandcomputing.blogspot.tw/
[23]Paltoglou, G., &; Thelwall, M. (2010, July). A study of information retrieval weighting schemes for sentiment analysis. In Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics (pp. 1386-1395). Association for Computational Linguistics.
[24]Pang, B., &; Lee, L. (2004, July). A sentimental education: Sentiment analysis using subjectivity summarization based on minimum cuts. InProceedings of the 42nd annual meeting on Association for Computational Linguistics (p. 271). Association for Computational Linguistics.
[25]Sparck Jones, K. (1972). A statistical interpretation of term specificity and its application in retrieval. Journal of documentation, 28(1), 11-21.
[26]Toutanova, K., Klein, D., Manning, C. D., &; Singer, Y. (2003, May). Feature-rich part-of-speech tagging with a cyclic dependency network. InProceedings of the 2003 Conference of the North American Chapter of the Association for Computational Linguistics on Human Language Technology-Volume 1 (pp. 173-180). Association for Computational Linguistics.