臺灣博碩士論文加值系統

語言測驗與語言教學和學習息息相關，然而，人工出題卻是相當費時費力的過程。近年來，電腦輔助產生試題的研究 (Computer-Assisted Item Generation，CAIG)，在自然語言處理領域裡形成重要的應用研究。綜觀目前的出題研究，大都著重字彙、聽寫、克漏詞、閱讀測驗題型，很少有針對英文文法的相關研究。本論文提出一個以網路為本的作法，可以半自動產生英文文法測驗考題。

在本論文中，我們提出一種電腦輔助英文文法出題的方法。我們先將文法出題的概念，撰寫成出題樣式 (test pattern)，輔以兩種出題策略 (test generation strategy)，讓電腦能將網路蒐集而來的資料，配合易讀性(readability)的自動分析，產生類似托福考試(Test of English as Foreign Language，TOEFL)的兩種文法題型：傳統單選題 (traditional multiple-choice)和偵錯題(error detection)。我們也根據這個作法，製作了電腦輔助線上文法測驗系統雛形，FAST。這個實作系統不僅能即時產生文法試題，以供老師採用，亦能提供文法練習題，幫助學習者自我學習。

動詞相關(verb-related)文法題的評估顯示，我們的方法可以產生約八成合宜的文法試題，此外，將這些文法題實施於考試中，分析結果證明電腦產生的文法試題亦能有不錯的試題鑑別度。總而言之，我們的方法結合了設計文法出題樣式、出題策略和運用網路資源，有極大的潛力，可以應用在適性化數位語言學習。

Testing has long been acknowledged as an integral part of language teaching and learning. However, manually designing language tests is not only time consuming but also labor intensive. Recently, due to the remarkable progress of computer technology, computer-assisted item generation (CAIG) has drawn considerable attention and becomes one of the active research areas in CALL (Computer Assisted Language Learning). CAIG provides an alternative and economical way for automatic generation of questions in relatively short time, effective establishment of item banks in large scale, and support for adaptive testing for incremental language learning. Previous work has explored generations of reading comprehension, vocabulary, listening dictation tests, but very little has been done on grammar tests. The purpose of this thesis is to address the issue of the computer-aided creation of English grammar tests.
We introduce a method for the semi-automatic generation of grammar test items by applying Natural Language Processing (NLP) techniques. Based on manually-designed patterns, sentences gathered from the Web are transformed into tests on grammaticality. The method involves representing test writing knowledge as test patterns, acquiring authentic sentences on the Web, and applying generation strategies to transform sentences into items. At runtime, sentences are converted into two types of TOEFL-style question: multiple-choice and error detection. We also describe a prototype system FAST (Free Assessment of Structural Tests). Evaluation on a set of generated questions indicates that the proposed method performs satisfactorily both in item facility and item discrimination. Our methodology provides a promising approach and offers significant potential for computer assisted language learning and assessment.

摘要
ABSTRACT
Acknowledgements
Table of Contents
List of Tables
List of Figures
Chapter 1 Introduction
1.1 Background
1.2 Motivation
1.3 Organization of the Thesis
Chapter 2 Related Work
2.1 CAIG in Language Learning
2.2 CAIG in Language Testing
Chapter 3 Method
3.1 Problem Statement
3.2 Question Generation Algorithm
3.3 Writing Test Patterns
3.3.1 Construction of Structural Patterns
3.3.2 Construction of Construct Patterns
3.3.3 Construction of Distractor Patterns
3.4 Web Crawl for Candidate Sentences
3.5 Test Generation Strategy
3.5.1 Test Generation Strategy for Multiple-Choice Questions
3.5.2 Test Generation Strategy for Error Detection Questions
Chapter 4 Experiments and Evaluation
4.1 Experimental Setting
4.2 Evaluation Metric
4.2.1 Evaluation Scale
4.2.2 Item Analysis
4.3 Evaluation Results
4.3.1 Question Generation Performance
4.3.2 Evaluation on Verb-Related Questions
4.3.3 Test Results Description
Chapter 5 Discussion
5.1 Performance Analysis
5.2 Error Analysis
5.3 Test Pattern Extension
5.4 Limitations of the Current Research
Chapter 6 Application
6.1 Teacher Interface of the System
6.2 Learner Interface of the System
Chapter 7 Conclusion and Future Work
References
Appendix A – Constructs for Quesestion Generation
Appendix B – Evaluation Scale
(a) Evaluation Criteria
(b) Verb-Related Test Patterns
(c) Construct-Pattern Relation Charts
Appendix C – Test Development Training Material
Appendix D – Manually-Designed Tests
(a) Manual Construction Questions-1
(b) Manual Construction Questions-2
Appendix E – Evaluation Results
(a) Traditional Multiple-Choice Questions
(b) Error Detection Questions
Appendix F – The Grammar Test
Appendix G – Test Results

Allen, M.J., & Yen, W.M. (1979). Introduction to measurement theory. Monterey, California: Brooks Cole.

Bachman, L.F., & Palmer, A. (2004). Language testing in practice. New York: Oxford University Press.

Bailey, K. M. (1998). Learning About Language Assessment: Dilemmas, Decisions, and Directions. Pacific Grove, California: Heinle and Heinle Publishers.

Bauer, H. (1991). Sore Finger Items in Multiple-choice Tests. System, 19(4), 453-458

Bolinger, D. (1977). Meaning and Form. London: Longman.

Brown, J.C., Frishkoff, G.A., & Eskenazi, M. (2005). Automatic Question Generation for Vocabulary Assessment. In Proceedings of Human Language Technology Conference and Conference on Empirical Methods in Natural Language Processing (HLT/EMNLP 2005) (pp. 819-826). Vancouver, Canada.

Brown, J. D. (1996). Testing in Language Programs: A Comprehensive Guide To English Language Assessment. NJ: Prentice Hall Regents.

Butler, J. (1991). Cloze procedures and concordances: The advantages of discourse level authenticity in testing expectancy grammar. System, 19(1/2), 29-38.

Chalhoub-Deville, M. (2001). Language Testing and Technology: Past and Future. Language Learning& Technology, 5(2), 95-98.

Chang, Y.L. (2000). Analysis of a Doctoral Students’ English Writing Proficiency Test at Chiao Tung University. Selected papers from the 9th International Symposium on English Teaching (pp.178-187). Taipei: Crane.

Close, R.A. (1982). English as a Foreign Language. London: George Allen and Unwin.

Coniam, D. (1997). A Preliminary Inquiry into Using Corpus Word Frequency Data in the Automatic Generation of English Cloze Tests. CALICO Journal, 14, 15- 33.

Dale, E., & Chall, J.S. (1948). A formula for predicting readability. Educational Research Bulletin, 27, 37-53.

Darian, S. (1979). The role of redundancy in language and language teaching. System, 7, 47-59.

Deane, P., & Sheehan, K. (2003). Automatic item generation via frame semantics. Retrieved
July 6, 2005 from http://www.ets.org/research/dload/ncme03-deane.pdf

Fairon, C. (1999). A Web-based System for Automatic Language Skill Assessment: EVALING. In Proceedings of Computer Mediated Language Assessment and Evaluation in Natural Language Processing Workshop (pp. 62-67).

Gamper, J., & Knapp, J. (2002). A review of intelligent CALL systems. CALL, 15(4), 329-342.

Gao, Z.M. (2000). AWETS: An Automatic Web-Based English Testing System. In Proceedings of the 8th Conference on Computers in Education/International Conference on Computer-Assisted Instruction (ICCE/ICCAI 2000) (pp. 25-31).

Gao, Z.M., & Liu, C.L. (2003). A Web-Based Assessment and Profiling System for College English. In Proceedings of the 11th International Conference on Computer Assisted Instruction (pp.24-26). Taipei, Taiwan.

Hoshino, A., & Nakagawa, H. (2005). A Real-Time Multiple-Choice Question Generation for Language Testing-A Preliminary Study-. In Proceedings of the Second Workshop on Building Educational Applications Using NLP (pp. 1-8). Ann Arbor, Michigan.

Huang, S. M., Liu, C. L.,&Gao, Z. M.(2003). Toward computer assisted learning for English dictation. In Proc. of the 2003 Joint Conf. on Artificial Intelligence, Fuzzy Systems, and Grey Systems, CD–ROM.

Irvine, S. H., & Kyllonen, P. C. (Eds.), (2002). Item generation for test development. Lawrence Erlbaum Associates.

Jian, J.Y., Chang, Y.C., & Chang, J.S. (2004). Collocational Translation Memory Extraction Based on Statistical and Linguistic Information. In Proc. of the 16th Conference on Computational Linguistics and Speech Processing (ROCLING XVI) (pp.111-120). Taipei, Taiwan.

Kathleen, K.S., & Kenji, K. (1996). Testing Grammar. The Internet TESL Journal, 2, 1-6

Kilgarriff, A., & Grefenstette, G. (2003). Special Issue on the WEB as Corpus. Computational Linguistics, 29 (3), 333-502.

Larsen-Freeman, D. (1997). Grammar and its teaching: Challenging the myths (ERIC Digest). Washington, DC: ERIC Clearinghouse on languages and Linguistics, Center for Applied Linguistics. Retrieved July 13, 2005, from http://www.vtaide.com/png/ERIC/Grammar.htm

Leech, G.N. (1980). Explorations in Semantics and Pragmatics. Amsterdam: J. Benjamins.

Liu, C.L., Wang, C.H., Gao, Z.M., & Huang, S.M. (2005). Applications of Lexical Information for Algorithmically Composing Multiple-Choice Cloze Items. In Proceedings of the Second Workshop on Building Educational Applications Using NLP (pp. 1-8). Ann Arbor, Michigan.

Liu, L. E. (2002). A corpus-based lexical semantic investigation of verb-noun miscollocations in Taiwan learners’ English. Unpublished master’s thesis. Tamkang University, Taipei.

Lynch, J. (2001). Guide to Grammar and Style. Retrieved March 10, 2006, from http://www.vtaide.com/png/ERIC/Grammar.htm

Milton J. (1998). WORDPILOT: enabling learners to navigate lexical universes. In S. Granger, & J. Hung (Eds.), Proceedings of the International Symposium on Computer Learner Corpora, Second Language Acquisition and Foreign Language Teaching (pp.97-98). The Chinese University of Hong Kong.

Mitkov, R., & Ha, L.A. (2003). Computer-Aided Generation of Multiple-Choice Tests. In Proceedings of the HLT-NAACL 2003 Workshop on Building Educational Applications Using Natural Language Processing (pp. 17 – 22). Edmonton, Canada.

Oller, J. W. (1979).Language Tests at School: A Pragmatic Approach. London: Longman.

Oranje, A. (2003). Automatic item generation applied to the national assessment of educational progress: Exploring a multilevel structural equation model for categorized variables. Retrieved July 6, 2005 from http://www.ets.org/research/dload/ncme03-andreas.pdf

Pentland, B.T. (1995). Grammatical models of organizational processes. Organization Science, 6(5), 541-556.

Poel, J., & Weatherly, S.D. (1997). A cloze look at placement testing. Shiken: JALT (Japanese Assoc. for Language Teaching) Testing & Evaluation SIG Newsletter, 1(1), 4-10.

Purpura, J.E. (2004). Assessing Grammar. New York: Cambridge University Press.

Rau, D. H. (1999). Validity and Reliability of Grammar Proficiency Test and Evaluation of the Effectiveness of Freshman English Composition Class. Providence Journal of Humanities, 12, 155-184

Rea-Dickins, P. M. (1991).What Makes a Grammar Test Communicative? In J.C., Alderson, & B. North (Eds.), Language Testing in the 1990s: The Communicative Legacy (pp.112-131). London: Macmillan.

Sharpe, P.J. (2004). How to Prepare for the TOEFL. Barrons’ Educational Series, Inc.

Shei, C. C. (2001). FollowYou! An automatic language lesson generation system. CALL, 14(2), 129-144.

Shih, F.M. (2005). An Automatic Quiz Generator for English Learning Using Natural Language Processing Technology (利用自然語言處理技術之英文試卷自動出題). CCL Technical Journal, 115, 70-74.

Stevens, V. (1991).Classroom concordancing: Vocabulary materials derived from relevant authentic text. English for Specific Purposes Journal, 10, 35-36.

Sumita, E., Sugaya, F., & Yamamoto, S. (2005). Measuring Non-Native Speakers' Proficiency of English by Using a Test with Automatically-Generated Fill-in-the-Blank Questions. In Proceedings of the Second Workshop on Building Educational Applications Using NLP (pp. 61-68). Ann Arbor, Michigan.

Wang, C.H., Liu, C. L.,& Gao, Z. M. (2003). Toward computer assisted item generation for English vocabulary tests (電腦輔助英文字彙出題系統之研究). In Proc. of the 2003 Joint Conf. on Artificial Intelligence, Fuzzy Systems, and Grey Systems, CD–ROM.

Wang, S., & Liou, H.C. (1991). An Item Analysis of an English Proficiency Test at Tsing Hua University: Grammar. Selected papers from the 7th on English Teaching and Learning in the Republic of China (pp.459-186). Taipei: Crane.

Wilson, E. (1997). The Automatic Generation of CALL exercises from general corpora. In, A. Wichmann, S. Fligelstone, T. McEnery, & G. Knowles (Eds.), Teaching and Language Corpora (pp.116-130). London: Longman.

Wolfe, J.H. (1976). Automatic question generation from text - an aid to independent study. ACM SIGCUE Bulletin, 2(1), 104-112.

Yang, Y.C., Yang, J.F., Chang, J.M., & Chang, J.S. (2005). Development of a Computer Assisted Reading Comprehension Test. In Proc. of the 17th Conference on Computational Linguistics and Speech Processing (ROCLING XVII) (pp.111-120).