臺灣博碩士論文加值系統

在算程式程用計機課採自動化批改作業工具既給予學生即時的反饋也降低教學人員評分的負擔。然而，當題目的範例輸出包含隨機資訊時，批改系統的制式流程常無法判定諸如此類的題目，使得佈署一道新題目超越了圖形介面的操作，導致修改系統原始碼變成週期性活動。

本研究在此實作一個線上程式批改系統，有別於演算法競賽型解題系統的架構，將每道題目的批改邏輯抽離系統，獨立實作於批改腳本，使得系統成為能夠容納任意批改規範的平台，以應付各種要求隨機輸出的練習題。批改腳本主要功能為：設定此題是否有測試資料(2個選項)、調整程式輸出要如何正規化(2個選項)、選擇此題的呈現方式(3個選項)，共12個面向以涵蓋所有需要文字輸出的練習題。

針對程式初學課程常見的經典練習題，本研究提供教學人員一套簡便的批改腳本客製化流程包含：首先，分辨題目是否需要測試資料；接著，依據輸出內容是否具有圖形意義(例如：印出兩個矩陣相加、印出金字塔)調整正規化程度；最後，將僅須透過比對學生程式輸出與解答程式輸出是否一致就能達成批改的題目歸納為「標準輸出比對題型」，而將輸出具有隨機內容(例如：當下時間、隨機數字、機率)的題目歸納為「分支輸出比對題型」與「後處理輸出比對題型」。歷經一個學期的測試，透過此流程所設計的批改腳本確實能夠批改所有程式作業，其中著名競賽式解題系統無法佈署而只有本研究能夠處理的題目佔45.95%。

受益於系統保留了所有學生繳交的程式碼，批改系統除了確實大幅降低了評分的工作量，也處理了偵測作業抄襲的工作，使得教學人員有更多的時間解答學生們的疑惑。而以一個課程輔助方案的角度，批改系統除了能即時地給予學生反饋與允許重複提交修正後的程式碼，亦提供教學人員一個平台在課後檢視學生們的程式碼品質與繳交情形，因而更能夠適時地補充或增強教學內容。

最後，對於系統後續的發展潛力，本研究分析了學生們留在系統的紀錄資料與編程能力的相關性，發現大量的編譯錯誤次數為後段學生的共通點(Max. |r|=0.52; p<0.01)，未來可以將每道練習題的編譯錯誤次數當作特徵值，以排定早期需要加強輔導的學生名單。

Implementing automatic assessment tools on laboratory or homework exercises in programming courses is a way to provide timely feedback for students as well as lower the instructors’ marking burden. However, if the exercises involve random outputs, which are not judged against a gold standard of comparison, system-level source code modification is often required on a ubiquitous online judging system.

This research develops an online judging system as a platform, aiming at adopting a variety of assessment specifications. This research also introduces a simple procedure for customizing judge scripts as the core of the online judging system and deals with all textual output problems in 12 aspects: two options for determining whether a problem requires input of test data; three options for classifying a problem with or without random outputs; and two options for matching special output symbols. By designing the judge scripts based on theses 12 aspects, this approach was an excellent solution that dealt with almost half (45.95%) of the exercises that other online judging systems could not handle.

With the advantage of preserving submitted source codes, the online judging system not only replaces instructors’ grading work but also enables plagiarism detection for each exercise. Thus, the online judging system frees instructors’ from tedious routine tasks, enabling them to offer further assistance to students in need. During the experiment in a programming course, students could successively correct and re-submit their source codes until their codes passed the test, while the instructors could inspect quality of students’ source codes after class for improving pedagogy.

To investigate the future potential of the online judging system, this research analyzed correlations between the students’ performance and data collected from the system. It was found that making a large number of repeated compiling errors is one of the main behaviors of the weak students (Max. |r|=0.52; p<0.01). Therefore, this research proposes that the instructor may set a compiling
error threshold to identify the weak students at an early stage so that extra tutoring can be arranged.

口試委員審定書 I
致謝 II
摘要 IV
Abstract V
圖目錄 VI
表目錄 VIII
第一章 緒論 1
1.1 研究背景 1
1.2 批改系統 2
1.3 研究目的 4
1.4 論文組織 6
第二章 需求分析 7
2.1 環境說明 7
2.2 題目範疇 8
2.3 題目輸出內容與格式 11
2.4 系統技術需求 17
2.5 使用案例分析 18
2.6 小結 20
第三章 系統設計與實作 22
3.1 系統概述 22
3.2 系統架構 23
3.3 功能設計 27
3.4 操作流程 36
3.5 資源配置 44
第四章 批改腳本設計與實作 45
4.1 批改腳本支援說明 45
4.2 正規表達式 46
4.3 批改腳本流程分析 49
4.4 批改腳本基本實作 54
4.5 題型快速分類方法 58
4.6 經典批改腳本範例 62
4.6.1 標準輸出比對題型 62
4.6.2 分支輸出比對題型 66
4.6.3 後處理輸出比對題型 70
4.7 探討題目的12個面向 76
第五章 系統使用分析 78
5.1 資料探勘 78
5.2 系統紀錄 79
5.3 相關程度 81
5.3.1平均嘗試繳交次數 81
5.3.2 平均花費時間 83
5.3.3 題目完成率 85
5.3.4 考試分數 86
5.4 預測模型 88
5.5 適用性討論 90
第六章 結論 91
6.1 研究總結 91
6.2 研究貢獻 92
6.3 未來工作 93
參考文獻 95
附錄 原始碼專案網址對照表 99

[1] Ahadi, A., R. Lister, H. Haapala, & A. Vihavainen (2015). “Exploring Machine Learning Methods to Automatically Identify Students in Need of Assistance.” In Proceedings of the 11th annual International Conference on International Computing Education Research (ICER ''15), Omaha, Nebraska, USA, 121-130.

[2] Ahadi, A., R. Lister, & A. Vihavainen (2016). “On the Number of Attempts Students Made on Some Online Programming Exercises During Semester and their Subsequent Performance on Final Exam Questions.” In Proceedings of the 2016 ACM Conference on Innovation and Technology in Computer Science Education (ITiCSE ''16), Arequipa, Peru, 218-223.

[3] Aiken, A. Moss [Online]. Available from http://theory.stanford.edu/~aiken/moss/

[4] Ala-Mutka, K. M. (2005). “A Survey of Automated Assessment Approaches for Programming Assignments.“ Computer Science Education, 15(2), 83-102.

[5] Albuquerque, D., B. Cafeo, A. Garcia, S. Barbosa, S. Abrahão, & A. Ribeiro (2015), “Quantifying Usability of Domain-Specific Languages: an Empirical Study on Software Maintenance.” Journal of Systems and Software, 101, 245-259.

[6] Association of Computing Machinery. ACM-ICPC [Online]. Available from http://icpc.baylor.edu/

[7] Bergin, S. & R. Reilly (2005). “Programming: Factors that Influence Success.” In Proceedings of the 36th SIGCSE Technical Symposium on Computer Science Education (SIGCSE ''05), St. Louis, Missouri, USA, 411-415.

[8] Bishop, J. & M. A. Verleger (2013). ”The Flipped Classroom: A Survey of the Research.” In Proceedings of the 2013 ASEE Annual Conference & Exposition, Atlanta, Georgia, 23.1200.1-23.1200.18.

[9] Booch, G., J. Rumbaugh, & I. Jacobson (2005). The Unified Modeling Language User Guide (2nd ed.). Addison-Wesley Professional.

[10] Breiman, L., J. Friedman, R. A. Olshen, & C. J. Stone (1984). Classification and Regression Trees. Belmont, CA: Wadsworth International Group.

[11] Constable, R. L. (1980). “The Role of Finite Automata in the Development of Modern Computing Theory.” In Proceedings of The Kleene Symposium, North-Holland Publishing Co., 61-83.

[12] Du, X., C. Yi, Y. Wei, S. Feng, & Z. Gong (2010). “Design of Automata Online Judge.” In Proceedings of the 2nd International Conference on Information Engineering and Computer Science (ICIECS), Wuhan, China, 1-4.

[13] Forišek, M. (2006). “Security of Programming Contest Systems.” In proceedings of the 2nd International Conference on Informatics in Secondary Schools: Evolution and Perspectives (ISSEP), Vilnius, Lietuvos, 553-563.

[14] Fowler, M. (2010). Domain-Specific Languages (1st ed.). Addison-Wesley Professional.

[15] Friedl, J. E. F. (2006). Mastering Regular Expressions (3rd ed.). O''Reilly Media, Inc.

[16] Google Inc. Google Code Jam [Online]. Available from http://code.google.com/codejam/

[17] Gronback, R. C. (2009). Eclipse Modeling Project: A Domain-Specific Language (DSL) Toolkit. Addison-Wesley Professional.

[18] Hanks, B. (2008). “Problems Encountered by Novice Pair Programmers.” Journal on Educational Resources in Computing (JERIC), 7(4), 1-13.

[19] Ihantola, P., T. Ahoniemi, V. Karavirta, & O. Seppälä (2010). “Review of Recent Systems for Automatic Assessment of Programming Assignments.” In Proceedings of the 10th Koli Calling International Conference on Computing Education Research (Koli Calling ''10), Koli, Finland, 86-93.

[20] Insa, D. & J. Silva (2015). “Semi-Automatic Assessment of Unrestrained Java Code: a Library, a DSL, and a Workbench to Assess Exams and Exercises.” In Proceedings of the 2015 ACM Conference on Innovation and Technology in Computer Science Education (ITiCSE ''15), Vilnius, Lithuania, 39-44.

[21] Jadud, M. C. (2006). “Methods and Tools for Exploring Novice Compilation Behaviour.” In Proceedings of the 2nd International Workshop on Computing Education Research (ICER''06), Canterbury, United Kingdom, 73-84.

[22] JUnit. JUnit 5 [Open source project]. Available from http://github.com/junit-team/junit5

[23] Kaohsiung Affilicated Senior High School. ZeroJudge [Online]. Available from http://judge.nccucs.org

[24] Kleene, S. C. (1956). “Representation of Events in Nerve Nets and Finite Automata.” In C. E. Shannon & J. McCarthy (Eds.), Automata Studies (pp. 3-41). Princeton, NJ: Princeton University Press.

[25] Kurnia, A., A. Lim, & B. Cheang (2001). “Online Judge.” Computers & Education, 36(4), 299-315.

[26] Liang, Y. D. (2015). Introduction to Java Programming, Comprehensive Version (10th ed.). Prentice Hall.

[27] McCulloch, W. S. & W. Pitts (1943). “A Logical Calculus of the Ideas Immanent in Nervous Activity.” Bulletin of Mathematical Biophysics, 5(4), 115-133.

[28] Mizuno, T. & S. Y. Nishizaki (2014). “Distributed Online Judge System for Interactive Theorem Provers.” EPJ Web of Conferences, 68, 00016.

[29] Pettit, R. S., J. D. Homer, K. M. Holcomb, N. Simone, & S. A. Mengel (2015). “Are Automated Assessment Tools Helpful in Programming Courses?” In Proceedings of the 2015 ASEE Annual Conference & Exposition, Seattle, Washington, 26.230.1-26.230.20.

[30] Resnick, M., J. Maloney, A. Monroy-Hernández, N. Rusk, E. Eastmond, K. Brennan, A. Millner, E. Rosenbaum, J. Silver, B. Silverman, & Y. Kafai (2009). “Scratch: Programming for All.” Communications of the ACM, 52(11), 60-67.

[31] Robins, A., P. Haden, & S. Garner (2006). “Problem Distributions in a CS1 Course.” In Proceedings of the 8th Australasian Conference on Computing Education - Volume 52 (ACE''06), Australian Computer Society, Inc., Darlinghurst, Australia, Australia, 165-173.

[32] Sams, A. & J. Bergmann (2013). “Flip Your Students'' Learning.” Educational Leadership, 70(6), 16-20.

[33] Sanders, W. (2013). “Using UMLs with Design Patterns.” In PHP Design Patterns (pp.59-78). Sebastopol, CA: O''Reilly Media, Inc.

[34] Solms, F. & V. Pieterse (2016). “Towards a Generic DSL for Automated Marking Systems.” In Proceedings of the 45th annual Conference of the Southern African Computer Lecturers'' Association (SACLA 2016), Springer International Publishing, 59-66.

[35] Suhailan, S., A. S. Shibghatullah, & M. A. Burhanuddin (2014). “Perspective of Automated Programming Error Feedback Approaches in Problem Solving Exercises.” Journal of Theoretical and Applied Information Technology, 70(1), 121-129.

[36] TestFX. TestFX [Open source project]. Available from http://github.com/TestFX/TestFX

[37] Therneau, T. M. & E. J. Atkinson (2015). An Introduction to Recursive Partitioning Using the RPART Routines. Mayo Foundation.

[38] Thompson, K. (1968). “Programming Techniques: Regular expression search algorithm.” Communications of the ACM, 11(6), 419-422.

[39] Tremblay, G., F. Guérin, A. Pons, & A. Salah (2008). “Oto, a Generic and Extensible Tool for Marking Programming Assignments.” Software: Practice and Experience, 38(3), 307–333.

[40] Tremblay, G. & P. Lessard (2011). “A Marking Language for the Oto Assignment Marking Tool.” In Proceedings of the 16th annual joint Conference on Innovation and Technology in Computer Science Education (ITiCSE ''11), Darmstadt, Germany, 148-152.

[41] University of Valladolid. UVa Online Judge [Online]. Available from http://uva.onlinejudge.org

[42] Watson, C., F. W. B. Li, & J. L. Godwin (2013). “Predicting Performance in an Introductory Programming Course by Logging and Analyzing Student Programming Behavior.” In 2013 IEEE 13th International Conference on Advanced Learning Technologies, Beijing, China, 319-323.

[43] Wilcox, C. (2016). “Testing Strategies for the Automated Grading of Student Programs.” In Proceedings of the 47th ACM Technical Symposium on Computing Science Education (SIGCSE ''16), Memphis, Tennessee, USA, 437-442.