(44.192.66.171) 您好!臺灣時間:2021/05/17 23:07
字體大小: 字級放大   字級縮小   預設字形  
回查詢結果

詳目顯示:::

: 
twitterline
研究生:黃聖傑
研究生(外文):Sheng-Chieh Huang
論文名稱(外文):GUI Component Detection for Cross-Platform Applications–Using Input Device and Image Change Synergistic Detection Method
指導教授:鄭永斌鄭永斌引用關係
指導教授(外文):Yung-Ping Cheng
學位類別:碩士
校院名稱:國立中央大學
系所名稱:軟體工程研究所
學門:電算機學門
學類:軟體發展學類
論文出版年:2020
畢業學年度:108
語文別:英文
論文頁數:63
中文關鍵詞:GUI自動化測試測試案例自動產生GUI模型GUI元件偵測圖形化使用者介面
外文關鍵詞:GUI automated testingtest case generationGUI modelGUI component detectionGUIGraphic User Interface
相關次數:
  • 被引用被引用:0
  • 點閱點閱:40
  • 評分評分:
  • 下載下載:3
  • 收藏至我的研究室書目清單書目收藏:0
在軟體工程領域,為了維持軟體的品質,「軟體測試」便是一門很重要的課題。軟體測試領域發展至今,回歸測試用來確保既有的功能不受系統修改與維護的影響。回歸測試自動化軟體技術相對成熟且種類多樣也實用化。但目前對於「測試案例自動產生」的實務應用目前仍然有很多實務上的困難。「GUI模型」的建立是目前針對GUI自動化測試的一項前置作業,如果能夠準確且快速的建立待測系統(如應用程式、網頁…等)的GUI模型,便能以GUI模型為基礎來達成測試案例產生的工作。

目前對於GUI模型的建立,大多都是針對特定平台或環境進行逆向工程式解構,或將程式的原始碼進行分析,以及使用開發框架提供的API進行GUI元件(或稱控制項)的提取及操作,這些方法的準確度雖然高但相對地對平台的依存度以及平台的開發知識也非常的高,而且往往要努力跟上該平台的底層升級。例如針對網頁開發的技術,就無法適用於Windows或是 Linux。更不用提手機的各式平台。

本論文將針對偵測待測系統的GUI元件提出新的方法,該方法將透過影像處理並與基本的通用輸入裝置進行協同識別,以偵測GUI元件;此方法具有高度與平台無關,並且無懼於平台的底層技術升級的特性。
In the field of software engineering, software testing is an important and major method to ensure software quality. In practical software testing, regression testing software tools have matured and varied. Regression test automation is to make sure that the system features that function correctly are not affected by software changes. On the other hand, “automated test case generation” remains to be very difficult in practice. The construction of the “GUI Model” is an important step toward GUI test case generation. If we can construct the GUI model of the system under test (such as application, webpage, etc.) accurately and quickly, it can be used to achieve GUI test case generation.
Most attempts to construct the GUI model are done by reverse engineering the program on specific platform, environment, or even analyzing the source code of the program when there is poor support from platforms. In some modern platforms, they can use the API provided by the platforms to extract and operate GUI components (called “control-items”). However, these approaches introduce very high dependency and coupling to the underlying platform and the results are not applicable to another platform. For example, approaches that aim for web technology are not applicable to Windows or Linux.
This paper proposes a new method for GUI component detection of the system under test using image processing and basic universal input devices. It is totally platform-independent. Finally, we will use this method to test different types of system under test to verify and evaluate the feasibility of the method.
摘 要 I
Abstract II
致 謝 III
Contents IV
Figures V
Tables VII
Chapter 1. Introduction 1
Chapter 2. Background and Related Work 4
2-1 GUI (Graphical User Interface) 4
2-2 Build GUI model 5
2-3 Image-based UI Automation 5
2-4 GUI component detection/extraction 6
2-5 Object detection 6
2-6 Feature extraction 7
Chapter 3. Problem Analysis 8
3-1 Limitations on previous GUI component analysis tools 8
3-1-1 Limitations on reverse engineering method 8
3-1-2 Limitations on GUI automation support framework 9
3-2 Why do we need the GUI-IDICS method and what it can achieve? 9
Chapter 4. Image Processing and Mouse Cursor - A Synergistic Approach 11
4-1 Area detection filtering method 11
4-2 Boundary detection by cursor's help 12
4-2-1 Image difference boundary detection (DIFF_Bound) 13
1. Image Difference Summation (DIFF_SUM) 13
2. Image Difference Edge Surrounding (DIFF_EDGE) 16
3. Image Difference Binarization (DIFF_BIN) 19
4. Image Difference Corner Reduction (DIFF_CORNER) 20
4-2-2 Cursor monitoring boundary detection (CUR_Monitor) 23
1. Cursor State Monitoring (CUR_CNG) 23
2. Cursor State and Pixel Color Monitoring (CUR_PIXEL) 26
Chapter 5. System Design and Architecture 29
5-1 Detection method modular architecture – PASS architecture 29
5-2 Detection program workflow 30
5-3 GUI component detection workflow 32
5-4 Results screenshot 33
5-5 User interface and functions of detection program 34
Chapter 6. Experiment and Evaluation 36
6-1 Experimental definition and premise 36
6-2 Experimental results 37
6-2-1 Type of system under test - general desktop GUI application 37
6-2-2 Type of system under test – Webpage 42
6-3 Summary and discussion 46
Chapter 7. Conclusion and Future Work 48
Chapter 8. Reference 50
[1] Memon, Atif, Ishan Banerjee, and Adithya Nagarajan. “GUI ripping: Reverse engineering of graphical user interfaces for testing.” 10th Working Conference on Reverse Engineering, 2003. WCRE 2003. Proceedings.. IEEE, 2003.
[2] Memon, Atif M. “An event‐flow model of GUI‐based applications for testing.” Software testing, verification and reliability 17.3 (2007): 137-157.
[3] Kull, Andres. “Automatic GUI model generation: State of the art.” 2012 IEEE 23rd International Symposium on Software Reliability Engineering Workshops. IEEE, 2012.
[4] Nguyen, Bao N., et al. “GUITAR: an innovative tool for automated testing of GUI-driven software.” Automated software engineering 21.1 (2014): 65-105.
[5] Lee, Shin-Jie, et al. “Test Command Auto-Wait Mechanisms for Record and Playback-Style Web Application Testing.” 2018 IEEE 42nd Annual Computer Software and Applications Conference (COMPSAC). Vol. 2. IEEE, 2018.
[6] Memon, Atif M., Mary Lou Soffa, and Martha E. Pollack. “Coverage criteria for GUI testing.” ACM SIGSOFT Software Engineering Notes 26.5 (2001): 256-267.
[7] Grilo, André MP, Ana CR Paiva, and João Pascoal Faria. “Reverse engineering of GUI models for testing.” 5th Iberian Conference on Information Systems and Technologies. IEEE, 2010.
[8] White, Lee, and Husain Almezen. “Generating test cases for GUI responsibilities using complete interaction sequences.” Proceedings 11th International Symposium on Software Reliability Engineering. ISSRE 2000. IEEE, 2000.
[9] Smokris. “How can I detect elements of GUI using OpenCV?”: Stack- Overflow. 7 March 2012. 2 April 2019 <https://dsp.stackexchange.com/questions/1657/how-can-i-detect-elements-of-gui-using-opencv>.
[10] Yun, Young-Sun, et al. "Detection of GUI Elements on Sketch Images Using Object Detector Based on Deep Neural Networks." International Conference on Green and Human Information Technology. Springer, Singapore, 2018.
[11] Redmon, Joseph, et al. "You only look once: Unified, real-time object detection." Proceedings of the IEEE conference on computer vision and pattern recognition. 2016.
[12] .NET Framework Documentation: Using UI Automation for Automated Testing. 30 March 2017 <https://docs.microsoft.com/en-us/dotnet/framework/ui-automation/using-ui-automation-for-automated-testing>.
[13] Yeh, Tom, Tsung-Hsiang Chang, and Robert C. Miller. "Sikuli: using GUI screenshots for search and automation." Proceedings of the 22nd annual ACM symposium on User interface software and technology. 2009.
[14] Cheng, Yung-Pin, Deron Liang, and Wei-Jen Wang. "KORAT—A platform independent test automation tool by emulating keyboard/mouse hardware signals." 2016 IEEE AUTOTESTCON. IEEE, 2016.
[15] Chang, Tsung-Hsiang, Tom Yeh, and Robert C. Miller. "GUI testing using computer vision." Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. 2010.
[16] Bakaev, Maxim, et al. "Auto-extraction and integration of metrics for web user interfaces." Journal of Web Engineering 17.6 (2018): 561-590.
[17] Li, Yingjun, S. U. N. Yingji, and Z. H. A. O. Qingyu. "Ui automation based on runtime image." U.S. Patent Application No. 13/787,801.
[18] OpenCV. “MSER Class Reference”: OpenCV Documentation. 2 April 2019 <https://docs.opencv.org/3.4/d3/d28/classcv_1_1MSER.html>.
[19] Windows File Explorer: Wikipedia. 11 April 2019 <https://en.wikipedia.org/wiki/File_Explorer>.
[20] TeamViewer. 11 April 2019 <https://www.teamviewer.com/>.
[21] PuTTY. 11 April 2019 <https://www.putty.org/>.
[22] Home: E-Invoice Platform. 11 April 2019 <https://www.einvoice.nat.gov.tw/>.
[23] Firefox Support. 11 April 2019 <https://support.mozilla.org/zh-TW/products/firefox>.
[24] Home: Central Weather Bureau. 11 April 2019 <https://www.cwb.gov.tw/>.
連結至畢業學校之論文網頁點我開啟連結
註: 此連結為研究生畢業學校所提供,不一定有電子全文可供下載,若連結有誤,請點選上方之〝勘誤回報〞功能,我們會盡快修正,謝謝!
QRCODE
 
 
 
 
 
                                                                                                                                                                                                                                                                                                                                                                                                               
第一頁 上一頁 下一頁 最後一頁 top