臺灣博碩士論文加值系統

在本論文中，我們提出一種能夠讓使用者互動輸入二維線條草圖並將其轉換為三維模型的方法。在以前的CAD與互動幾何建模方法中，大多是基於啟發式的規則來解決三維模型的重建，往往限制線條圖的繪畫方式，且圖上的線條必須為輪廓線，如果繪畫不符合規則可能會導致重建失敗。然而，要求一般使用者描繪如此制式化的圖是有困難而且不直覺的。
為了讓使用者如同平日所描繪草圖的方式輸入圖片，我們採用深度學習的方法來辨認使用者所繪製的任意二維線條草圖。首先，我們用快速的區域辨識之卷積神經網路來偵測及分解完整二維建築線條圖的感興趣區域，獲得感興趣區域以及標記後，我們再利用直方圖上亮度分布情況分類出每塊區域的類型，最後，藉由辨識與分類的結果，我們可以找出不同標記及種類的三維子模型集合以及相對應的替代分割語法，再依照分割語法並利用子模型作為建構塊來建構完整的三維模型。經由和現有方法比較的結果顯示，我們的系統可以更容易的讓使用者透過互動式繪製來建造三維建築模型。

In this thesis, we propose a method of allowing users to interactively enter 2D sketches and converting the sketches into 3D models. 3D model reconstruction from 2D sketches remains an important research in graphics modeling such as CAD and vision fields. Existing approaches usually established based on heuristic rules or with various restrictions. For instance, the line sketch must be orthogonal projection. However, they may fail due to the imprecision of strokes. Besides, it is difficult and not intuitive for users to sketch such a complex and standardized drawing.
In our system, users can sketch a building as they usually do. We use deep learning approach to recognize the components drawn in their sketches. First, we use the faster R-CNN model to detect and decompose the 2D building line sketch into different parts and each part has a corresponding label. After acquiring the Region of Interests (RoIs) and the labels, we use the histogram of the intensity distribution to classify the types of each RoI. By means of the deep learning approach and the classifier, we can find out a collection of 3D sub-models with substitute split grammars of each part. Our system is based on the results to construct the complete 3D model. The reports of comparing with existing methods show that our system makes it easier for users to model a 3D building by interactive sketching.

摘要 I
Abstract II
Acknowledgement III
Table of Contents IV
List of Figures VI
Lists of Tables VIII
Chapter1. Introduction 1
Chapter2. Related Work 4
2.1 3D object reconstruction 4
2.2 Sketch-based and example-based modeling 4
2.3 Procedural modeling and inverse procedural modeling 5
2.4 Deep learning 6
2.5 Similar work 6
Chapter3. System Overview and Collection of Datasets 8
3.1 System overview 8
3.2 Collection of datasets 9
Chapter4. Part detector and type classifier 11
4.1 Part detector - faster R-CNN 11
4.2 CNN 17
4.3 Type classifier 20
Chapter 5. Online Stage 22
Chapter 6. Experiment 31
6.1 Performance of the part detector 31
6.2 Comparison with state-of-art approaches 35
6.3 Comparison of type classifiers 36
6.4 Comparison with existing 3D modeling software 37
6.5 Results 39
Chapter 7. Conclusion 42
Reference 43

1. Gen Nishida, Ignacio Garcia-Dorado, Daniel G. Aliaga, Bedrich Benes, and Adrien Bousseau. 2016. Interactive Sketching of Urban Procedural Models. ACM Transactions on Graphics (SIGGRAPH Conference Proceedings). vol 35, no 4
2. Haibin Huang, Evangelos Kalogerakis, Ersin Yumer, and Radomir Mech. 2016. Shape Synthesis from Sketches via Procedural Models and Convolutional Networks. IEEE TRANSACTIONS ON VISUALIZATION AND COMPUTER GRAPHICS. vol 23, no 8: 2003 - 2013
3. Liangliang Cao, Jianzhuang Liu, and Xiaoou Tang. 2008. What the back of the object looks like: 3D reconstruction from line drawings without hidden lines. IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE. vol 30, no 3: 507-517
4. Jianzhuang Liu, Yu Chen, and Xiaoou Tang. 2011. Decomposition of Complex Line Drawings with Hidden Lines for 3D Planar-Faced Manifold Object Reconstruction. IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI). vol 33, no 1: 3-15
5. Tianfan Xue, Jianzhuang Liu, and Xiaoou Tang. 2012. Example-Based 3D Object Reconstruction from Line Drawings. IEEE Conference on Computer Vision and Pattern Recognition (CVPR). vol 10: 302-309
6. Changqing Zou, Tianfan Xue, Xiaojiang Peng, Honghua Li, Baochang Zhang, Ping Tan, and Jianzhuang Liu. 2016. An example-based approach to 3D man-made object reconstruction from line drawings. Pattern Recognition. vol 60, no C: 543-553
7. Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. 2015. Faster R-CNN: Towards real-time object detection with region proposal networks. Neural Information Processing Systems (NIPS).
8. Linjie Yang, Jianzhuang Liu, and Xiaoou Tang. 2013. Complex 3d general object reconstruction from line drawings. IEEE International Conference on Computer Vision (ICCV).
9. Fang Wang, Le Kang, and Yi Li. 2015. Sketch-based 3D Shape Retrieval using Convolutional Neural Networks.
10. Peter Wonka, Michael Wimmer, François Sillion, and William Ribarsky. 2003. Instant architecture. ACM Transactions on Graphics (TOG). vol 22, no 3: 669-677
11. Ruben M. Smelik, Tim Tutenel, Rafael Bidarra, and Bedrich Benes. 2014. A Survey on Procedural Modelling for Virtual Worlds. Computer Graphics Forum. vol 33, no 6: 31-50
12. O. Stava, S. Pirk, J. Kratt, B. Chen, R. Mźch, O. Deussen, and B. Benes. 2014. Inverse procedural modelling of trees. Computer Graphics Forum. vol 33, no 6: 118-131
13. Carlos A. Vanegas, Ignacio Garcia-Dorado, Daniel G. Aliaga, Bedrich Benes, and Paul Waddell. 2012. Inverse design of urban procedural models. ACM Transactions on Graphics (TOG). vol 31, no 6
14. Jonathan Long, Evan Shelhamer, and Trevor Darrell. 2015. Fully convolutional networks for semantic segmentation. IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 3431-3440
15. Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton. 2012. ImageNet classification with deep convolutional neural networks. Neural Information Processing Systems 25 (NIPS 2012).
16. Giorgos Tolias, Ronan Sicre, and Hervé Jégou. 2015. Particular object retrieval with integral max-pooling of CNN activations. International Conference on Learning Representations (ICLR). vol abs/1511.05879,
17. Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep Residual Learning for Image Recognition. IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 770 - 778
18. Ross Girshick. 2015. Fast R-CNN. IEEE International Conference on Computer Vision (ICCV). 1440 - 1448
19. J. R. R. Uijlings, K. E. A. van de Sande, T. Gevers, and A. W. M. Smeulders. 2013. Selective search for object recognition. International Journal of Computer Vision. vol 104, no 2: 154-171
20. Matthew D Zeiler and Rob Fergus. 2013. Visualizing and understanding convolutional neural networks. European Conference on Computer Vision (ECCV).
21. Ross Girshick, Jeff Donahue, Trevor Darrell, and Jitendra Malik. 2014. Rich feature hierarchies for accurate object detection and semantic segmentation. IEEE Conference on Computer Vision and Pattern Recognition. 580 - 587
22. Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. 2014. Dropout: A Simple Way to Prevent Neural Networks from Overfitting. Journal of Machine Learning Research. vol 15: 1929-1958
23. Yangqing Jia, Evan Shelhamer, Jeff Donahue, Sergey Karayev, Jonathan Long, Ross Girshick, Sergio Guadarrama, and Trevor Darrell. 2014. Caffe: Convolutional architecture for fast feature embedding. MM '14 Proceedings of the 22nd ACM international conference on Multimedia.