臺灣博碩士論文加值系統

近年來在許多不同的理論以及應用上，深度學習透過大量的標記資料的監督式學習發展得相當迅速。然而，收集大量標記資料實際上是相當耗費時間以及資源的。在現實世界中我們並無法總是取得含標記資料，有時候我們只能取得少量的標記資料甚至只有無標記資料。如何利用數據增強(data augmentation)和改進模型的正規化(regularization)是相當關鍵的問題。近年來對抗式學習被廣泛運用於生成近乎真實的資料並且沒有傳統生成模型需要處理馬可夫鍊的收斂問題(mixing problem). 本篇論文主要探討如何利用深層對抗式學習再產生新的訓練資料。 我們的目標在於利用對抗式網路產生新的訓練資料並且應用在產品製造過程的缺陷檢測上。 為了改進系統的效能，我們建立了一個條件式對抗生成網路(conditional generative adversarial networks)並且引入了額外的隱藏代碼(latent code)並且最大化生成樣本與其的互信息(mutual information)。 我們將此非監督式學習模型應用在銅箔缺陷的產品檢測上。
另一方面，轉移式學習也提供了另一種方法來處理標記資料不足的問題而不需透過產生新樣本。轉移式學習根據不同的設定可被細分為各種不同的議題，在本篇論文我們著重的議題為領域調適(domain adaptation)。領域調適主要透過取的源領域(source domain)與標的領域(target domain)資料的共同特徵(shared feature)來將源領域模型調適到標的領域並實現知識的轉移。傳統領域調適方法專注於取得源領域與標的領域的共同特徵而獨有特徵(individual feature)則未被考量。在本篇論中我們提出一個深層混合對抗式學習的架構，此架構可以同時取出源領域與標的領域的共同特徵以及獨有特徵。我們的概念是透過取得含分類類別資訊的共有特徵以及含各領域獨有資訊的獨有特徵來幫助領域調適的學習。我們利用調適網路來取出共有特徵以及分離網路來取出獨有資訊。調適網路和分離網路都是對抗式網路。我們透過在調適網路和分離網路引入混和對抗式學習(hybrid adversarial learning)來分別最佳化分離損失和領域依賴性(domain dependency)上的最小最大化(minimax)問題。調適網路的概念在於我們透過取出無法被一個最佳化的判別器正確分類的特徵作為源領域和標的領域的共同特徵。分離網路的概念在於我們透過取出可以被一個較差的判別器正確的分類的特徵作為獨有特徵。換言之，我們取出來的特徵必須要足夠好來強迫判別器正確分類。
在對抗式生成模型的試驗中，我們比較了在生成銅箔缺陷影像上不同非監督式學習方法的表現。一般而言，缺陷檢測需要相當高的正確率然而產品的缺陷率會式相對較低了，這意味著在鑑別含缺陷以及不含缺陷上，兩者的資料量將相當的不平均，而這將影響分類器的準確性。在領域調適的問題上，我們比較了我們提出的模型在不同的工作上的表現並且展示了對抗式領域分離與調適在文章分類與影像辨識上的效果。

Deep learning has been rapidly developing from different aspects of theories and applications where a large amount of labeled data are available for supervised training. However, in practice, it is time-consuming to collect a large set of labeled data. In real world, we may only observe a limited set of labeled data and unlabeled data. How to perform data augmentation and improve model regularization is a crucial research topic. Recently, adversarial learning has been discovering to generate or synthesize realistic data without the mixing problem in traditional model based on Markov chain. This thesis deals with the generation of new training samples based on deep adversarial learning. Our goal is to carry out the adversarial generation of new samples and apply it for defect classification in manufacturing process. To improve system performance, we introduce the additional latent codes and maximize the mutual information between generative samples and latent codes to build a conditional generative adversarial model. This model is capable of generating a variety of samples under the same class. We evaluate the performance of this unsupervised model by detecting the defect conditions in production process of copper foil images.
On the other hand, transfer learning provides an alternative method to handle the problem of insufficient labeled data where data generation is not required. Transfer learning involves several issues owing to different setups. The issue we concern is mainly on domain adaptation. Domain adaptation aims to adapt a model from source domain to target domain through learning the shared representation that allows knowledge transfer across domains. Traditional domain adaptation methods are specialized to learn the shared representation for distribution matching between source domain and target domain where the individual information in both domains is missing. In this thesis, we present a deep hybrid adversarial learning framework which captures the shared information and the individual information simultaneously. Our idea is to estimate the shared feature which is informative for classification and the individual feature which contains the domain specific information. We use adaptation network to extract the shared feature and separation network to extract individual feature. Both adaptation and separation network are seen as an adversarial network. A hybrid adversarial learning is incorporated in the separation network as well as the adaptation network where the according to the minimax optimization over separation loss and domain discrepancy, respectively. The idea in the adaptation network is that we want to extract shared feature that an optimal discriminator cannot tell where feature come from. The idea in the separation network is that we want to extract feature including shared and individual feature which can be separated even by a bad discriminator. In other words the features have to be good enough to force the discriminator to classify them correctly.
For the experiment on generative adversarial model, we investigate different unsupervised learning methods for defect detection in presence of copper foil images. In general, defect detection requires very high accuracy but the defect rate usually is relatively low which means the images with and without defect are very unbalanced. We generate the defective images to balance the training data between defective images and non-defective images conditioned on different classes. For the experiments on domain adaptation problem, we evaluate the proposed method on different tasks and show the merit of using the proposed adversarial domain separation and adaptation in the tasks of sentiment classification and image recognition.

Contents
Chinese Abstract iii
English Abstract v
Contents vii
List of Tables x
List of Figures xi
List of Notations xii
List of Abbreviations xv
1. Introduction 1
1.1 Background Survey 1
1.2 Application of Domain Adaptation 2
1.3 Motivation 3
1.4 Contribution 4
1.5 Outline of Thesis 4
2. Deep Learning and Domain Adaptation 6
2.1 Transfer Learning 6
2.1.1 Introduction 7
2.1.2 Semi-Supervised Learning 11
2.1.3 Distribution Matching 12
2.2 Deep Neural Network 15
2.2.1 Introduction 16
2.2.2 Deep Transfer Learning 19
3. Adversarial Generative Model 23
3.1 Generative Adversarial Networks 23
3.1.1 Introduction 23
3.1.2 Deep Convolution Adversarial Networks 25
3.1.3 Gradient Vanish and Mode Collapse 25
3.1.4 Model Regularization in GAN 27
3.2 Advances in GANs 31
3.2.1 Coupled Generative Adversarial Networks 31
3.2.2 Information Generative Adversarial Networks 33
4. Adversarial Domain Adaptation 35
4.1 Adversarial Learning for Domain Adaptation 35
4.2 Model Structure 38
4.2.1 Classification Network 40
4.2.2 Adaptation Network 41
4.2.3 Separation Network 41
4.3 Objective Function and Optimization Procedure 42
4.3.1 Hybrid Adversarial Learning 43
5. Experiments on Adversarial Generative Model 46
5.1 Experiment for Defect Detection 46
5.2 Evaluation for Quality of Generated Data 47
5.3 Evaluation for Diversity of Generated Data 49
6. Experiments on Adversarial Domain Adaptation 53
6.1 Data Description 53
6.2 Evaluation for Optimization Strategies 55
6.2.1 Experiment Setup 56
6.2.2 Experiment Results 57
6.3 Sentiment Classification 58
6.3.1 Experiments Setup 58
6.3.2 Experimental Result 59
6.4 Digit Classification 62
6.4.1 Experiment Setup 62
6.4.2 Experimental Result 62
7. Conclusion and Future Works 67
7.1 Conclusions 67
7.2 Future Works 67
Bibliography 69

[1] S. J. Pan and Q. Yang, “A survey on transfer learning,” IEEE Transactions on Knowledge and Data Engineering, vol. 22, no. 10, pp. 1345-1359, 2010.
[2] M. Sugiyama, M. Krauledat, and K.-R. MÞller, “Covariate shift adaptation by importance weighted cross validation,” Journal of Machine Learning Research, vol. 8, no. May, pp. 985-1005, 2007.
[3] J. Blitzer, R. McDonald, and F. Pereira, “Domain adaptation with structural correspondence learning,” in Proceedings of Conference on Empirical Methods in Natural Language Processing, pp. 120-128, 2006.
[4] H. Daumé III, “Frustratingly easy domain adaptation,” in Proceedings of Conference of the Association for Computational Linguistics (ACL), 2007.
[5] S. J. Pan, J. T. Kwok, and Q. Yang, “Transfer learning via dimensionality reduction,” in Proceedings of Advancement of Artificial Intelligence (AAAI), vol. 8, pp. 677-682, 2008.
[6] S. J. Pan, I. W. Tsang, J. T. Kwok, Q. Yang, et al., “Domain adaptation via transfer component analysis,” IEEE Transactions on Neural Networks, vol. 22, no. 2, pp. 199-210, 2011.
[7] M. Chen, Z. Xu, K. Weinberger, and F. Sha, “Marginalized denoising autoencoders for domain adaptation,” 2012.
[8] E. Tzeng, J. Hoffman, N. Zhang, K. Saenko, and T. Darrell, “Deep domain confusion: Maximizing for domain invariance,” arXiv preprint arXiv:1412.3474, 2014.
[9] Y. Ganin, E. Ustinova, H. Ajakan, P. Germain, H. Larochelle, F. Laviolette, M. Marchand, and V. Lempitsky, “Domain-adversarial training of neural networks,” Journal of Machine Learning Research, vol. 17, no. 59, pp. 1-35, 2016.
[10] C. Louizos, K. Swersky, Y. Li, M. Welling, and R. Zemel, “The variational fair auto encoder,” in International Conference on Learning Representations(ICLR), 2016.
[11] M. Long, H. Zhu, J. Wang, and M. I. Jordan, “Unsupervised domain adaptation
with residual transfer networks,” in Advances in Neural Information Processing
Systems, pp. 136-144, 2016.
[12] K. Bousmalis, G. Trigeorgis, N. Silberman, D. Krishnan, and D. Erhan, “Domain
separation networks,” in Advances in Neural Information Processing Systems,
pp. 343-351, 2016.
[13] R. Raina, A. Battle, H. Lee, B. Packer, and A. Y. Ng, “Self-taught learning:
transfer learning from unlabeled data,” in Proceedings of International Conference on Machine Learning (ICML), pp. 759-766, ACM, 2007.
[14] J. Huang, A. Gretton, K. M. Borgwardt, B. Sch¨olkopf, and A. J. Smola, “Correcting sample selection bias by unlabeled data,” in Advances in Neural Information
Processing Systems (NIPS), pp. 601-608, 2006.
[15] M. Sugiyama, S. Nakajima, H. Kashima, P. V. Buenau, and M. Kawanabe, “Direct importance estimation with model selection and its application to covariate shift adaptation,” in Advances in Neural Information Processing Systems
(NIPS), pp. 1433-1440, 2008
[16] T. Kanamori, S. Hido, and M. Sugiyama, “A least-squares approach to direct importance estimation,” Journal of Machine Learning Research, vol. 10, pp. 1391-
1445, 2009.
[17] S. Bickel, M. Br¨uckner, and T. Scheffer, “Discriminative learning under covariate
shift,” Journal of Machine Learning Research, vol. 10, pp. 2137-2155, 2009.
[18] F. Zhuang, X. Cheng, S. J. Pan, W. Yu, Q. He, and Z. Shi, “Transfer learning
with multiple sources via consensus regularized autoencoders,” in Joint European Conference on Machine Learning and Knowledge Discovery in Databases,
pp. 417-431, Springer, 2014.
[19] A. T. Selvan, S. Samanta, and S. Das, “Domain adaptation using weighted subspace sampling for object categorization,” in Advances in Pattern Recognition
(ICAPR), 2015 Eighth International Conference on, pp. 1-5, IEEE, 2015.
[20] J. Jiang and C. Zhai, “Instance weighting for domain adaptation in nlp," in ACL,
vol. 7, pp. 264{271, 2007.
[21] B. Zadrozny, “Learning and evaluating classifiers under sample selection bias,”
in Proceedings of International Conference on Machine Learning (ICML), p. 114,
ACM, 2004.
[22] A. Gretton, K. M. Borgwardt, M. Rasch, B. Sch¨olkopf, and A. J. Smola, “A
kernel method for the two-sample-problem,” in Advances in Neural Information
Processing Systems (NIPS), pp. 513-520, 2007.
[23] K. M. Borgwardt, A. Gretton, M. J. Rasch, H.-P. Kriegel, B. SchÖlkopf, and
A. J. Smola, “Integrating structured biological data by kernel maximum mean
discrepancy,” Bioinformatics, vol. 22, no. 14, pp. e49-e57, 2006.
[24] I. Steinwart, “On the influence of the kernel on the consistency of support vector
machines,” Journal of Machine Learning Research, vol. 2, pp. 67-93, 2002.
[25] D. E. Rumelhart, G. E. Hinton, and R. J. Williams, “Learning internal representations by error propagation,” in Parallel Distributed Processing: Explorations inthe Microstructure of Cognition (D. E. Rumelhart, J. L. McClelland, and C. PDP
Research Group, eds.), vol. 1, pp. 318-362, MIT Press, 1986.
[26] Q. V. Le, J. Ngiam, A. Coates, A. Lahiri, B. Prochnow, and A. Y. Ng, “On optimization methods for deep learning,” in Proceedings of International Conference
on Machine Learning (ICML), pp. 265-272, 2011.
[27] J. Martens, “Deep learning via Hessian-free optimization,” in Proceedings of
International Conference on Machine Learning (ICML), pp. 735-742, 2010.
[28] D. Kingma and J. Ba, “Adam: A method for stochastic optimization,” 2015.
[29] N. Qian, “On the momentum term in gradient descent learning algorithms,”
Neural Networks, vol. 12, no. 1, pp. 145-151, 1999.
[30] M. Kan, S. Shan, and X. Chen, “Bi-shifting auto-encoder for unsupervised domain adaptation,” in Proceedings of the IEEE International Conference on Computer Vision, pp. 3846-3854, 2015.
[31] F. Zhuang, X. Cheng, P. Luo, S. J. Pan, and Q. He, “Supervised representation
learning: transfer learning with deep autoencoders,” in Int. Joint Conf. Artif.
Intell, 2015.
[32] W. Jiang, H. Gao, F.-l. Chung, and H. Huang, “The l2, 1-norm stacked robust
autoencoders for domain adaptation,” in Thirtieth AAAI Conference on Artificial Intelligence, 2016.
[33] Y. Ganin and V. Lempitsky, “Unsupervised domain adaptation by backpropagation,” arXiv preprint arXiv:1409.7495, 2014.
[34] R. Collobert and J. Weston, “A unified architecture for natural language processing: Deep neural networks with multitask learning,” in Proceedings of the
25th international conference on Machine learning, pp. 160-167, ACM, 2008.
[35] X. Glorot, A. Bordes, and Y. Bengio, “Domain adaptation for large-scale sentiment classification: A deep learning approach,” in Proceedings of International
Conference on Machine Learning (ICML), pp. 513-520, 2011.
[36] T. H. Nguyen and R. Grishman, “Event detection and domain adaptation with
convolutional neural networks,” in Proceedings of the 53rd Annual Meeting of
the Association for Computational Linguistics and the 7th International Joint
Conference on Natural Language Processing, vol. 2, pp. 365-371, 2015.
[37] R. Johnson and T. Zhang, “Semi-supervised convolutional neural networks for
text categorization via region embedding,” in Advances in neural information
processing systems, pp. 919-927, 2015.
[38] J. Hu, J. Lu, and Y.-P. Tan, “Deep transfer metric learning,” in Proceedings of
the IEEE Conference on Computer Vision and Pattern Recognition, pp. 325-333,
2015.
[39] J. Yosinski, J. Clune, Y. Bengio, and H. Lipson, “How transferable are features
in deep neural networks?,” in Advances in neural information processing systems,
pp. 3320-3328, 2014.
[40] H.-Y. Chen and J.-T. Chien, “Deep semi-supervised learning for domain adaptation,” in 2015 IEEE 25th International Workshop on Machine Learning for
Signal Processing (MLSP), pp. 1-6, IEEE, 2015.
[41] M. Long and J. Wang, “Learning transferable features with deep adaptation
networks,” CoRR, abs/1502.02791, vol. 1, p. 2, 2015.
[42] A. Rahimi and B. Recht, “Weighted sums of random kitchen sinks: Replacing
minimization with randomization in learning,” in Advances in neural information
processing systems, pp. 1313-1320, 2009.
[43] I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair,
A. Courville, and Y. Bengio, “Generative adversarial nets,” in Advances in Neural Information Processing Systems, pp. 2672-2680, 2014.
[44] A. Radford, L. Metz, and S. Chintala, “Unsupervised representation learning with deep convolutional generative adversarial networks,” arXiv preprint
arXiv:1511.06434, 2015.
[45] S. Nowozin, B. Cseke, and R. Tomioka, “f-gan: Training generative neural samplers using variational divergence minimization,” in Advances in Neural Information Processing Systems, pp. 271-279, 2016.
[46] M. Arjovsky, S. Chintala, and L. Bottou, “Wasserstein gan,” arXiv preprint
arXiv:1701.07875, 2017.
[47] X. Mao, Q. Li, H. Xie, R. Y. Lau, Z. Wang, and S. P. Smolley, “Least squares
generative adversarial networks,” arXiv preprint ArXiv:1611.04076, 2016.
[48] M.-Y. Liu and O. Tuzel, “Coupled generative adversarial networks,” in Advances
in Neural Information Processing Systems, pp. 469-477, 2016.
[49] X. Chen, Y. Duan, R. Houthooft, J. Schulman, I. Sutskever, and P. Abbeel, “InfoGAN: Interpretable representation learning by information maximizing generative adversarial nets,” in Advances in Neural Information Processing Systems,
pp. 2172{2180, 2016.
[50] G. Salton and C. Buckley, “Term-weighting approaches in automatic text retrieval,” Information processing & management, vol. 24, no. 5, pp. 513-523,
1988.
[51] F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel,
M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay, “Scikit-learn:
Machine learning in Python,” Journal of Machine Learning Research, vol. 12,
pp. 2825-2830, 2011.
[52] P. Arbelaez, M. Maire, C. Fowlkes, and J. Malik, “Contour detection and hierarchical image segmentation,” IEEE Transactions on Pattern Analysis and
Machine Intelligencen, vol. 33, no. 5, pp. 898-916, 2011.
[53] M. Long, J. Wang, G. Ding, J. Sun, and P. S. Yu, “Transfer feature learning
with joint distribution adaptation,” in Proceedings of the IEEE International
Conference on Computer Vision, pp. 2200-2207, 2013.
[54] D. L. Davies and D. W. Bouldin, “A cluster separation measure," IEEE Transactions on Pattern Analysis and Machine Intelligence, no. 2, pp. 224-227, 1979.
[55] J. Blitzer, M. Dredze, F. Pereira, et al., “Biographies, bollywood, boom-boxes
and blenders: Domain adaptation for sentiment classification,” in Proceedings
of Conference of the Association for Computational Linguistics (ACL), vol. 7,
pp. 440-447, 2007.
[56] I. Sutskever, J. Martens, G. Dahl, and G. Hinton, “On the importance of initialization and momentum in deep learning,” in International Conference on
Machine Learning, pp. 1139-1147, 2013.