臺灣博碩士論文加值系統

通常收集有標記的資料所耗費的成本是高昂的. 而另一方面, 有時候
我們在相關領域會擁有較多的標記資料. 如果沒有足夠的訓練資料, 一
些分類器如最鄰近結點演算法(kNN) 或者支持向量機(SVM) 就不容易
達到較佳的分類效果. 在這一篇論文裡, 我們研究利用少許的目標領域
的標記資料與數量相對較多的來源領域的標記資料, 來提昇對於目標領
域的分類結果. 我們假設來源領域與目標領域擁有不同的特徵空間. 此
外, 這兩個領域也假設沒有明顯的共同特徵, 但擁有相同的標記空間.
利用其他領域資料的一個關鍵技術在於找出兩個映射的函數, 使得
在來源空間與目標空間的資料可以被映射至一個共同的空間. 在這一篇
論文, 我們提出一個簡易又直覺的方法, 稱為線性分辨式映射, 來處理這
個問題. 首先, 我們使用分辨方法如線性判別分析(LDA), 將來源領域的
資料依據標記做分群. 接著使用回歸技術(regression) 將目標領域的標
記資料盡可能地映射到與自己相同標記的來源資料群體的中心. 最後,
我們再一次使用分辨方法將所有的標記資料都依據標記做分群. 實驗結
果顯示, 在重要的資料集中, 我們的方法於少量標記資料的監督式分類
上學習具分辨能力的特徵是有效益的.

It is often expensive to collect labeled data and we sometimes have large
amounts of labeled data in a related domain. Without enough training data,
some classifiers such as k-Nearest Neighbor (kNN) or Support Vector Machine
(SVM) may fail to achieve good classification performance. In this
thesis, we consider the problem of utilizing few labeled data samples in a
target domain and the data samples in a source domain to improve data classification
in the target domain. We assume that the source and target domains
have different feature spaces. In addition, the two domains are assumed to
share no explicit common features but have the same set of class labels.
A key technique for leveraging the data from another domain is to find
two mapping functions so that the source and target spaces can be projected
on a common space. In this thesis, we present a simple and intuitive technique
called linear discriminative projections to address the problem. First, we separate
the source data of distinct classes by using a discriminative method such
as Linear Discriminative Analysis (LDA). We then apply a regression technique
to map each labeled target data instance as close as possible to the center
of the source data group with the same class label. Finally, we again use a
discriminative method to separate all the data of distinct classes. Experimental
results on some benchmark datasets clearly demonstrate that our approach
is effective for learning discriminative features for supervised classification
with few training target data.

中文摘要iii
Abstract v
1 Introduction 1
2 Literature Review 3
2.1 Background and Topic Development . . . . . . . . . . . . . . . . . . . . 3
2.1.1 Distance Metric Learning . . . . . . . . . . . . . . . . . . . . . 3
2.1.2 Domain Adaptation . . . . . . . . . . . . . . . . . . . . . . . . . 7
2.1.3 Manifold Learning . . . . . . . . . . . . . . . . . . . . . . . . . 9
2.1.4 Feature Augmentation . . . . . . . . . . . . . . . . . . . . . . . 10
2.1.5 Other Related Literature . . . . . . . . . . . . . . . . . . . . . . 11
2.2 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
3 Heterogeneous Domain Adaption Problem 13
3.1 Supervised Classification with Few Training Data . . . . . . . . . . . . . 13
3.2 Min-Max Principle . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14
3.3 A Hypothesis for Generalization . . . . . . . . . . . . . . . . . . . . . . 15
3.3.1 Discriminative Clusters . . . . . . . . . . . . . . . . . . . . . . . 15
3.3.2 Pairwise Disjointness of Clusters . . . . . . . . . . . . . . . . . . 16
3.3.3 Existence of A Hypothesis . . . . . . . . . . . . . . . . . . . . . 17
4 Discriminative Projections Framework 18
4.1 High Level Explanation . . . . . . . . . . . . . . . . . . . . . . . . . . . 18
4.2 Learning Discriminative Features . . . . . . . . . . . . . . . . . . . . . . 20
4.2.1 Source Projection . . . . . . . . . . . . . . . . . . . . . . . . . . 20
4.2.2 Target Projection . . . . . . . . . . . . . . . . . . . . . . . . . . 20
4.2.3 Overall Projection . . . . . . . . . . . . . . . . . . . . . . . . . 22
4.3 Feature augmentation . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22
4.4 Discriminative Projections . . . . . . . . . . . . . . . . . . . . . . . . . 23
4.4.1 Linear Discriminative Analysis . . . . . . . . . . . . . . . . . . 23
4.4.2 Graph Embedding . . . . . . . . . . . . . . . . . . . . . . . . . 24
4.5 Framework . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27
5 Experimental Results 28
5.1 Datasets . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28
5.1.1 Object Recognition Dataset . . . . . . . . . . . . . . . . . . . . 28
5.1.2 Text Categorization Dataset . . . . . . . . . . . . . . . . . . . . 29
5.1.3 Handwritten Digits Datasets . . . . . . . . . . . . . . . . . . . . 30
vii
5.2 Comparative Studies . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31
5.2.1 Object Recognition Dataset . . . . . . . . . . . . . . . . . . . . 32
5.2.2 Text Categorization Dataset . . . . . . . . . . . . . . . . . . . . 32
5.3 Comparison in Different Settings . . . . . . . . . . . . . . . . . . . . . . 33
5.3.1 Object Recognition Dataset . . . . . . . . . . . . . . . . . . . . 34
5.3.2 Text Categorization Dataset . . . . . . . . . . . . . . . . . . . . 35
5.3.3 Handwritten Digits Dataset . . . . . . . . . . . . . . . . . . . . . 35
5.4 Comparison with Different Projections . . . . . . . . . . . . . . . . . . . 36
5.4.1 Object Recognition Dataset . . . . . . . . . . . . . . . . . . . . 36
5.4.2 Text Categorization Dataset . . . . . . . . . . . . . . . . . . . . 37
5.5 The Influence of the Number of Target Training Samples per Class . . . . 37
6 Conclusion 41
Bibliography 43
Appendices 49
Appendix A: Ridge Regression 50

[1] Massih-Reza Amini, Nicolas Usunier, and Cyril Goutte. Learning from multiple
partially observed views - an application to multilingual text categorization. In Pro-
ceedings of the Twenty-Third Annual Conference on Advances in Neural Information
Processing Systems (NIPS-09), pages 28–36, 2009.
[2] K. Bache andM. Lichman. UCI machine learning repository. http://archive.
ics.uci.edu/ml, 2013. University of California, Irvine, School of Information
and Computer Sciences.
[3] Herbert Bay, Tinne Tuytelaars, and Luc J. Van Gool. Surf: Speeded up robust features.
In Proceedings of 9th European Conference on Computer Vision (ECCV’06),
pages 404–417, 2006.
[4] Mikhail Belkin and Partha Niyogi. Laplacian eigenmaps and spectral techniques
for embedding and clustering. In Proceedings of the Fourteenth Annual Conference
on Advances in Neural Information Processing Systems (NIPS-01), pages 585–591,
2001.
[5] Mikhail Belkin and Partha Niyogi. Using manifold structure for partially labelled
classification. In Proceedings of the Sixteenth Annual Conference on Advances in
Neural Information Processing Systems (NIPS-03), pages 929–936, 2003.
[6] Gilles Bisson and Clement Grimal. Co-clustering of multi-view datasets: A parallelizable
approach. In Proceedings of 2012 IEEE 12th International Conference on
Data Mining (ICDM’12), pages 828–833, 2012.
[7] John Blitzer, RyanMcDonald, and Fernando Pereira. Domain adaptation with structural
correspondence learning. In Proceedings of Conference on Empirical Methods
in Natural Language Processing (EMNLP 06), pages 120–128, 2006.
[8] Yi-ting Chiang,Wen-Chieh Fang, and Jane Yung-jen Hsu. Knowledge source selection
by estimating distance between datasets. In Proceedings of the 2012 Conference on Technologies and Applications of Artificial Intelligence (TAAI-2012), pages 44 –
49, 2012.
[9] Wenyuan Dai, Yuqiang Chen, Gui-Rong Xue, Qiang Yang, and Yong Yu. Translated
learning: Transfer learning across different feature spaces. In Proceedings of the
Twenty-Second Annual Conference on Advances in Neural Information Processing
Systems (NIPS-08), pages 353–360, 2008.
[10] Wenyuan Dai, Qiang Yang, Gui-Rong Xue, and Yong Yu. Boosting for transfer
learning. In Proceedings of the Twenty-Fourth International Conference Machine
Learning (ICML’ 07), pages 193–200, 2007.
[11] Halneigh Daum′e III. Frustratingly easy domain adaptation. In Proceedings of the
45th Annual Meeting of the Association of Computational Linguistics, pages 256–
263, 2007.
[12] Lixin Duan, Dong Xu, and Ivor W. Tsang. Learning with augmented features for
heterogeneous domain adaptation. In Proceedings of the 29th International Confer-
ence on Machine Learning (ICML-12), 2012.
[13] Andrea Frome, Yoram Singer, and JitendraMalik. Image retrieval and classification
using local distance functions. In Proceedings of the Eighteenth Annual Conference
on Advances in Neural Information Processing Systems (NIPS-06), pages 417–424,
2006.
[14] Andrea Frome, Yoram Singer, Fei Sha, and Jitendra Malik. Learning globallyconsistent
local distance functions for shape-based image retrieval and classification.
In Proceedings of IEEE Eleventh International Conference on Computer Vision
(ICCV 07), pages 1–8, 2007.
[15] Keinosuke Fukunaga. Introduction to statistical pattern recognition. Academic
Press Professional, Inc., 1990.
[16] Jacob Goldberger, Sam Roweis, Geoffrey Hinton, and Ruslan Salakhutdinov. Neighbourhood
components analysis. In Proceedings of the Seventeenth Annual Con-
ference on Advances in Neural Information Processing Systems (NIPS-05), pages
513–520, 2005.
[17] Jihun Ham, Daniel D. Lee, and Lawrence K. Saul. Semisupervised alignment of
manifolds. In Proceedings of 10th International Workshop on Artificial Intelligence
and Statistics (AISTATS 2005), pages 120–127, 2005.
[18] S&;oslash;ren Hauberg, Oren Freifeld, and Michael J. Black. A geometric take on metric
learning. In P. Bartlett, F.C.N. Pereira, C.J.C. Burges, L. Bottou, and K.Q. Weinberger,
editors, Proceedings of the Twenty-Six Annual Conference on Advances in
Neural Information Processing Systems (NIPS-12), pages 2033–2041. MIT Press,
2012.
[19] Xiaofei He and Partha Niyogi. Locality preserving projections. In Sebastian Thrun,
Lawrence Saul, and Bernhard Sch‥olkopf, editors, Advances in Neural Information
Processing Systems 16. MIT Press, Cambridge, MA, 2004.
[20] Steven C. H. Hoi, Wei Liu, and Shih-Fu Chang. Semi-supervised distance metric
learning for collaborative image retrieval. In Proceedings of the IEEE Conference
on Computer Vision and Pattern Recognition (CVPR-08), 2008.
[21] Rong Jin, ShijunWang, and Yang Zhou. Regularized distance metric learning: Theory
and algorithm. In Proceedings of the Twenty-Third Annual Conference on Ad-
vances in Neural Information Processing Systems (NIPS-09), pages 862–870, 2009.
[22] Brian Kulis, Prateek Jain, and Kristen Grauman. Fast similarity search for learned
metrics. IEEE Trans. Pattern Anal. Mach. Intell., 31(12):2143–2157, 2009.
[23] Brian Kulis, Kate Saenko, and Trevor Darrell. What you saw is not what you get:
Domain adaptation using asymmetric kernel transforms. In Proceedings of the IEEE
Conference on Computer Vision and Pattern Recognition (CVPR-11), pages 1785–
1792, 2011.
[24] John Langford. Tutorial on practical prediction theory for classification. Journal of
Machine Learning Research, 6:273–306, 2005.
[25] Xiao Ling, Gui-Rong Xue, Wenyuan Dai, Yun Jiang, Qiang Yang, and Yong Yu.
Can chinese web pages be classified with english data source? In Proceedings of the
17th International Conference on World Wide Web (WWW 2008), pages 969–978,
2008.
[26] H. Liu, H. Motoda, and L. Yu. Feature extraction, selection, and construction. In
N. Ye, editor, The Handbook of Data Mining, pages 409–423. Lawrence Erlbaum,
2003.
[27] Wei Liu, ShiqianMa, Dacheng Tao, Jianzhuang Liu, and Peng Liu. Semi-supervised
sparse metric learning using alternating linearization optimization. In Proceedings
of the 16th ACM SIGKDD international conference on Knowledge discovery and
data mining (KDD-10), pages 1139–1148, 2010.
[28] Wei Liu, Xinmei Tian, Dacheng Tao, and Jianzhuang Liu. Constrained metric learning
via distance gap maximization. In Proceedings of the Twenty-Fourth AAAI Con-
ference on Artificial Intelligence (AAAI-10), 2010.
[29] Yang Mu, Wei Ding, and Dacheng Tao. Local discriminative distance metrics ensemble
learning. Pattern Recognition, 46(8):2337–2349, 2013.
[30] Sinno Jialin Pan, Xiaochuan Ni, Jian-Tao Sun, Qiang Yang, and Zheng Chen. Crossdomain
sentiment classification via spectral feature alignment. In Proceedings of
the 19th international conference on World Wide Web (WWW’10), pages 751–760,
2010.
[31] Sinno Jialin Pan, Ivor W. Tsang, James T. Kwok, and Qiang Yang. Domain adaptation
via transfer component analysis. In Proceedings of the 21st international jont
conference on Artifical intelligence, IJCAI’09, pages 1187–1192, San Francisco,
CA, USA, 2009. Morgan Kaufmann Publishers Inc.
[32] Sinno Jialin Pan and Qiang Yang. A survey on transfer learning. IEEE Transactions
on Knowledge and Data Engineering, 22(10):1345–1359, 2010.
[33] Deva Ramanan and Simon Baker. Local distance functions: A taxonomy, new algorithms,
and an evaluation. IEEE Transactions on Pattern Analysis and Machine
Intelligence (PAMI), 33(4):794–806, 2011.
[34] Kate Saenko, Brian Kulis, Mario Fritz, and Trevor Darrell. Adapting visual category
models to new domains. In Proceedings of the 11th European conference on
Computer vision: Part IV, pages 213–226, 2010.
[35] Viktoriia Sharmanska, Novi Quadrianto, and Christoph H. Lampert. Augmented
attribute representations. In Proceedings of the 12th European Conference on Com-
puter Vision (ECCV -12), pages 242–255, 2012.
[36] John Shawe-Taylor and Nello Cristianini. Kernel Methods for Pattern Analysis.
Cambridge University Press, 2004.
[37] Xiaoxiao Shi, Qi Liu, Wei Fan, Philip S. Yu, and Ruixin Zhu. Transfer learning on
heterogenous feature spaces via spectral transformation. In Proceedings of the 2010
IEEE International Conference on Data Mining, pages 1049–1054, 2010.
[38] Yuan Shi and Fei Sha. Information-theoretical learning of discriminative clusters
for unsupervised domain adaptation. In Proceedings of the 29th International Con-
ference on Machine Learning (ICML-12), pages 1079–1086, 2012.
[39] Ulrike von Luxburg. A tutorial on spectral clustering. Statistics and Computing,
pages 395–416, 2007.
[40] ChangWang and SridharMahadevan. Heterogeneous domain adaptation usingmanifold
alignment. In Proceedings of the Twenty-Second international joint conference
on Artificial Intelligence - Volume Volume Two, pages 1541–1546, 2011.
[41] Chang Wang and Sridhar Mahadevan. Manifold alignment preserving global geometry.
Technical Report UM-CS-2012-031, UMass Computer Science Department,
2012.
[42] Jun Wang, Alexandros Kalousis, and Adam Woznica. Parametric local metric
learning for nearest neighbor classification. In Proceedings of the Twenty-Six An-
nual Conference on Advances in Neural Information Processing Systems (NIPS-12),
pages 1610–1618, 2012.
[43] Xiang Wang, Buyue Qian, and Ian Davidson. Improving document clustering using
automated machine translation. In Proceedings of International Conference on
Information and Knowledge Management (CIKM’12), pages 645–653, 2012.
[44] Kilian Q.Weinberger, John Blitzer, and Lawrence K. Saul. Distance metric learning
for large margin nearest neighbor classification. In Proceedings of the Eighteenth
Annual Conference on Advances in Neural Information Processing Systems (NIPS-
05), 2005.
[45] Kilian Q.Weinberger and Lawrence K. Saul. Distance metric learning for large margin
nearest neighbor classification. Journal of Machine Learning Research, 10:207–
244, 2009.
[46] Gang Wu, Edward Y. Chang, and Navneet Panda. Formulating context-dependent
similarity functions. In Proceedings of the 13th ACM International Conference on
Multimedia, pages 725–734, 2005.
[47] Yejun Wu and Douglas W. Oard. Bilingual topic aspect classification with a few
training examples. In Proceedings of the 31st Annual International ACM SIGIR
Conference on Research and Development in Information Retrieval (SIGIR 2008),
pages 203–210, 2008.
[48] Eric P. Xing, Andrew Y. Ng, Michael I. Jordan, and Stuart J. Russell. Distance
metric learning with application to clustering with side-information. In Proceedings
of the Fifteenth Annual Conference on Advances in Neural Information Processing
Systems (NIPS-02), pages 505–512, 2002.
[49] Shuicheng Yan, Dong Xu, Benyu Zhang, Hong-Jiang Zhang, Qiang Yang, and
Stephe Lin. Graph embedding and extensions: A general framework for dimensionality
reduction. IEEE Transactions on Pattern Analysis and Machine Intelligence,
pages 40–51, 2007.
[50] Qiang Yang, Yuqiang Chen, Gui-Rong Xue, Wenyuan Dai, and Yong Yu. Heterogeneous
transfer learning for image clustering via the social web. In Proceedings
of the 47th Annual Meeting of the ACL and the 4th IJCNLP of the AFNLP (ACL-
IJCNLP-09), ACL ’09, 2009.
[51] Yin Zhu, Yuqiang Chen, Zhongqi Lu, Sinno Jialin Pan, Gui-Rong Xue, Yong Yu,
and Qiang Yang. Heterogeneous transfer learning for image classification. In Pro-
ceedings of the Twenty-Fifth AAAI Conference on Artificial Intelligence, 2011.