本論文探討具備語音功能的兩項雲端應用技術：個人化的語言模型
及互動式語音文件檢索。
在語音辨識中，模型的不匹配對辨識率一向有很大的損害。由於個
人化手機普及，個人化辨識系統成為可行，而考量到每個個人語言使
用習慣的差異，語言模型的個人化有其必要。過去個人語料庫建立不
易，但如今，越來越多人習慣性地在社群網站上留下大量的文章與留
言，故個人化語料庫較以前容易取得許多。但資料稀疏的問題仍然不
易解決。在本論文中，我們提出以各種方式估計社群網站上不同使用
者間的用語相似度，並據以加入不同使用者的語料庫來幫助估計更強
健的個人化語言模型。我們並比較了用N 連文法語言模型及遞迴式類
神經網路語言模型來實做的效能表現，並驗證了新提出的方法確實提
升了對個人語言的預測能力。
在第二部分裡，我們探討互動式語音文件檢索。由於語音文件很難
呈現且瀏覽耗時，而過差的辨識率更可能使檢索結果不如人意，因此
藉由與使用者互動使系統對使用者想找的資訊有更多瞭解，是一個有
效改善此問題的方法。在本論文中，我們用馬可夫決策模型(Markov
Decision Process, MDP) 來模擬互動式檢索的問題，並採用強化學習
(Reinforcement Learning) 演算法學習出最佳系統決策，亦採用不同的檢
索模型來實作檢索系統。實驗顯示，我們提出的方法確實能夠輔助檢
索進行，幫助使用者更有效的找到所要找的資訊。

This thesis considers voice access of cloud applications with two parts: (1)
Personalized Language Model and (2) Interactive spoken document retrieval.
Model mismatch has been a major problem in speech recognition. With
hand-held devices widely used today, personalized models become possible.
A huge quantities of posts and comments with known owners emerged on
social network websites, personal corpora become practically available but
with data sparseness problem unsolved. In the first part of this thesis, we
proposed personalized language modeling approaches by estimating the language
similarities between different social network users and integrating the
corresponding personal corpora accordingly. We studied both N-gram language
models as well as recurrent neural network language models, and the
experimental results support the concept.
In the second part of this thesis, we studied interactive spoken document
retrieval. Interactive retrieval is helpful to spoken content retrieval because
retrieved spoken items are difficult to be shown on screen and browsed by
the user, in addition to the speech recognition uncertainty. We model the
interaction process by a Markov Decision Process and train the policy with
Reinforcement Learning. Experimental results demonstrate the retrieval performance
can be improved with the interactions.

口試委員會審定書 i
致謝 ii
中文摘要 iv
Abstract v
Contents vi
List of Figures x
List of Tables xii
1 導論 1
1.1 具備語音界面的雲端應用平台. . . . . . . . . . . . . . . . . . . . . . 1
1.1.1 個人化語音辨識系統. . . . . . . . . . . . . . . . . . . . . . . 3
1.1.2 互動式搜尋系統. . . . . . . . . . . . . . . . . . . . . . . . . . 4
1.2 章節安排. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
2 基礎背景簡介 7
2.1 個人化語音辨識系統. . . . . . . . . . . . . . . . . . . . . . . . . . . 7
2.1.1 聲學模型與調適. . . . . . . . . . . . . . . . . . . . . . . . . . 8
2.1.2 N 連文法語言模型(N-gram) . . . . . . . . . . . . . . . . . . . 9
2.1.3 類神經網路語言模型(NNLM) . . . . . . . . . . . . . . . . . . 10
2.1.4 語言模型調適. . . . . . . . . . . . . . . . . . . . . . . . . . . 13
2.2 互動式搜尋系統. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15
2.2.1 語音文件搜尋. . . . . . . . . . . . . . . . . . . . . . . . . . . 15
2.2.2 語音文件索引. . . . . . . . . . . . . . . . . . . . . . . . . . . 16
2.2.3 向量空間檢索模型(VSM) . . . . . . . . . . . . . . . . . . . . 18
2.2.4 語言模型檢索模型(LM) . . . . . . . . . . . . . . . . . . . . . 19
2.2.5 對話管理者- 馬可夫決策模型(MDP) . . . . . . . . . . . . . . 20
I 個人化語音辨識系統 23
3 社群網路群力模式 24
3.1 群力模式概念介紹. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24
3.2 具備語音界面的社群網路瀏覽器. . . . . . . . . . . . . . . . . . . . . 26
3.2.1 系統特點. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26
3.2.2 臉書認證機制. . . . . . . . . . . . . . . . . . . . . . . . . . . 27
3.2.3 系統生態系. . . . . . . . . . . . . . . . . . . . . . . . . . . . 28
4 個人化語言模型 29
4.1 N 連文法語言模型(N-gram LM) 個人化. . . . . . . . . . . . . . . . . 29
4.1.1 模型調適整體架構. . . . . . . . . . . . . . . . . . . . . . . . 31
4.1.2 基於社群網路互動關係之相似度估計. . . . . . . . . . . . . . 32
4.1.3 基於潛藏式主題模型之相似度估計. . . . . . . . . . . . . . . 33
4.1.4 基於隨機漫步演算法之相似度重估. . . . . . . . . . . . . . . 35
4.1.5 語句集群法. . . . . . . . . . . . . . . . . . . . . . . . . . . . 36
4.2 個人化N 連文法語言模型評估. . . . . . . . . . . . . . . . . . . . . . 37
4.2.1 實驗設定. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37
4.2.2 混淆度實驗結果與結果分析. . . . . . . . . . . . . . . . . . . 39
4.2.3 語音辨識實驗結果. . . . . . . . . . . . . . . . . . . . . . . . 41
4.3 遞迴式類神經網路語言模型(RNNLM) 的個人化. . . . . . . . . . . . 43
4.3.1 模型結構. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43
4.3.2 最佳化演算法- 沿時間反向傳播(BPTT) . . . . . . . . . . . . 45
4.3.3 上下文相關的遞迴式類神經網路與詞輔助特徵. . . . . . . . 46
4.3.4 三步驟調適機制. . . . . . . . . . . . . . . . . . . . . . . . . . 47
4.3.5 使用者導向的詞特徵. . . . . . . . . . . . . . . . . . . . . . . 50
4.4 個人化遞迴式類神經網路語言模型評估. . . . . . . . . . . . . . . . . 52
4.4.1 實驗設定. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52
4.4.2 混淆度實驗結果與結果分析. . . . . . . . . . . . . . . . . . . 53
4.4.3 前N 最佳結果重評分實驗結果. . . . . . . . . . . . . . . . . . 55
4.5 個人化語言模型結論. . . . . . . . . . . . . . . . . . . . . . . . . . . 58
5 個人化語言模型與個人化聲學模型整合 59
5.1 個人化聲學模型. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 59
5.2 整合系統架構. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 61
II 互動式搜尋系統 62
6 互動式語音搜尋系統 63
6.1 簡介. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 63
6.1.1 研究動機. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 63
6.1.2 系統組成. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 64
6.2 語音檢索的相關回饋. . . . . . . . . . . . . . . . . . . . . . . . . . . 65
6.2.1 向量空間模型相關回饋. . . . . . . . . . . . . . . . . . . . . . 66
6.2.2 語言模型相關性回饋. . . . . . . . . . . . . . . . . . . . . . . 67
7 馬可夫決策模型作為對話管理員 70
7.1 馬可夫決策模型模擬互動式搜尋系統. . . . . . . . . . . . . . . . . . 70
7.1.1 狀態(State) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 70
7.1.2 行動(Action) . . . . . . . . . . . . . . . . . . . . . . . . . . . . 71
7.1.3 獎勵(Reward) 與回報(Return) . . . . . . . . . . . . . . . . . . 73
7.2 強化學習演算法. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 73
7.2.1 值迭代學習法. . . . . . . . . . . . . . . . . . . . . . . . . . . 74
7.2.2 貼合值迭代學習法. . . . . . . . . . . . . . . . . . . . . . . . 74
7.3 狀態估計. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 76
7.3.1 狀態特徵值. . . . . . . . . . . . . . . . . . . . . . . . . . . . 76
7.3.2 離散狀態估計- 多類別支持向量機. . . . . . . . . . . . . . . 78
7.3.3 連續狀態估計- 正規化線性回歸法. . . . . . . . . . . . . . . 80
7.4 系統評估. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 80
7.4.1 實驗設定. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 81
7.4.2 模擬使用者. . . . . . . . . . . . . . . . . . . . . . . . . . . . 81
7.4.3 離散空間向量模型初始實驗. . . . . . . . . . . . . . . . . . . 82
7.4.4 離散、連續空間語言模型實驗與結果比較. . . . . . . . . . . 84
7.5 互動式搜尋系統結論. . . . . . . . . . . . . . . . . . . . . . . . . . . 86
8 結論與未來展望 87
8.1 雲端化的應用程式平台. . . . . . . . . . . . . . . . . . . . . . . . . . 87
8.2 個人化語音辨識系統. . . . . . . . . . . . . . . . . . . . . . . . . . . 87
8.3 互動式搜尋系統. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 88
Bibliography 90

[1] D. Hakkani-Tur, G. Tur, and L. Heck. Research challenges and opportunities in mobile applications. Signal Processing Magazine, IEEE, 2011.
[2] G. Hinton, Li Deng, Dong Yu, G.E. Dahl, A. Mohamed, N. Jaitly, A. Senior, V. Vanhoucke, P. Nguyen, T.N. Sainath, and B. Kingsbury. Deep neural networks for acoustic modeling in speech recognition: The shared views of four research groups. Signal Processing Magazine, IEEE, 2012.
[3] Ronan Collobert and Jason Weston. A unified architecture for natural language processing: deep neural networks with multitask learning. In Proceedings of the 25th international conference on Machine learning, ICML ''08. ACM, 2008.
[4] G.E. Dahl, Dong Yu, Li Deng, and A. Acero. Context-dependent pre-trained deep neural networks for large-vocabulary speech recognition. Audio, Speech, and Language Processing, IEEE Transactions on, 2012.
[5] Tee Kiah Chia, Khe Chai Sim, Haizhou Li, and Hwee Tu Ng. Statistical lattice-based spoken document retrieval. ACM Trans. Inf. Syst., 2010.
[6] Yi-Chen Pan, Hung-Yi Lee, and Lin-Shan Lee. Interactive spoken document retrieval with suggested key terms ranked by a markov decision process. Audio, Speech, and Language Processing, 2012.
[7] S. E. Johnson, P. Jourlin, G.L. Moore, K.S. Jones, and P.C. Woodland. The cambridge university spoken document retrieval system. In Acoustics, Speech, and
Signal Processing, 1999. Proceedings., 1999 IEEE International Conference on, 1999.
[8] Diane J. Litman and Scott Silliman. Itspoke: an intelligent tutoring spoken dialogue system. In Demonstration Papers at HLT-NAACL 2004, 2004.
[9] Stephanie Seneff and Joseph Polifroni. Dialogue management in the mercury flight reservation system. In Proceedings of the 2000 ANLP/NAACL Workshop on
Conversational systems, 2000.
[10] Teruhisa Misu and Tatsuya Kawahara. Bayes risk-based dialogue management for document retrieval system with speech interface. Speech Commun., January 2010.
[11] Berlin Chen and Yi-Ting Chen. Extractive spoken document summarization for information retrieval. Pattern Recognition Letters, 2008.
[12] Lin shan Lee and B. Chen. Spoken document understanding and organization. Signal Processing Magazine, IEEE, 2005.
[13] Jerome R. Bellegarda. Statistical language model adaptation: review and perspectives. Speech Communication, 2004.
[14] Aaron Heidel and Lin-Shan Lee. Robust topic inference for latent semantic language model adaptation. In Proc. on ASRU, 2007.
[15] Tsung-Hsien Wen, Hung-Yi Lee, Tai-Yuan Chen, and Lin-Shan Lee. Personalized language modeling by crowd sourcing with social network data for voice access of cloud applications. In Proc. on IEEE SLT workshop, 2012.
[16] Hung-Yi Lee, Tsung-Hsien Wen, and Lin-Shan Lee. Improved semantic retrieval of spoken content by language models enhanced with acoustic similarity graph. In SLT, 2012.
[17] Yun-Nung Chen, Chia-Ping Chen, Hung-Yi Lee, Chun-An Chan, and Lin-Shan Lee. Improved spoken term detection with graph-based re-ranking in feature space.
In Acoustics, Speech and Signal Processing (ICASSP), 2011 IEEE International Conference on, pages 5644--5647. IEEE, 2011.
[18] Hung-yi Lee and Lin-shan Lee. Integrating recognition and retrieval with user feedback: A new framework for spoken term detection. In Acoustics Speech and Signal Processing (ICASSP), 2010 IEEE International Conference on, pages 5290--5293. IEEE, 2010.
[19] Tsung-Hsien Wen, Hung-Yi Lee, and Lin-Shan Lee. Interactive spoken content retrieval with different types of actions optimized by a markov decision process. In
Interspeech, 2012.
[20] Tsung-Hsien Wen, Hung-yi Lee, Pei-hao Su, and Lin-Shan Lee. Interactive spoken content retrieval by extended query model and continuous state space markov decision process. In ICASSP, 2013.
[21] C. J. Leggetter and P. C. Woodland. Maximum likelihood linear regression for speaker adaptation of continuous density hidden markov models. Computer Speech and Language, 1995.
[22] luc Gauvain Jean and Chin-Hui Lee. Maximum a posteriori estimation for multivariate gaussian mixture observations of markov chains. IEEE Transactions on Speech and Audio Processing, 1994.
[23] P. C. Woodland. Speaker adaptation for continuous density hmms: A review. In Proc. on ITRW on Adaptation Methods for Speech Recognition, 2001.
[24] Peter F. Brown, Peter V. deSouza, Robert L. Mercer, Vincent J. Della Pietra, and Jenifer C. Lai. Class-based n-gram models of natural language. Computational Linguistics, 1992.
[25] William Gale. Good-Turing Smoothing Without Tears. Technical report, AT&T Bell Laboratories, 1994.
[26] Frankie James. Modified kneser-ney smoothing of n-gram models modified kneserney smoothing of n-gram models. Technical report, 2000.
[27] Georey Hinton. A Practical Guide to Training Restricted Boltzmann Machines. Technical report, 2010.
[28] Yoshua Bengio, R ejean Ducharme, Pascal Vincent, and Christian Janvin. A neural probabilistic language model. Journal of Machine Learning Research, 2003.
[29] Junho Park, Xunying Liu, Mark J. F. Gales, and Philip C. Woodland. Improved neural network based language modeling and adaptation. In Proc. on InterSpeech, 2010.
[30] Hai-Son Le, Ilya Oparin, Alexandre Allauzen, Jean-Luc Gauvain, and Francois Yvon. Structured Output Layer neural network language model. In Proc. on ICASSP, 2011.
[31] Tomas Mikolov, Martin Karafiat, Lukas Burget, Jan Cernocky, and Sanjeev Khudanpur.
Recurrent neural network based language model. In Proc. on InterSpeech,
2010.
[32] T. Mikolov, S. Kombrink, L. Burget, J.H. Cernocky, and S. Khudanpur. Extensions of recurrent neural network language model. In Proc. on ICASSP, 2011.
[33] T. Mikolov and G. Zweig. Context dependent recurrent neural network language model. In Proc. on IEEE SLT workshop, 2012.
[34] R.M. Iyer and M. Ostendorf. Modeling long distance dependence in language: topic mixtures versus dynamic cache models. IEEE Transactions on Speech and Audio Processing, 1999.
[35] Aaron Heidel, Hung-An Chang, and Lin-Shan Lee. Language model adaptation using latent dirichlet allocation and an efficient topic inference algorithm. In Proc. on InterSpeech, 2007.
[36] Marcello Federico. Efficient language model adaptation through mdi estimation. In Proc. on EuroSpeech, 1999.
[37] Ciprian Chelba and Frederick Jelinek. Structured language modeling. Computer Speech and Language, 2000.
[38] Hung yi Lee and Lin-Shan Lee. Enhanced spoken term detection using support vector machines and weighted pseudo examples. IEEE Transactions on Audio, Speech & Language Processing, 2013.
[39] Hung yi Lee, Tsung wei Tu, Chia-Ping Chen, Chao-Yu Huang, and Lin-Shan Lee. Improved spoken term detection using support vector machines based on lattice context consistency. In ICASSP, 2011.
[40] John Lafferty and Chengxiang Zhai. Document language models, query models, and risk minimization for information retrieval. SIGIR ''01. ACM, 2001.
[41] Richard S. Sutton and Andrew G. Barto. Reinforcement learning: An introduction. Cambridge Journal, 1999.
[42] Richard Bellman. Dynamic programming. 1957.
[43] Stuart Dreyfus. Richard bellman on the birth of dynamic programming. Oper. Res., January 2002.
[44] Anhai Doan, Raghu Ramakrishnan, and Alon Y. Halevy. Crowdsourcing systems on the world-wide web. Communications of the ACM, 2011.
[45] Munro and Robert. Crowdsourcing and language studies: the new generation of linguistic data. In Proc. on NAACL, 2010.
[46] Jingjing Liu, Scott Cyphers, Panupong Pasupat, Ian McGraw, and Jim Glass. A conversational movie search system based on conditional random field. In Proc. on InterSpeech, 2012.
[47] Ian McGraw, Scott Cyphers, Panupong Pasupat, Jingjing Liu, and Jim Glass. Automating crowd-supervised learning for spoken language systems. In Proc. on InterSpeech, 2012.
[48] Tsung-Hsien Wen, Aaron Heidel, Hung-Yi Lee, Yu Tsao, and Lin-Shan Lee. Recurrent neural network based language model personalization by social network crowdsourcing. In to be published.
[49] John Paolillo. The virtual speech community: Social network and language variation on irc. Journal of Computer-Mediated Communication, 1999.
[50] Devan Rosen and Margaret Corbit. Social network analysis in virtual environments. In Proc. on ACM Hypertext, 2009.
[51] Thomas L. Griffiths and Mark Steyvers. Finding scientific topics. Proceedings of the National Academy of Sciences of the United States of America, 2004.
[52] Hsu Bo-June and James Glass. Style and topic language model adaptation using hmm-lda. In Proc. on EMNLP, 2006.
[53] Tam Yik-Cheung and Tanja Schultz. Unsupervised language model adaptation using latent semantic marginals. In Proc. on InterSpeech, 2006.
[54] David M. Blei, Andrew Y. Ng, and Michael I. Jordan. Latent dirichlet allocation. J. Mach. Learn. Res., 2003.
[55] Gregor Heinrich. Parameter estimation for text analysis. Technical report, 2004.
[56] Ian Porteous, David Newman, Alexander Ihler, Arthur Asuncion, Padhraic Smyth, and Max Welling. Fast collapsed gibbs sampling for latent dirichlet allocation. In KDD. ACM, 2008.
[57] Teh Yee Whye, David Newman, and Max Welling. A collapsed variational bayesian inference algorithm for latent dirichlet allocation. In NIPS, 2006.
[58] Hsu Winston H., Lyndon S. Kennedy, and Chang Shih-Fu. Video search reranking through random walk over document-level context graph. In MULTIMEDIA. ACM, 2007.
[59] Chen Yun-Nung, Chen Chia-Ping, Lee Hung-Yi, Chan Chun-An, and Lee Lin-Shan. Improved spoken term detection with graph-based re-ranking in feature space. In ICASSP, 2011.
[60] Chen Yun-Nung, Yu Huang, Ching-Feng Yeh, and Lee Lin-Shan. Spoken lecture summarization by random walk over a graph constructed with automatically extracted key terms. In Interspeech, 2011.
[61] Andreas Stolcke. Srilm - an extensible language modeling toolkit. In Proc. on Spoken Language Processing, 2002.
[62] Steve J. Young, D. Kershaw, J. Odell, D. Ollason, V. Valtchev, and P. Woodland. The HTK Book Version 3.4. Cambridge University Press, 2006.
[63] Mikael Bod’en. A guide to recurrent neural networks and backpropagation. 2002.
[64] Yangyang Shi, Pascal Wiggers, and Catholijn M. Jonker. Towards recurrent neural networks language models with linguistic and contextual features. In Proc. on InterSpeech, 2012.
[65] Yoshua Bengio, Patrice Simard, and Paolo Frasconi. Learning long-term dependencies with gradient descent is difficult, 1994.
[66] Xunying Liu, Mark J. F. Gales, and Philip C. Woodland. Improving lvcsr system combination using neural network language model cross adaptation. In Proc. on InterSpeech, 2011.
[67] Yehuda Koren, Robert Bell, and Chris Volinsky. Matrix factorization techniques for recommender systems. Computer, 2009.
[68] Thomas K. Landauer, Peter W. Foltz, and Darrell Laham. An Introduction to Latent Semantic Analysis. Discourse Processes, 1998.
[69] Keh-Jiann Chen and Shing-Huan Liu. Word identification for mandarin chinese sentences. In Proceedings of the 14th conference on Computational linguistics - Volume 1, COLING ''92. Association for Computational Linguistics, 1992.
[70] Wei-Yun Ma and Keh-Jiann Chen. A bottom-up merging algorithm for chinese unknown word extraction. In Proceedings of the second SIGHAN workshop on Chinese language processing - Volume 17, SIGHAN ''03. Association for Computational Linguistics, 2003.
[71] Tomas Mikolov, Stefan Kombrink, Anoop Deoras, Lukas Burget, and Jan Cernocky. Rnnlm - recurrent neural network language modeling toolkit. In Proc. on ASRU, 2011.
[72] George A. Miller. Wordnet: A lexical database for english. Communications of the ACM, 1995.
[73] Yi-Lun Yang. Plutalk: A spoken social network application with speaker adaptation and error correction. 2012.
[74] L. Bahl, P. Brown, P.V. De Souza, and R. Mercer. Maximum mutual information estimation of hidden markov model parameters for speech recognition. In Acoustics, Speech, and Signal Processing, IEEE International Conference on ICASSP ''86.,1986.
[75] D. Povey and P.C. Woodland. Minimum phone error and i-smoothing for improved discriminative training. In Acoustics, Speech, and Signal Processing (ICASSP), 2002 IEEE International Conference on, 2002.
[76] David Robins. Interactive information retrieval: Context and basic notions. Informing Science Journal, 2000.
[77] Ian Ruthven. Interactive information retrieval. Annual Review of Information Science and Technology, 2008.
[78] Teruhisa Misu and Tatsuya Kawahara. Speech-based interactive information guidance
system using question-answering technique. In ICASSP, 2007.
[79] Chengxiang Zhai and John Lafferty. Model-based feedback in the language modeling approach to information retrieval. CIKM ''01. ACM, 2001.
[80] Tao Tao and ChengXiang Zhai. Regularized estimation of mixture models for robust pseudo-relevance feedback. In SIGIR''06, 2006.
[81] Thomas Hofmann. Probabilistic latent semantic indexing. In ACM SIGIR, 1999.
[82] Richard Bellman and Sherman Dreyfus. Functional approximation and dynamic programming. Mathematical Tables and Other Aids to Computation, 1959.
[83] R emi Munos and Csaba Szepesv ari. Finite-time bounds for fitted value iteration. Journal of Machine Learning Research, 2008.
[84] Csaba Szepesv ari and R emi Munos. Finite time bounds for sampling based fitted value iteration. ICML ''05. ACM, 2005.
[85] Ben He and Iadh Ounis. Query performance prediction. Inf. Syst., 2006.
[86] Ying Zhao, Falk Scholer, and Yohannes Tsegay. Effective pre-retrieval query performance prediction using similarity and variability evidence. In Advances in Information Retrieval. 2008.
[87] Yun Zhou and W. Bruce Croft. Query performance prediction in web search environments.
SIGIR ''07. ACM, 2007.