Query-Document Topic Mismatch Detection�
Sahil Chelaramani, Ankush Chatterjee, Sonam Damani, Kedhar Nath Narahari, Meghana Joshi, Manish Gupta and Puneet Agrawal�
gmanish@microsoft.com
Why is query-document topic mismatch detection important?
Why topic mismatch happens?
VU=Very Unsatisfactory
U=Unsatisfactory
N=Neutral
S=Satisfactory
VS=Very Satisfactory
Topic mismatch detection (TMD) problem
Naïve first-cut solutions
Deep learning based semantic methods
BiLSTMs
Better pre-trained BERT model?
Correlation between the ranking task and topic mismatch prediction task (in percentages).
Point-wise input vs smoothed input
BERT with Smoothed Input (SI)
BERT with Key-phrases (KP) and Topic Distribution
BERT with Key-phrases (KP) and Topic Distribution
TMD Dataset
Traditional Baselines
Overall Main Results
Overall Main Results
Overall Main Results
Error analysis
Examples of query-URL instances correctly (top) / incorrectly (bottom) predicted by our model.
Related Work
Take-aways
Refererences
1. Bahdanau, D., Cho, K., Bengio, Y.: Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.0473 (2014)
2. Blei, D.M., Ng, A.Y., Jordan, M.I.: Latent dirichlet allocation. JMLR 3(Jan), 993–1022 (2003)
3. Brenes, D.J., Gayo-Avello, D., Perez-Gonz ´ alez, K.: Survey and evaluation of query intent detection methods. In: Workshop on Web Search Click Data. pp. 1–7 (2009)
4. Cao, Z., Qin, T., Liu, T.Y., Tsai, M.F., Li, H.: Learning to rank: from pairwise approach to listwise approach. In: Proceedings of the 24th international conference on Machine learning. pp. 129–136 (2007)
5. Cer, D., Yang, Y., Kong, S.y., Hua, N., Limtiaco, N., John, R.S., Constant, N., GuajardoCespedes, M., Yuan, S., Tar, C., et al.: Universal sentence encoder. arXiv preprint arXiv:1803.11175 (2018)
6. Chapelle, O., Chang, Y.: Yahoo! learning to rank challenge overview. In: Proceedings of the learning to rank challenge. pp. 1–24 (2011)
7. Conneau, A., Kiela, D., Schwenk, H., Barrault, L., Bordes, A.: Supervised learning of universal sentence representations from natural language inference data. In: Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing. pp. 670–680 (2017)
8. Devlin, J., Chang, M.W., Lee, K., Toutanova, K.: Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805 (2018)
9. Gayo-Avello, D.: A survey on session detection methods in query logs and a proposal for future evaluation. Information Sciences 179(12), 1822–1843 (2009)
10. Gupta, M., Agrawal, P.: Compression of deep learning models for text: A survey. arXiv preprint arXiv:2008.05221 (2020)
11. Hashemi, H.B., Asiaee, A., Kraft, R.: Query intent detection using convolutional neural networks. In: Intl. Conf. on Web Search and Data Mining, Workshop on Query Understanding (2016)
Refererences
12. Hochreiter, S., Schmidhuber, J.: Long short-term memory. Neural computation 9(8), 1735– 1780 (1997)
13. Jelodar, H., Wang, Y., Yuan, C., Feng, X., Jiang, X., Li, Y., Zhao, L.: Latent dirichlet allocation (lda) and topic modeling: models, applications, a survey. Multimedia Tools and Applications 78(11), 15169–15211 (2019)
14. Jones, R., Klinkner, K.L.: Beyond the session timeout: automatic hierarchical segmentation of search topics in query logs. In: CIKM. pp. 699–708 (2008)
15. Le, Q., Mikolov, T.: Distributed representations of sentences and documents. In: International conference on machine learning. pp. 1188–1196 (2014)
16. Li, J., Luong, M.T., Jurafsky, D.: A hierarchical neural autoencoder for paragraphs and documents. arXiv preprint arXiv:1506.01057 (2015)
17. Li, X., de Rijke, M.: Do topic shift and query reformulation patterns correlate in academic search? In: ECIR. pp. 146–159. Springer (2017)
18. Loshchilov, I., Hutter, F.: Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101 (2017)
19. Mikolov, T., Sutskever, I., Chen, K., Corrado, G.S., Dean, J.: Distributed representations of words and phrases and their compositionality. In: Advances in neural information processing systems. pp. 3111–3119 (2013)
20. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, Ł., Polosukhin, I.: Attention is all you need. In: NIPS. pp. 5998–6008 (2017)
21. Xiong, L., Hu, C., Xiong, C., Campos, D., Overwijk, A.: Open domain web keyphrase extraction beyond language modeling. In: EMNLP-IJCNLP. pp. 5178–5187 (2019)
22. Yang, Z., Yang, D., Dyer, C., He, X., Smola, A., Hovy, E.: Hierarchical attention networks for document classification. In: Proceedings of the 2016 conference of the North American chapter of the association for computational linguistics: human language technologies. pp. 1480–1489 (2016)
Thanks!