Author Interview - Transformer Memory as a Differentiable Search Index

#neuralsearch #interview #google

This is an interview with the authors Yi Tay and Don Metzler. Paper Review Video: https://youtu.be/qlB0TPBQ7YY

Search engines work by building an index and then looking up things in it. Usually, that index is a separate data structure. In keyword search, we build and store reverse indices. In neural search, we build nearest-neighbor indices. This paper does something different: It directly trains a Transformer to return the ID of the most relevant document. No similarity search over embeddings or anything like this is performed, and no external data structure is needed, as the entire index is essentially captured by the model's weights. The paper experiments with various ways of representing documents and training the system, which works surprisingly well!

OUTLINE: 0:00 - Intro 0:50 - Start of Interview 1:30 - How did this idea start? 4:30 - How does memorization play into this? 5:50 - Why did you not compare to cross-encoders? 7:50 - Instead of the ID, could one reproduce the document itself? 10:50 - Passages vs documents 12:00 - Where can this model be applied? 14:25 - Can we make this work on large collections? 19:20 - What's up with the NQ100K dataset? 23:55 - What is going on inside these models? 28:30 - What's the smallest scale to obtain meaningful results? 30:15 - Investigating the document identifiers 34:45 - What's the end goal? 38:40 - What are the hardest problems currently? 40:40 - Final comments & how to get started

Paper: https://arxiv.org/abs/2202.06991

Abstract: In this paper, we demonstrate that information retrieval can be accomplished with a single Transformer, in which all information about the corpus is encoded in the parameters of the model. To this end, we introduce the Differentiable Search Index (DSI), a new paradigm that learns a text-to-text model that maps string queries directly to relevant docids; in other words, a DSI model answers queries directly using only its parameters, dramatically simplifying the whole retrieval process. We study variations in how documents and their identifiers are represented, variations in training procedures, and the interplay between models and corpus sizes. Experiments demonstrate that given appropriate design choices, DSI significantly outperforms strong baselines such as dual encoder models. Moreover, DSI demonstrates strong generalization capabilities, outperforming a BM25 baseline in a zero-shot setup.

Authors: Yi Tay, Vinh Q. Tran, Mostafa Dehghani, Jianmo Ni, Dara Bahri, Harsh Mehta, Zhen Qin, Kai Hui, Zhe Zhao, Jai Gupta, Tal Schuster, William W. Cohen, Donald Metzler

Links: Merch: http://store.ykilcher.com TabNine Code Completion (Referral): http://bit.ly/tabnine-yannick YouTube: https://www.youtube.com/c/yannickilcher Twitter: https://twitte ... https://www.youtube.com/watch?v=C7mUYocWdG0