Introduction to Information Retrieval

N/A

DRAFT! © April 1, 2009 Cambridge University Press. Feedback welcome. 19 2 The term vocabulary and postings lists Recall the major steps in inverted index construction: 1. Collect the documents to be indexed. 2. Tokenize the text. 3. Do linguistic preprocessing of tokens. 4. Index the documents that each term occurs in. In this chapter we first briefly mention how the basic unit of a document can be defined and how the character sequence that it…

related reading

• The Anatomy of a Search Engineinfolab.stanford.edu
• [2112.10508] Between words and characters: A Brief History of Open-Vocabulary Modeling and Tokenization in NLParxiv.org
• Fast regex search: indexing text for agent tools · Cursorcursor.com
• Bag-of-words model - Wikipediaen.wikipedia.org
• The Bitter Lesson is coming for Tokenization – ⛰️ lucalplucalp.dev
• [2605.22821] Tokenisation via Convex Relaxationsarxiv.org
• What Are Stemming and Lemmatization? | IBMibm.com
• 1301.3781arxiv.org
• Okapi BM25 - Wikipediaen.wikipedia.org
• Stemming and lemmatizationnlp.stanford.edu
• Lexical analysis - Wikipediaen.wikipedia.org
• Regular Expression Matching with a Trigram Indexswtch.com