✳flâneur — a map of the web's best reading

Stemming and lemmatization

nlp.stanford.edu · 1,089 words · saved by 1 readers

For grammatical reasons, documents are going to use different forms of a word, such as organize, organizes, and organizing. Additionally, there are families of derivationally related words with similar meanings, such as democracy, democratic, and democratization. In many situations, it seems as if it would be useful for a search for one of these words to return documents that contain another word in the set. The goal of both stemming and lemmatization is to reduce inflectional forms and sometimes derivationally related forms of a word to a common base form. For instance: The most common algorithm for stemming English, and one that has repeatedly been shown to be empirically very effective, is Porter's algorithm (Porter, 1980). The entire algorithm is too long and intricate to present here, but we will indicate its general nature. Porter's algorithm consists of 5 phases of word reductions, applied sequentially. Within each phase there are various conventions to select rules, such as sel

Stemming and lemmatization Next: Faster postings list intersection Up: Determining the vocabulary of Previous: Other languages. Contents Index Stemming and lemmatization For grammatical reasons, documents are going to use different forms of a word, such as organize , organizes , and organizing . Additionally, there are families of derivationally related words with similar meanings, such as democracy , democratic , and democratization . In many situations, it seems as if it would be useful for a search for one of these words to return documents that contain another word in the set. The goal of

Explore this link on the map →

related reading