✳flâneur — a map of the web's best reading

Text Normalization for Natural Language Processing (NLP) | by Diego Lopez Yse | Towards Data Science

towardsdatascience.com · 2,831 words · saved by 1 readers

Natural Language Processing (NLP) is probably the hottest topic in Artificial Intelligence (AI) right now. After the breakthrough of GPT-3 with its ability to write essays, code and also create images from text, Google announced its new trillion-parameter AI language model that’s almost 6 times bigger than GPT-3. These are massive advances in the discipline that keep pushing the boundaries to new limits. How is this possible? How can machines interact with human language? There are dozens of subfields in NLP, but we must start with the basics. In another post I went through some tips on how to begin the NLP journey. Now it’s time to talk about normalizing text. Why do we need text normalization? When we normalize text, we attempt to reduce its randomness, bringing it closer to a predefined “standard”. This helps us to reduce the amount of different information that the computer has to deal with, and therefore improves efficiency. The goal of normalization techniques like stemming and l

Text Normalization | Towards Data Science Skip to content Data Science Text Normalization Why, what and how. Tiago Duque Apr 2, 2020 13 min read Share Image by Markus Winkler from Unsplash , edited by the author. In the last few articles we spent some time explaining and implementing some of the most important preprocessing techniques in NLP. However, we played too little with real text situations. Now it is the time to work a little with that. We talked about Text Normalization in the article about stemming. However, stemming is not the most important (and even used) task in Text Normalizatio

Explore this link on the map →

related reading