flâneur

Lexical analysis

en.wikipedia.org · 4,036 words · saved by 1 readers

Lexical tokenization is conversion of a text into (semantically or syntactically) meaningful lexical tokens belonging to categories defined by a "lexer" program. In case of a natural language, those categories include nouns, verbs, adjectives, punctuations etc. In case of a programming language, the categories include identifiers, operators, grouping symbols, data types and language keywords. Lexical tokenization is related to the type of tokenization used in large language models (LLMs) but with two differences. First, lexical tokenization is usually based on a lexical grammar, whereas LLM tokenizers are usually probability-based. Second, LLM tokenizers perform a second step that converts the tokens into numerical values.

Lexical analysis - Wikipedia Jump to content From Wikipedia, the free encyclopedia "Lexer" redirects here. For people with this name, see Lexer (surname) . "Lexing" redirects here. For the Chinese guerrilla leader, see Zhang Lexing . For the mathematician, see Lexing Ying . Conversion of character sequences into token sequences in computer science Lexical tokenization is conversion of a text into (semantically or syntactically) meaningful lexical tokens belonging to categories defined by a "lexer" program. In case of a natural language, those categories include nouns, verbs, adjectives, punctu

saved by

related reading