Scanning · Crafting Interpreters

The first step in any compiler or interpreter is scanning. The scanner takes in raw source code as a series of characters and groups it into a series of chunks we call tokens. These are the meaningful “words” and “punctuation” that make up the language’s grammar.

4 Scanning Take big bites. Anything worth doing is worth overdoing. Robert A. Heinlein, Time Enough for Love The first step in any compiler or interpreter is scanning . The scanner takes in raw source code as a series of characters and groups it into a series of chunks we call tokens . These are the meaningful "words" and "punctuation" that make up the language's grammar. This task has been variously called "scanning" and "lexing" (short for "lexical analysis") over the years. Way back when computers were as big as Winnebagos but had less memory than your watch, some people used "scanner" only

related reading

• Lexical analysis - Wikipediaen.wikipedia.org
• Let’s Build A Simple Interpreter. Part 1. - Ruslan's Blogruslanspivak.com
• PA1.pdfweb.stanford.edu
• Introduction · Crafting Interpreterscraftinginterpreters.com
• 1. Kaleidoscope: Kaleidoscope Introduction and the Lexer — LLVM 23.0.0git documentationllvm.org
• Statements and State · Crafting Interpreterscraftinginterpreters.com
• Parsing Expressions · Crafting Interpreterscraftinginterpreters.com
• 2. Kaleidoscope: Implementing a Parser and AST — LLVM 23.0.0git documentationllvm.org
• A Map of the Territory · Crafting Interpreterscraftinginterpreters.com
• Reading 12: Regular Expressions & Grammarsweb.mit.edu
• A Map of the Territory · Crafting Interpreterscraftinginterpreters.com
• Lexical Analysis With Flex, for Flex 2.6.2: Simple Exampleswestes.github.io