flâneur

Beating the fastest lexer generator in Rust

alic.dev · 2,639 words · saved by 1 readers

I had my doubts about the second part. I was aware of the efficiency of state machine driven lexers, but most generators have one problem: they can't be arbitrarily generic and consistently optimal at the same time. There will always be some assumptions about your data that are either impossible to express, or outside the scope of the generator's optimizations. Either way, I was curious to find out how my hand-rolled implementation would fare. Each invocation of next_token() begins at the start state, continually performing state transitions by looking up the next character, until we hit a terminal state. The terminal state contains the information about the token variant. If the number of language keywords is sufficiently small, we could even build a trie for matching against a static set of keywords. The implementation of the state lookup is input-dependent. For ASCII input, we can let each state define a full jump table for every possible character, and use the raw bytes as lookup i

Beating the fastest lexer generator in Rust Blog Repos Github Linkedin Beating the Fastest Lexer Generator in Rust Jun 2023 - RSS I was recently made aware of a crate for writing efficient lexers in Rust called logos . According to its documentation, the project has two main goals. To make it easy to create a lexer, so you can focus on more complex problems. To make the generated lexer faster than anything you'd write by hand. I had my doubts about the second part. I was aware of the efficiency of state machine driven lexers, but most generators have one problem: they can't be arbitrarily gene

saved by

related reading