Struct Lexer

pub struct Lexer<'src> { /* private fields */ }

Just language lexer

The lexer proceeds character-by-character, as opposed to using regular expressions to lex tokens or semi-tokens at a time. As a result, it is verbose and straightforward. Just used to have a regex-based lexer, which was slower and generally godawful. However, this should not be taken as a slight against regular expressions, the lexer was just idiosyncratically bad.

Implementations

impl<'src> Lexer<'src>

pub fn lex( path: &'src Path, src: &'src str, ) -> Result<Vec<Token<'src>>, CompileError<'src>>

Lex src

pub fn is_identifier(text: &str) -> bool

True if text could be an identifier

pub fn is_identifier_start(c: char) -> bool

True if c can be the first character of an identifier

pub fn is_identifier_continue(c: char) -> bool

True if c can be a continuation character of an identifier