Struct Lexer

pub struct Lexer<'a> {
    pub filenames: Vec<Arc<str>>,
    pub file_texts: Vec<Arc<str>>,
    /* private fields */
}

The lexer.

Breaks source text up into a sequence of tokens (with source positions).

Fields

filenames: Vec<Arc<str>>

Arena of filenames from the input source.

Indexed via Pos::file.

file_texts: Vec<Arc<str>>

Arena of file source texts.

Indexed via Pos::file.

Implementations

impl<'a> Lexer<'a>

pub fn from_str(s: &'a str, filename: &'a str) -> Result<Lexer<'a>, Errors>

Create a new lexer for the given source contents and filename.

pub fn from_files<P>( file_paths: impl IntoIterator<Item = P>, ) -> Result<Lexer<'a>, Errors>

where P: AsRef<Path>,

Create a new lexer from the given files.

pub fn from_file_contents( files: Vec<(PathBuf, String)>, ) -> Result<Lexer<'a>, Errors>

Create a new lexer from the given files and contents.

pub fn pos(&self) -> Pos

Get the lexer’s current source position.

pub fn next(&mut self) -> Result<Option<(Pos, Token)>, Errors>

Get the next token from this lexer’s token stream, if any.

pub fn peek(&self) -> Option<&(Pos, Token)>

Peek ahead at the next token.

pub fn eof(&self) -> bool

Are we at the end of the source input?

Trait Implementations

impl<'a> Clone for Lexer<'a>

fn clone(&self) -> Lexer<'a>

Returns a copy of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl<'a> Debug for Lexer<'a>

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.