pub trait TokenStream {
// Required methods
fn advance(&mut self) -> bool;
fn token(&self) -> &Token;
fn token_mut(&mut self) -> &mut Token;
// Provided methods
fn next(&mut self) -> Option<&Token> { ... }
fn process(&mut self, sink: &mut dyn FnMut(&Token)) { ... }
}
Expand description
TokenStream is the result of the tokenization.
It consists consumable stream of Tokens.
Advance to the next token
Returns false if there are no other tokens.
Returns a reference to the current token.
Returns a mutable reference to the current token.
Helper to iterate over tokens. It
simply combines a call to .advance()
and .token().
Helper function to consume the entire TokenStream
and push the tokens to a sink function.
Returns a reference to the current token.
Returns a mutable reference to the current token.
Helper to iterate over tokens. It
simply combines a call to .advance()
and .token().
Helper function to consume the entire TokenStream
and push the tokens to a sink function.