Enum makepad_live_tokenizer::tokenizer::State

pub enum State {
    Initial(InitialState),
    BlockCommentTail(BlockCommentTailState),
    DoubleQuotedStringTail(DoubleQuotedStringTailState),
    RawDoubleQuotedStringTail(RawDoubleQuotedStringTailState),
}

The state of the tokenizer.

Variants

Initial(InitialState)

BlockCommentTail(BlockCommentTailState)

DoubleQuotedStringTail(DoubleQuotedStringTailState)

RawDoubleQuotedStringTail(RawDoubleQuotedStringTailState)

Implementations

impl State

pub fn next(self, cursor: &mut Cursor<'_>) -> (State, Option<TokenWithLen>)

Given the current state of the tokenizer and a cursor over a slice of chars, finds the next token in in that string, and moves the cursor forward by the number of characters in the token. Returns the new state of the tokenizer and the token recognised, or None if there are no more tokens in the string.

Examples

use makepad_live_tokenizer::{
    full_token::{FullToken, TokenWithLen},
    tokenizer::{Cursor, InitialState, State}
};
 
let mut state = State::default();
let mut scratch = String::new();
let mut cursor = Cursor::new(&['1', '2', '3'], &mut scratch);
assert_eq!(
    state.next(&mut cursor),
    (
        State::Initial(InitialState),
        Some(TokenWithLen {
           len: 3,
           token: FullToken::Int(123),
        })
    )
);

Trait Implementations

impl Clone for State

fn clone(&self) -> State

Returns a copy of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for State

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Default for State

fn default() -> State

Returns the “default value” for a type.

impl Hash for State

fn hash<__H: Hasher>(&self, state: &mut __H)

Feeds this value into the given Hasher.

fn hash_slice<H>(data: &[Self], state: &mut H)where H: Hasher, Self: Sized,

Feeds a slice of this type into the given Hasher.

impl PartialEq<State> for State

fn eq(&self, other: &State) -> bool

This method tests for self and other values to be equal, and is used by ==.

fn ne(&self, other: &Rhs) -> bool

This method tests for !=. The default implementation is almost always sufficient, and should not be overridden without very good reason.

impl Copy for State

impl Eq for State

impl StructuralEq for State

impl StructuralPartialEq for State