Enum TokenKind

pub enum TokenKind {
    LineComment {
        doc_style: Option<DocStyle>,
    },
    BlockComment {
        doc_style: Option<DocStyle>,
        terminated: bool,
    },
    Whitespace,
    Ident,
    InvalidIdent,
    RawIdent,
    UnknownPrefix,
    UnknownPrefixLifetime,
    RawLifetime,
    GuardedStrPrefix,
    Literal {
        kind: LiteralKind,
        suffix_start: u32,
    },
    Lifetime {
        starts_with_number: bool,
    },
    Semi,
    Comma,
    Dot,
    OpenParen,
    CloseParen,
    OpenBrace,
    CloseBrace,
    OpenBracket,
    CloseBracket,
    At,
    Pound,
    Tilde,
    Question,
    Colon,
    Dollar,
    Eq,
    Bang,
    Lt,
    Gt,
    Minus,
    And,
    Or,
    Plus,
    Star,
    Slash,
    Caret,
    Percent,
    Unknown,
    Eof,
}

Enum representing common lexeme types.

Variants

LineComment

A line comment, e.g. // comment.

Fields

doc_style: Option<DocStyle>

BlockComment

A block comment, e.g. /* block comment */.

Block comments can be recursive, so a sequence like /* /* */ will not be considered terminated and will result in a parsing error.

Fields

doc_style: Option<DocStyle>

terminated: bool

Whitespace

Any whitespace character sequence.

Ident

An identifier or keyword, e.g. ident or continue.

InvalidIdent

An identifier that is invalid because it contains emoji.

RawIdent

A raw identifier, e.g. “r#ident”.

UnknownPrefix

An unknown literal prefix, like foo#, foo', foo". Excludes literal prefixes that contain emoji, which are considered “invalid”.

Note that only the prefix (foo) is included in the token, not the separator (which is lexed as its own distinct token). In Rust 2021 and later, reserved prefixes are reported as errors; in earlier editions, they result in a (allowed by default) lint, and are treated as regular identifier tokens.

UnknownPrefixLifetime

An unknown prefix in a lifetime, like 'foo#.

Like UnknownPrefix, only the ' and prefix are included in the token and not the separator.

RawLifetime

A raw lifetime, e.g. 'r#foo. In edition < 2021 it will be split into several tokens: 'r and # and foo.

GuardedStrPrefix

Guarded string literal prefix: #" or ##.

Used for reserving “guarded strings” (RFC 3598) in edition 2024. Split into the component tokens on older editions.

Literal

Literals, e.g. 12u8, 1.0e-40, b"123". Note that _ is an invalid suffix, but may be present here on string and float literals. Users of this type will need to check for and reject that case.

See LiteralKind for more details.

Fields

kind: LiteralKind

suffix_start: u32

Lifetime

A lifetime, e.g. 'a.

Fields

starts_with_number: bool

Semi

;

Comma

,

Dot

.

OpenParen

(

CloseParen

)

OpenBrace

{

CloseBrace

}

OpenBracket

[

CloseBracket

]

At

@

Pound

#

Tilde

~

Question

?

Colon

:

Dollar

$

Eq

=

Bang

!

Lt

<

Gt

>

Minus

-

And

&

Or

|

Plus

+

Star

*

Slash

/

Caret

^

Percent

%

Unknown

Unknown token, not expected by the lexer, e.g. “№”

Eof

End of input.

Trait Implementations

impl Clone for TokenKind

fn clone(&self) -> TokenKind

Returns a copy of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for TokenKind

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl PartialEq for TokenKind

fn eq(&self, other: &TokenKind) -> bool

Tests for self and other values to be equal, and is used by ==.

fn ne(&self, other: &Rhs) -> bool

Tests for !=. The default implementation is almost always sufficient, and should not be overridden without very good reason.

impl Copy for TokenKind

impl Eq for TokenKind

impl StructuralPartialEq for TokenKind