Enum TokenKind

#[non_exhaustive]
pub enum TokenKind {
    WhiteSpace,
    Keyword(Keyword),
    Algorithm(Algorithm),
    Language(Language),
    Distance(DistanceKind),
    VectorType(VectorTypeKind),
    Operator(Operator),
    OpenDelim(Delim),
    CloseDelim(Delim),
    Qoute(QouteKind),
    Parameter,
    Identifier,
    LeftChefron,
    RightChefron,
    Star,
    Question,
    Dollar,
    ArrowRight,
    ForwardSlash,
    Dot,
    DotDot,
    DotDotDot,
    SemiColon,
    PathSeperator,
    Colon,
    Comma,
    Vert,
    At,
    Eof,
    Digits,
    NaN,
    Glued(Glued),
    Invalid,
}

The type of token

Variants (Non-exhaustive)

Non-exhaustive enums could have additional variants added in future. Therefore, when matching against variants of non-exhaustive enums, an extra wildcard arm must be added to account for any future variants.

WhiteSpace

Keyword(Keyword)

Algorithm(Algorithm)

Language(Language)

Distance(DistanceKind)

VectorType(VectorTypeKind)

Operator(Operator)

OpenDelim(Delim)

CloseDelim(Delim)

Qoute(QouteKind)

a token denoting the opening of a string, i.e. r"

Parameter

A parameter like $name.

Identifier

LeftChefron

<

RightChefron

>

Star

*

Question

?

Dollar

$

ArrowRight

->

ForwardSlash

‘/’

Dot

.

DotDot

..

DotDotDot

... or …

SemiColon

;

PathSeperator

::

Colon

:

Comma

,

Vert

|

At

@

Eof

A token which indicates the end of the file.

Digits

A token consiting of one or more ascii digits.

NaN

The Not-A-Number number token.

Glued(Glued)

A token which is a compound token which has been glued together and then put back into the token buffer. This is required for some places where we need to look past possible compound tokens.

Invalid

A token which could not be properly lexed.

Implementations

impl TokenKind

pub fn has_data(&self) -> bool

pub fn as_str(&self) -> &'static str

Trait Implementations

impl Clone for TokenKind

fn clone(&self) -> TokenKind

Returns a copy of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for TokenKind

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Display for TokenKind

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Hash for TokenKind

fn hash<__H: Hasher>(&self, state: &mut __H)

Feeds this value into the given Hasher.

fn hash_slice<H>(data: &[Self], state: &mut H)

where H: Hasher, Self: Sized,

Feeds a slice of this type into the given Hasher.

impl PartialEq for TokenKind

fn eq(&self, other: &TokenKind) -> bool

Tests for self and other values to be equal, and is used by ==.

fn ne(&self, other: &Rhs) -> bool

Tests for !=. The default implementation is almost always sufficient, and should not be overridden without very good reason.

impl Copy for TokenKind

impl Eq for TokenKind

impl StructuralPartialEq for TokenKind