Crate msiz_rustc_ap_rustc_lexer

Source

Modules§

unescape
Utilities for validating string and char literals and turning them into values they represent.

Structs§

Token

Enums§

Base
LiteralKind
TokenKind

Functions§

first_token
is_id_continue
True if c is valid as a non-first character of an identifier.
is_id_start
True if c is valid as a first character of an identifier.
is_whitespace
True if c is considered a whitespace according to Rust language definition.
strip_shebang
tokenize