fast_rustc_ap_rustc_lexer

Function tokenize

Source
pub fn tokenize(input: &str) -> impl Iterator<Item = Token> + '_
Expand description

Creates an iterator that produces tokens from the input string.