Module moore_svlog::cat [−][src]
Expand description
The categorizing lexer. Tokenizes an input stream of characters, yielding a stream of newline, whitespace, comment, symbol, and text tokens.
Example
extern crate moore_svlog_syntax;
let input = "Löwe 老虎 Léopard\n";
let mut cat = moore_svlog_syntax::cat::Cat::new(Box::new(input.char_indices()));
let tokens: Vec<_> = cat.collect();Structs
The categorizing lexer. Divides an input stream of characters (unicode) into coarse groups of tokens. These include whitespace, comments, symbols, and text. The strings contained in the emitted tokens can be concatenated to arrive at the original file, i.e. no information is lost.
A token emitted by the categorizing lexer.
Enums
The different kinds of tokens the categorizing lexer can emit.