Module lexer

Functions

• get_token_stream
  Tokenize the input, associating with each token a leading and trailing comment if they are present. Also returns a string containing any comments that may be present at the end of input after all tokens are consumed.