pub unsafe extern "C" fn llama_detokenize(
model: *const llama_model,
tokens: *const llama_token,
n_tokens: i32,
text: *mut c_char,
text_len_max: i32,
remove_special: bool,
unparse_special: bool,
) -> i32
Expand description
@details Convert the provided tokens into text (inverse of llama_tokenize()). @param text The char pointer must be large enough to hold the resulting text. @return Returns the number of chars/bytes on success, no more than text_len_max. @return Returns a negative number on failure - the number of chars/bytes that would have been returned. @param remove_special Allow to remove BOS and EOS tokens if model is configured to do so. @param unparse_special If true, special tokens are rendered in the output.