Enum VocabType

#[repr(u32)]
pub enum VocabType {
    BPE = 2,
    SPM = 1,
}

a rusty equivalent of llama_vocab_type

Variants

BPE = 2

Byte Pair Encoding

SPM = 1

Sentence Piece Tokenizer

Trait Implementations

impl Clone for VocabType

fn clone(&self) -> VocabType

Returns a copy of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for VocabType

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl PartialEq for VocabType

fn eq(&self, other: &VocabType) -> bool

Tests for self and other values to be equal, and is used by ==.

fn ne(&self, other: &Rhs) -> bool

Tests for !=. The default implementation is almost always sufficient, and should not be overridden without very good reason.

impl TryFrom<u32> for VocabType

type Error = LlamaTokenTypeFromIntError

The type returned in the event of a conversion error.

fn try_from(value: llama_vocab_type) -> Result<Self, Self::Error>

Performs the conversion.

impl Copy for VocabType

impl Eq for VocabType

impl StructuralPartialEq for VocabType