Struct TokenCollection

pub struct TokenCollection(/* private fields */);

A type-safe, enum-backed collection of tokens.

TokenCollection can hold a collection of i32 or usize tokens, ensuring type safety and efficient storage.

Implementations

impl TokenCollection

pub fn as_i32(self) -> Result<Vec<i32>, TokenizerError>

Converts the TokenCollection into a vector of i32, if it contains i32 values. Returns None otherwise.

pub fn as_usize(self) -> Result<Vec<usize>, TokenizerError>

Converts the TokenCollection into a vector of usize, if it contains usize values. Returns None otherwise.

pub fn len(&self) -> usize

Returns the number of tokens in the token collection

pub fn is_empty(&self) -> bool

Returns true if the TokenCollection is empty

pub fn slice(&self, start: usize, end: usize) -> Self

Slices the token collection between start and end.

Trait Implementations

impl Debug for TokenCollection

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl From<Vec<i32>> for TokenCollection

Enables the conversion from a vector of i32 into a TokenCollection.

fn from(v: Vec<i32>) -> Self

Converts to this type from the input type.

impl From<Vec<usize>> for TokenCollection

Enables the conversion from a vector of usize into a TokenCollection.

fn from(v: Vec<usize>) -> Self

Converts to this type from the input type.