Struct KVCacheView

pub struct KVCacheView<'a> { /* private fields */ }

An updateable view of the KV cache. (use only for debugging purposes)

Implementations

impl KVCacheView<'_>

pub fn update(&mut self)

Update the KV cache view structure with the current state of the KV cache. (use only for debugging purposes)

pub fn n_cells(&self) -> i32

Number of KV cache cells. This will be the same as the context size.

pub fn token_count(&self) -> i32

Number of tokens in the cache. For example, if there are two populated cells, the first with 1 sequence id in it and the second with 2 sequence ids then you’ll have 3 tokens.

pub fn used_cells(&self) -> i32

Number of populated cache cells.

pub fn max_contiguous(&self) -> i32

Maximum contiguous empty slots in the cache.

pub fn max_contiguous_idx(&self) -> i32

Index to the start of the max_contiguous slot range. Can be negative when cache is full.

pub fn cells(&self) -> impl Iterator<Item = KVCacheViewCell>

Information for individual cells.

Panics

• if n_cells does not fit into usize.

pub fn cells_sequences(&self) -> impl Iterator<Item = &[llama_seq_id]>

The sequences for each cell. There will be n_max_seq items per cell.

Panics

• if n_cells * n_max_seq does not fit into usize.
• if n_max_seq does not fit into usize.

Trait Implementations

impl<'a> Debug for KVCacheView<'a>

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Drop for KVCacheView<'_>

fn drop(&mut self)

Executes the destructor for this type.