Struct LlamaBatch

pub struct LlamaBatch { /* private fields */ }

A safe wrapper around llama_batch.

Implementations

impl LlamaBatch

pub fn clear(&mut self)

Clear the batch. This does not free the memory associated with the batch, but it does reset the number of tokens to 0.

pub fn add( &mut self, LlamaToken: LlamaToken, pos: llama_pos, seq_ids: &[i32], logits: bool, ) -> Result<(), BatchAddError>

add a token to the batch for sequences seq_ids at position pos. If logits is true, the token will be initialized and can be read from after the next decode.

Panics

• [self.llama_batch.n_tokens] does not fit into a usize
• [seq_ids.len()] does not fit into a llama_seq_id

Errors

returns a error if there is insufficient space in the buffer

pub fn add_sequence( &mut self, tokens: &[LlamaToken], seq_id: i32, logits_all: bool, ) -> Result<(), BatchAddError>

Add a sequence of tokens to the batch for the given sequence id. If logits_all is true, the tokens will be initialized and can be read from after the next decode.

Either way the last token in the sequence will have its logits set to true.

Errors

Returns an error if there is insufficient space in the buffer

Panics

• [self.llama_batch.n_tokens] does not fit into a usize
• [n_tokens - 1] does not fit into a llama_pos

pub fn new(n_tokens: usize, n_seq_max: i32) -> Self

Create a new LlamaBatch that can contain up to n_tokens tokens.

Arguments

• n_tokens: the maximum number of tokens that can be added to the batch
• n_seq_max: the maximum number of sequences that can be added to the batch (generally 1 unless you know what you are doing)

Panics

Panics if n_tokens is greater than i32::MAX.

pub fn get_one(tokens: &[LlamaToken]) -> Result<Self, BatchAddError>

llama_batch_get_one Return batch for single sequence of tokens

NOTE: this is a helper function to facilitate transition to the new batch API

Errors

If the provided token buffer is empty.

Panics

If the number of tokens in tokens exceeds i32::MAX.

pub fn n_tokens(&self) -> i32

Returns the number of tokens in the batch.

Trait Implementations

impl Debug for LlamaBatch

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Drop for LlamaBatch

fn drop(&mut self)

Drops the LlamaBatch.

let batch = LlamaBatch::new(512, 1);
// frees the memory associated with the batch. (allocated by llama.cpp)
drop(batch);