Trait VectorStore

pub trait VectorStore:
    Send
    + Sync
    + Sized
    + Clone {
    type DistanceCalculator<'a>: DistCalculator
       where Self: 'a;

    // Required methods
    fn try_from_batch(
        batch: RecordBatch,
        distance_type: DistanceType,
    ) -> Result<Self>;
    fn as_any(&self) -> &dyn Any;
    fn schema(&self) -> &SchemaRef;
    fn to_batches(&self) -> Result<impl Iterator<Item = RecordBatch> + Send>;
    fn len(&self) -> usize;
    fn distance_type(&self) -> DistanceType;
    fn row_id(&self, id: u32) -> u64;
    fn row_ids(&self) -> impl Iterator<Item = &u64>;
    fn append_batch(
        &self,
        batch: RecordBatch,
        vector_column: &str,
    ) -> Result<Self>;
    fn dist_calculator(&self, query: ArrayRef) -> Self::DistanceCalculator<'_>;
    fn dist_calculator_from_id(&self, id: u32) -> Self::DistanceCalculator<'_>;
    fn distance_between(&self, a: u32, b: u32) -> f32;

    // Provided methods
    fn remap(&self, mapping: &HashMap<u64, Option<u64>>) -> Result<Self> { ... }
    fn is_empty(&self) -> bool { ... }
    fn dist_calculator_from_native(
        &self,
        _query: ArrayRef,
    ) -> Self::DistanceCalculator<'_> { ... }
}

Vector Storage is the abstraction to store the vectors.

It can be in-memory or on-disk, raw vector or quantized vectors.

It abstracts away the logic to compute the distance between vectors.

TODO: should we rename this to “VectorDistance”?;

Internal API

API stability is not guaranteed

Required Associated Types

type DistanceCalculator<'a>: DistCalculator where Self: 'a

Required Methods

fn try_from_batch( batch: RecordBatch, distance_type: DistanceType, ) -> Result<Self>

Create a VectorStore from a RecordBatch. The batch should consist of row IDs and quantized vector.

fn as_any(&self) -> &dyn Any

fn schema(&self) -> &SchemaRef

fn to_batches(&self) -> Result<impl Iterator<Item = RecordBatch> + Send>

fn len(&self) -> usize

fn distance_type(&self) -> DistanceType

Return DistanceType.

fn row_id(&self, id: u32) -> u64

Get the lance ROW ID from one vector.

fn row_ids(&self) -> impl Iterator<Item = &u64>

fn append_batch(&self, batch: RecordBatch, vector_column: &str) -> Result<Self>

Append Raw RecordBatch into the Storage. The storage implement will perform quantization if necessary.

fn dist_calculator(&self, query: ArrayRef) -> Self::DistanceCalculator<'_>

Create a DistCalculator to compute the distance between the query.

Using dist calculator can be more efficient as it can pre-compute some values.

fn dist_calculator_from_id(&self, id: u32) -> Self::DistanceCalculator<'_>

fn distance_between(&self, a: u32, b: u32) -> f32

Provided Methods

fn remap(&self, mapping: &HashMap<u64, Option<u64>>) -> Result<Self>

fn is_empty(&self) -> bool

Returns true if this graph is empty.

fn dist_calculator_from_native( &self, _query: ArrayRef, ) -> Self::DistanceCalculator<'_>

Dyn Compatibility

This trait is not dyn compatible.

In older versions of Rust, dyn compatibility was called "object safety", so this trait is not object safe.

Implementors

impl VectorStore for FlatBinStorage

type DistanceCalculator<'a> = FlatDistanceCal<'a, UInt8Type>

impl VectorStore for FlatFloatStorage

type DistanceCalculator<'a> = FlatDistanceCal<'a, Float32Type>

impl VectorStore for ProductQuantizationStorage

type DistanceCalculator<'a> = PQDistCalculator

impl VectorStore for ScalarQuantizationStorage

type DistanceCalculator<'a> = SQDistCalculator<'a>