Struct ProductQuantizationStorage

pub struct ProductQuantizationStorage { /* private fields */ }

Product Quantization Storage

It stores PQ code, as well as the row ID to the original vectors.

It is possible to store additional metadata to accelerate filtering later.

TODO: support f16/f64 later.

Implementations

impl ProductQuantizationStorage

pub fn new( codebook: FixedSizeListArray, batch: RecordBatch, num_bits: u32, num_sub_vectors: usize, dimension: usize, distance_type: DistanceType, transposed: bool, ) -> Result<Self>

pub fn batch(&self) -> &RecordBatch

pub async fn build( quantizer: ProductQuantizer, batch: &RecordBatch, vector_col: &str, ) -> Result<Self>

Build a PQ storage from ProductQuantizer and a RecordBatch.

Parameters

quantizer: ProductQuantizer The quantizer used to transform the vectors. batch: RecordBatch The batch of vectors to be transformed. vector_col: &str The name of the column containing the vectors.

pub async fn load(object_store: &ObjectStore, path: &Path) -> Result<Self>

Load full PQ storage from disk.

Parameters

object_store: &ObjectStore The object store to load the storage from. path: &Path The path to the storage.

Returns

Self

Currently it loads everything in memory. TODO: support lazy loading later.

pub fn schema(&self) -> SchemaRef

pub fn get_row_ids(&self, ids: &[u32]) -> Vec<u64>

pub async fn write_partition( &self, writer: &mut FileWriter<ManifestDescribing>, ) -> Result<usize>

Write the PQ storage as a Lance partition to disk, and returns the number of rows written.

pub async fn write_full( &self, writer: &mut FileWriter<ManifestDescribing>, ) -> Result<()>

Write the PQ storage to disk.

Trait Implementations

impl Clone for ProductQuantizationStorage

fn clone(&self) -> ProductQuantizationStorage

Returns a copy of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for ProductQuantizationStorage

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl DeepSizeOf for ProductQuantizationStorage

fn deep_size_of_children(&self, _context: &mut Context) -> usize

Returns an estimation of the heap-managed storage of this object. This does not include the size of the object itself.

fn deep_size_of(&self) -> usize

Returns an estimation of a total size of memory owned by the object, including heap-managed storage.

impl PartialEq for ProductQuantizationStorage

fn eq(&self, other: &Self) -> bool

Tests for self and other values to be equal, and is used by ==.

fn ne(&self, other: &Rhs) -> bool

Tests for !=. The default implementation is almost always sufficient, and should not be overridden without very good reason.

impl QuantizerStorage for ProductQuantizationStorage

fn load_partition<'life0, 'life1, 'async_trait>( reader: &'life0 FileReader, range: Range<usize>, distance_type: DistanceType, metadata: &'life1 Self::Metadata, ) -> Pin<Box<dyn Future<Output = Result<Self>> + Send + 'async_trait>>

where Self: 'async_trait, 'life0: 'async_trait, 'life1: 'async_trait,

Load a partition of PQ storage from disk.

Parameters

• *reader: &FileReader

type Metadata = ProductQuantizationMetadata

impl VectorStore for ProductQuantizationStorage

type DistanceCalculator<'a> = PQDistCalculator

fn try_from_batch( batch: RecordBatch, distance_type: DistanceType, ) -> Result<Self>

where Self: Sized,

Create a VectorStore from a RecordBatch. The batch should consist of row IDs and quantized vector.

fn to_batches(&self) -> Result<impl Iterator<Item = RecordBatch>>

fn append_batch( &self, _batch: RecordBatch, _vector_column: &str, ) -> Result<Self>

Append Raw RecordBatch into the Storage. The storage implement will perform quantization if necessary.

fn schema(&self) -> &SchemaRef

fn as_any(&self) -> &dyn Any

fn len(&self) -> usize

fn distance_type(&self) -> DistanceType

Return DistanceType.

fn row_id(&self, id: u32) -> u64

Get the lance ROW ID from one vector.

fn row_ids(&self) -> impl Iterator<Item = &u64>

fn dist_calculator(&self, query: ArrayRef) -> Self::DistanceCalculator<'_>

Create a DistCalculator to compute the distance between the query.

fn dist_calculator_from_id(&self, _: u32) -> Self::DistanceCalculator<'_>

fn distance_between(&self, _: u32, _: u32) -> f32

fn remap(&self, mapping: &HashMap<u64, Option<u64>>) -> Result<Self>

fn is_empty(&self) -> bool

Returns true if this graph is empty.

fn dist_calculator_from_native( &self, _query: ArrayRef, ) -> Self::DistanceCalculator<'_>