Enum DataBlock

pub enum DataBlock {
    Empty(),
    Constant(ConstantDataBlock),
    AllNull(AllNullDataBlock),
    Nullable(NullableDataBlock),
    FixedWidth(FixedWidthDataBlock),
    FixedSizeList(FixedSizeListBlock),
    VariableWidth(VariableWidthBlock),
    Opaque(OpaqueBlock),
    Struct(StructDataBlock),
    Dictionary(DictionaryDataBlock),
}

A DataBlock is a collection of buffers that represents an “array” of data in very generic terms

The output of each decoder is a DataBlock. Decoders can be chained together to transform one DataBlock into a different kind of DataBlock.

The DataBlock is somewhere in between Arrow’s ArrayData and Array and represents a physical layout of the data.

A DataBlock can be converted into an Arrow ArrayData (and then Array) for a given array type. For example, a FixedWidthDataBlock can be converted into any primitive type or a fixed size list of a primitive type. This is a zero-copy operation.

In addition, a DataBlock can be created from an Arrow array or arrays. This is not a zero-copy operation as some normalization may be required.

Variants

Empty()

Constant(ConstantDataBlock)

AllNull(AllNullDataBlock)

Nullable(NullableDataBlock)

FixedWidth(FixedWidthDataBlock)

FixedSizeList(FixedSizeListBlock)

VariableWidth(VariableWidthBlock)

Opaque(OpaqueBlock)

Struct(StructDataBlock)

Dictionary(DictionaryDataBlock)

Implementations

impl DataBlock

pub fn into_arrow( self, data_type: DataType, validate: bool, ) -> Result<ArrayData>

Convert self into an Arrow ArrayData

pub fn into_buffers(self) -> Vec<LanceBuffer>

Convert the data block into a collection of buffers for serialization

The order matters and will be used to reconstruct the data block at read time.

pub fn borrow_and_clone(&mut self) -> Self

Converts the data buffers into borrowed mode and clones the block

This is a zero-copy operation but requires a mutable reference to self and, afterwards, all buffers will be in Borrowed mode.

pub fn try_clone(&self) -> Result<Self>

Try and clone the block

This will fail if any buffers are in owned mode. You can call borrow_and_clone() to ensure that all buffers are in borrowed mode before calling this method.

pub fn name(&self) -> &'static str

pub fn num_values(&self) -> u64

pub fn data_size(&self) -> u64

pub fn remove_validity(self) -> Self

Removes any validity information from the block

This does not filter the block (e.g. remove rows). It only removes the validity bitmaps (if present). Any garbage masked by null bits will now appear as proper values.

pub fn make_builder( &self, estimated_size_bytes: u64, ) -> Box<dyn DataBlockBuilderImpl>

impl DataBlock

pub fn as_all_null(self) -> Option<AllNullDataBlock>

pub fn as_nullable(self) -> Option<NullableDataBlock>

pub fn as_fixed_width(self) -> Option<FixedWidthDataBlock>

pub fn as_fixed_size_list(self) -> Option<FixedSizeListBlock>

pub fn as_variable_width(self) -> Option<VariableWidthBlock>

pub fn as_struct(self) -> Option<StructDataBlock>

pub fn as_dictionary(self) -> Option<DictionaryDataBlock>

pub fn as_all_null_ref(&self) -> Option<&AllNullDataBlock>

pub fn as_nullable_ref(&self) -> Option<&NullableDataBlock>

pub fn as_fixed_width_ref(&self) -> Option<&FixedWidthDataBlock>

pub fn as_fixed_size_list_ref(&self) -> Option<&FixedSizeListBlock>

pub fn as_variable_width_ref(&self) -> Option<&VariableWidthBlock>

pub fn as_struct_ref(&self) -> Option<&StructDataBlock>

pub fn as_dictionary_ref(&self) -> Option<&DictionaryDataBlock>

pub fn as_all_null_ref_mut(&mut self) -> Option<&mut AllNullDataBlock>

pub fn as_nullable_ref_mut(&mut self) -> Option<&mut NullableDataBlock>

pub fn as_fixed_width_ref_mut(&mut self) -> Option<&mut FixedWidthDataBlock>

pub fn as_fixed_size_list_ref_mut(&mut self) -> Option<&mut FixedSizeListBlock>

pub fn as_variable_width_ref_mut(&mut self) -> Option<&mut VariableWidthBlock>

pub fn as_struct_ref_mut(&mut self) -> Option<&mut StructDataBlock>

pub fn as_dictionary_ref_mut(&mut self) -> Option<&mut DictionaryDataBlock>

impl DataBlock

pub fn from_arrays(arrays: &[ArrayRef], num_values: u64) -> Self

pub fn from_array<T: Array + 'static>(array: T) -> Self

Trait Implementations

impl ComputeStat for DataBlock

fn compute_stat(&mut self)

impl Debug for DataBlock

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl From<Arc<dyn Array>> for DataBlock

fn from(array: ArrayRef) -> Self

Converts to this type from the input type.

impl GetStat for DataBlock

fn get_stat(&self, stat: Stat) -> Option<Arc<dyn Array>>

fn expect_stat(&self, stat: Stat) -> Arc<dyn Array>

fn expect_single_stat<T: ArrowPrimitiveType>(&self, stat: Stat) -> T::Native