Struct PrimitiveStructuralEncoder

pub struct PrimitiveStructuralEncoder { /* private fields */ }

An encoder for primitive (leaf) arrays

This encoder is fairly complicated and follows a number of paths depending on the data.

First, we convert the validity & offsets information into repetition and definition levels. Then we compress the data itself into a single buffer.

If the data is narrow then we encode the data in small chunks (each chunk should be a few disk sectors and contains a buffer of repetition, a buffer of definition, and a buffer of value data). This approach is called “mini-block”. These mini-blocks are stored into a single data buffer.

If the data is wide then we zip together the repetition and definition value with the value data into a single buffer. This approach is called “zipped”.

If there is any repetition information then we create a repetition index (TODO)

In addition, the compression process may create zero or more metadata buffers. For example, a dictionary compression will create dictionary metadata. Any mini-block approach has a metadata buffer of block sizes. This metadata is stored in a separate buffer on disk and read at initialization time.

TODO: We should concatenate metadata buffers from all pages into a single buffer at (roughly) the end of the file so there is, at most, one read per column of metadata per file.

Implementations

impl PrimitiveStructuralEncoder

pub fn try_new( options: &EncodingOptions, compression_strategy: Arc<dyn CompressionStrategy>, column_index: u32, field: Field, ) -> Result<Self>

Trait Implementations

impl FieldEncoder for PrimitiveStructuralEncoder

fn maybe_encode( &mut self, array: ArrayRef, _external_buffers: &mut OutOfLineBuffers, repdef: RepDefBuilder, row_number: u64, num_rows: u64, ) -> Result<Vec<EncodeTask>>

Buffer the data and, if there is enough data in the buffer to form a page, return an encoding task to encode the data.

fn flush( &mut self, _external_buffers: &mut OutOfLineBuffers, ) -> Result<Vec<EncodeTask>>

Flush any remaining data from the buffers into encoding tasks

fn num_columns(&self) -> u32

The number of output columns this encoding will create

fn finish( &mut self, _external_buffers: &mut OutOfLineBuffers, ) -> BoxFuture<'_, Result<Vec<EncodedColumn>>>

Finish encoding and return column metadata