lance_encoding

Module encoder

Source

Structs§

BatchEncoder
A batch encoder that encodes RecordBatch objects by delegating to field encoders for each top-level field in the batch.
BitpackingBufferMeta
ColumnIndexSequence
Keeps track of the current column index and makes a mapping from field id to column index
CoreArrayEncodingStrategy
The core array encoding strategy is a set of basic encodings that are generally applicable in most scenarios.
CoreFieldEncodingStrategy
The core field encoding strategy is a set of basic encodings that are generally applicable in most scenarios.
EncodedArray
An encoded array
EncodedBatch
An encoded batch of data and a page table describing it
EncodedBufferMeta
EncodedColumn
EncodedPage
An encoded page of data
EncodingOptions
Options that control the encoding process
MiniBlockChunk
Describes the size of a mini-block chunk of data
MiniBlockCompressed
Page data that has been compressed into a series of chunks put into a single buffer.
OutOfLineBuffers
A tool to reserve space for buffers that are not in-line with the data
StructuralEncodingStrategy
An encoding strategy used for 2.1+ files

Enums§

PerValueDataBlock
Per-value compression must either:

Constants§

MAX_MINIBLOCK_BYTES
MAX_MINIBLOCK_VALUES
MIN_PAGE_BUFFER_ALIGNMENT
The minimum alignment for a page buffer. Writers must respect this.

Traits§

ArrayEncoder
Encodes data from one format to another (hopefully more compact or useful) format
ArrayEncodingStrategy
A trait to pick which encoding strategy to use for a single page of data
BlockCompressor
Trait for compression algorithms that compress an entire block of data into one opaque and self-described chunk.
CompressionStrategy
A trait to pick which compression to use for given data
FieldEncoder
Top level encoding trait to code any Arrow array type into one or more pages.
FieldEncodingStrategy
A trait to pick which kind of field encoding to use for a field
MiniBlockCompressor
Trait for compression algorithms that are suitable for use in the miniblock structural encoding
PerValueCompressor
Trait for compression algorithms that are suitable for use in the zipped structural encoding
VariablePerValueCompressor
Trait for compression algorithms that are suitable for use in the zipped structural encoding

Functions§

default_encoding_strategy
encode_batch
Helper method to encode a batch of data into memory
values_column_encoding

Type Aliases§

EncodeTask
A task to create a page of data