Module write

APIs to write to Parquet format.

Arrow/Parquet Interoperability

As of parquet-format v2.9 there are Arrow DataTypes which do not have a parquet representation. These include but are not limited to:

• ArrowDataType::Timestamp(TimeUnit::Second, _)
• ArrowDataType::Int64
• ArrowDataType::Duration
• ArrowDataType::Date64
• ArrowDataType::Time32(TimeUnit::Second)

The use of these arrow types will result in no logical type being stored within a parquet file.

Re-exports

pub use crate::parquet::compression::BrotliLevel;

pub use crate::parquet::compression::CompressionOptions;

pub use crate::parquet::compression::GzipLevel;

pub use crate::parquet::compression::ZstdLevel;

pub use crate::parquet::encoding::Encoding;

pub use crate::parquet::metadata::Descriptor;

pub use crate::parquet::metadata::FileMetadata;

pub use crate::parquet::metadata::SchemaDescriptor;

pub use crate::parquet::page::CompressedDataPage;

pub use crate::parquet::page::CompressedPage;

pub use crate::parquet::page::Page;

pub use crate::parquet::schema::types::FieldInfo;

pub use crate::parquet::schema::types::ParquetType;

pub use crate::parquet::schema::types::PhysicalType as ParquetPhysicalType;

pub use crate::parquet::write::compress;

pub use crate::parquet::write::write_metadata_sidecar;

pub use crate::parquet::write::Compressor;

pub use crate::parquet::write::DynIter;

pub use crate::parquet::write::DynStreamingIterator;

pub use crate::parquet::write::RowGroupIterColumns;

pub use crate::parquet::write::Version;

pub use crate::parquet::fallible_streaming_iterator;

Structs

FileWriter

An interface to write a parquet to a Write

KeyValue

Wrapper struct to store key values

RowGroupIterator

An iterator adapter that converts an iterator over RecordBatchT into an iterator of row groups. Use it to create an iterator consumable by the parquet’s API.

StatisticsOptions

The statistics to write

ThriftFileMetadata

Description for file metadata

WriteOptions

Currently supported options to write to parquet

Enums

EncodeNullability

Options to encode an array

Nested

Descriptor of nested information of a field

Traits

FallibleStreamingIterator

A fallible, streaming iterator.

Functions

array_to_columns

Returns a vector of iterators of Page, one per leaf column in the array

array_to_page

Converts an Array to a CompressedPage based on options, descriptor and encoding.

array_to_page_simple

Converts an Array to a CompressedPage based on options, descriptor and encoding.

array_to_pages

Returns an iterator of Page.

arrays_to_columns

get_max_length

Get the length of Array that should be sliced.

num_values

return number values of the nested

row_group_iter

Maps a RecordBatchT and parquet-specific options to an RowGroupIterColumns used to write to parquet

slice_nested_leaf

returns offset and length to slice the leaf values

slice_parquet_array

Slices the Array to Box<dyn Array> and Vec<Nested>.

to_leaves

Convert Array to a Vec<Box<dyn Array>> leaves in DFS order.

to_nested

Constructs the necessary Vec<Vec<Nested>> to write the rep and def levels of array to parquet

to_parquet_leaves

Convert ParquetType to Vec<ParquetPrimitiveType> leaves in DFS order.

to_parquet_schema

Creates a parquet SchemaDescriptor from a ArrowSchema.

to_parquet_type

Creates a ParquetType from a Field.

transverse

Transverses the dtype up to its (parquet) columns and returns a vector of items based on map.

write_def_levels

writes the def levels to a Vec<u8> and returns it.

write_rep_and_def

Write repetition_levels and definition_levels to buffer.