Struct DeltaWriter

pub struct DeltaWriter { /* private fields */ }

A parquet writer implementation tailored to the needs of writing data to a delta table.

Implementations

impl DeltaWriter

pub fn new(object_store: ObjectStoreRef, config: WriterConfig) -> Self

Create a new instance of DeltaWriter

pub fn with_writer_properties(self, writer_properties: WriterProperties) -> Self

Apply custom writer_properties to the underlying parquet writer

pub async fn write_partition( &mut self, record_batch: RecordBatch, partition_values: &IndexMap<String, Scalar>, ) -> DeltaResult<()>

Write a batch to the partition induced by the partition_values. The record batch is expected to be pre-partitioned and only contain rows that belong into the same partition. However, it should still contain the partition columns.

pub async fn write(&mut self, batch: &RecordBatch) -> DeltaResult<()>

Buffers record batches in-memory per partition up to appx. target_file_size for a partition. Flushes data to storage once a full file can be written.

The close method has to be invoked to write all data still buffered and get the list of all written files.

pub async fn close(self) -> DeltaResult<Vec<Add>>

Close the writer and get the new Add actions.

This will flush all remaining data.

Trait Implementations

impl Debug for DeltaWriter

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.