Struct PartitionWriter

pub struct PartitionWriter { /* private fields */ }

Partition writer implementation This writer takes in table data as RecordBatches and writes it out to partitioned parquet files. It buffers data in memory until it reaches a certain size, then writes it out to optimize file sizes. When you complete writing you get back a list of Add actions that can be used to update the Delta table commit log.

Implementations

impl PartitionWriter

pub fn try_with_config( object_store: ObjectStoreRef, config: PartitionWriterConfig, num_indexed_cols: i32, stats_columns: Option<Vec<String>>, ) -> DeltaResult<Self>

Create a new instance of PartitionWriter from PartitionWriterConfig

pub async fn write(&mut self, batch: &RecordBatch) -> DeltaResult<()>

Buffers record batches in-memory up to appx. target_file_size. Flushes data to storage once a full file can be written.

The close method has to be invoked to write all data still buffered and get the list of all written files.

pub async fn close(self) -> DeltaResult<Vec<Add>>

Close the writer and get the new Add actions.

Trait Implementations

impl Debug for PartitionWriter

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.