Struct RecordBatchWriter

pub struct RecordBatchWriter { /* private fields */ }

Writes messages to a delta lake table.

Implementations

impl RecordBatchWriter

pub fn try_new( table_uri: impl AsRef<str>, schema: ArrowSchemaRef, partition_columns: Option<Vec<String>>, storage_options: Option<HashMap<String, String>>, ) -> Result<Self, DeltaTableError>

Create a new RecordBatchWriter instance

pub fn for_table(table: &DeltaTable) -> Result<Self, DeltaTableError>

Creates a RecordBatchWriter to write data to provided Delta Table

pub fn buffer_len(&self) -> usize

Returns the current byte length of the in memory buffer. This may be used by the caller to decide when to finalize the file write.

pub fn buffered_record_batch_count(&self) -> usize

Returns the number of records held in the current buffer.

pub fn reset(&mut self)

Resets internal state.

pub fn arrow_schema(&self) -> ArrowSchemaRef

Returns the arrow schema representation of the delta table schema defined for the wrapped table.

pub async fn write_partition( &mut self, record_batch: RecordBatch, partition_values: &IndexMap<String, Scalar>, mode: WriteMode, ) -> Result<ArrowSchemaRef, DeltaTableError>

Write a batch to the specified partition

pub fn with_writer_properties(self, writer_properties: WriterProperties) -> Self

Sets the writer properties for the underlying arrow writer.

Trait Implementations

impl Debug for RecordBatchWriter

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl DeltaWriter<RecordBatch> for RecordBatchWriter

fn write<'life0, 'async_trait>( &'life0 mut self, values: RecordBatch, ) -> Pin<Box<dyn Future<Output = Result<(), DeltaTableError>> + Send + 'async_trait>>

where Self: 'async_trait, 'life0: 'async_trait,

Write a chunk of values into the internal write buffers with the default write mode

fn write_with_mode<'life0, 'async_trait>( &'life0 mut self, values: RecordBatch, mode: WriteMode, ) -> Pin<Box<dyn Future<Output = Result<(), DeltaTableError>> + Send + 'async_trait>>

where Self: 'async_trait, 'life0: 'async_trait,

Divides a single record batch into into multiple according to table partitioning. Values are written to arrow buffers, to collect data until it should be written to disk.

fn flush<'life0, 'async_trait>( &'life0 mut self, ) -> Pin<Box<dyn Future<Output = Result<Vec<Add>, DeltaTableError>> + Send + 'async_trait>>

where Self: 'async_trait, 'life0: 'async_trait,

Writes the existing parquet bytes to storage and resets internal state to handle another file.

fn flush_and_commit<'life0, 'life1, 'async_trait>( &'life0 mut self, table: &'life1 mut DeltaTable, ) -> Pin<Box<dyn Future<Output = Result<i64, DeltaTableError>> + Send + 'async_trait>>

where Self: 'async_trait, 'life0: 'async_trait, 'life1: 'async_trait,

Flush the internal write buffers to files in the delta table folder structure. and commit the changes to the Delta log, creating a new table version.