Struct FileWriter

pub struct FileWriter { /* private fields */ }

Implementations

impl FileWriter

pub fn try_new( object_writer: ObjectWriter, schema: LanceSchema, options: FileWriterOptions, ) -> Result<Self>

Create a new FileWriter with a desired output schema

pub fn new_lazy(object_writer: ObjectWriter, options: FileWriterOptions) -> Self

Create a new FileWriter without a desired output schema

The output schema will be set based on the first batch of data to arrive. If no data arrives and the writer is finished then the write will fail.

pub async fn create_file_with_batches( store: &ObjectStore, path: &Path, schema: Schema, batches: impl Iterator<Item = RecordBatch> + Send, options: FileWriterOptions, ) -> Result<usize>

Write a series of record batches to a new file

Returns the number of rows written

pub fn version(&self) -> LanceFileVersion

Returns the format version that will be used when writing the file

pub async fn write_batches( &mut self, batches: impl Iterator<Item = &RecordBatch>, ) -> Result<()>

Schedule batches of data to be written to the file

pub async fn write_batch(&mut self, batch: &RecordBatch) -> Result<()>

Schedule a batch of data to be written to the file

Note: the future returned by this method may complete before the data has been fully flushed to the file (some data may be in the data cache or the I/O cache)

pub fn add_schema_metadata( &mut self, key: impl Into<String>, value: impl Into<String>, )

Add a metadata entry to the schema

This method is useful because sometimes the metadata is not known until after the data has been written. This method allows you to alter the schema metadata. It must be called before finish is called.

pub async fn add_global_buffer(&mut self, buffer: Bytes) -> Result<u32>

Adds a global buffer to the file

The global buffer can contain any arbitrary bytes. It will be written to the disk immediately. This method returns the index of the global buffer (this will always start at 1 and increment by 1 each time this method is called)

pub async fn finish(&mut self) -> Result<u64>

Finishes writing the file

This method will wait until all data has been flushed to the file. Then it will write the file metadata and the footer. It will not return until all data has been flushed and the file has been closed.

Returns the total number of rows written

pub async fn tell(&mut self) -> Result<u64>

pub fn field_id_to_column_indices(&self) -> &[(u32, u32)]