Struct StreamDecoder

Source

pub struct StreamDecoder { /* private fields */ }

Expand description

A low-level interface for reading RecordBatch data from a stream of bytes

See StreamReader for a higher-level interface

Implementations§

Source §

impl StreamDecoder

Source

pub fn new() -> Self

Create a new StreamDecoder

Source

pub fn with_require_alignment(self, require_alignment: bool) -> Self

Specifies whether or not array data in input buffers is required to be properly aligned.

If require_alignment is true, this decoder will return an error if any array data in the input buf is not properly aligned. Under the hood it will use arrow_data::ArrayDataBuilder::build to construct arrow_data::ArrayData.

If require_alignment is false (the default), this decoder will automatically allocate a new aligned buffer and copy over the data if any array data in the input buf is not properly aligned. (Properly aligned array data will remain zero-copy.) Under the hood it will use arrow_data::ArrayDataBuilder::build_aligned to construct arrow_data::ArrayData.

Source

pub fn decode( &mut self, buffer: &mut Buffer, ) -> Result<Option<RecordBatch>, ArrowError>

Try to read the next RecordBatch from the provided Buffer

Buffer::advance will be called on buffer for any consumed bytes.

The push-based interface facilitates integration with sources that yield arbitrarily delimited bytes ranges, such as a chunked byte stream received from object storage

fn print_stream<I>(src: impl Iterator<Item = Buffer>) -> Result<(), ArrowError> {
    let mut decoder = StreamDecoder::new();
    for mut x in src {
        while !x.is_empty() {
            if let Some(x) = decoder.decode(&mut x)? {
                println!("{x:?}");
            }
        }
    }
    decoder.finish().unwrap();
    Ok(())
}