Struct DecodeBatchScheduler

pub struct DecodeBatchScheduler {
    pub root_fields: Fields,
    /* private fields */
}

The scheduler for decoding batches

Lance decoding is done in two steps, scheduling, and decoding. The scheduling tends to be lightweight and should quickly figure what data is needed from the disk issue the appropriate I/O requests. A decode task is created to eventually decode the data (once it is loaded) and scheduling moves on to scheduling the next page.

Meanwhile, it’s expected that a decode stream will be setup to run at the same time. Decode tasks take the data that is loaded and turn it into Arrow arrays.

This approach allows us to keep our I/O parallelism and CPU parallelism completely separate since those are often two very different values.

Backpressure should be achieved via the I/O service. Requests that are issued will pile up if the decode stream is not polling quickly enough. The crate::EncodingsIo::submit_request function should return a pending future once there are too many I/O requests in flight.

TODO: Implement backpressure

Fields

root_fields: Fields

Implementations

impl DecodeBatchScheduler

pub async fn try_new<'a>( schema: &'a Schema, column_indices: &[u32], column_infos: &[Arc<ColumnInfo>], file_buffer_positions_and_sizes: &'a Vec<(u64, u64)>, num_rows: u64, _decoder_plugins: Arc<DecoderPlugins>, io: Arc<dyn EncodingsIo>, cache: Arc<FileMetadataCache>, filter: &FilterExpression, ) -> Result<Self>

Creates a new decode scheduler with the expected schema and the column metadata of the file.

pub fn from_scheduler( root_scheduler: Arc<dyn FieldScheduler>, root_fields: Fields, cache: Arc<FileMetadataCache>, ) -> Self

pub fn schedule_ranges_to_vec( &mut self, ranges: &[Range<u64>], filter: &FilterExpression, io: Arc<dyn EncodingsIo>, priority: Option<Box<dyn PriorityRange>>, ) -> Result<Vec<DecoderMessage>>

pub fn schedule_ranges( &mut self, ranges: &[Range<u64>], filter: &FilterExpression, sink: UnboundedSender<Result<DecoderMessage>>, scheduler: Arc<dyn EncodingsIo>, )

Schedules the load of multiple ranges of rows

Ranges must be non-overlapping and in sorted order

Arguments

• ranges - The ranges of rows to load
• sink - A channel to send the decode tasks
• scheduler An I/O scheduler to issue I/O requests

pub fn schedule_range( &mut self, range: Range<u64>, filter: &FilterExpression, sink: UnboundedSender<Result<DecoderMessage>>, scheduler: Arc<dyn EncodingsIo>, )

Schedules the load of a range of rows

Arguments

• range - The range of rows to load
• sink - A channel to send the decode tasks
• scheduler An I/O scheduler to issue I/O requests

pub fn schedule_take( &mut self, indices: &[u64], filter: &FilterExpression, sink: UnboundedSender<Result<DecoderMessage>>, scheduler: Arc<dyn EncodingsIo>, )

Schedules the load of selected rows

Arguments

• indices - The row indices to load (these must be in ascending order!)
• sink - A channel to send the decode tasks
• scheduler An I/O scheduler to issue I/O requests