Struct ListFieldScheduler

pub struct ListFieldScheduler { /* private fields */ }

A page scheduler for list fields that encodes offsets in one field and items in another

The list scheduler is somewhat unique because it requires indirect I/O. We cannot know the ranges we need simply by looking at the metadata. This means that list scheduling doesn’t fit neatly into the two-thread schedule-loop / decode-loop model. To handle this, when a list page is scheduled, we only schedule the I/O for the offsets and then we immediately launch a new tokio task. This new task waits for the offsets, decodes them, and then schedules the I/O for the items. Keep in mind that list items can be lists themselves. If that is the case then this indirection will continue. The decode task that is returned will only finish waiting when all of the I/O has completed.

Whenever we schedule follow-up I/O like this the priority is based on the top-level row index. This helps ensure that earlier rows get finished completely (including follow up tasks) before we perform I/O for later rows.

Implementations

impl ListFieldScheduler

pub fn new( offsets_scheduler: Arc<dyn FieldScheduler>, items_scheduler: Arc<dyn FieldScheduler>, items_field: Arc<Field>, offset_type: DataType, offset_page_info: Vec<OffsetPageInfo>, ) -> Self

Trait Implementations

impl Debug for ListFieldScheduler

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl FieldScheduler for ListFieldScheduler

fn schedule_ranges<'a>( &'a self, ranges: &[Range<u64>], filter: &FilterExpression, ) -> Result<Box<dyn SchedulingJob + 'a>>

Schedules I/O for the requested portions of the field.

fn num_rows(&self) -> u64

The number of rows in this field

fn initialize<'a>( &'a self, _filter: &'a FilterExpression, _context: &'a SchedulerContext, ) -> BoxFuture<'a, Result<()>>

Called at the beginning of scheduling to initialize the scheduler