Struct Scanner

pub struct Scanner { /* private fields */ }

Dataset Scanner

let dataset = Dataset::open(uri).await.unwrap();
let stream = dataset.scan()
    .project(&["col", "col2.subfield"]).unwrap()
    .limit(10)
    .into_stream();
stream
  .map(|batch| batch.num_rows())
  .buffered(16)
  .sum()

Implementations

impl Scanner

pub fn new(dataset: Arc<Dataset>) -> Self

pub fn from_fragment(dataset: Arc<Dataset>, fragment: Fragment) -> Self

pub fn with_fragments(&mut self, fragments: Vec<Fragment>) -> &mut Self

Set which fragments should be scanned.

If scan_in_order is set to true, the fragments will be scanned in the order of the vector.

pub fn project<T: AsRef<str>>(&mut self, columns: &[T]) -> Result<&mut Self>

Projection.

Only select the specified columns. If not specified, all columns will be scanned.

pub fn project_with_transform( &mut self, columns: &[(impl AsRef<str>, impl AsRef<str>)], ) -> Result<&mut Self>

Projection with transform

Only select the specified columns with the given transform.

pub fn prefilter(&mut self, should_prefilter: bool) -> &mut Self

Should the filter run before the vector index is applied

If true then the filter will be applied before the vector index. This means the results will be accurate but the overall query may be more expensive.

If false then the filter will be applied to the nearest results. This means you may get back fewer results than you ask for (or none at all) if the closest results do not match the filter.

pub fn materialization_style( &mut self, style: MaterializationStyle, ) -> &mut Self

Set the materialization style for the scan

This controls when columns are fetched from storage. The default should work well for most cases.

If you know (in advance) a query will return relatively few results (less than 0.1% of the rows) then you may want to experiment with applying late materialization to more (or all) columns.

If you know a query is going to return many rows then you may want to experiment with applying early materialization to more (or all) columns.

pub fn filter(&mut self, filter: &str) -> Result<&mut Self>

Apply filters

The filters can be presented as the string, as in WHERE clause in SQL.

let dataset = Dataset::open(uri).await.unwrap();
let stream = dataset.scan()
    .project(&["col", "col2.subfield"]).unwrap()
    .filter("a > 10 AND b < 200").unwrap()
    .limit(10)
    .into_stream();

Once the filter is applied, Lance will create an optimized I/O plan for filtering.

pub fn full_text_search( &mut self, query: FullTextSearchQuery, ) -> Result<&mut Self>

Filter by full text search The column must be a string column. The query is a string to search for. The search is case-insensitive, BM25 scoring is used.

let dataset = Dataset::open(uri).await.unwrap();
let stream = dataset.scan()
   .project(&["col", "col2.subfield"]).unwrap()
   .full_text_search("col", "query").unwrap()
   .limit(10)
   .into_stream();

pub fn filter_substrait(&mut self, filter: &[u8]) -> Result<&mut Self>

Set a filter using a Substrait ExtendedExpression message

The message must contain exactly one expression and that expression must be a scalar expression whose return type is boolean.

pub fn batch_size(&mut self, batch_size: usize) -> &mut Self

Set the batch size.

pub fn io_buffer_size(&mut self, size: u64) -> &mut Self

Set the I/O buffer size

This is the amount of RAM that will be reserved for holding I/O received from storage before it is processed. This is used to control the amount of memory used by the scanner. If the buffer is full then the scanner will block until the buffer is processed.

Generally this should scale with the number of concurrent I/O threads. The default is 2GiB which comfortably provides enough space for somewhere between 32 and 256 concurrent I/O threads.

This value is not a hard cap on the amount of RAM the scanner will use. Some space is used for the compute (which can be controlled by the batch size) and Lance does not keep track of memory after it is returned to the user.

Currently, if there is a single batch of data which is larger than the io buffer size then the scanner will deadlock. This is a known issue and will be fixed in a future release.

pub fn batch_readahead(&mut self, nbatches: usize) -> &mut Self

Set the prefetch size.

pub fn fragment_readahead(&mut self, nfragments: usize) -> &mut Self

Set the fragment readahead.

This is only used if scan_in_order is set to false.

pub fn scan_in_order(&mut self, ordered: bool) -> &mut Self

Set whether to read data in order (default: true)

A scan will always read from the disk concurrently. If this property is true then a ready batch (a batch that has been read from disk) will only be returned if it is the next batch in the sequence. Otherwise, the batch will be held until the stream catches up. This means the sequence is returned in order but there may be slightly less parallelism.

If this is false, then batches will be returned as soon as they are available, potentially increasing throughput slightly

If an ordering is defined (using Self::order_by) then the scan will always scan in parallel and any value set here will be ignored.

pub fn use_scalar_index(&mut self, use_scalar_index: bool) -> &mut Self

Set whether to use scalar index.

By default, scalar indices will be used to optimize a query if available. However, in some corner cases, scalar indices may not be the best choice. This option allows users to disable scalar indices for a query.

pub fn limit( &mut self, limit: Option<i64>, offset: Option<i64>, ) -> Result<&mut Self>

Set limit and offset.

If offset is set, the first offset rows will be skipped. If limit is set, only the provided number of rows will be returned. These can be set independently. For example, setting offset to 10 and limit to None will skip the first 10 rows and return the rest of the rows in the dataset.

pub fn nearest( &mut self, column: &str, q: &Float32Array, k: usize, ) -> Result<&mut Self>

Find k-nearest neighbor within the vector column.

pub fn nprobs(&mut self, n: usize) -> &mut Self

pub fn ef(&mut self, ef: usize) -> &mut Self

pub fn fast_search(&mut self) -> &mut Self

Only search the data being indexed.

Default value is false.

This is essentially a weak consistency search, only on the indexed data.

pub fn refine(&mut self, factor: u32) -> &mut Self

Apply a refine step to the vector search.

A refine improves query accuracy but also makes search slower, by reading extra elements and using the original vector values to re-rank the distances.

• factor - the factor of extra elements to read. For example, if factor is 2, then the search will read 2x more elements than the requested k before performing the re-ranking. Note: even if the factor is 1, the results will still be re-ranked without fetching additional elements.

pub fn distance_metric(&mut self, metric_type: MetricType) -> &mut Self

Change the distance MetricType, i.e, L2 or Cosine distance.

pub fn order_by( &mut self, ordering: Option<Vec<ColumnOrdering>>, ) -> Result<&mut Self>

Sort the results of the scan by one or more columns

If Some, then the resulting stream will be sorted according to the given ordering. This may increase the latency of the first result since all data must be read before the first batch can be returned.

pub fn use_index(&mut self, use_index: bool) -> &mut Self

Set whether to use the index if available

pub fn with_row_id(&mut self) -> &mut Self

Instruct the scanner to return the _rowid meta column from the dataset.

pub fn with_row_address(&mut self) -> &mut Self

Instruct the scanner to return the _rowaddr meta column from the dataset.

pub fn use_stats(&mut self, use_stats: bool) -> &mut Self

Set whether to use statistics to optimize the scan (default: true)

This is used for debugging or benchmarking purposes.

pub async fn schema(&self) -> Result<SchemaRef>

The Arrow schema of the output, including projections and vector / _distance

pub fn get_filter(&self) -> Result<Option<Expr>>

Fetches the currently set filter

Note that this forces the filter to be evaluated and the result will depend on the current state of the scanner (e.g. if with_row_id has been called then _rowid will be available for filtering but not otherwise) and so you may want to call this after setting all other options.

pub async fn try_into_stream(&self) -> Result<DatasetRecordBatchStream>

Create a stream from the Scanner.

pub async fn try_into_batch(&self) -> Result<RecordBatch>

pub async fn count_rows(&self) -> Result<u64>

Scan and return the number of matching rows

pub async fn create_plan(&self) -> Result<Arc<dyn ExecutionPlan>>

Create ExecutionPlan for Scan.

An ExecutionPlan is a graph of operators that can be executed.

The following plans are supported:

• Plain scan without filter or limits.

Scan(projections)

• Scan with filter and/or limits.

Scan(filtered_cols) -> Filter(expr)
   -> (*LimitExec(limit, offset))
   -> Take(remaining_cols) -> Projection()

• Use KNN Index (with filter and/or limits)

KNNIndex() -> Take(vector) -> FlatRefine()
    -> Take(filtered_cols) -> Filter(expr)
    -> (*LimitExec(limit, offset))
    -> Take(remaining_cols) -> Projection()

• Use KNN flat (brute force) with filter and/or limits

Scan(vector) -> FlatKNN()
    -> Take(filtered_cols) -> Filter(expr)
    -> (*LimitExec(limit, offset))
    -> Take(remaining_cols) -> Projection()

In general, a plan has 5 stages:

1. Source (from dataset Scan or from index, may include prefilter)
2. Filter
3. Sort
4. Limit / Offset
5. Take remaining columns / Projection

pub async fn explain_plan(&self, verbose: bool) -> Result<String>