Struct DataFile

pub struct DataFile {
    pub path: String,
    pub fields: Vec<i32>,
    pub column_indices: Vec<i32>,
    pub file_major_version: u32,
    pub file_minor_version: u32,
}

Lance Data File

Fields

path: String

Relative path to the root.

fields: Vec<i32>

The ids of the fields/columns in this file.

-1 is used for “unassigned” while in memory. It is not meant to be written to disk. -2 is used for “tombstoned”, meaningful a field that is no longer in use. This is often because the original field id was reassigned to a different data file.

In Lance v1 IDs are assigned based on position in the file, offset by the max existing field id in the table (if any already). So when a fragment is first created with one file of N columns, the field ids will be 1, 2, …, N. If a second, fragment is created with M columns, the field ids will be N+1, N+2, …, N+M.

In Lance v1 there is one field for each field in the input schema, this includes nested fields (both struct and list). Fixed size list fields have only a single field id (these are not considered nested fields in Lance v1).

This allows column indices to be calculated from field IDs and the input schema.

In Lance v2 the field IDs generally follow the same pattern but there is no way to calculate the column index from the field ID. This is because a given field could be encoded in many different ways, some of which occupy a different number of columns. For example, a struct field could be encoded into N + 1 columns or it could be encoded into a single packed column. To determine column indices the column_indices property should be used instead.

In Lance v1 these ids must be sorted but might not always be contiguous.

column_indices: Vec<i32>

The top-level column indices for each field in the file.

If the data file is version 1 then this property will be empty

Otherwise there must be one entry for each field in fields.

Some fields may not correspond to a top-level column in the file. In these cases the index will -1.

For example, consider the schema:

• dimension: packed-struct (0):
  • x: u32 (1)
  • y: u32 (2)
• path: list (3)
• embedding: fsl<768> (4)
  • fp64
• borders: fsl<4> (5)
  • simple-struct (6)
    • margin: fp64 (7)
    • padding: fp64 (8)

One possible column indices array could be: [0, -1, -1, 1, 3, 4, 5, 6, 7]

This reflects quite a few phenomenon:

• The packed struct is encoded into a single column and there is no top-level column for the x or y fields
• The variable sized list is encoded into two columns
• The embedding is encoded into a single column (common for FSL of primitive) and there is not “FSL column”
• The borders field actually does have an “FSL column”

The column indices table may not have duplicates (other than -1)

file_major_version: u32

The major file version used to create the file

file_minor_version: u32

The minor file version used to create the file

If both file_major_version and file_minor_version are set to 0, then this is a version 0.1 or version 0.2 file.

Trait Implementations

impl Clone for DataFile

fn clone(&self) -> DataFile

Returns a copy of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for DataFile

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Default for DataFile

fn default() -> Self

Returns the “default value” for a type.

impl From<&DataFile> for DataFile

fn from(df: &DataFile) -> Self

Converts to this type from the input type.

impl Message for DataFile

fn encoded_len(&self) -> usize

Returns the encoded length of the message without a length delimiter.

fn clear(&mut self)

Clears the message, resetting all fields to their default.

fn encode(&self, buf: &mut impl BufMut) -> Result<(), EncodeError>

where Self: Sized,

Encodes the message to a buffer.

fn encode_to_vec(&self) -> Vec<u8>

where Self: Sized,

Encodes the message to a newly allocated buffer.

fn encode_length_delimited( &self, buf: &mut impl BufMut, ) -> Result<(), EncodeError>

where Self: Sized,

Encodes the message with a length-delimiter to a buffer.

fn encode_length_delimited_to_vec(&self) -> Vec<u8>

where Self: Sized,

Encodes the message with a length-delimiter to a newly allocated buffer.

fn decode(buf: impl Buf) -> Result<Self, DecodeError>

where Self: Default,

Decodes an instance of the message from a buffer.

fn decode_length_delimited(buf: impl Buf) -> Result<Self, DecodeError>

where Self: Default,

Decodes a length-delimited instance of the message from the buffer.

fn merge(&mut self, buf: impl Buf) -> Result<(), DecodeError>

where Self: Sized,

Decodes an instance of the message from a buffer, and merges it into self.

fn merge_length_delimited(&mut self, buf: impl Buf) -> Result<(), DecodeError>

where Self: Sized,

Decodes a length-delimited instance of the message from buffer, and merges it into self.

impl Name for DataFile

const NAME: &'static str = "DataFile"

Simple name for this Message. This name is the same as it appears in the source .proto file, e.g. FooBar.

const PACKAGE: &'static str = "lance.table"

Package name this message type is contained in. They are domain-like and delimited by ., e.g. google.protobuf.

fn full_name() -> String

Fully-qualified unique name for this Message. It’s prefixed with the package name and names of any parent messages, e.g. google.rpc.BadRequest.FieldViolation. By default, this is the package name followed by the message name. Fully-qualified names must be unique within a domain of Type URLs.

fn type_url() -> String

Type URL for this Message, which by default is the full name with a leading slash, but may also include a leading domain name, e.g. type.googleapis.com/google.profile.Person. This can be used when serializing into the google.protobuf.Any type.

impl PartialEq for DataFile

fn eq(&self, other: &DataFile) -> bool

Tests for self and other values to be equal, and is used by ==.

fn ne(&self, other: &Rhs) -> bool

Tests for !=. The default implementation is almost always sufficient, and should not be overridden without very good reason.

impl TryFrom<DataFile> for DataFile

type Error = Error

The type returned in the event of a conversion error.

fn try_from(proto: DataFile) -> Result<Self>

Performs the conversion.

impl StructuralPartialEq for DataFile