Crate rkyv

rkyv

rkyv (archive) is a zero-copy deserialization framework for Rust.

It’s similar to other zero-copy deserialization frameworks such as Cap’n Proto and FlatBuffers. However, while the former have external schemas and heavily restricted data types, rkyv allows all serialized types to be defined in code and can serialize a wide variety of types that the others cannot. Additionally, rkyv is designed to have little to no overhead, and in most cases will perform exactly the same as native types.

Design

Like serde, rkyv uses Rust’s powerful trait system to serialize data without the need for reflection. Despite having a wide array of features, you also only pay for what you use. If your data checks out, the serialization process can be as simple as a memcpy! Like serde, this allows rkyv to perform at speeds similar to handwritten serializers.

Unlike serde, rkyv produces data that is guaranteed deserialization free. If you wrote your data to disk, you can just mmap your file into memory, cast a pointer, and your data is ready to use. This makes it ideal for high-performance and IO-bound applications.

Limited data mutation is supported through Pin APIs, and archived values can be truly deserialized with Deserialize if full mutation capabilities are needed.

The book has more details on the design and capabilities of rkyv.

Type support

rkyv has a hashmap implementation that is built for zero-copy deserialization, so you can serialize your hashmaps with abandon. The implementation performs perfect hashing with the compress, hash and displace algorithm to use as little memory as possible while still performing fast lookups.

It also comes with a B+ tree implementation that is built for maximum performance by splitting data into easily-pageable 4KB segments. This makes it perfect for building immutable databases and structures for bulk data.

rkyv also has support for contextual serialization, deserialization, and validation. It can properly serialize and deserialize shared pointers like Rc and Arc, and can be extended to support custom contextual types.

Finally, rkyv makes it possible to serialize trait objects and use them as trait objects without deserialization. See the archive_dyn crate for more details.

Tradeoffs

While rkyv is a great format for final data, it lacks a full schema system and isn’t well equipped for data migration and schema upgrades. If your use case requires these capabilities, you may need additional libraries the build these features on top of rkyv. You can use other serialization frameworks like serde with the same types as rkyv conflict-free.

Features

• alloc: Enables types that require the alloc crate. Enabled by default.

• arbitrary_enum_discriminant: Enables the arbitrary_enum_discriminant feature for stable multibyte enum discriminants using archive_le and archive_be. Requires nightly.

• archive_be: Forces archives into a big-endian format. This guarantees cross-endian compatibility optimized for big-endian architectures.

• archive_le: Forces archives into a little-endian format. This guarantees cross-endian compatibility optimized for little-endian architectures.

• copy: Enables copy optimizations for packed copyable data types. Requires nightly.

• copy_unsafe: Automatically opts all potentially copyable types into copy optimization. This broadly improves performance but may cause uninitialized bytes to be copied to the output. Requires nightly.

• size_16: Archives integral *size types as 16-bit integers. This is intended to be used only for small archives and may not handle large, more general data.

• size_32: Archives integral *size types as 32-bit integers. Enabled by default.

• size_64: Archives integral *size types as 64-bit integers. This is intended to be used only for very large archives and may cause unnecessary data bloat.

• std: Enables standard library support. Enabled by default.

• strict: Guarantees that types will have the same representations across platforms and compilations. This is already the case in practice, but this feature provides a guarantee along with C type compatibility.

  Note: Enabling strict will disable Archive implementations for tuples, as tuples do not have a C type layout. Making a generic Tuple<T1, T2> and deriving Archive for it should provide similar functionality.

• validation: Enables validation support through bytecheck.

Crate support

Some common crates need to be supported by rkyv before an official integration has been made. Support is provided by rkyv for these crates, but in the future crates should depend on rkyv and provide their own implementations. The crates that already have support provided by rkyv should work toward integrating the implementations into themselves.

Crates supported by rkyv:

• indexmap
• rend Enabled automatically when using endian-specific archive features.
• tinyvec
• uuid

Support for each of these crates can be enabled with a feature of the same name. Additionally, the following external crate features are available:

• tinyvec_alloc: Supports types behind the alloc feature in tinyvec.
• uuid_std: Enables the std feature in uuid.

Examples

• See Archive for examples of how to use rkyv through the derive macro and manual implementation.
• For more details on the derive macro and its capabilities, see Archive.
• Fully worked examples using rkyv are available in the examples directory of the source repo.

Re-exports

• pub use rend;
• pub use bytecheck;
• pub use validation::check_archived_root_with_context;
• pub use validation::check_archived_value_with_context;
• pub use validation::validators::check_archived_root;
• pub use validation::validators::check_archived_value;
• pub use validation::validators::from_bytes;
• pub use util::*;

Modules

• boxed
  An archived version of Box.
• collections
  Archived versions of standard library containers.
• de
  Deserialization traits, deserializers, and adapters.
• ffi
  Archived versions of FFI types.
• net
  Archived versions of network types.
• niche
  Manually niched type replacements.
• ops
  Archived versions of ops types.
• option
  An archived version of Option.
• rc
  Archived versions of shared pointers.
• rel_ptr
  Relative pointer implementations and options.
• result
  An archived version of Result.
• ser
  Serialization traits, serializers, and adapters.
• string
  Archived versions of string types.
• time
  Archived versions of time types.
• util
  Utilities for common archive operations.
• validation
  Validation implementations and helper types.
• vec
  An archived version of Vec.
• with
  Wrapper type support and commonly used wrappers.

Macros

• from_archived
  Returns the unarchived value of the given archived primitive.
• out_field
  Returns a tuple of (field_pos, field_out), where field_pos is the “position”, i.e. offset in bytes, of the field relative to the base address of the struct and field_out is a *mut that points to the field directly.
• to_archived
  Returns the archived value of the given archived primitive.

Structs

• Infallible
  A fallible type that cannot produce errors.

Traits

• Archive
  A type that can be used without deserializing.
• ArchivePointee
  An archived type with associated metadata for its relative pointer.
• ArchiveUnsized
  A counterpart of Archive that’s suitable for unsized types.
• CheckBytes
  A type that can check whether a pointer points to a valid value.
• Deserialize
  Converts a type back from its archived form.
• DeserializeUnsized
  A counterpart of Deserialize that’s suitable for unsized types.
• Fallible
  A type that can produce an error.
• Serialize
  Converts a type to its archived form.
• SerializeUnsized
  A counterpart of Serialize that’s suitable for unsized types.

Type Definitions

• Archived
  Alias for the archived version of some Archive type.
• ArchivedMetadata
  Alias for the archived metadata for some ArchiveUnsized type.
• FixedIsize
  The native type that isize is converted to for archiving.
• FixedUsize
  The native type that usize is converted to for archiving.
• MetadataResolver
  Alias for the metadata resolver for some ArchiveUnsized type.
• RawRelPtr
  The default raw relative pointer.
• RelPtr
  The default relative pointer.
• Resolver
  Alias for the resolver for some Archive type.

Derive Macros

• Archive
  Derives Archive for the labeled type.
• CheckBytes
  Derives CheckBytes for the labeled type.
• Deserialize
  Derives Deserialize for the labeled type.
• Serialize
  Derives Serialize for the labeled type.