Crate bitpacking

source ·
Expand description

Fast Bitpacking algorithms

This crate is a Rust port of Daniel Lemire’s simdcomp C library. It contains different flavor of integers compression via bitpacking : BitPacker1x, BitPacker4x, and BitPacker8x.

Each produces different formats, and are incompatible one with another, and requires integers to be encoded in block of different size..

BitPacker4x and BitPacker8x are designed specifically to leverage SSE3 and AVX2 instructions respectively.

The library will fallback to a scalar implementation if these instruction sets are not available. For instance :

  • because your compilation target architecture is not x86_64
  • because the CPU you use is from an older generation

I recommend using BitPacker4x if you are in doubt.

See the BitPacker trait for example usage.

Structs

  • BitPacker1x
    BitPacker1x is standard bitpacking : the integer representation over b bits are simply concatenated one after the other.
  • BitPacker4x
    BitPacker4x packs integers in groups of 4. This gives an opportunity to leverage SSE3 instructions to encode and decode the stream.
  • BitPacker8x
    BitPacker8x packs integers in groups of 8. This gives an opportunity to leverage AVX2 instructions to encode and decode the stream. One block must contain 256 integers.

Traits