Data compression

Funders

Supporters & Past funders

Building a data compression ecosystem

Compression algorithms are used in a vast number of protocols and file formats throughout all of computing. Implemented in C, these libraries encounter regular security issues despite receiving extensive industry-wide scrutiny.

Our initiative aims to create memory-safe implementations of compression libraries:

  • zlib: a widely-used compression library, used primarily on the web to provide gzip compression to the text/html/js/css we send around.
  • zstd: a modern successor to zlib, providing better compression faster.
  • bzip2: a file compression program that is widely deployed and supported e.g. as part of zip.
  • xz: a compression format that provides very good compression, but is comparatively slow. Commonly used for large file downloads.

Please get in touch with us, if you are interested in financially supporting the development of memory-safe zlib, zstd or xz. We offer technical support services for organizations wanting to adopt our data compression software.

What We've Done

For zlib, we've created an implementation based on zlib-ng, called zlib-rs, with a focus on maintaining excellent performance while introducing memory safety. The initial development of zlib-rs was started and partly funded by Prossimo and Tweede golf.

In April 2024, an early release of zlib-rs was integrated in flate2. In Nov 2024 an audit by ISRG was succesfully completed, and optimizations for Webassembly were included in a new release.

The development of bzip2, the 2nd project in this initiative, started Oct 2024. Unlike in zlib-rs we use c2rust to translate the original bzip2 C code to Rust. Early releases are available on GitHub, and also through the bzip2 crate we now maintain.

What's Next

We're currently seeking funding to complete work on zlib and to start work on xz and zstd.

The high level goals for the four projects are:

  • provide on-par performance with C/C++ counterparts
  • provide a dynamic library that is a drop-in replacement, but has compiled memory-safe rust code inside
  • dramatically reduce attack surface through memory safety, improved tooling and a robust build system
  • provide a pure rust implementation to rust users that integrates with the existing ecosystem

Work plan

For per project details, see the workplan.

Please get in touch with us, if you are interested in financially supporting the development of memory-safe zlib, zstd or xz.

Blog and news

All news