hysnappy
Version:
fast and tiny snappy compression in wasm
100 lines (67 loc) • 3.99 kB
Markdown
# HySnappy
[](https://www.npmjs.com/package/hysnappy)
[](https://www.npmjs.com/package/hysnappy)
[](https://github.com/hyparam/hysnappy/actions)
[](https://opensource.org/licenses/MIT)
[](https://www.npmjs.com/package/hysnappy?activeTab=dependencies)
**HySnappy** is a lightweight, high-performance Snappy compression/decompression library compiled to WebAssembly. It provides:
- Very fast Snappy compression/decompression suitable for web and Node.js environments.
- A minimal footprint with no external dependencies.
- Seamless integration with tools like [Hyparquet](https://github.com/hyparam/hyparquet).
The Snappy compression format, originally released by Google, is designed for high-speed and reasonable compression ratios. HySnappy leverages these strengths by providing a WebAssembly build that can be included directly in your JavaScript bundle for optimal performance.
## Usage
### Decompress Snappy Data
The `snappyUncompress` function requires arguments:
- `compressed`: a `Uint8Array` with compressed data.
- `outputLength`: the uncompressed size of the data.
The length is needed to know how much wasm memory to allocate.
For formats like parquet, this length will generally be known in advance.
To decompress a `Uint8Array` with known output length:
```javascript
const { snappyUncompress } = await import('hysnappy')
const compressed = new Uint8Array([
0x0a, 0x24, 0x68, 0x79, 0x70, 0x65, 0x72, 0x70, 0x61, 0x72, 0x61, 0x6d
])
const outputLength = 10
const output = snappyUncompress(compressed, outputLength) // hyperparam
```
### Compress Snappy Data
Use the `snappyCompress` function to compress a `Uint8Array`:
```javascript
const { snappyCompress } = await import('hysnappy')
const input = new Uint8Array([
0x68, 0x79, 0x70, 0x61, 0x72, 0x61, 0x6d
])
const compressed = snappyCompress(input)
```
## Hyparquet Integration
Hysnappy was built specifically to accelerate the [hyparquet](https://github.com/hyparam/hyparquet) parquet parsing library.
Hysnappy exports a loader function `snappyUncompressor()` which loads the WASM module once, and returns a pre-loaded version of `snappyUncompress` function.
To use hysnappy with hyparquet:
```javascript
import { parquetQuery } from 'hyparquet'
import { snappyUncompressor } from 'hysnappy'
await parquetQuery({
file,
compressors: {
SNAPPY: snappyUncompressor(),
},
})
```
Alternatively, check out [hyparquet-compressors](https://github.com/hyparam/hyparquet-compressors) which includes hysnappy decompression.
Hysnappy can also be used to accelerate compression in the [hyparquet-writer](https://github.com/hyparam/hyparquet-writer) library (read the [advanced usage section](https://github.com/hyparam/hyparquet-writer#advanced-usage) for details).
## Development
The build uses clang _without_ emscripten, in order to produce the smallest possible binary.
Run `make` to build from source. The build process consists of:
1. Compile from c to wasm using `clang`.
2. Encode wasm as base64 to `uncompress.wasm.base64` and `compress.wasm.base64`.
3. Insert base64 strings into `uncompress.js` and `compress.js` for distribution.
## WASM Loading
By keeping wasm files under 4kb, we can include it directly in the javascript files and load the WASM blob synchronously, which is faster than loading a separate `.wasm` file. [[web.dev]](https://web.dev/articles/loading-wasm)
## References
- https://en.wikipedia.org/wiki/Snappy_(compression)
- https://github.com/andikleen/snappy-c
- https://github.com/google/snappy
- https://github.com/zhipeng-jia/snappyjs
- https://web.dev/articles/loading-wasm