react-native-quick-crypto
Version:
A fast implementation of Node's `crypto` module written in C/C++ JSI
1,050 lines (892 loc) • 138 kB
Markdown
# simdutf: Text processing at billions of characters per second
<!-- if(github) { (this is used by the github pages export, don't modify!) -->
[](https://github.com/simdutf/simdutf/actions/workflows/alpine.yml) [](https://github.com/simdutf/simdutf/actions/workflows/msys2-clang.yml) [](https://github.com/simdutf/simdutf/actions/workflows/ubuntu22-cxx20.yml)
<img src="doc/logo.svg" width="25%" style="float: right">
- [Real-World Usage](#real-world-usage)
- [How fast is it?](#how-fast-is-it)
- [Requirements](#requirements)
- [Usage (Usage)](#usage-usage)
- [Usage (CMake)](#usage-cmake)
- [Single-header version](#single-header-version)
- [Single-header version with limited features](#single-header-version-with-limited-features)
- [Packages](#packages)
- [Example](#example)
- [API](#api)
- [Base64](#base64)
- [Find](#find)
- [C++20 and std::span usage in simdutf](#c20-and-stdspan-usage-in-simdutf)
- [C++23 and constexpr support](#c23-and-constexpr-support)
- [The sutf command-line tool](#the-sutf-command-line-tool)
- [Manual implementation selection](#manual-implementation-selection)
- [Thread safety](#thread-safety)
- [References](#references)
- [License](#license)
<!-- } (this is used by the github pages export, don't modify!) -->
<!-- base64 KioqUGxlYXNlIHZpc2l0IGh0dHBzOi8vZ2l0aHViLmNvbS9zaW1kdXRmL3NpbWR1dGYgZm9yIHNvdXJjZSBjb2RlIGFuZCBpc3N1ZSB0cmFja2luZyEqKioK (this is used by the github pages export, don't modify!) -->
Most modern software relies on the [Unicode standard](https://en.wikipedia.org/wiki/Unicode).
In memory, Unicode strings are represented using either
UTF-8 or UTF-16. The UTF-8 format is the de facto standard on the web (JSON, HTML, etc.) and it has been adopted as the default in many popular
programming languages (Go, Zig, Rust, Swift, etc.). The UTF-16 format is standard in Java, C# and in many Windows technologies.
Not all sequences of bytes are valid Unicode strings. It is unsafe to use Unicode strings in UTF-8 and UTF-16LE without first validating them. Furthermore, we often need to convert strings from one encoding to another, by a process called [transcoding](https://en.wikipedia.org/wiki/Transcoding). For security purposes, such transcoding should be validating: it should refuse to transcode incorrect strings.
This library provide fast Unicode functions such as
- ASCII, UTF-8, UTF-16LE/BE and UTF-32 validation, with and without error identification,
- Latin1 to UTF-8 transcoding,
- Latin1 to UTF-16LE/BE transcoding
- Latin1 to UTF-32 transcoding
- UTF-8 to Latin1 transcoding, with or without validation, with and without error identification,
- UTF-8 to UTF-16LE/BE transcoding, with or without validation, with and without error identification,
- UTF-8 to UTF-32 transcoding, with or without validation, with and without error identification,
- UTF-16LE/BE to Latin1 transcoding, with or without validation, with and without error identification,
- UTF-16LE/BE to UTF-8 transcoding, with or without validation, with and without error identification,
- UTF-32 to Latin1 transcoding, with or without validation, with and without error identification,
- UTF-32 to UTF-8 transcoding, with or without validation, with and without error identification,
- UTF-32 to UTF-16LE/BE transcoding, with or without validation, with and without error identification,
- UTF-16LE/BE to UTF-32 transcoding, with or without validation, with and without error identification,
- From an UTF-8 string, compute the size of the Latin1 equivalent string,
- From an UTF-8 string, compute the size of the UTF-16 equivalent string,
- From an UTF-8 string, compute the size of the UTF-32 equivalent string (equivalent to UTF-8 character counting),
- From an UTF-16LE/BE string, compute the size of the Latin1 equivalent string,
- From an UTF-16LE/BE string, compute the size of the UTF-8 equivalent string,
- From an UTF-32 string, compute the size of the UTF-8 or UTF-16LE equivalent string,
- From an UTF-16LE/BE string, compute the size of the UTF-32 equivalent string (equivalent to UTF-16 character counting),
- UTF-8 and UTF-16LE/BE character counting,
- UTF-16 endianness change (UTF16-LE/BE to UTF-16-BE/LE),
- [WHATWG forgiving-base64](https://infra.spec.whatwg.org/#forgiving-base64-decode) (with or without URL encoding) to binary,
- Binary to base64 (with or without URL encoding).
The functions are accelerated using SIMD instructions (e.g., ARM NEON, SSE, AVX, AVX-512, RISC-V Vector Extension, LoongSon, POWER, etc.). When your strings contain hundreds of characters, we can often transcode them at speeds exceeding a billion characters per second. You should expect high speeds not only with English strings (ASCII) but also Chinese, Japanese, Arabic, and so forth. We handle the full character range (including, for example, emojis).
The library compiles down to a small library of a few hundred kilobytes. Our functions are exception-free and non allocating. We have extensive tests and extensive benchmarks.
We have exhaustive tests, including an elaborate fuzzing setup. The library has been used in production systems for years.
If using C++23 or newer, there is experimental support for using the library at compile time (constexpr).
## Real-World Usage
The simdutf library is used by:
- [Node.js](https://nodejs.org/en/) (19.4.0 or better, 20.0 or better, 18.15 or better), a standard JavaScript runtime environment,
- [Bun](https://bun.sh), a fast JavaScript runtime environment,
- [WebKit](https://github.com/WebKit/WebKit/pull/9990), the Web engine behind the Safari browser (iOS, macOS),
- [Chromium](https://chromium-review.googlesource.com/c/chromium/src/+/6054817), the Web engine behind the Google Chrome, Microsoft Edge and Brave,
- [StarRocks](https://www.starrocks.io), an Open-Source, High-Performance Analytical Database,
- [Oracle GraalVM JavaScript](https://github.com/oracle/graaljs), a JavaScript implementation by Oracle,
- [Couchbase](https://www.couchbase.com), a popular database system,
- [Ladybird](https://ladybird.org), an independent Web browser,
- [Cloudflare workerd](https://github.com/cloudflare/workerd), a JavaScript/Wasm Runtime,
- [haskell/text](https://github.com/haskell/text), a library for fast operations over Unicode text,
- [klogg](https://github.com/variar/klogg), a Really fast log explorer,
- [Pixie](https://github.com/pixie-io/pixie), observability tool for Kubernetes applications,
- [vte](https://gitlab.gnome.org/GNOME/vte) (0.81.0 or better), a virtual terminal widget for GTK applications.
## How fast is it?
The adoption of the simdutf library by the popular Node.js JavaScript runtime lead to a significant
performance gain:
> Decoding and Encoding becomes considerably faster than in Node.js 18. With the addition of simdutf for UTF-8 parsing the observed benchmark, results improved by 364% (an extremely impressive leap) when decoding in comparison to Node.js 16. ([State of Node.js Performance 2023](https://blog.rafaelgss.dev/state-of-nodejs-performance-2023))
<img src="doc/node2023.png" width="70%" />
Over a wide range of realistic data sources, the simdutf library transcodes a billion characters per second or more. Our approach can be 3 to 10 times faster than the popular ICU library on difficult (non-ASCII) strings. We can be 20x faster than ICU when processing easy strings (ASCII). Our good results apply to both recent x64 and ARM processors.
To illustrate, we present a benchmark result with values are in billions of characters processed by second. Consider the following figures.
<img src="doc/utf8utf16.png" width="70%" />
<img src="doc/utf16utf8.png" width="70%" />
If your system supports AVX-512, the simdutf library can provide very high performance. We get the following speed results on an Ice Lake Intel processor (both AVX2 and AVX-512) are simdutf kernels:
<img src="doc/avx512.png" width="70%" />
Datasets: https://github.com/lemire/unicode_lipsum
Please refer to our benchmarking tool for a proper interpretation of the numbers. Our results are reproducible.
## Requirements
- C++11 compatible compiler. We support LLVM clang, GCC, Visual Studio. (Our tests and benchmark tools requires C++17.) Be aware that GCC under Windows is buggy and thus unsupported.
- For high speed, you should have a recent 64-bit system (e.g., ARM, x64, RISC-V with vector extensions, Loongson, POWER). On Loongson processors, LASX runtime dispatching is only enabled on GCC 15+, not on LLVM or older versions of GCC.
- If you rely on CMake, you should use a recent CMake (at least 3.15); otherwise you may use the [single header version](#single-header-version). The library is also available from [Microsoft's vcpkg](https://github.com/simdutf/simdutf-vcpkg), from [conan](https://conan.io/center/recipes/simdutf), from [FreeBSD's port](https://cgit.freebsd.org/ports/tree/converters/simdutf), from [brew](https://formulae.brew.sh/formula/simdutf), and many other systems.
- AVX-512 support require a processor with AVX512-VBMI2 (Ice Lake or better, AMD Zen 4 or better) and a recent compiler (GCC 8 or better, Visual Studio 2022 or better, LLVM clang 6 or better). You need a correspondingly recent assembler such as gas (2.30+) or nasm (2.14+): recent compilers usually come with recent assemblers. If you mix a recent compiler with an incompatible/old assembler (e.g., when using a recent compiler with an old Linux distribution), you may get errors at build time because the compiler produces instructions that the assembler does not recognize: you should update your assembler to match your compiler (e.g., upgrade binutils to version 2.30 or better under Linux) or use an older compiler matching the capabilities of your assembler.
- To benefit from RISC-V Vector Extensions on RISC-V systems, you should compile specifically for the desired architecture. E.g., add `-march=rv64gcv` as a compiler flag when using a version of GCC or LLVM which supports these extensions (such as GCC 14 or better). The command `CXXFLAGS=-march=rv64gcv cmake -B build` may suffice.
- We recommend that Visual Studio users compile with LLVM (ClangCL). Using LLVM as a front-end inside Visual Studio provides faster release builds and better runtime performance.
## Usage (Usage)
We made a video to help you get started with the library.
[](https://www.youtube.com/watch?v=H9NZtb7ykYs)<br />
### Quick Start
Linux or macOS users might follow the following instructions if they have a recent C++ compiler installed and the standard utilities (`wget`, `unzip`, etc.)
1. Pull the library in a directory
```
wget https://github.com/simdutf/simdutf/releases/download/v8.2.0/singleheader.zip
unzip singleheader.zip
```
You can replace `wget` by `curl -OL https://...` if you prefer.
2. Compile
```shell
c++ -std=c++17 -o amalgamation_demo amalgamation_demo.cpp
```
3. `./amalgamation_demo`
```
valid UTF-8
wrote 4 UTF-16LE words.
valid UTF-16LE
wrote 4 UTF-8 words.
1234
perfect round trip
```
*We strongly discourage working from our main git branch. You should never use our main branch
in production. [Use our releases](https://github.com/simdutf/simdutf/releases/). They are tagged as `vX.Y.Z`.*
### Usage (CMake)
```shell
cmake -B build
cmake --build build
cd build
ctest .
```
Visual Studio users must specify whether they want to build the Release or Debug version.
To use the library as a CMake dependency in your project, please see `tests/installation_tests/from_fetch` for
an example.
You may also use a package manager. E.g., [we have a complete example using vcpkg](https://github.com/simdutf/simdutf-vcpkg).
## Single-header version
You can create a single-header version of the library where
all of the code is put into two files (`simdutf.h` and `simdutf.cpp`).
We publish a zip archive containing these files, e.g., see
https://github.com/simdutf/simdutf/releases/download/v8.2.0/singleheader.zip
You may generate it on your own using a Python script.
```shell
python3 ./singleheader/amalgamate.py
```
We require Python 3 or better.
Under Linux and macOS, you may test it as follows:
```shell
cd singleheader
c++ -o amalgamation_demo amalgamation_demo.cpp -std=c++17
./amalgamation_demo
```
### Single-header version with limited features
When creating a single-header version, it is possible to limit which
features are enabled. Then the API of library is limited too and the
amalgamated sources do not include code related to disabled features.
The script `singleheader/amalgamate.py` accepts the following parameters:
* `--with-utf8` - procedures related only to UTF-8 encoding (like string validation);
* `--with-utf16` - likewise: only UTF-16 encoding;
* `--with-utf32` - likewise: only UTF-32 encoding;
* `--with-ascii` - procedures related to ASCII encoding;
* `--with-latin1` - convert between selected UTF encodings and Latin1;
* `--with-base64` - procedures related to Base64 encoding, includes 'find';
* `--with-detect-enc` - enable detect encoding.
If we need conversion between different encodings, like UTF-8 and UTF-32, then
these two features have to be enabled.
The amalgamated sources set to 1 the following preprocessor defines:
* `SIMDUTF_FEATURE_UTF8`,
* `SIMDUTF_FEATURE_UTF16`,
* `SIMDUTF_FEATURE_UTF32`,
* `SIMDUTF_FEATURE_ASCII`,
* `SIMDUTF_FEATURE_LATIN1`,
* `SIMDUTF_FEATURE_BASE64`,
* `SIMDUTF_FEATURE_DETECT_ENCODING`.
Thus, when it is needed to make sure the correct set of features are
enabled, we may test it using preprocessor:
```cpp
#if SIMDUTF_FEATURE_UTF16 || SIMDUTF_FEATURE_UTF32
#error "Please amalagamate simdutf without UTF-16 and UTF-32"
#endif
```
### Packages
------
[](https://repology.org/project/simdutf/versions)
### Example
Using the single-header version, you could compile the following program.
```cpp
#include <iostream>
#include <memory>
#include "simdutf.cpp"
#include "simdutf.h"
int main(int argc, char *argv[]) {
const char *source = "1234";
// 4 == strlen(source)
bool validutf8 = simdutf::validate_utf8(source, 4);
if (validutf8) {
std::cout << "valid UTF-8" << std::endl;
} else {
std::cerr << "invalid UTF-8" << std::endl;
return EXIT_FAILURE;
}
// We need a buffer of size where to write the UTF-16LE code units.
size_t expected_utf16words = simdutf::utf16_length_from_utf8(source, 4);
std::unique_ptr<char16_t[]> utf16_output{new char16_t[expected_utf16words]};
// convert to UTF-16LE
size_t utf16words =
simdutf::convert_utf8_to_utf16le(source, 4, utf16_output.get());
std::cout << "wrote " << utf16words << " UTF-16LE code units." << std::endl;
// It wrote utf16words * sizeof(char16_t) bytes.
bool validutf16 = simdutf::validate_utf16le(utf16_output.get(), utf16words);
if (validutf16) {
std::cout << "valid UTF-16LE" << std::endl;
} else {
std::cerr << "invalid UTF-16LE" << std::endl;
return EXIT_FAILURE;
}
// convert it back:
// We need a buffer of size where to write the UTF-8 code units.
size_t expected_utf8words =
simdutf::utf8_length_from_utf16le(utf16_output.get(), utf16words);
std::unique_ptr<char[]> utf8_output{new char[expected_utf8words]};
// convert to UTF-8
size_t utf8words = simdutf::convert_utf16le_to_utf8(
utf16_output.get(), utf16words, utf8_output.get());
std::cout << "wrote " << utf8words << " UTF-8 code units." << std::endl;
std::string final_string(utf8_output.get(), utf8words);
std::cout << final_string << std::endl;
if (final_string != source) {
std::cerr << "bad conversion" << std::endl;
return EXIT_FAILURE;
} else {
std::cerr << "perfect round trip" << std::endl;
}
return EXIT_SUCCESS;
}
```
## API
Our API is made of a few non-allocating functions. They typically take a pointer and a length as a parameter,
and they sometimes take a pointer to an output buffer. Users are responsible for memory allocation.
We use three types of data pointer types:
- `char*` for UTF-8 or indeterminate Unicode formats,
- `char16_t*` for UTF-16 (both UTF-16LE and UTF-16BE),
- `char32_t*` for UTF-32. UTF-32 is primarily used for internal use, not data interchange. Thus, unless otherwise stated, `char32_t` refers to the native type and is typically UTF-32LE since virtually all systems are little-endian today.
In generic terms, we refer to `char`, `char16_t`, and `char32_t` as *code units*. A *character* may use several *code units*: between 1 and 4 code units in UTF-8, and between
1 and 2 code units in UTF-16LE and UTF-16BE.
Our functions and declarations are all in the `simdutf` namespace. Thus you should prefix our functions
and types with `simdutf::` as required.
If using C++20, all functions which take a pointer and a size (which is almost all of them)
also have a span overload. Here is an example:
```cpp
std::vector<char> data{1, 2, 3, 4, 5};
// C++11 API
auto cpp11 = simdutf::autodetect_encoding(data.data(), data.size());
// C++20 API
auto cpp20 = simdutf::autodetect_encoding(data);
```
The span overloads use std::span for UTF-16 and UTF-32. For latin1, UTF-8,
"binary" (used by the base64 functions) anything that has a `.size()` and
`.data()` that returns a pointer to a byte-like type will be accepted as a
span. This makes it possible to directly pass std::string, std::string_view,
std::vector, std::array and std::span to the functions. The reason for allowing
all byte-like types in the api (as opposed to only `std::span<char>`) is to
make it easy to interface with whatever data the user may have, without having
to resort to casting.
We have basic functions to detect the type of an input. They return an integer defined by
the following `enum`.
```cpp
enum encoding_type {
UTF8 = 1, // BOM 0xef 0xbb 0xbf
UTF16_LE = 2, // BOM 0xff 0xfe
UTF16_BE = 4, // BOM 0xfe 0xff
UTF32_LE = 8, // BOM 0xff 0xfe 0x00 0x00
UTF32_BE = 16, // BOM 0x00 0x00 0xfe 0xff
unspecified = 0
};
```
```cpp
/**
* Autodetect the encoding of the input, a single encoding is recommended.
* E.g., the function might return simdutf::encoding_type::UTF8,
* simdutf::encoding_type::UTF16_LE, simdutf::encoding_type::UTF16_BE, or
* simdutf::encoding_type::UTF32_LE.
*
* @param input the string to analyze.
* @param length the length of the string in bytes.
* @return the detected encoding type
*/
simdutf_warn_unused simdutf::encoding_type autodetect_encoding(const char *input, size_t length) noexcept;
/**
* Autodetect the possible encodings of the input in one pass.
* E.g., if the input might be UTF-16LE or UTF-8, this function returns
* the value (simdutf::encoding_type::UTF8 | simdutf::encoding_type::UTF16_LE).
*
* @param input the string to analyze.
* @param length the length of the string in bytes.
* @return the detected encoding type
*/
simdutf_warn_unused int detect_encodings(const char *input, size_t length) noexcept;
```
For validation and transcoding, we also provide functions that will stop on error and return a result struct which is a pair of two fields:
```cpp
struct result {
error_code error; // see `struct error_code`.
size_t count; // In case of error, indicates the position of the error in the input in code units.
// In case of success, indicates the number of code units validated/written.
};
```
On error, the `error` field indicates the type of error encountered and the `count` field indicates the position of the error in the input in code units or the number of characters validated/written.
We report six types of errors related to Latin1, UTF-8, UTF-16 and UTF-32 encodings:
```cpp
enum error_code {
SUCCESS = 0,
HEADER_BITS, // Any byte must have fewer than 5 header bits.
TOO_SHORT, // The leading byte must be followed by N-1 continuation bytes,
// where N is the UTF-8 character length This is also the error
// when the input is truncated.
TOO_LONG, // We either have too many consecutive continuation bytes or the
// string starts with a continuation byte.
OVERLONG, // The decoded character must be above U+7F for two-byte characters,
// U+7FF for three-byte characters, and U+FFFF for four-byte
// characters.
TOO_LARGE, // The decoded character must be less than or equal to
// U+10FFFF,less than or equal than U+7F for ASCII OR less than
// equal than U+FF for Latin1
SURROGATE, // The decoded character must be not be in U+D800...DFFF (UTF-8 or
// UTF-32)
// OR
// a high surrogate must be followed by a low surrogate
// and a low surrogate must be preceded by a high surrogate
// (UTF-16)
// OR
// there must be no surrogate at all and one is
// found (Latin1 functions)
// OR
// *specifically* for the function
// utf8_length_from_utf16_with_replacement, a surrogate (whether
// in error or not) has been found (I.e., whether we are in the
// Basic Multilingual Plane or not).
INVALID_BASE64_CHARACTER, // Found a character that cannot be part of a valid
// base64 string. This may include a misplaced padding character ('=').
BASE64_INPUT_REMAINDER, // The base64 input terminates with a single
// character, excluding padding (=). It is also used
// in strict mode when padding is not adequate.
BASE64_EXTRA_BITS, // The base64 input terminates with non-zero
// padding bits.
OUTPUT_BUFFER_TOO_SMALL, // The provided buffer is too small.
OTHER // Not related to validation/transcoding.
};
```
On success, the `error` field is set to `SUCCESS` and the `position` field indicates either the number of code units validated for validation functions or the number of written
code units in the output format for transcoding functions. In ASCII, Latin1 and UTF-8, code units occupy 8 bits (they are bytes); in UTF-16LE and UTF-16BE, code units occupy 16 bits; in UTF-32, code units occupy 32 bits.
Generally speaking, functions that report errors always stop soon after an error is
encountered and might therefore be faster on inputs where an error occurs early in the input.
The functions that return a boolean indicating whether or not an error has been encountered
are meant to be used in an *optimistic setting*---when we expect that inputs will almost always
be correct.
You may use functions that report an error to indicate where the problem happens during, as follows:
```cpp
std::string bad_ascii = "\x20\x20\x20\x20\x20\xff\x20\x20\x20";
simdutf::result res = simdtuf::validate_ascii_with_errors(bad_ascii.data(), bad_ascii.size());
if(res.error != simdutf::error_code::SUCCESS) {
std::cerr << "error at index " << res.count << std::endl;
}
```
Or as follows:
```cpp
std::string bad_utf8 = "\xc3\xa9\xc3\xa9\x20\xff\xc3\xa9";
simdutf::result res = simdtuf::validate_utf8_with_errors(bad_utf8.data(), bad_utf8.size());
if(res.error != simdutf::error_code::SUCCESS) {
std::cerr << "error at index " << res.count << std::endl;
}
res = simdtuf::validate_utf8_with_errors(bad_utf8.data(), res.count);
// will be successful in this case
if(res.error == simdutf::error_code::SUCCESS) {
std::cerr << "we have " << res.count << "valid bytes" << std::endl;
}
```
We have fast validation functions.
```cpp
/**
* Validate the ASCII string.
*
* @param buf the ASCII string to validate.
* @param len the length of the string in bytes.
* @return true if and only if the string is valid ASCII.
*/
simdutf_warn_unused bool validate_ascii(const char *buf, size_t len) noexcept;
/**
* Validate the ASCII string and stop on error.
*
* @param buf the ASCII string to validate.
* @param len the length of the string in bytes.
* @return a result pair struct (of type simdutf::result containing the two fields error and count) with an error code and either position of the error (in the input in code units) if any, or the number of code units validated if successful.
*/
simdutf_warn_unused result validate_ascii_with_errors(const char *buf, size_t len) noexcept;
/**
* Validate the ASCII string as a UTF-16 sequence.
* An UTF-16 sequence is considered an ASCII sequence
* if it could be converted to an ASCII string losslessly.
*
* @param buf the UTF-16 string to validate.
* @param len the length of the string in bytes.
* @return true if and only if the string is valid ASCII.
*/
simdutf_warn_unused bool validate_utf16_as_ascii(const char16_t *buf,
size_t len) noexcept;
/**
* Validate the ASCII string as a UTF-16BE sequence.
* An UTF-16 sequence is considered an ASCII sequence
* if it could be converted to an ASCII string losslessly.
*
* @param buf the UTF-16BE string to validate.
* @param len the length of the string in bytes.
* @return true if and only if the string is valid ASCII.
*/
simdutf_warn_unused bool validate_utf16be_as_ascii(const char16_t *buf,
size_t len) noexcept;
/**
* Validate the ASCII string as a UTF-16LE sequence.
* An UTF-16 sequence is considered an ASCII sequence
* if it could be converted to an ASCII string losslessly.
*
* @param buf the UTF-16LE string to validate.
* @param len the length of the string in bytes.
* @return true if and only if the string is valid ASCII.
*/
simdutf_warn_unused bool validate_utf16le_as_ascii(const char16_t *buf,
size_t len) noexcept;
/**
* Validate the UTF-8 string. This function may be best when you expect
* the input to be almost always valid. Otherwise, consider using
* validate_utf8_with_errors.
*
* @param buf the UTF-8 string to validate.
* @param len the length of the string in bytes.
* @return true if and only if the string is valid UTF-8.
*/
simdutf_warn_unused bool validate_utf8(const char *buf, size_t len) noexcept;
/**
* Validate the UTF-8 string and stop on error. It might be faster than
* validate_utf8 when an error is expected to occur early.
*
* @param buf the UTF-8 string to validate.
* @param len the length of the string in bytes.
* @return a result pair struct (of type simdutf::result containing the two fields error and count) with an error code and either position of the error (in the input in code units) if any, or the number of code units validated if successful.
*/
simdutf_warn_unused result validate_utf8_with_errors(const char *buf, size_t len) noexcept;
/**
* Using native endianness; Validate the UTF-16 string.
* This function may be best when you expect the input to be almost always valid.
* Otherwise, consider using validate_utf16_with_errors.
*
* This function is not BOM-aware.
*
* @param buf the UTF-16 string to validate.
* @param len the length of the string in number of 2-byte code units (char16_t).
* @return true if and only if the string is valid UTF-16.
*/
simdutf_warn_unused bool validate_utf16(const char16_t *buf, size_t len) noexcept;
/**
* Validate the UTF-16LE string. This function may be best when you expect
* the input to be almost always valid. Otherwise, consider using
* validate_utf16le_with_errors.
*
* This function is not BOM-aware.
*
* @param buf the UTF-16LE string to validate.
* @param len the length of the string in number of 2-byte code units (char16_t).
* @return true if and only if the string is valid UTF-16LE.
*/
simdutf_warn_unused bool validate_utf16le(const char16_t *buf, size_t len) noexcept;
/**
* Validate the UTF-16BE string. This function may be best when you expect
* the input to be almost always valid. Otherwise, consider using
* validate_utf16be_with_errors.
*
* This function is not BOM-aware.
*
* @param buf the UTF-16BE string to validate.
* @param len the length of the string in number of 2-byte code units (char16_t).
* @return true if and only if the string is valid UTF-16BE.
*/
simdutf_warn_unused bool validate_utf16be(const char16_t *buf, size_t len) noexcept;
/**
* Using native endianness; Validate the UTF-16 string and stop on error.
* It might be faster than validate_utf16 when an error is expected to occur early.
*
* This function is not BOM-aware.
*
* @param buf the UTF-16 string to validate.
* @param len the length of the string in number of 2-byte code units (char16_t).
* @return a result pair struct (of type simdutf::result containing the two fields error and count) with an error code and either position of the error (in the input in code units) if any, or the number of code units validated if successful.
*/
simdutf_warn_unused result validate_utf16_with_errors(const char16_t *buf, size_t len) noexcept;
/**
* Validate the UTF-16LE string and stop on error. It might be faster than
* validate_utf16le when an error is expected to occur early.
*
* This function is not BOM-aware.
*
* @param buf the UTF-16LE string to validate.
* @param len the length of the string in number of 2-byte code units (char16_t).
* @return a result pair struct (of type simdutf::result containing the two fields error and count) with an error code and either position of the error (in the input in code units) if any, or the number of code units validated if successful.
*/
simdutf_warn_unused result validate_utf16le_with_errors(const char16_t *buf, size_t len) noexcept;
/**
* Validate the UTF-16BE string and stop on error. It might be faster than
* validate_utf16be when an error is expected to occur early.
*
* This function is not BOM-aware.
*
* @param buf the UTF-16BE string to validate.
* @param len the length of the string in number of 2-byte code units (char16_t).
* @return a result pair struct (of type simdutf::result containing the two fields error and count) with an error code and either position of the error (in the input in code units) if any, or the number of code units validated if successful.
*/
simdutf_warn_unused result validate_utf16be_with_errors(const char16_t *buf, size_t len) noexcept;
/**
* Validate the UTF-32 string.
*
* This function is not BOM-aware.
*
* @param buf the UTF-32 string to validate.
* @param len the length of the string in number of 4-byte code units (char32_t).
* @return true if and only if the string is valid UTF-32.
*/
simdutf_warn_unused bool validate_utf32(const char32_t *buf, size_t len) noexcept;
/**
* Validate the UTF-32 string and stop on error.
*
* This function is not BOM-aware.
*
* @param buf the UTF-32 string to validate.
* @param len the length of the string in number of 4-byte code units (char32_t).
* @return a result pair struct (of type simdutf::result containing the two fields error and count) with an error code and either position of the error (in the input in code units) if any, or the number of code units validated if successful.
*/
simdutf_warn_unused result validate_utf32_with_errors(const char32_t *buf, size_t len) noexcept;
```
Given a potentially invalid UTF-16 input, you may want to make it correct, by using
a replacement character whenever needed. We have fast functions for this purpose
(`to_well_formed_utf16`, `to_well_formed_utf16le`, and `to_well_formed_utf16be`).
They can either copy the string while fixing it, or they can be used to fix
a string in-place.
```cpp
/**
* Fixes an ill-formed UTF-16LE string by replacing mismatched surrogates with
* the Unicode replacement character U+FFFD. If input and output points to
* different memory areas, the procedure copies string, and it's expected that
* output memory is at least as big as the input. It's also possible to set
* input equal output, that makes replacements an in-place operation.
*
* @param input the UTF-16LE string to correct.
* @param len the length of the string in number of 2-byte code units
* (char16_t).
* @param output the output buffer.
*/
void to_well_formed_utf16le(const char16_t *input, size_t len,
char16_t *output) noexcept;
/**
* Fixes an ill-formed UTF-16BE string by replacing mismatched surrogates with
* the Unicode replacement character U+FFFD. If input and output points to
* different memory areas, the procedure copies string, and it's expected that
* output memory is at least as big as the input. It's also possible to set
* input equal output, that makes replacements an in-place operation.
*
* @param input the UTF-16BE string to correct.
* @param len the length of the string in number of 2-byte code units
* (char16_t).
* @param output the output buffer.
*/
void to_well_formed_utf16be(const char16_t *input, size_t len,
char16_t *output) noexcept;
/**
* Fixes an ill-formed UTF-16 string by replacing mismatched surrogates with the
* Unicode replacement character U+FFFD. If input and output points to different
* memory areas, the procedure copies string, and it's expected that output
* memory is at least as big as the input. It's also possible to set input equal
* output, that makes replacements an in-place operation.
*
* @param input the UTF-16 string to correct.
* @param len the length of the string in number of 2-byte code units
* (char16_t).
* @param output the output buffer.
*/
void to_well_formed_utf16(const char16_t *input, size_t len,
char16_t *output) noexcept;
```
Given a valid UTF-8 or UTF-16 input, you may count the number Unicode characters using
fast functions. For UTF-32, there is no need for a function given that each character
requires a flat 4 bytes. Likewise for Latin1: one byte will always equal one character.
```cpp
/**
* Count the number of code points (characters) in the string assuming that
* it is valid.
*
* This function assumes that the input string is valid UTF-16 (native endianness).
* It is acceptable to pass invalid UTF-16 strings but in such cases
* the result is implementation defined.
*
* This function is not BOM-aware.
*
* @param input the UTF-16 string to process
* @param length the length of the string in 2-byte code units (char16_t)
* @return number of code points
*/
simdutf_warn_unused size_t count_utf16(const char16_t * input, size_t length) noexcept;
/**
* Count the number of code points (characters) in the string assuming that
* it is valid.
*
* This function assumes that the input string is valid UTF-16LE.
* It is acceptable to pass invalid UTF-16 strings but in such cases
* the result is implementation defined.
*
* This function is not BOM-aware.
*
* @param input the UTF-16LE string to process
* @param length the length of the string in 2-byte code units (char16_t)
* @return number of code points
*/
simdutf_warn_unused size_t count_utf16le(const char16_t * input, size_t length) noexcept;
/**
* Count the number of code points (characters) in the string assuming that
* it is valid.
*
* This function assumes that the input string is valid UTF-16BE.
* It is acceptable to pass invalid UTF-16 strings but in such cases
* the result is implementation defined.
*
* This function is not BOM-aware.
*
* @param input the UTF-16BE string to process
* @param length the length of the string in 2-byte code units (char16_t)
* @return number of code points
*/
simdutf_warn_unused size_t count_utf16be(const char16_t * input, size_t length) noexcept;
/**
* Count the number of code points (characters) in the string assuming that
* it is valid.
*
* This function assumes that the input string is valid UTF-8.
* It is acceptable to pass invalid UTF-8 strings but in such cases
* the result is implementation defined.
*
* @param input the UTF-8 string to process
* @param length the length of the string in bytes
* @return number of code points
*/
simdutf_warn_unused size_t count_utf8(const char * input, size_t length) noexcept;
```
Prior to transcoding an input, you need to allocate enough memory to receive the result.
We have fast function that scan the input and compute the size of the output. These functions
are fast and non-validating.
```cpp
/**
* Return the number of bytes that this Latin1 string would require in UTF-8 format.
*
* @param input the Latin1 string to convert
* @param length the length of the string bytes
* @return the number of bytes required to encode the Latin1 string as UTF-8
*/
simdutf_warn_unused size_t utf8_length_from_latin1(const char * input, size_t length) noexcept;
/**
* Compute the number of bytes that this UTF-8 string would require in Latin1 format.
*
* This function does not validate the input. It is acceptable to pass invalid UTF-8 strings but in such cases
* the result is implementation defined.
*
* This function is not BOM-aware.
*
* @param input the UTF-8 string to convert
* @param length the length of the string in byte
* @return the number of bytes required to encode the UTF-8 string as Latin1
*/
simdutf_warn_unused size_t latin1_length_from_utf8(const char * input, size_t length) noexcept;
/**
* Compute the number of bytes that this UTF-16 string would require in Latin1 format.
*
* @param length the length of the string in Latin1 code units (char)
* @return the length of the string in Latin1 code units (char) required to encode the UTF-16 string as Latin1
*/
simdutf_warn_unused size_t latin1_length_from_utf16(size_t length) noexcept;
/*
* Compute the number of bytes that this UTF-16LE/BE string would require in Latin1 format.
*
* This function does not validate the input. It is acceptable to pass invalid UTF-16 strings but in such cases
* the result is implementation defined.
*
* This function is not BOM-aware.
*
* @param length the length of the string in 2-byte code units (char16_t)
* @return the number of bytes required to encode the UTF-16LE string as Latin1
*/
simdutf_warn_unused size_t latin1_length_from_utf16(size_t length) noexcept;
/**
* Compute the number of bytes that this UTF-32 string would require in Latin1 format.
*
* This function does not validate the input. It is acceptable to pass invalid UTF-32 strings but in such cases
* the result is implementation defined.
*
* This function is not BOM-aware.
*
* @param length the length of the string in 4-byte code units (char32_t)
* @return the number of bytes required to encode the UTF-32 string as Latin1
*/
simdutf_warn_unused size_t latin1_length_from_utf32(size_t length) noexcept;
/**
* Compute the number of 2-byte code units that this UTF-8 string would require in UTF-16 format.
*
* This function does not validate the input. It is acceptable to pass invalid UTF-8 strings but in such cases
* the result is implementation defined.
*
* @param input the UTF-8 string to process
* @param length the length of the string in bytes
* @return the number of char16_t code units required to encode the UTF-8 string as UTF-16
*/
simdutf_warn_unused size_t utf16_length_from_utf8(const char * input, size_t length) noexcept;
/**
* Compute the number of 4-byte code units that this UTF-8 string would require in UTF-32 format.
*
* This function is equivalent to count_utf8
*
* This function does not validate the input. It is acceptable to pass invalid UTF-8 strings but in such cases
* the result is implementation defined.
*
* This function is not BOM-aware.
*
* @param input the UTF-8 string to process
* @param length the length of the string in bytes
* @return the number of char32_t code units required to encode the UTF-8 string as UTF-32
*/
simdutf_warn_unused size_t utf32_length_from_utf8(const char * input, size_t length) noexcept;
/**
* Using native endianness; Compute the number of bytes that this UTF-16
* string would require in UTF-8 format.
*
* This function does not validate the input. It is acceptable to pass invalid UTF-16 strings but in such cases
* the result is implementation defined.
*
* @param input the UTF-16 string to convert
* @param length the length of the string in 2-byte code units (char16_t)
* @return the number of bytes required to encode the UTF-16LE string as UTF-8
*/
simdutf_warn_unused size_t utf8_length_from_utf16(const char16_t * input, size_t length) noexcept;
/**
* Using native endianness; compute the number of bytes that this UTF-16
* string would require in UTF-8 format even when the UTF-16 content contains mismatched
* surrogates that have to be replaced by the replacement character (0xFFFD).
*
* @param input the UTF-16 string to convert
* @param length the length of the string in 2-byte code units (char16_t)
* @return the number of bytes required to encode the UTF-16 string as UTF-8
* @return a result pair struct (of type simdutf::result containing the two fields error and count)
* where the count is the number of bytes required to encode the UTF-16 string as UTF-8, and the
* error code is either SUCCESS or SURROGATE. The count is correct regardless of the error field.
* When SURROGATE is returned, it does not indicate an error in the case of this function:
* it indicates that at least one surrogate has been encountered: the surrogates may be matched
* or not (thus this function does not validate). If the returned error code is SUCCESS,
* then the input contains no surrogate, is in the Basic Multilingual Plane, and is necessarily valid.
*/
simdutf_warn_unused result utf8_length_from_utf16_with_replacement(const char16_t *input,
size_t length) noexcept;
/**
* Compute the number of bytes that this UTF-16LE string would require in UTF-8 format.
*
* This function does not validate the input. It is acceptable to pass invalid UTF-16 strings but in such cases
* the result is implementation defined.
*
* @param input the UTF-16LE string to convert
* @param length the length of the string in 2-byte code units (char16_t)
* @return the number of bytes required to encode the UTF-16LE string as UTF-8
*/
simdutf_warn_unused size_t utf8_length_from_utf16le(const char16_t * input, size_t length) noexcept;
/**
* Compute the number of bytes that this UTF-16BE string would require in UTF-8 format.
*
* This function does not validate the input. It is acceptable to pass invalid UTF-16 strings but in such cases
* the result is implementation defined.
*
* @param input the UTF-16BE string to convert
* @param length the length of the string in 2-byte code units (char16_t)
* @return the number of bytes required to encode the UTF-16BE string as UTF-8
*/
simdutf_warn_unused size_t utf8_length_from_utf16be(const char16_t * input, size_t length) noexcept;
/**
* Compute the number of bytes that this UTF-16LE string would require in UTF-8
* format even when the UTF-16LE content contains mismatched surrogates
* that have to be replaced by the replacement character (0xFFFD).
*
* @param input the UTF-16LE string to convert
* @param length the length of the string in 2-byte code units (char16_t)
* @return the number of bytes required to encode the UTF-16LE string as UTF-8
* @return a result pair struct (of type simdutf::result containing the two fields error and count)
* where the count is the number of bytes required to encode the UTF-16LE string as UTF-8, and the
* error code is either SUCCESS or SURROGATE. The count is correct regardless of the error field.
* When SURROGATE is returned, it does not indicate an error in the case of this function:
* it indicates that at least one surrogate has been encountered: the surrogates may be matched
* or not (thus this function does not validate). If the returned error code is SUCCESS,
* then the input contains no surrogate, is in the Basic Multilingual Plane, and is necessarily valid.
*/
simdutf_warn_unused result utf8_length_from_utf16le_with_replacement(
const char16_t *input, size_t length) noexcept;
/**
* Compute the number of bytes that this UTF-16BE string would require in UTF-8
* format even when the UTF-16BE content contains mismatched surrogates
* that have to be replaced by the replacement character (0xFFFD).
*
* @param input the UTF-16BE string to convert
* @param length the length of the string in 2-byte code units (char16_t)
* @return a result pair struct (of type simdutf::result containing the two fields error and count)
* where the count is the number of bytes required to encode the UTF-16LE string as UTF-8, and
* the error code is either SUCCESS or SURROGATE. The count is correct regardless of the error field.
* When SURROGATE is returned, it does not indicate an error in the case of this function:
* it indicates that at least one surrogate has been encountered: the surrogates may be matched
* or not (thus this function does not validate). If the returned error code is SUCCESS,
* then the input contains no surrogate, is in the Basic Multilingual Plane, and is necessarily valid.
*/
simdutf_warn_unused result utf8_length_from_utf16be_with_replacement(
const char16_t *input, size_t length) noexcept;
/**
* Compute the number of bytes that this UTF-16LE string would require in UTF-8
* format even when the UTF-16LE content contains mismatched surrogates
* that have to be replaced by the replacement character (0xFFFD).
*
* @param input the UTF-16LE string to convert
* @param length the length of the string in 2-byte code units (char16_t)
* @return a result pair struct (of type simdutf::result containing the two fields error and count) where the count is the number of bytes required to encode the UTF-16LE string as UTF-8, and the error code is either SUCCESS or SURROGATE. The count is correct regardless of the error field.
*/
simdutf_warn_unused result utf8_length_from_utf16le_with_replacement(
const char16_t *input, size_t length) noexcept;
/**
* Compute the number of bytes that this UTF-16BE string would require in UTF-8
* format even when the UTF-16BE content contains mismatched surrogates
* that have to be replaced by the replacement character (0xFFFD).
*
* @param input the UTF-16BE string to convert
* @param length the length of the string in 2-byte code units (char16_t)
* @return a result pair struct (of type simdutf::result containing the two fields error and count) where the count is the number of bytes required to encode the UTF-16LE string as UTF-8, and the error code is either SUCCESS or SURROGATE. The count is correct regardless of the error field.
*/
simdutf_warn_unused result utf8_length_from_utf16be_with_replacement(
const char16_t *input, size_t length) noexcept;
/**
* Compute the number of bytes that this UTF-32 string would require in UTF-8 format.
*
* This function does not validate the input. It is acceptable to pass invalid UTF-32 strings but in such cases
* the result is implementation defined.
*
* @param input the UTF-32 string to convert
* @param length the length of the string in 4-byte code units (char32_t)
* @return the number of bytes required to encode the UTF-32 string as UTF-8
*/
simdutf_warn_unused size_t utf8_length_from_utf32(const char32_t * input, size_t length) noexcept;
/**
* Compute the number of two-byte code units that this UTF-32 string would require in UTF-16 format.
*
* This function does not validate the input. It is acceptable to pass invalid UTF-32 strings but in such cases
* the result is implementation defined.
*
* @param input the UTF-32 string to convert
* @param length the length of the string in 4-byte code units (char32_t)
* @return the number of bytes required to encode the UTF-32 string as UTF-16
*/
simdutf_warn_unused size_t utf16_length_from_utf32(const char32_t * input, size_t length) noexcept;
/**
* Compute the number of code units that this Latin1 string would require in UTF-16 format.
*
* @param length the length of the string in Latin1 code units (char)
* @return the length of the string in 2-byte code units (char16_t) required to encode the Latin1 string as UTF-16
*/
simdutf_warn_unused size_t utf16_length_from_latin1(size_t length) noexcept;
/**
* Using native endianness; Compute the number of bytes that this UTF-16
* string would require in UTF-32 format.
*
* This function is equivalent to count_utf16.
*
* This function does not validate the input. It is acceptable to pass invalid UTF-16 strings but in such cases
* the result is implementation defined.
*
* This function is not BOM-aware.
*
* @param input the UTF-16 string to convert
* @param length the length of the string in 2-byte code units (char16_t)
* @return the number of bytes required to encode the UTF-16LE string as UTF-32
*/
simdutf_warn_unused size_t utf32_length_from_utf16(const char16_t * input, size_t length) noexcept;
/**
* Compute the number of bytes that this UTF-16LE string would require in UTF-32 format.
*
* This function is equivalent to count_utf16le.
*
* This function does not validate the input. It is acceptable to pass invalid UTF-16 strings but in such cases
* the result is implementation defined.
*
* This function is not BOM-aware.
*
* @param input the UTF-16LE string to convert
* @param length the length of the string in 2-byte code units (char16_t)