@flourd/toml

# @flourd/toml Better TOML parsing and stringifying all in that familiar JSON interface. Now includes a basic command line interface for converting files and STDIN input between JSON and TOML syntaxes (with support for YAML in the works!). [![Coverage Status](https://coveralls.io/repos/github/flourd/toml/badge.svg)](https://coveralls.io/github/flourd/toml) ## Getting Started <h3>Install globally to use the CLI</h3> <details open> #### ››› [Skip to: Command Line Interface Usage and Examples](#command-line-interface) ```bash yarn global add @flourd/toml ``` </details> - - - ### 1. Add locally to a project's dependencies ```bash yarn add --dev @flourd/toml ``` ### 2. Include the library via `import` or `require` #### CommonJS (`require`) ```js // CommonJS pseudo-equivalent to default imports const TOML = require('@flourd/toml') // CommonJS's attempt at named imports const { parse, stringify } = require('@flourd/toml') // and the pseudo-equivalent to aliased named imports const { parse: fromTOML, stringify: toTOML } = require('@flourd/toml') ``` <details> <summary>Some personal thoughts/comments on CommonJS vs. ECMAScript</summary> CommonJS is ***supported***, but the ES syntax is ***recommended***. CommonJS is the target this was originally written for (see: [@iarna/toml](https://github.com/iarna/toml)). It's also still the official Node.js format. However... it's 2021, this library is 5+ years old, and standards/specifications/best practices have changed quite a bit. A major problem with CommonJS, and the main reason why the above code **is not equal** to the below code, is the lack of tree-shaking support. In the above examples, we have included the ***entire @flourd/toml library's code** not once, but **3 times**. All so we could cherry-pick a couple methods. In ECMAScript, we can import individual methods and such, and when using the right bundler, like [Rollup](https://rollupjs.org) or [esbuild](https://esbuild.js.org), **only the code we require is bundled in our final production code**. This means a *huge* performance increase, and a *drastic* decrease in build-time (which can get quite expensive on large projects with frequent pushes to CI/CD pipelines). </details> #### ECMAScript (`import`) ```es6 // namespaced imports.... or default imports import * as TOML from '@flourd/toml'; import TOML from '@flourd/toml'; // named imports import { parse, stringify } from '@flourd/toml'; // aliased named imports import { parse as fromTOML, stringify as toTOML } from '@flourd/toml'; ``` ##### Direct exports (when import isn't necessary) If you aren't going to use any of the libraries methods in the current working file, it may make more sense to *directly export* the desired methods, rather than importing them only to export right after. ```es6 // export the full `toml` namespace export * as toml from '@flourd/toml'; // export into the default namespace export * as default from '@flourd/toml'; // named exports export { parse, stringify } from '@flourd/toml'; // aliased named exports export { parse as fromTOML, stringify as toTOML } from '@flourd/toml'; ``` - - - ### Usage Examples #### Programmatic Parsing and Stringifying ```js const obj = TOML.parse(`[abc] foo = 123 bar = [1,2,3]`) /* obj = {abc: {foo: 123, bar: [1,2,3]}} */ const str = TOML.stringify(obj) /* str = [abc] foo = 123 bar = [ 1, 2, 3 ] */ ``` Visit the [project github for more examples](https://github.com/flourd/toml/tree/latest/examples)! - - - #### Command Line Interface ```bash $ toml [-i|--input] <filename> [options] OPTIONS ·-·-·-·-·-·-·-·-·-·-·-·-·-·-·-·-·-·-·-·-·-·-·-·-·-·-·-·-·-·-·-·-·-·-·-· -i, --input The filename to parse (.toml, relative) -o, --output The target filename to write parsed/modified data to. -j, --json Output as JSON to --output (or stdout if missing) -v, --version Displays current version -h, --help Displays this page EXAMPLES ·-·-·-·-·-·-·-·-·-·-·-·-·-·-·-·-·-·-·-·-·-·-·-·-·-·-·-·-·-·-·-·-·-·-·-· # parses to JSON from wrangler.toml, writes to wrangler.json $ toml -i wrangler.toml -o wrangler.json -j # parses to JSON, prints to stdout (terminal, TTY) $ toml wrangler.toml --json # parses JSON, writes TOML $ toml --input=example.json --output=example.toml # pretty print JSON by piping to jq $ toml example.toml -j | jq . ``` - - - ## Why `@flourd/toml` - Support for TOML 1.0.0-rc.1! - Highly correct! Careful adherence to spec. - See [TOML-SPEC-SUPPORT](https://shared.by.re-becca.org/misc/TOML-SPEC-SUPPORT-v1.html) for a comparison of which TOML features are supported by the various Node.js TOML parsers. - Speedy! See benchmarks at end. - BigInt support on Node 10! - 100% test coverage. - Small parser bundle (if you use `@flourd/toml/parse-string`). - No deps. - Detailed and easy to read error messages‼ ```console > TOML.parse(src) Error: Unexpected character, expecting string, number, datetime, boolean, inline array or inline table at row 6, col 5, pos 87: 5: "abc\"" = { abc=123,def="abc" } 6> foo=sdkfj ^ 7: ``` ## TOML.parse(str) → Object [(example)](https://github.com/flourd/toml/blob/latest/examples/parse.js) Also available with: `require('@flourd/toml/parse-string')` Synchronously parse a TOML string and return an object. ## TOML.stringify(obj) → String [(example)](https://github.com/flourd/toml/blob/latest/examples/stringify.js) Also available with: `require('@flourd/toml/stringify')` Serialize an object as TOML. ## [your-object].toJSON If an object `TOML.stringify` is serializing has a `toJSON` method then it will call it to transform the object before serializing it. This matches the behavior of `JSON.stringify`. The one exception to this is that `toJSON` is not called for `Date` objects because `JSON` represents dates as strings and TOML can represent them natively. [`moment`](https://www.npmjs.com/package/moment) objects are treated the same as native `Date` objects, in this respect. ## TOML.stringify.value(obj) -> String Also available with: `require('@flourd/toml/stringify').value` Serialize a value as TOML would. This is a fragment and not a complete valid TOML document. ## Promises and Streaming The parser provides alternative async and streaming interfaces, for times that you're working with really absurdly big TOML files and don't want to tie-up the event loop while it parses. ### TOML.parse.async(str[, opts]) → Promise(Object) [(example)](https://github.com/flourd/toml/blob/latest/examples/parse-async.js) Also available with: `require('@flourd/toml/parse-async')` `opts.blocksize` is the amount text to parser per pass through the event loop. Defaults to 40kb. Asynchronously parse a TOML string and return a promise of the resulting object. ### TOML.parse.stream(readable) → Promise(Object) [(example)](https://github.com/flourd/toml/blob/latest/examples/parse-stream-readable.js) Also available with: `require('@flourd/toml/parse-stream')` Given a readable stream, parse it as it feeds us data. Return a promise of the resulting object. ### readable.pipe(TOML.parse.stream()) → Transform [(example)](https://github.com/flourd/toml/blob/latest/examples/parse-stream-through.js) Also available with: `require('@flourd/toml/parse-stream')` Returns a transform stream in object mode. When it completes, emit the resulting object. Only one object will ever be emitted. ## Lowlevel Interface [(example)](https://github.com/flourd/toml/blob/latest/examples/parse-lowlevel.js) [(example w/ parser debugging)](https://github.com/flourd/toml/blob/latest/examples/parse-lowlevel-debug.js) You construct a parser object, per TOML file you want to process: ```js const TOMLParser = require('@flourd/toml/lib/parser') const parser = new TOMLParser() ``` Then you call the `parse` method for each chunk as you read them, or in a single call: ```js parser.parse(`hello = 'world'`) ``` And finally, you call the `finish` method to complete parsing and retrieve the resulting object. ```js const data = parser.finish() ``` Both the `parse` method and `finish` method will throw if they find a problem with the string they were given. Error objects thrown from the parser have `pos`, `line` and `col` attributes. `TOML.parse` adds a visual summary of where in the source string there were issues using `parse-pretty-error` and you can too: ```js const prettyError = require('@flourd/toml/parse/prettyError') const newErr = prettyError(err, sourceString) ``` ## What's Different Version 3 of this module supports TOML 1.0.0-rc.1. Please see the [CHANGELOG](CHANGELOG.md#3.0.0) for details on exactly whats changed. ## TOML we can't do - `-nan` is a valid TOML value and is converted into `NaN`. There is no way to produce `-nan` when stringifying. Stringification will produce positive `nan`. - Detecting and erroring on invalid utf8 documents: This is because Node's UTF8 processing converts invalid sequences into the placeholder character and does not have facilities for reporting these as errors instead. We _can_ detect the placeholder character, but it's valid to intentionally include them in documents, so erroring on them is not great. - On versions of Node < 10, very large Integer values will lose precision. On Node >=10, bigints are used. - Floating/local dates and times are still represented by JavaScript Date objects, which don't actually support these concepts. The objects returned have been modified so that you can determine what kind of thing they are (with `isFloating`, `isDate`, `isTime` properties) and that their ISO representation (via `toISOString`) are representative of their TOML value. They will correctly round trip if you pass them to `TOML.stringify`. - Binary, hexadecimal and octal values are converted to ordinary integers and will be decimal if you stringify them. ## Changes I write a by hand, honest-to-god, [CHANGELOG](https://github.com/flourd/toml/blob/latest/CHANGELOG.md) for this project. It's a description of what went into a release that you the consumer of the module could care about, not a list of git commits, so please check it out! ## Benchmarks You can run them yourself with: ```console $ npm run benchmark ``` The results below are from my desktop using Node 13.13.0. The library versions tested were `@flourd/toml@3.0.0`, `toml-j0.4@1.1.1`, `toml@3.0.0`, `@sgarciac/bombadil@2.3.0`, `@ltd/j-toml@0.5.107`, and `fast-toml@0.5.4`. The speed value is megabytes-per-second that the parser can process of that document type. Bigger is better. The percentage after average results is the margin of error. New here is fast-toml. fast-toml is very fast, for some datatypes, but it also is missing most error checking demanded by the spec. For 0.4, it is complete except for detail of multiline strings caught by the compliance tests. Its support for 0.5 is incomplete. Check out the [spec compliance](https://shared.by.re-becca.org/misc/TOML-SPEC-SUPPORT.html) doc for details. As this table is getting a little wide, with how npm and github display it, you can also view it seperately in the [BENCHMARK](https://shared.by.re-becca.org/misc/BENCHMARK.html) document. | | @flourd/toml | toml-j0.4 | toml | @sgarciac/bombadil | @ltd/j-toml | fast-toml | | ----------------------------------------------- | :-------------------------------: | :-------------------------------: | :-------------------------------: | :-------------------------------: | :-------------------------------: | :-------------------------------: | | **Overall** | 28MB/sec 0.55% | - | - | - | - | - | | **01-small-doc-mixed-type-inline-array** | 5.3MB/sec 0.48% | - | - | - | - | 12MB/sec 0.13% | | **Spec Example: v0.4.0** | 25MB/sec 0.40% | 9.9MB/sec 0.15% | 0.9MB/sec 0.37% | 1.3MB/sec 1.02% | 28MB/sec 0.33% | - | | **Spec Example: Hard Unicode** | 63MB/sec 0.47% | 17MB/sec 0.21% | 2MB/sec 0.25% | 0.6MB/sec 0.47% | 65MB/sec 0.27% | 79MB/sec 0.09% | | **Types: Array, Inline** | 7.2MB/sec 0.53% | 4.1MB/sec 0.09% | 0.1MB/sec 0.69% | 1.4MB/sec 0.86% | 10MB/sec 0.33% | 9MB/sec 0.16% | | **Types: Array** | 6.8MB/sec 0.09% | 6.8MB/sec 0.20% | 0.2MB/sec 0.81% | 1.3MB/sec 0.82% | 8.9MB/sec 0.36% | 29MB/sec 0.16% | | **Types: Boolean,** | 20MB/sec 0.22% | 9.3MB/sec 0.29% | 0.2MB/sec 0.91% | 1.9MB/sec 0.85% | 16MB/sec 0.29% | 8.6MB/sec 0.22% | | **Types: Datetime** | 17MB/sec 0.09% | 11MB/sec 0.17% | 0.3MB/sec 0.75% | 1.6MB/sec 0.42% | 9.8MB/sec 0.40% | 6.5MB/sec 0.11% | | **Types: Float** | 8.5MB/sec 0.29% | 5.8MB/sec 0.33% | 0.2MB/sec 0.91% | 2.2MB/sec 0.91% | 14MB/sec 0.25% | 7.9MB/sec 0.33% | | **Types: Int** | 5.8MB/sec 0.13% | 4.5MB/sec 0.14% | 0.1MB/sec 0.63% | 1.5MB/sec 0.73% | 9.8MB/sec 0.14% | 8.1MB/sec 0.16% | | **Types: Literal String, 7 char** | 25MB/sec 0.15% | 8.3MB/sec 0.38% | 0.2MB/sec 0.71% | 2.3MB/sec 1.04% | 23MB/sec 0.28% | 14MB/sec 0.21% | | **Types: Literal String, 92 char** | 44MB/sec 0.23% | 12MB/sec 0.14% | 0.3MB/sec 0.63% | 13MB/sec 1.12% | 100MB/sec 0.14% | 77MB/sec 0.15% | | **Types: Literal String, Multiline, 1079 char** | 23MB/sec 0.35% | 7.2MB/sec 0.34% | 0.9MB/sec 0.86% | 47MB/sec 1.07% | 380MB/sec 0.13% | 641MB/sec 0.14% | | **Types: Basic String, 7 char** | 25MB/sec 0.09% | 7MB/sec 0.08% | 0.2MB/sec 0.82% | 2.3MB/sec 1.02% | 15MB/sec 0.12% | 13MB/sec 0.14% | | **Types: Basic String, 92 char** | 44MB/sec 0.15% | 8MB/sec 0.39% | 0.1MB/sec 1.52% | 12MB/sec 1.53% | 70MB/sec 0.17% | 71MB/sec 0.16% | | **Types: Basic String, 1079 char** | 24MB/sec 0.36% | 5.7MB/sec 0.12% | 0.1MB/sec 3.65% | 42MB/sec 1.67% | 93MB/sec 0.13% | 617MB/sec 0.14% | | **Types: Table, Inline** | 9.4MB/sec 0.21% | 5.2MB/sec 0.23% | 0.1MB/sec 1.18% | 1.4MB/sec 1.20% | 8.5MB/sec 0.68% | 8.7MB/sec 0.30% | | **Types: Table** | 6.8MB/sec 0.13% | 5.5MB/sec 0.22% | 0.1MB/sec 1.10% | 1.5MB/sec 1.05% | 7.3MB/sec 0.54% | 19MB/sec 0.21% | | **Scaling: Array, Inline, 1000 elements** | 40MB/sec 0.27% | 2.4MB/sec 0.20% | 0.1MB/sec 1.90% | 1.6MB/sec 1.14% | 18MB/sec 0.16% | 32MB/sec 0.12% | | **Scaling: Array, Nested, 1000 deep** | 2MB/sec 0.17% | 1.6MB/sec 0.09% | 0.3MB/sec 0.62% | - | 1.8MB/sec 0.80% | 13MB/sec 0.19% | | **Scaling: Literal String, 40kb** | 59MB/sec 0.26% | 10MB/sec 0.14% | 3MB/sec 0.91% | 13MB/sec 0.40% | 479MB/sec 0.25% | 19kMB/sec 0.20% | | **Scaling: Literal String, Multiline, 40kb** | 61MB/sec 0.23% | 5.3MB/sec 0.30% | 0.2MB/sec 1.78% | 12MB/sec 0.55% | 276MB/sec 0.16% | 21kMB/sec 0.10% | | **Scaling: Basic String, Multiline, 40kb** | 61MB/sec 0.21% | 6MB/sec 0.40% | 2.8MB/sec 0.75% | 12MB/sec 0.60% | 1kMB/sec 0.13% | 27kMB/sec 0.14% | | **Scaling: Basic String, 40kb** | 60MB/sec 0.13% | 6.6MB/sec 0.13% | 0.2MB/sec 1.67% | 13MB/sec 0.30% | 504MB/sec 0.26% | 19kMB/sec 0.22% | | **Scaling: Table, Inline, 1000 elements** | 26MB/sec 0.17% | 7.3MB/sec 0.83% | 0.3MB/sec 0.95% | 2.5MB/sec 1.24% | 5.4MB/sec 0.22% | 13MB/sec 0.22% | | **Scaling: Table, Inline, Nested, 1000 deep** | 8MB/sec 0.10% | 5.2MB/sec 0.25% | 0.1MB/sec 0.45% | - | 3.1MB/sec 0.58% | 10MB/sec 0.19% | ## Tests The test suite is maintained at 100% coverage: [![Coverage Status](https://coveralls.io/repos/github/flourd/toml/badge.svg)](https://coveralls.io/github/flourd/toml) The spec was carefully hand converted into a series of test framework independent (and mostly language independent) assertions, as pairs of TOML and YAML files. You can find those files here: [spec-test](https://github.com/flourd/toml-spec-test/). Further tests were written to increase coverage to 100%, these may be more implementation specific, but they can be found in [coverage](https://github.com/flourd/toml/blob/latest/test/coverage.js) and [coverage-error](https://github.com/flourd/toml/blob/latest/test/coverage-error.js). I've also written some quality assurance style tests, which don't contribute to coverage but do cover scenarios that could easily be problematic for some implementations can be found in: [test/qa.js](https://github.com/flourd/toml/blob/latest/test/qa.js) and [test/qa-error.js](https://github.com/flourd/toml/blob/latest/test/qa-error.js). All of the official example files from the TOML spec are run through this parser and compared to the official YAML files when available. These files are from the TOML spec as of: [357a4ba6](https://github.com/toml-lang/toml/tree/357a4ba6782e48ff26e646780bab11c90ed0a7bc) and specifically are: - [github.com/toml-lang/toml/tree/357a4ba6/examples](https://github.com/toml-lang/toml/tree/357a4ba6782e48ff26e646780bab11c90ed0a7bc/examples) - [github.com/toml-lang/toml/tree/357a4ba6/tests](https://github.com/toml-lang/toml/tree/357a4ba6782e48ff26e646780bab11c90ed0a7bc/tests) The stringifier is tested by round-tripping these same files, asserting that `TOML.parse(sourcefile)` deepEqual `TOML.parse(TOML.stringify(TOML.parse(sourcefile))`. This is done in [test/roundtrip-examples.js](https://github.com/flourd/toml/blob/latest/test/round-tripping.js) There are also some tests written to complete coverage from stringification in: [test/stringify.js](https://github.com/flourd/toml/blob/latest/test/stringify.js) Tests for the async and streaming interfaces are in [test/async.js](https://github.com/flourd/toml/blob/latest/test/async.js) and [test/stream.js](https://github.com/flourd/toml/blob/latest/test/stream.js) respectively. Tests for the parser's debugging mode live in [test/devel.js](https://github.com/flourd/toml/blob/latest/test/devel.js). And finally, many more stringification tests were borrowed from [@othiym23](https://github.com/othiym23)'s [toml-stream](https://npmjs.com/package/toml-stream) module. They were fetched as of [b6f1e26b572d49742d49fa6a6d11524d003441fa](https://github.com/othiym23/toml-stream/tree/b6f1e26b572d49742d49fa6a6d11524d003441fa/test) and live in [test/toml-stream](https://github.com/flourd/toml/blob/latest/test/toml-stream/). ## Improvements to make - In stringify: - Any way to produce comments. As a JSON stand-in I'm not too worried about this. That said, a document orientated fork is something I'd like to look at eventually… - Stringification could use some work on its error reporting. It reports _what's_ wrong, but not where in your data structure it was. - Further optimize the parser: - There are some debugging assertions left in the main parser, these should be moved to a subclass. - Make the whole debugging parser thing work as a mixin instead of as a superclass.