UNPKG

@spmeesseman/json

Version:

a 'json' command for massaging and processing JSON on the command line

958 lines (658 loc) 27.3 kB
# json Changelog ## not yet released (nothing yet) ## 9.0.6 - [issue #107] Fix man page installation with `npm install -g json`. ## 9.0.5 - [issue #112] Improve streaming (json -ga) performance for very long lines. For example, using a 35 MB JSON object on one line gave a 50x speed improvement. However, this is restricted to streaming of newline-separated JSON as opposed to adjacent JSON objects not separated by newlines ({"a":1}{"a":2}). The former case is expected to be much more common, and the latter may take a slight performance hit from this change. ## 9.0.4 - [issue #108] Fix a crash on `json foo.bar` if "foo" is null. ## 9.0.3 - [issue #82] Fix a race in `-I/--in-place` temp file creation. By https://github.com/inator ## 9.0.2 - [pull #72] Correct examples in docs for conditional filtering. ## 9.0.1 - [issue #71] Support `-o json-tab` and `-o jsony-tab` for TAB (i.e. `\t`) indentation of emitted JSON. ## 9.0.0 - [issue #52] Fix termination on EPIPE in some cases. - Add `-0`, `-2`, `-4` options to more conveniently set the JSON indentation without changing the mode. - [pull #64] Add `-M, --items` option for "itemizing" key/value pairs in an object for easy iteration. For example: $ echo '{"trent":{"age":38}, "ewan": {"age":4}}' | json -M [ { "key": "trent", "value": { "age": 38 } }, { "key": "ewan", "value": { "age": 4 } } ] $ echo '{"trent":{"age":38}, "ewan": {"age":4}}' | json -Ma key value.age trent 38 ewan 4 # List people that can vote. $ echo '{"trent":{"age":38}, "ewan": {"age":4}}' | json -M -c 'this.value.age > 18' -a key trent Thanks to [AndrewO](https://github.com/AndrewO) for providing this! - **Backward incompatible change to `-c CODE` and `-e CODE`** changing their implementation to use a JS function for processing rather than `vm.runInNewContext`. This is the technique for which the `-C CODE` and `-E CODE` options were added in version 7.0.0. Basically: This technique is obviously better because it is 10x faster, so it is being made the only supported way. `-C` and `-E`, then, become synonyms and may be removed in a later release. Unfortunately this does mean a few semantic differences in the `CODE`, the most noticeable of which is that **`this` is required to access the object fields:** # Bad. Works with json < v9... $ echo '{"green": "eggs"}' | json-v8 -e 'green="ham"' { "green": "ham" } # ... does *not* work with json v9. $ echo '{"green": "eggs"}' | json -e 'green="ham"' { "green": "eggs" } # Good. Works with all versions of json. $ echo '{"green": "eggs"}' | json -e 'this.green="ham"' { "green": "ham" } The old behaviour of `-c` and `-e` can be restored with the `JSON_EXEC=vm` environment variable: $ echo '{"green": "eggs"}' | JSON_EXEC=vm json -e 'green="ham"' { "green": "ham" } See the notes on [json 7.0.0](#json-700) below for full details on the performance improvements and semantic changes. ## 8.0.0 - [pull #70] Move from 'jsontool' to 'json' in the npm registry! Thanks to <https://github.com/zpoley> for graciously giving up the name, and to @izs for driving. `npm install json` FTW. Here after `jsontool` will stagnate at version 7.0.2. ## 7.0.2 - [issue #68] Fix `--keys, -k` handling in streaming mode, i.e. `json -gak`. ## 7.0.1 - [pull #60, issue #59] Fix not having a `json` on the PATH from 'npm install -g jsontool'. ## 7.0.0 - [issue #49] New `-C CODE` and `-E CODE` options to replace `-c CODE` and `-e CODE`. The new options can be **10x or more faster**. An example processing a large log of newline-separated JSON object as a stream: $ ls -al big.log -rw-r--r-- 1 trentm staff 156M Oct 25 23:31 big.log $ time json -f big.log -gac 'this.audit' req.method req.url >/dev/null real 0m21.380s user 0m21.051s sys 0m0.526s $ time json -f big.log -gaC 'this.audit' req.method req.url >/dev/null real 0m3.336s user 0m3.124s sys 0m0.295s For comparison with `jq` (a C-based fast JSON processor tool): $ time cat big.log | jq '.req.method, .req.url' >/dev/null real 0m3.307s user 0m3.249s sys 0m0.136s The speed difference in `json` is in how the given `CODE` is executed: the new implementation uses a JS function, while the `-c/-e` options use node.js's `vm.runInNewContext`. This change means some semantic changes to the given `CODE`. Some examples to show the semantic differences: 1. `this` is required to access the object fields: $ echo '{"foo": "bar"}' | json -e 'foo="baz"' { "foo": "baz" } $ echo '{"foo": "bar"}' | json -e 'this.foo="baz"' { "foo": "baz" } $ echo '{"foo": "bar"}' | json -E 'foo="baz"' # doesn't work { "foo": "bar" } $ echo '{"foo": "bar"}' | json -E 'this.foo="baz"' { "foo": "baz" } 2. Explicit `return` is required with `-C` when using multiple statements: $ echo '{"a": 2, "b": 6}' | json -C 'sum = this.a + this.b; sum > 5' undefined:2 return (sum = this.a + this.b; sum > 5) ^ SyntaxError: Unexpected token ; $ echo '{"a": 2, "b": 6}' | json -C 'sum = this.a + this.b; return sum > 5' { "a": 2, "b": 6 } 3. Some operations on the input object are more as you'd expect: $ echo '["a", "b"]' | json -AE 'this.push("c")' [ "a", "b", "c" ] $ echo '{"a":1,"b":2}' | json -E 'delete this.a' { "b": 2 } 4. `CODE` is no longer run in a sandbox, so you can shoot yourself in the foot. Security warning: *Don't run untrusted code with '-E' or '-C'.* $ echo '{}' | json -C 'process.stdout.end()' [Error: process.stdout cannot be closed.] $ echo '{}' | json -c 'process.stdout.end()' vm.js:41 return ns[f].apply(ns, arguments); ^ ReferenceError: process is not defined Overall: (1) is a annoying, (2) is likely rare but at least is explicit, (3) is a minor win, (4) is something to just be aware of. The major win is a ~10x speed improvement! See <https://github.com/nfitch/node-test/blob/master/test/vmalterns.js> for analysis on the perf of various options for running user-given code. Thanks to Nate for pushing me on this! - Major perf win on simple lookups, and with no behaviour change(!). Similarly this was achieved by avoiding `vm.runInNewContext`: $ time json6 -f big.log -ga time >/dev/null # v6 real 0m28.892s user 0m26.839s sys 0m2.295s $ time json -f big.log -ga time >/dev/null # v7 real 0m2.427s user 0m2.289s sys 0m0.212s The changes for this have changed `jsontool.parseLookup` and `jsontool.handleLookup` incompatibly. However, using "jsontool.js" is experimental and extremely rare. Please contact me if this impacts you. - Support for node 0.11.x -- basically stop using `util.puts`. Note that apparently `vm.runInNewContext` has changed in node 0.11.x such that the following no longer works: $ echo '["a", "b"]' | json -A -e 'this[0]="A"' [ "A", "b" ] The result with node 0.11.x is actually: $ echo '["a", "b"]' | json -A -e 'this[0]="A"' [ "a", "b" ] Using the new `-E` works: $ echo '["a", "b"]' | json -A -E 'this[0]="A"' [ "A", "b" ] All the more reason to use the new `-E CODE`. - Change to 4-space indents. 'make check' clean. No functional change. - Include project url (and my name) in `json --version`. Also show *JSON* formatted version info with `-j`. The point here isn't self-agrandization but to help differentiate from the other `json` tool out there (`npm home json`). $ json --version json 7.0.0 written by Trent Mick https://github.com/trentm/json $ json --version -j { "version": "7.0.0", "author": "Trent Mick", "project": "https://github.com/trentm/json" } - Validation failures with a given filename will now show the filename, e.g.: $ json -nf foo.json json: error: "foo.json" is not JSON: Expected ',' instead of '"' at line 3, column 5: "baz": "car" ....^ - Move json.1 to "man/man1" and set "directories.man" in package.json to have "man json" work after "npm install -g jsontool" with the coming npm 1.3.3 work. ## 6.0.0 - [Backwards incompatibility, issue #55] Drop support for grouping of adjacent arrays (via `-g`, `--group`) **separated by no space**: # Before echo '["one"]["two"]' | json -g [ "one", "two" ] # After $ echo '["one"]["two"]' | json -g json: error: input is not JSON: Syntax error at line 1, column 8: ["one"]["two"] .......^ ["one"]["two"] We still allow grouping of arrays separated by a newline: # Before and after $ echo '["one"] ["two"]' | json -g [ "one", "two" ] This was dropped because the current regex technique used for grouping can fail with a JSON string that contains ']['. Not worth it. - New `-I/--in-place` option for in-place editing of given files with: $ cat foo.json {"foo":1} $ json -I -f foo.json # format the file json: updated "foo.json" in-place $ cat foo.json { "foo": 1 } $ json -I -f foo.json -e 'this.bar=42' # add bar field json: updated "foo.json" in-place $ cat foo.json { "foo": 1, "bar": 42 } Note: I'd have loved to have used `-i` a la sed, but that is unfortunately taken in this tool. ## 5.1.3 - Fix an issue with option parsing that resulted in this failing: json -f foo.json -- -1.someKey ## 5.1.2 - [pull #43] Add '-n' as a short alias for '--validate'. (By Bill Pijewski, github.com/pijewski). ## 5.1.1 - [issue #42] Fix an edge case where a blank line would be emitted for `... | json -ga -c COND` where the `COND` resulted in no matches. - [issue #40] Improve "Lookups" section of docs to show how to lookup non-identifier keys. ## 5.1.0 - [pull #39, issue #34] `json -ga` streams. (Largely by Fred Kuo, github.com/fkuo) This means you can use `json` with an input stream of JSON objects. E.g.: yes '{"foo":"bar"}' | json -ga Limitations: As was already a limitation of the '-g' switch, this only supports JSON objects delimited by newlines or with no space: {"a":1}{"b":2} # good {"a":1}\n{"b":2} # good {"a":1} {"b":2} # bad Additionally, currently only a stream of *objects* is supported, not a stream of arrays. Such are the typical use cases I've encountered. ## 5.0.0 - [**backward incompatible**, issue #35] Special case the output for **a single lookup AND JSON output** (i.e. `-j` or `-o json*`) to only output the value instead of the more general array or table that is necessary for multiple lookups. For objects: # Before: $ echo '{"one": "un", "two": "deux", "three": "troix"}' | json -j one { "one": "un" } # After: $ echo '{"one": "un", "two": "deux", "three": "troix"}' | json -j one "un" # Unchanged: $ echo '{"one": "un", "two": "deux", "three": "troix"}' | json -j one two { "one": "un", "two": "deux" } For arrays: # Before: $ echo '["a", "b", "c"]' | json -j 0 [ "a" ] # After: $ echo '["a", "b", "c"]' | json -j 0 "a" # Unchanged: $ echo '["a", "b", "c"]' | json -j 0 1 [ "a", "b" ] The motivation for this change was (a) the WTF of the first example above and (b) issue #36 where one could no longer extract a single value using '-j' to explicitly get JSON string quoting. ## 4.0.1 - [issue #36] Turn off coloring for inspect output (`json -i`, `json -o inspect`) if stdout is not a TTY. ## 4.0.0 - Add `--validate` option to just validate (no processing and output) $ echo '{"foo" "bar"}' | json json: error: input is not JSON: Expected ':' instead of '"' at line 1, column 8: {"foo" "bar"} .......^ {"foo" "bar"} $ echo '{"foo" "bar"}' | json --validate json: error: input is not JSON: Expected ':' instead of '"' at line 1, column 8: {"foo" "bar"} .......^ $ echo '{"foo" "bar"}' | json --validate -q $ echo $? 1 - Add `-f FILE` option for specifying an input file (or files) instead of stdin: $ json -f foo.json { "foo": "bar" } $ json -f foo.json foo bar - [Backward incompatible] Move "auto-arrayification" to require explicitly using the "-g" or "--group" option: $ echo '{"one":"two"}{"three":"four"}' | json json: error: input is not JSON: Syntax error at line 1, column 14: {"one":"two"}{"three":"four"} .............^ {"one":"two"}{"three":"four"} $ echo '{"one":"two"}{"three":"four"}' | json -g -o json-0 [{"one":"two"},{"three":"four"}] This is to avoid auto-arrayification accidentally making an invalid JSON object (with a missing comma) be transformed to a valid array: $ cat oops.json { "a": { "b": [ {"foo": "bar"} {"foo": "bar"} ] } } $ cat oops.json | json3 -o json-0 [{"a":{"b":[{"foo":"bar"},{"foo":"bar"}]}}] Basically the jusitification for this breaking change is that the invariant of `json` validating the input JSON is more important than the occassional convenience of grouping. - Use 8 space indent for syntax error message on stderr instead of '\t'. Minor change. Tabs are evil. ## 3.3.0 - Add `-k|--keys` option to output the input objects keys: $ echo '{"name": "trent", "age": 38}' | json -k [ "name", "age" ] $ echo '{"name": "trent", "age": 38}' | json -ka name age - Drop jsontool v2 dependency. This had been added for the first few json3 releases to provide a `json2` for comparison. `json` v3 is fairing well enough now to not bother. ## 3.2.0 - Support negative array indeces (a la Python list indeces), e.g.: $ echo '["a", "b", "c"]' | json -- -1 c ## 3.1.2 - Update man page and move bulk examples from README to man page. Use ronn (the ruby one) instead of ronnjs: better and more reliable formatting. Add 'make docs' and 'make publish' (the latter to push to GH pages at <http://trentm.com/json>). - [issue #31] Fix error message for `json -o`. ## 3.1.1 - [issue #32] Fix '-D' option processing so `json -D/` works (no space). ## 3.1.0 - [pull #29] Add '-D' option to set a delimiter for lookups (default is '.'), so that this example works: $ echo '{"a.b": {"b": 1}}' | json -D / a.b/b 1 By Yaniv Aknin. ## 3.0.3 - [issue #30] Fix lookup strings with multiple double-quotes. - [issue #28] Don't error on a multi-level lookup where one of the components is undefined. E.g., the following is no longer an error: $ echo '{"foo": "bar"}' | json not_foo.bar ## 3.0.2 - [issue #27] Fix issue handling multi-level lookups (e.g. 'json foo.bar'). ## 3.0.1 - Fix a bogus 'json' dep. ## 3.0.0 - Switched to json 3.x dev on master. "2.x" branch created for any necessary 2.x releases. See the [2.x changelog here](https://github.com/trentm/json/blob/2.x/CHANGES.md). - [Backward incompatible] A significant change to 'jsony' default output and some use cases to increase utility. These changes necessitated a few backward incompatible changes. However, care was take to only break compat for (a) rare use cases and (b) where utility was much improved. See <https://github.com/trentm/json/wiki/backward-incompat-json-3-changes> for full details of backward incompatible changes. - New "conditional filtering" via the `-c CODE` option. If the input is an array, then `-c` will automatically switch to processing each element of the array. Example: $ echo '[{"name":"trent", "age":38}, {"name":"ewan", "age":4}]' \ | json3 -c 'age>21' [ { "name": "trent", "age": 38 } ] - Change `-e CODE` option to automatically switch to array processing if the input is an array. This matches the behaviour of the new `-c CODE` option. Example: $ echo '[{"name":"trent", "age":38}, {"name":"ewan", "age":4}]' \ | json3 -e 'age++' -o json-0 [{"name":"trent","age":39},{"name":"ewan","age":5}] - New '-A' option to force `-e` and `-c` to process an input array as a single item. - Add [ansidiff](https://github.com/trentm/ansidiff)-based colored diffs for test suite failures. - [issue #26] Add support for escapes in the delimiter given by `-d DELIM`: $ echo '[{"one":"un","two":"deux"},{"one":"uno","two":"dos"}]' \ | json -a -d'\t' one two un deux uno dos ## 2.2.1 - Hack workaround for issue #24 to not get a spurious "process.stdout cannot be closed" from current node 0.6 versions. Note: currently this guard is only applied for node v0.6.0..v0.6.8 inclusive. ## 2.2.0 - New "-e CODE" option to execute the given code on the input object; or, if '-a/--array' is given, then on each item in the input array. Execution is done before filtering. $ echo '{"age": 38}' | json -e 'this.age++' { "age": 39 } ## 2.1.0 - Improve error message when input is not JSON to include context and line and column position. This is implemented using a JSON parser from (<https://github.com/douglascrockford/JSON-js>). Example: $ echo "[1,,2]" | json json: error: input is not JSON: Unexpected ',' at line 1, column 4: [1,,2] ...^ [1,,2] ## 2.0.3 - Auto-arrayification: Drop support for arrayifying an array adjacent to an object. I.e. only arrayify adjacent objects *or* adjacent arrays. - Auto-arrayification: Change "arrayification" of adjacent *arrays* to be a single flat arrays of the input arrays' elements. Before: $ echo '[1,2][3,4]' | bin/json [ [ 1, 2, ], [ 3, 4 ] ] and now: $ echo '[1,2][3,4]' | bin/json [ 1, 2, 3, 4 ] This is expected to be more useful in practice. - Auto-arrayification: Allow JSON objects (or arrays) to be "arrayified" if not separated by any space. Previously a newline (at least) separation was required. So, for example, the following now works: $ echo '{"a":1}{"b":2}' | bin/json -o json-0 [{"a":1},{"b":2}] The rules for auto-arrayification then are: Objects and arrays only, separated by no space or space including a newline. - Fix stdout flushing in some cases. ## 2.0.2 - Add node v0.6 support. Drop v0.2 and v0.3 support. ## 2.0.1 - [issue#23] Fix output in '-a|--array' mode if one or more keys don't exist in one or more of the array items. ## 2.0.0 - '-o | --output MODE' support. Supported modes: jsony (default): JSON with string quotes elided json: JSON output, 2-space indent json-N: JSON output, N-space indent, e.g. 'json-4' inspect: node.js `util.inspect` output - '-a|--array' for independently processing each element of an input array. $ echo '[ { "name": "Trent", "id": 12, "email": "trent@example.com" }, { "name": "Mark", "id": 13, "email": "mark@example.com" } ]' | json -a name email Trent trent@example.com Mark mark@example.com This example shows that '-a' results in tabular output. The '-d' option can be used to specify a delimiter other than the default single space, e.g.: json -d, -a field1 field2 [Backward Incompatibility] This is a replacement for the experimental '*' syntax in the lookup strings (previously enabled via '-x|--experimental'). That syntax and option has been removed. - Add '--' option processing support and error out if an unknown option is given. - Support multiple top-level JSON objects as input to mean a list of these object: $ echo '{"one": 1} {"two": 1}' | ./lib/jsontool.js [ { "one": 1 }, { "two": 1 } ] This can be nice to process a stream of JSON objects generated from multiple calls to another tool or `cat *.json | json`. Rules: - Only JS objects and arrays. Don't see strong need for basic JS types right now and this limitation simplifies. - The break between JS objects has to include a newline. I.e. good: {"one": 1} {"two": 2} bad: {"one": 1}{"two": 2} This condition should be fine for typical use cases and ensures no false matches inside JS strings. ## 1.4.1 - [issue #9] Gracefully handle EPIPE (i.e. stdout being closed on json before it is finished writing). ## 1.4.0 - [issue #19] Allow multiple lookup arguments: $ echo '{"one": 1, "two": 2}' | json one two 1 2 WARNING: This involve a backward incompatible change in the JS APIs `jsontool.processDatum` and `jsontool.processDatumExperimental`. ## 1.3.4 - [issue #18] Fix `json --version` for standalone mode again (was broken in json 1.3.3). ## 1.3.3 - WARNING: `json --version` is broken when running outside the source (or npm install'd) tree. I.e. this is a bad release for standalone. - [issue #17] Ensure stdout is flushed on exit. ## 1.3.2 - [issue #16] Fix to use `<regex object>.exec` instead of using the regex object as a function -- no longer allowed in the v8 used in node v0.5.x. ## 1.3.1 - Make "jsontool" require'able as a module. For example, you can now: $ npm install jsontool $ node > var jsontool = require('jsontool') > jsontool.parseLookup('a.b.c') [ 'a', 'b', 'c' ] > jsontool.parseLookup('my-key.0["bar"]') [ 'my-key', '0', '["bar"]' ] > jsontool.main(['', '', '--help']) Usage: <something generating JSON on stdout> | json [options] [lookup] ... Currently other exported API is experimental and will likely change to be more generally useful (e.g. the current `processDatum` isn't all handy for module usage). Note: For command-line usage, the main module has moved from "json" to "lib/jsontool.js". So, if you are not using npm, you can setup the `json` command via something like: alias json='.../json/lib/jsontool.js' ## 1.3.0 - package.json and publish to npm as "jsontool" ("json" name is taken) - Add experimental support for '*' in the lookup. This will extract all the elements of an array. Examples: $ echo '["a", "b", "c"]' | json -x '*' a b c $ echo '[{"one": "un"}, {"two": "deux"}]' | json -x '*' { "one": "un" } { "two": "deux" } $ echo '[{"foo": "bar"}, {"foo": "baz"}]' | json -x '*.foo' bar baz This is still experimental because I want to feel it out (is it useful? does it cause problems for regular usage?) and it is incomplete. The second example above shows that with '\*', json can emit multiple JSON documents. `json` needs to change to support *accepting* multiple JSON documents. Also, a limitation: How to extract *multiple* fields from a list of objects? Is this even a necessary feature? Thinking out loud: '*.{name,version}' # a la bash. Josh likes it. What else do you need? - Add '-x|--experimental' option to turn on incomplete/experimental features. ## 1.2.1 - [issue #12] Fix handling of output when result of lookup is `undefined`. ## 1.2.0 - [issue #10] Fix for node v0.5. ## 1.1.9 - [Issue 8] Don't emit a newline for empty output. ## 1.1.8 - [Issue 7] Handle "HTTP/1.1 100 Continue" leading header block. - [Issue 4] Add a man page (using ronnjs). ## 1.1.7 - [Issue 5] Fix getting a key with a period. E.g.: echo '{"foo.bar": 42}' | json '["foo.bar"]' `json` is now doing much better lookup string parsing. Because escapes are now handled properly you can do the equivalent a little more easily: $ echo '{"foo.bar": 42}' | json foo\\.bar 42 ## 1.1.6 - [Issue 6] Error exit value if invalid JSON. ## 1.1.4 - [Issue 2] Fix bracket notation: `echo '{"foo-bar": "baz"}' | json '["foo-bar"]'` (Started maintaining this log 19 March 2011. For earlier change information you'll have to dig into the commit history.)