stated-js
Version:
JSONata embedded in JSON
1,123 lines (1,078 loc) • 113 kB
Markdown
# Stated
<!-- TOC -->
* [Stated](#stated)
* [Licensing and documentation](#licensing-and-documentation)
* [API docs](#api-docs)
* [Testing](#testing)
* [Intro](#intro)
* [Motivation](#motivation)
* [Getting Started](#getting-started)
* [Installation](#installation)
* [Running](#running)
* [REPL Mode](#repl-mode)
* [Oneshot mode](#oneshot-mode)
* [stated-js lib](#stated-js-lib)
* [API documentation](#api-documentation)
* [Basic usage](#basic-usage)
* [Stated-js package.json exports](#stated-js-packagejson-exports)
* [Node.js based projects](#nodejs-based-projects)
* [REPL Commands](#repl-commands)
* [.open](#open)
* [All commands](#all-commands)
* [.out](#out)
* [.color](#color)
* [.log](#log)
* [.set](#set-)
* [REPL command arguments](#repl-command-arguments)
* [--options](#--options)
* [strict](#strict)
* [--xf](#--xf)
* [--ctx](#--ctx)
* [Language & Syntax](#language--syntax)
* [Expressions and Variables](#expressions-and-variables)
* [Dollars-Moustache](#dollars-moustache)
* [Dollars-Variables](#dollars-variables)
* [Temporary Expressions](#temporary-expressions)
* [References](#references)
* [Expression Scope](#expression-scope)
* [Rerooting Expressions](#rerooting-expressions)
* [Tags](#tags)
* [Generative Templates](#generative-templates)
* [Reactive Behavior](#reactive-behavior)
* [SVG command](#svg-command)
* [YAML](#yaml)
* [Setting Values in the stated REPL](#setting-values-in-the-stated-repl)
* [Expression Scope](#expression-scope-1)
* [Rerooting Expressions](#rerooting-expressions-1)
* [DAG](#dag)
* [visualizing the plan with .svg command](#visualizing-the-plan-with-svg-command)
* [Concurrency](#concurrency)
* [Serialized Mutations](#serialized-mutations)
* [Atomic State Updates](#atomic-state-updates)
* [Multi Version Concurrency Control (MVCC) and $forked](#multi-version-concurrency-control-mvcc-and-forked)
* [YAML](#yaml-1)
* [Complex Data Processing](#complex-data-processing)
* [Functions](#functions)
* [JSONata built-in](#jsonata-built-in)
* [Stated functions](#stated-functions)
* [$timeout](#timeout)
* [$interval](#interval)
* [$fetch](#fetch)
* [$import](#import)
* [Importing bits of other templates](#importing-bits-of-other-templates)
* [Setting up local imports with --importPath](#setting-up-local-imports-with---importpath)
* [Import Javascript ES modules](#import-javascript-es-modules)
* [The __init sidecar function](#the-__init-sidecar-function)
* [$open](#open-1)
* [$set](#set)
* [$debounce](#debounce)
* [$defer](#defer)
* [$rateLimit](#ratelimit)
* [Custom Functions](#custom-functions)
* [Simple Custom Function Example](#simple-custom-function-example)
* [More Complex Function Example](#more-complex-function-example)
* [Error Handling](#error-handling)
* [The error object](#the-error-object)
* [Error Reporting](#error-reporting)
* [$errorReport function](#errorreport-function)
* [TemplateProcessor Snapshots](#templateprocessor-snapshots)
* [Understanding Plans](#understanding-plans)
* [Planning](#planning)
* [MetaInfo Graph](#metainfo-graph)
* [DFS Tree Traversal](#dfs-tree-traversal)
<!-- TOC -->
# Licensing and documentation
Stated is a cisco-open, Apache 2 Licensed, Open Source project at https://github.com/cisco-open/stated, our github
main page.
# API docs
If you would like to see a table of contents and developer API docs, jump over to our API docs page which is
generated in CI and [published here](https://cisco-open.github.io/stated/index.html). Most developers will need to
interact with the TemplateProcessor. It's [API Docs are here](https://cisco-open.github.io/stated/classes/TemplateProcessor.default.html)
# Testing
Are the examples in this README
correct/reliable? Every markdown codeblock in this readme is
[tested on every commit](https://github.com/cisco-open/stated/blob/main/README.test.js).
# Intro
Stated is an engine of reactive state (a State Daemon). State is expressed as JSON or YAML. Rules for evolving state are
written via embedded [JSONata](http://docs.jsonata.org/) expressions in classic shell variable syntax `${}`.
```shell
cat examples/hello.json
{
"to": "world",
"msg": "${'hello ' & to}"
}
```
You can use stated as a 'one shot' template processor that computes all bound states once. If we run the template above
through stated, we see the `${}` expression has been evaluated:
```shell
stated example/hello.json
{
"to": "world",
"msg": "hello world"
}
```
Variables can be passed in:
```shell
cat example/helloVar.json
{
"msg": "${'hello ' & $TO}"
}
stated example/helloVar.json --ctx.TO=world
{
"msg": "hello world"
}
```
The more interesting use of Stated is an engine of reactive state. Load a template in the REPL, `.set` a value
and watch the state evolve.
```json
stated
> .init -f example/hello.json
{
"to": "world",
"msg": "${'hello ' & to}"
}
> .set /to "Joshua"
{
"to": "Joshua",
"msg": "hello Joshua"
}
> .set /to "David Lightman"
{
"to": "David Lightman",
"msg": "hello David Lightman"
}
```
Stated templates can contain expressions, reusable functions, and can even use JS timeouts and intervals. Let's see
a template that increments a counter every 10 ms, forever, as long it is running in the Stated
engine. We will use the `--tail` command to tail the `count` variable until it reaches 100, then
automatically disconnect the tail.
```json ["data = 100"]
cat example/infiniteCount.json
{
"count": 0,
"counter": "${ $setInterval(function(){$set('/count', count+1)}, 10) }",
}
stated
> .init -f example/infiniteCount.json --tail "/count until $=100"
Started tailing... Press Ctrl+C to stop.
100
```
Stated is written in JS, and runs in the browser and in Node. Stated's REPL extends the Node.js REPL and allows you to
interact with running templates. Stated uses asynchronous event loop I/O and can be used to orchestrate complex workflows:
```json
> .init -f example/homeworlds.json
{
"lukePersonUrl": "${ $fetch('https://swapi.tech/api/people/?name=luke').json().**.url}",
"lukePersonDetails": "${ $fetch(lukePersonUrl).json().result[0]}",
"lukeHomeworldURL": "${ lukePersonDetails.**.homeworld }",
"homeworldDetails": "${ $fetch(lukeHomeworldURL).json() }",
"homeworldName": "${ homeworldDetails.**.name }"
}
> .out /homeworldName
"Tatooine"
```
Unlike an ordinary program, Stated templates can be kept "alive" indefinitely. A change to any of the independent fields
causes change propagation throughout the DAG. Stated includes a node REPL, `stated.ts`, for testing Stated json templates, and a JS library for embedding stated
in applications. A typical REPL session consists of loading a template with the `init` command, viewing the computed
output with the `.out` command and then setting values with the `.set` command and observing the changed output.
```json [false, false, "a=87 and e=42"]
falken$ stated
> .init -f "example/ex08.json"
{
"a": "${c}",
"b": "${d+1+e}",
"c": "${b+1}",
"d": "${e+1}",
"e": 1
}
> .out
{
"a": 5,
"b": 4,
"c": 5,
"d": 2,
"e": 1
}
> .set /e 42
{
"a": 87,
"b": 86,
"c": 87,
"d": 43,
"e": 42
}
```
Stated templates are modular and can be imported from a URL:
```json
> .init -f "example/ex18.json"
{
"noradCommander": "${ norad.commanderDetails }",
"norad": "${ $import('https://raw.githubusercontent.com/geoffhendrey/jsonataplay/main/norad.json')}"
}
> .out
{
"noradCommander": {
"fullName": "Jack Beringer",
"salutation": "General Jack Beringer",
"systemsUnderCommand": 4
},
"norad": {
"commanderDetails": {
"fullName": "Jack Beringer",
"salutation": "General Jack Beringer",
"systemsUnderCommand": 4
},
"organization": "NORAD",
"location": "Cheyenne Mountain Complex, Colorado",
"commander": {
"firstName": "Jack",
"lastName": "Beringer",
"rank": "General"
},
"purpose": "Provide aerospace warning, air sovereignty, and defense for North America",
"systems": [
"Ballistic Missile Early Warning System (BMEWS)",
"North Warning System (NWS)",
"Space-Based Infrared System (SBIRS)",
"Cheyenne Mountain Complex"
]
}
}
```
# Motivation
Consider this ordinary program:
```js
let a=1;
let b=a;
a=42;
console.log(b); //prints out 1
```
In an ordinary sequential program the value of `b` is not affected by changes to the value of `a`
at any point after the value of `b` has been assigned. But there are many situations where we
do NOT want a sequential program execution Instead, we actually want `b` to change when `a` changes. Broadly, these cases fall under the rubric
of "reactive" or "state driven" applications. When we try to build reactive applications
upon a sequential execution model we are forced to code the data flow graph ourselves and things become
very complex quickly. How could we make `b` change any time `a` changes in a sequential world? Perhaps naively like this?
```js
let a=1;
let b=a;
function setA(val){
a=val;
b=a;
}
```
...or perhaps more generally like this:
```js
let data = {
a: 1,
b: 1
};
let handler = {
set: function(target, property, value) {
if (property === 'a') {
target.b = value; // When 'a' changes, also change 'b'
}
target[property] = value;
return true; // The set operation was successful
}
};
let proxy = new Proxy(data, handler);
proxy.a = 2; // Setting a new value for 'a'
console.log(proxy.a); // Outputs: 2
console.log(proxy.b); // Outputs: 2
```
Every "coding" approach requires us to understand and implement code for propagating data dependencies. Stated solves
for this by natively parsing and understanding dependencies.
```json
{
"a": 1,
"b$": "a"
}
```
`b$` is now declared to be continuously dependent upon `a` and reactive to any changes in `a`. This greatly
simplifies the development of systems that *are* naturally reactive or dependency heavy, such as:
* dynamic/continuous reactive UI state
* lambda-like computations
* workflows
* configuration file
# Getting Started
## Installation
To install the `stated-js` package, you can use yarn or npm. Open your terminal and run one of the following commands:
Using Yarn:
```shell
yarn global add stated-js
````
Using Npm:
```shell
npm install -g stated-js
````
## Running
### REPL Mode
To use the Stated REPL (Read-Eval-Print Loop) it is recommended to have at least version 19.2 or higher of node.js. The
Stated REPL is built on [Node REPL](https://nodejs.org/api/repl.html#repl).
You can start the REPL by running the `stated` command in your terminal:
```shell
stated
```
The REPL will launch, allowing you to interact with the stated-js library. For example you can enter this command in the
REPL:
```bash
> .init -f "example/ex01.json"
```
### Oneshot mode
in oneshot mode, stated.ts simply computes the output template and exits. This is useful when you do not intend to
change any if the fields after the initial output render
```bash
falken$ stated example/ex01.json
{
"to": "world",
"msg": "hello world"
}
```
# stated-js lib
## API documentation
API documentation is generated in CI and published to the project github page at https://cisco-open.github.io/stated/
## Basic usage
Much more detailed progammer usage can be found by perusing https://cisco-open.github.io/stated/classes/TemplateProcessor.default.html
To use the stated-js library in your own projects, you can require it as a dependency.
Here's an example of how you can import it into your JavaScript file:
```js
import TemplateProcessor from 'stated-js'
async function tryIt() {
const tp = new TemplateProcessor({
"a": "aaa",
"b": "${a}"
});
await tp.initialize();
await tp.setData("/a", 42);
console.log(JSON.stringify(tp.output));
// console will print...
// {
// "a": 42,
// "b": 42
// }
}
tryIt().catch(err => console.error(err));
```
#### Stated-js package.json exports
The distribution of stated-js exports two different webpacks, one for ES Module (ie "import") which is in `./dist/bundle.mjs`,
and one for Common JS (ie "require") which is in `./dist/bundle-common-js.cjs`. Using `import` or `require` in your JS
code from a project with a package.json will automatically choose the correct dist target. Both the ES and Common JS
exports are designed for use in the browser. Here is an excerpt of package.json showing exactly what is exported.
```json
"exports": {
".": {
"import": "./dist/bundle.mjs",
"require": "./dist/bundle-common-js.cjs"
},
"./dist/src/TemplateProcessor.js": "./dist/src/TemplateProcessor.js",
"./dist/src/CliCore.js": "./dist/src/CliCore.js",
"./dist/src/StatedREPL.js": "./dist/src/StatedREPL.js"
}
```
#### Node.js based projects
If you are building for a Node.js environment you should not use the webpacked exports for `import` or `require`.
Instead, use the 'raw' `TemplateProcessor.js`, `CliCore.js`, and `StatedREPL.js`. Write your import statement like this
for Node.js :
```js
import TemplateProcessor from 'stated-js/dist/src/TemplateProcessor.js';
```
When you run Node.js, you must pass `--experimental-vm-modules` flag to the Node.js runtime. This is due to the
fact that stated-js is written using ES Module syntax.
# REPL Commands
REPL commands can be used once you start the REPL by running `stated` from your prompt. All examples in this
doc assume your working directory is the stated git repo's root folder.
## .open
Stated provides a set of REPL commands to interact with the system. The `.open` command is the easiest way to get started
opening an example template:
<img width="1000" src="https://media.giphy.com/media/v1.Y2lkPTc5MGI3NjExdnl6NDgzdnE0bWlwbzU0NjBlOTNtMmE0OHJ1NjRpdmJxYTdtb3FleiZlcD12MV9pbnRlcm5hbF9naWZfYnlfaWQmY3Q9Zw/0kmQtLaWvuthTU1f2o/giphy.gif"/>
## All commands
| Command | Description | flags & args | Example |
|------------|----------------------------------------------------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| `.open` | Interactive command to open a template (defaults to examples folder) | | `.open` |
| `.cd` | Change directory (then use .open command) | | `.cd ..` |
| `.init` | Initialize the template from a JSON file. | • `-f <path>` <br> • `--tags=<taglist>`<br>•`--options=<json>` <br> • `--xf=<path>`<br> • `--importPath=<path>` <br> • `--tail "<tailargs>" --ctx.<dotted-path>=val` | `.init -f "example/hello.json" --tags=FOO,BAR --xf=~/falken/myEnv.json --options={"strict":{"refs":true}} --importPath=~/falken/mytemplates --tail "/ until msg='hello world'" --ctx.TO=world` |
| `.set` | Set data to a JSON pointer path. | `<path> <data>` | `.set /to "jsonata"` |
| `.from` | Show the dependents of a given JSON pointer. | `<path>` | `.from /a` |
| `.to` | Show the dependencies of a given JSON pointer. | `<path>` | `.to /b` |
| `.in` | Show the input template. | `None` | `.in` |
| `.out` | Show the current state of the template. | `[<jsonPtr>]` | `.out` <br>`.out /data/accounts` |
| `.state` | Show the current state of the template metadata. | `None` | `.state` |
| `.plan` | Show the execution plan for rendering the template. | `None` | `.plan` |
| `.note` | Show a separator line with a comment in the REPL output. | `<comment>` | `.note "Example 8"` |
| `.log` | Set the logging level | `[silent, error, warn, info, verbose, debug]` | `.log silent` |
| `.color` | Enable Colors | `[on,off]` | `.color on` |
| `.tail` | Tail part of the document for changes | `<jsonPointer> (until <jsonata_expr>)?` | `.tail /` <br> `.tail "/ until foo='bar'"` |
| `.svg` | Serve an SVG diagram of the DAG | `--port <portnumber>` (defaults to 4242) | `.svg --port 3000` |
| `.restore` | Restore from a snapshot | • `-f <path>` <br> • `--tags=<taglist>`<br>• `--xf=<path>`<br> • `--importPath=<path>` <br> • `--tail "<tailargs>"` | `.restore -f "example/restoreSnapshot.json" --tail "/count until $=10"` |
## .out
The stated repl lets you experiment with templates. The simplest thing to do in the REPL is load a json file. The REPL
parses the input, builds an execution plan, and executes the result. To see the result you have to use the `.out`
```json
> .init -f "example/ex09.json"
{
"a": [
0,
1,
"${ $[0] + $[1] }"
]
}
> .out
{
"a": [
0,
1,
1
]
}
```
## .color
You can set terminal colors for enhanced readability from a terminal
```json
> .note color does not appear in this markdown file
"============================================================="
> .color on
> .init -f "example/ex01.json"
{
"a": 42,
"b": "${a}",
"c": "${'the answer is: '& b}"
}
```
## .log
the `.log` command can set the log level to `[silent, error, warn, info, verbose, debug]`. Enabling high
log levels like debug can help you track down problems with expressions.
```json
> .log debug
{
"log level": "debug"
}
```
```shell
.init -f "example/errors.json"
arguments: {"_":[],"f":"example/errors.json","filepath":"example/errors.json","tags":[],"oneshot":false,"options":{}}
verbose: initializing...
debug: tags: {}
verbose: evaluating template...
error: Error evaluating expression at /b
error: The right side of the "+" operator must evaluate to a number {"code":"T2002","position":3,"stack":"Error\n at evaluateNumericExpression (/Users/ghendrey/proj/jsonataexperiments/node_modules/jsonata/jsonata.js:4122:25)\n at evaluateBinary (/Users/ghendrey/proj/jsonataexperiments/node_modules/jsonata/jsonata.js:3900:30)\n at async evaluate (/Users/ghendrey/proj/jsonataexperiments/node_modules/jsonata/jsonata.js:3490:26)\n at async Object.evaluate (/Users/ghendrey/proj/jsonataexperiments/node_modules/jsonata/jsonata.js:5558:26)\n at async TemplateProcessor._evaluateExprNode (file:///Users/ghendrey/proj/jsonataexperiments/src/TemplateProcessor.ts:637:25)\n at async TemplateProcessor._evaluateExpression (file:///Users/ghendrey/proj/jsonataexperiments/src/TemplateProcessor.ts:556:28)\n at async TemplateProcessor.evaluateJsonPointersInOrder (file:///Users/ghendrey/proj/jsonataexperiments/src/TemplateProcessor.ts:515:31)\n at async TemplateProcessor.evaluateDependencies (file:///Users/ghendrey/proj/jsonataexperiments/src/TemplateProcessor.ts:358:16)\n at async TemplateProcessor.evaluate (file:///Users/ghendrey/proj/jsonataexperiments/src/TemplateProcessor.ts:123:9)\n at async TemplateProcessor.initialize (file:///Users/ghendrey/proj/jsonataexperiments/src/TemplateProcessor.ts:113:9)","token":"+","value":" is not a string"}
debug: Expression: a + ' is not a string'
debug: Target: {
"a": 42,
"b": "${a + ' is not a string'}"
}
debug: Result: null
verbose: _evaluateExpression at /b completed in 13 ms.
verbose: evaluation complete in 13 ms...
verbose: initialization complete...
{
"a": 42,
"b": "${a + ' is not a string'}"
}
```
## .set
The stated REPL also allows you to dynamically set values in your templates, further aiding in debugging and development.
In the example below `.set /a/0 100` sets a[0] to 100. The syntax of `/a/0` is [RFC-6901 JSON Pointer](https://datatracker.ietf.org/doc/html/rfc6901).
```json
> .init -f "example/ex09.json"
{
"a": [
0,
1,
"${ $[0] + $[1] }"
]
}
> .set /a/0 100
{
"a": [
100,
1,
101
]
}
```
## REPL command arguments
### --options
The cli and REPL both support `--options` which can be provided as arguments to other commands
Options can also be used in oneshot mode. Note the use of backslashes to escape quotes in the JSON on the CLI
```shell
falken$ stated --options={\"strict\":{\"refs\":true}} example/strictref.json
error: /z does not exist, referenced from /c (strict.refs option enabled)
{
a: 42,
b: 42,
c: {
error: {
name: 'strict.refs',
message: '/z does not exist, referenced from /c (strict.refs option enabled)'
}
}
}
```
#### strict
The `strict` option currently supports the `refs` property. Setting `{"strict":{"refs":true}}` will cause templates
to throw an Exception when references in the template cannot be resolved. Reference checking is only performed against
the template itself; it is not performed agains variables that are injected into the execution context by the TemplateProcessor
library.
```json
> .log silent
{
"log level": "silent"
}
> .init -f "example/strictref.json" --options={"strict":{"refs":true}}
{
"a": 42,
"b": "${a}",
"c": "${z}"
}
> .out
{
"a": 42,
"b": 42,
"c": {
"error": {
"name": "strict.refs",
"message": "/z does not exist, referenced from /c (strict.refs option enabled)"
}
}
}
```
### --xf
The `--xf` argument can be used to provide a context file. Context is used
to provide [JSONata Bindings](https://docs.jsonata.org/embedding-extending#expressionevaluateinput-bindings-callback)
```shell
> .note here is a regular json file
"============================================================="
> .init -f "example/env.json"
{
"env": {
"name": "Dr. Stephen Falken",
"addr": "Goose Island, OR, USA"
}
}
> .note let's use it as context to a stated template
"============================================================="
> .init -f "example/useEnv.json" --xf=example/env.json
{
"name": "${$env.name}",
"address": "${$env.addr}"
}
> .out
{
"name": "Dr. Stephen Falken",
"address": "Goose Island, OR, USA"
}
```
### --ctx
The `--ctx` argument can be used to inject context variables into the template. Variables with a sinle `$` like `$foo`
refer to the JSONata Context. You can inject variables into the context using `--ctx.<dot-path>=val`.
```shell
> .init -f example/ctx.json --ctx.name david --ctx.games.choice1=chess --ctx.games.choice2 "global thermonuclear war"
{
"msg": "${'Hello, ' & $name & ', how about a nice game of ' & $games.choice1}",
"games": "${$games}"
}
> .out
{
"msg": "Hello, david, how about a nice game of chess",
"games": {
"choice1": "chess",
"choice2": "global thermonuclear war"
}
}
```
# Language & Syntax
## Expressions and Variables
What makes a Stated template different from an ordinary JSON file? JSONata Expressions of course! Stated analyzes the
Abstract Syntax Tree of every JSONata expression in the file, and learns what _references_ are made by each expression
to other fields of the document. The _references_ of an expression become the _dependencies_ of the field, which are
used to build a DAG. The DAG allows Stated to know what expressions to compute if any fields of the document change.
Fields of the document are changed either via the REPL `.set` function, or by calling the equivalent library function.
Many classes of _reactive_ application need to maintain state, and need to propagate state changes through the _dependees_
of a particular field (a _dependee_ of foo is a field that _depends_ on foo). Stated can be used as state store for
reactive applications.
## Dollars-Moustache
returning to our `example/hello.json`, the `msg` field is a simple example of a dollars-moustache.
Stated allows JSONata _expressions_ to be embedded in a JSON document using `${<...JSONata here...>}` syntax. The `${}`
tells stated that a field such as `msg` is not an ordinary string field, but rather a JSONata expression that has to be
evaluated in order to _compute_ the value of the field.
```json
falken$ cat example/hello.json
{
"to": "world",
"msg": "${'hello ' & to}"
}
```
## Dollars-Variables
There is also a more concise alternative to `${}`. The field can simply be named with a trailing
dollars sign.
```json
{
"to": "world",
"msg$": "'hello ' & to"
}
```
However the `foo$` style can only be used when the expression is being assigned to a field. It won't work for array
elements like this, where there is no field name. For array elements the `${}` must be used:
```json
[1, 2, "${$[0]+$[1]}"]
```
## Temporary Expressions
The `!` symbol is used mark fields as temporary. The `!` can be used both as a prefix to the expression, and as a suffix
to a key. Temporary fields are removed from the output.
Notice how /b and /c! are removed from the output. Also notice that when an expression like ```${`c!`.c1}``` refers to `c!`
that backtics must be used.
```json
> .init -f "example/tempVars.json"
{
"a": 42,
"b": "!${a}",
"c!": {
"c1": "../${a + 1}"
},
"d": "${`c!`.c1}"
}
> .out
{
"a": 42,
"d": 43
}
```
## References
The most important thing stated does is analyze your embedded jsonata programs to understand their _references_. This
means that as the user you don't have to _tell_ stated what an expression depends on, and
consequently you don't have to instruct Stated on how it should react to changes. It _knows_. In the example below, a JSONata [_block_](https://docs.jsonata.org/programming) is used to produce `defcon$`. It
defines local variables like `$tmp` which are pure JSONata constructs. The JSONata program also references fields
of the input like `MAX_DEFCON` and `threatLevel`. States understands that if `threatLevel changes`,
`defcon$` must be recalculated. As shown below after viewing the output with the `.out` commnand, we
mutate the `threatLevel` field which results in `defcon$` changing from 3 to 5.
```json
> .init -f "example/ex20.json"
{
"defcon$": "($tmp:=$floor(intelLevel * threatLevel);$tmp:= $tmp<=MAX_DEFCON?$tmp:MAX_DEFCON;$tmp>=MIN_DEFCON?$tmp:MIN_DEFCON)",
"MAX_DEFCON": 5,
"MIN_DEFCON": 1,
"intelLevel": 1.45,
"threatLevel": 2.2
}
> .out
{
"defcon$": 3,
"MAX_DEFCON": 5,
"MIN_DEFCON": 1,
"intelLevel": 1.45,
"threatLevel": 2.2
}
> .set /threatLevel 3.5
{
"defcon$": 5,
"MAX_DEFCON": 5,
"MIN_DEFCON": 1,
"intelLevel": 1.45,
"threatLevel": 3.5
}
```
## Expression Scope
Individual JSONata programs are embedded in JSON files between `${..}`. What is the input to the JSONata program?
The input, by default, is the object or array that the expression resides in. For instance in the example **above**, you can see that the JSONata `$` variable refers to the array itself. Therefore, expressions like `$[0]`
refer to the first element of the array.
## Rerooting Expressions
In Stated templates, one way to declare a JSONata expression is by surrounding it by "dollars moustaches".
E.g `${...some expression...}`. JSONata expressions always have a [context](https://docs.jsonata.org/programming#built-in-variables).
The `$` variable always points to the current context. The `$$` variable always points to the input (root context) for an
expression.
In a Stated template, the root context for an expression is the object in which the expression is contained. For
Example:
```json
> .init -f "example/context.json"
{
"a": {
"b": "${[c,' and ',$.c,' and ',$$.c,' are the same thing. $ (current context) is /a, the object in which this expression resides']~>$join}",
"c": "hello"
}
}
> .out
{
"a": {
"b": "hello and hello and hello are the same thing. $ (current context) is /a, the object in which this expression resides",
"c": "hello"
}
}
```
Now we will show how we can change the context of an expression using 'rerooting.' Rerooting allows the expression's root
context to be pointed anywhere in the json document.
In the example below, consider `greeting & ', ' & player1'`. We want `player1` to refer to the content at json pointer `/player1` (the field named 'player1' at the root of the document).
But our expression `greeting & ', ' & player1` is located deep in the document at `/dialog/partI`. So how can we cause
the root of the document to be the context for the JSONata expression `greeting & ', ' & player1`?
You can reroot an expression in a different part of the document using relative rooting `../${<expr>}` syntax or you can root an
at the absolute doc root with `/${<expr>}`. The example below shows how expressions located below the root object, can
explicitly set their context using the rooting syntax. Both absolute rooting, `/${...}` and relative rooting `../${...}`
are shown.
```json
> .init -f "example/ex04.json"
{
"greeting": "Hello",
"player1": "Joshua",
"player2": "Professor Falken",
"dialog": {
"partI": [
"../../${greeting & ', ' & player1}",
"../../${greeting & ', ' & player2}"
],
"partII": {
"msg3": "/${player1 & ', would you like to play a game?'}",
"msg4": "/${'Certainly, '& player2 & '. How about a nice game of chess?'}"
}
}
}
> .out
{
"greeting": "Hello",
"player1": "Joshua",
"player2": "Professor Falken",
"dialog": {
"partI": [
"Hello, Joshua",
"Hello, Professor Falken"
],
"partII": {
"msg3": "Joshua, would you like to play a game?",
"msg4": "Certainly, Professor Falken. How about a nice game of chess?"
}
}
}
```
An advanced rerooting operator is the `//` absolute root operator. The `/` rooting operator, that we showed above, will never allow the expression
to 'escape' outside of the template it was defined in. But what if we intend for a template to be imported into another template
and we expect there to be a variable defined in the other template that we should use? This is where the `//` absolute root
operator can be used. The `//` operator will set the expression context to the absolute root of whatever the final document is
after all imports have been performed.
```json
> .init -f "example/absoluteRoot.json"
{
"to": "!${'Professor Falken'}",
"greeting": "//${'Hello, ' & to}"
}
> .out
{
"greeting": "Hello, Professor Falken"
}
> .init -f "example/importsAbsoluteRoot.json"
{
"to": "Joshua",
"message": "${$import('example/absoluteRoot.json')}"
}
> .out
{
"to": "Joshua",
"message": {
"greeting": "Hello, Joshua"
}
}
```
## Tags
JSONata "dollars moustache" expressions can be _tagged_ with `@tag` syntax. In the example below our template
declares tags WAR and PEACE. Expressions with a tag will only be executed when the corresponding tag is
provided in the `--tags` flag. When tags are provided with `--tags` untagged expressions will be ignored.
Tags such as `@WAR` and `@PEACE` are automatically propagated transitively to all dependent expressions.
Therefore, in the example below only `peace` requires the `@PEACE` tag, which is transitively applies to
`peaceMsg` and `warAndPeaceMsg`. In large templates this is extrememly desirable to as it alleviates the
developer of having to maintain the tag by continually checking that any expressions that depend on a tagged
expression are also tagged.
```json
> .init -f "example/ex23.json"
{
"peace": "@PEACE ${'DEFCON 1'}",
"war": "@WAR ${'DEFCON 5'}",
"warMsg": "${'war is ' & war}",
"peaceMsg": "${'Peace is ' & peace}",
"warAndPeaceMsg": "${ [warMsg,peaceMsg]~>$join(' ') }",
"untaggedExpr": "${'A strange game. The only winning move is not to play.'}"
}
> .out
{
"peace": "@PEACE ${'DEFCON 1'}",
"war": "@WAR ${'DEFCON 5'}",
"warMsg": "${'war is ' & war}",
"peaceMsg": "${'Peace is ' & peace}",
"warAndPeaceMsg": "${ [warMsg,peaceMsg]~>$join(' ') }",
"untaggedExpr": "A strange game. The only winning move is not to play."
}
> .init -f "example/ex23.json" --tags=PEACE
{
"peace": "@PEACE ${'DEFCON 1'}",
"war": "@WAR ${'DEFCON 5'}",
"warMsg": "${'war is ' & war}",
"peaceMsg": "${'Peace is ' & peace}",
"warAndPeaceMsg": "${ [warMsg,peaceMsg]~>$join(' ') }",
"untaggedExpr": "${'A strange game. The only winning move is not to play.'}"
}
> .out
{
"peace": "DEFCON 1",
"war": "@WAR ${'DEFCON 5'}",
"warMsg": "${'war is ' & war}",
"peaceMsg": "Peace is DEFCON 1",
"warAndPeaceMsg": "${ [warMsg,peaceMsg]~>$join(' ') }",
"untaggedExpr": "${'A strange game. The only winning move is not to play.'}"
}
> .init -f "example/ex23.json" --tags=PEACE,WAR
{
"peace": "@PEACE ${'DEFCON 1'}",
"war": "@WAR ${'DEFCON 5'}",
"warMsg": "${'war is ' & war}",
"peaceMsg": "${'Peace is ' & peace}",
"warAndPeaceMsg": "${ [warMsg,peaceMsg]~>$join(' ') }",
"untaggedExpr": "${'A strange game. The only winning move is not to play.'}"
}
> .out
{
"peace": "DEFCON 1",
"war": "DEFCON 5",
"warMsg": "war is DEFCON 5",
"peaceMsg": "Peace is DEFCON 1",
"warAndPeaceMsg": "war is DEFCON 5 Peace is DEFCON 1",
"untaggedExpr": "${'A strange game. The only winning move is not to play.'}"
}
```
## Blocked Properties with ⛔
Properties whose names start with "⛔" are completely ignored during template processing. This is useful for:
1. Adding notes or comments that should not be evaluated
2. Temporarily disabling parts of a template
3. Keeping sensitive or debug information in the template but preventing its evaluation
When a property name starts with "⛔", the MetaInfoProducer will skip that property and all of its children, effectively making them invisible to the template processor.
```json
> .init -f "example/ex25.json"
{
"a": 42,
"⛔note": "${$string('should not be evaluated: ') & a}",
"c": "${a}"
}
> .out
{
"a": 42,
"⛔note": "${$string('should not be evaluated: ') & a}",
"c": 42
}
> .init -f "example/ex26.json"
{
"a": 42,
"⛔note": "${$string('Will NOT be evaluated: ') & a}",
"note": "${$string('Will be evaluated: ') & a}",
"config": {
"⛔productionDb": "${$env('DB_PROD_URL', 'mysql://prod.example.com:3306')}",
"productionDb": "${$env('DB_PROD_URL', 'mysql://prod.example.com:3306')}",
"localDb": "mysql://localhost:3306"
},
"features": {
"⛔experimental": {
"enabled": true,
"endpoint": "${$string('https://api.example.com/config/') & 'experimental'}"
},
"experimental": {
"enabled": true,
"endpoint": "${$string('https://api.example.com/config/') & 'experimental'}"
}
}
}
> .out
{
"a": 42,
"⛔note": "${$string('Will NOT be evaluated: ') & a}",
"note": "Will be evaluated: 42",
"config": {
"⛔productionDb": "${$env('DB_PROD_URL', 'mysql://prod.example.com:3306')}",
"productionDb": "mysql://prod.example.com:3306",
"localDb": "mysql://localhost:3306"
},
"features": {
"⛔experimental": {
"enabled": true,
"endpoint": "${$string('https://api.example.com/config/') & 'experimental'}"
},
"experimental": {
"enabled": true,
"endpoint": "https://api.example.com/config/experimental"
}
}
}
```
In these examples, `⛔...` will be completely ignored during template processing - the expression will not be evaluated.
Note that array elements with "⛔" in their values will still be processed, as array indices are numbers, not property names. Only object properties that start with "⛔" are blocked.
# Generative Templates
Templates can contain generative expressions that cause their content to change over time.
For instance the `$setInterval` function behaves exactly as it does in Javascript. Below,
it causes the `incr` function to be called forever, every 10 ms.
```json
> .init -f "example/tail.json"
{
"incr": "${function(){$set('/counter',counter+1)}}",
"counter": 0,
"upCount": "${ $setInterval(incr, 10) }"
}
```
The `tail` command can be used to watch changes on a particular field. Below we use `--tail "/ until counter=5"` to tail
the document root until the value of counter reaches 5, at which point `--tail` stops
producing output, though the `incr` function is still being called every 10 ms.
field:
```json ["data.counter = 5"]
> .init -f "example/tail.json" --tail "/ until counter=5"
Started tailing... Press Ctrl+C to stop.
{
"incr": "{function:}",
"counter": 5,
"upCount": "--interval/timeout--"
}
```
Here was have a more complex example, again using $setInterval to increment a counter
that drives rotation and scrolling of arrays and strings.
```json
> .init -f "example/tailgraphs.yaml"
{
"i": 0,
"mod": "${i%10}",
"ms": 30,
"chars": [
"この世界は広いですね。いろいろな人がいます。",
"日本語はとても美しい言語です。詩を書くには最適。",
"明日もきっといい日になる。希望を持って生きよう。",
"プログラミングは楽しいですね。論理的思考が必要。",
"本を読むことは、心を豊かにする。知識は無限だ。",
"花が咲いて、山は緑が多い。自然は素晴らしい。",
"春夏秋冬、日本の四季ははっきりしている。",
"友達と一緒に遊ぶ時が最も楽しい時間だ。",
"美しい海を見ていると、心が穏やかになる。",
"映画を見るのは、私の趣味の一つです。"
],
"wave": [
"................==................................................................==................................................",
".............o......o..........................................................o......o.............................................",
"..........o............o....................................................o............o..........................................",
"........o................o................................................o................o........................................",
"......o....................o............................................o....................o......................................",
".....o......................o..........................................o......................o.....................................",
"...o..........................o......................................o..........................o...................................",
"..o............................o....................................o............................o..................................",
".o...............................o.................................o...............................o................................",
"..................................o..............................o..................................o..............................o",
"...................................o............................o....................................o............................o.",
"....................................o..........................o......................................o..........................o..",
"......................................o......................o..........................................o......................o....",
".......................................o....................o............................................o....................o.....",
".........................................o................o................................................o................o.......",
"...........................................o............o....................................................o............o.........",
"..............................................o......o..........................................................o......o............",
".................................................==................................................................==..............."
],
"chart": [
"║█████████████████████████████",
"║███████████████████████████████",
"║██████████████████████",
"║████████████",
"║█████████",
"║████",
"║███████",
"║██████████",
"║██████████████████████",
"║████████████████████████"
],
"rot": "${ function($arr){$arr.($substring($, $$.i%$length($)) & $substring($, 0, $$.i%$length($)))} }",
"incr": "${ function(){ $set('/i', i+1) } }",
"upCount": "${ $setInterval(incr, ms) }",
"scroll": "${ function($arr){ $arr[[$$.mod..9]] ~> $append($arr[[0..$$.mod]])} }",
"stars": "${[1..$$.i%20].($=19?'🚀':'⭐')~>$join}",
"wavy": "${ (i;rot(wave)) }",
"rotate": "${ (i;rot(chars)) }",
"scrolled": "${ (i;scroll(chars))}",
"moveBars": "${ (i;scroll(chart))}"
}
```
Setting `--tail /` instructs the REPL to tail the root of the
document "in place" so the screen does not scroll and you can observe
live changes to the json document.
`> .init -f "example/tailgraphs.yaml" --tail /`
<img src="https://raw.githubusercontent.com/geoffhendrey/jsonataplay/main/tailgraphs.gif" width="800" height="400" alt="Alt text for the GIF" />
# Reactive Behavior
Stated is naturally reactive. In the example below, `story` will evaluate when the promises for `partI` and `partII` have both
resolved, simply because `story` has references to `partI` and `partII`, each of which respectively is triggered by the
resolution of the two fetch functions they each depend on.
```json
> .init -f "example/ex21.json"
{
"call": "${function($url){$fetch($url) ~> handleRes}}",
"falcon": "${ call('https://swapi.tech/api/starships/?name=Millennium').result[0].properties.name}",
"han": "${ call('https://swapi.tech/api/people/?name=han').result[0].properties.name}",
"handleRes": "${ function($res){$res.ok? $res.json():$res.status?{'status': $res.status}:$res} }",
"luke": "${ call('https://swapi.tech/api/people/?name=luke').result[0].properties.name}",
"partI": "${ [han, 'piloted the', falcon] ~> $join(' ')}",
"partII": "${ [luke, 'piloted the', xwing] ~> $join(' ')}",
"story": "${ [partI, 'then', partII]~>$join(' ')}",
"xwing": "${ call('https://swapi.tech/api/starships/?name=x-wing').result[0].properties.name}"
}
> .plan
[
"/handleRes",
"/call",
"/falcon",
"/han",
"/luke",
"/partI",
"/xwing",
"/partII",
"/story"
]
> .out
{
"story": "Han Solo piloted the Millennium Falcon then Luke Skywalker piloted the X-wing",
"handleRes": "{function:}",
"call": "{function:}",
"partI": "Han Solo piloted the Millennium Falcon",
"luke": "Luke Skywalker",
"xwing": "X-wing",
"partII": "Luke Skywalker piloted the X-wing",
"han": "Han Solo",
"falcon": "Millennium Falcon"
}
```
## SVG command
The .svg command serves an SVG diagram of the DAG
```json [false, "$='http://localhost:4042'"]
> .init -f "example/ex21.json"
{
"story": "${ [partI, 'then', partII]~>$join(' ')}",
"handleRes": "${ function($res){$res.ok? $res.json():$res.status?{'status': $res.status}:$res} }",
"call": "${function($url){$fetch($url) ~> handleRes}}",
"partI": "${ [han, 'piloted the', falcon] ~> $join(' ')}",
"luke": "${ call('https://swapi.tech/api/people/?name=luke').result[0].properties.name}",
"xwing": "${ call('https://swapi.tech/api/starships/?name=x-wing').result[0].properties.name}",
"partII": "${ [luke, 'piloted the', xwing] ~> $join(' ')}",
"han": "${ call('https://swapi.tech/api/people/?name=han').result[0].properties.name}",
"falcon": "${ call('https://swapi.tech/api/starships/?name=Millennium').result[0].properties.name}"
}
> .svg --port=4042
Server is running on port 4042
"http://localhost:4042"
```
Access the URL from your web browser to view the SVG diagram.
## YAML
Input can be provided in YAML. YAML is convenient because JSONata prorgrams are often multi-line, and json does not
support text blocks with line returns in a way that is readable. For instance if we compare ex12.json and ex12.yaml,
which is more readable?
```json
falken$ cat ex12.json
{
"url": "https://raw.githubusercontent.com/geoffhendrey/jsonataplay/main/games.json",
"selectedGame": "${game.selected}",
"respHandler": "${ function($res){$res.ok? $res.json():{'error': $res.status}} }",
"game": "${ $fetch(url) ~> respHandler ~> |$|{'player':'dlightman'}| }"
}
```
In YAML the `respHandler` function can be written as a text block, whereas in JSON it must appear on a single line.
```bash
falken$ cat ex12.yaml
```
```yaml
url: "https://raw.githubusercontent.com/geoffhendrey/jsonataplay/main/games.json"
se