integreat/README.md

Node.js integration layer

# ![Integreat](media/logo.png)

An integration layer written in TypeScript.

[![npm Version](https://img.shields.io/npm/v/integreat.svg)](https://www.npmjs.com/package/integreat)
[![Maintainability](https://api.codeclimate.com/v1/badges/a5bd9841a47ff9f74577/maintainability)](https://codeclimate.com/github/integreat-io/integreat/maintainability)

The basic idea of Integreat is to make it easy to define how to send data to
and receive data from a set of [**services**](#services), and expose them
through a well defined interface, abstracting away the specifics of each
service.

There are a few concepts that makes this possible:

- [**Transporters**](#transporters) and [**adapters**](#adapters) speak the
  language of different types of services and standards of data exchange, and
  does the basic translation to and from the structures used by Integreat. You
  deal with familiar JavasScript objects, arrays, and primitive data types,
  regardless of what the service expects.
- [**Mutation pipelines**](#mutations) let you define how the data coming from
  or going to a service should be transformed. This includes changing the
  overal structure, renaming properties, transforming and filtering values with
  transformer functions, etc. You may also provide your own transformer
  functions.
- [**Schemas**](#schemas) serve as a common normalization of data between
  services. You define your own schemas and mutate data to and from them,
  enabling inter-service sharing of data. If you have data in one schema, you
  may send it to any service where you have set up the right mutations for this
  schema, again abstracting away all service details.

All configuration is done through basic JSON-friendly structures, and you
define your services with different endpoints, mutation pipelines,
authentication schemes, etc.

Your configuration is spun up as an Integreat instance. To send and retrieve
data, you dispatch [**actions**](#actions) to your instance and get
[**response**](#action-response) objects back. You may define [jobs](#jobs) to
run simple actions or longer "flows" consisting of several actions with
conditions and logic. You may also configure [queues](#queues) to have actions
run in sequence or on a later time.

```
                   ____________________________________________________
                  |                                                   |
                  |                Integreat instance                 |
Action ----|      |                                                   |
           |-> Dispatch <-> Schema <-> Mutation <-> Adapter <-> Transporter <-> Service
Response <-|      |                                                   |
                  |___________________________________________________|
```

To deal with security and permissions, Integreat has a concept of an ident.
Other authentication schemes may be mapped to Integreat's ident scheme, to
provide data security from a service to another service or to the dispatched
action. A ground principle is that nothing that enters Integreat from an
authenticated service, will leave Integreat unauthenticated. What this means,
though, depends on how you define your services.

# Table of contents

1. [Usage](#usage)
   1. [Install](#install)
   2. [Basic example](#basic-example)
2. [Integreat concepts](#integreat-concepts)
   1. [Services](#services)
   2. [Transporters](#transporters)
   3. [Adapters](#adapters)
   4. [Authenticators](#authenticators)
   5. [Mutations](#mutations)
   6. [Schemas](#schemas)
   7. [Actions](#actions)
   8. [Jobs](#jobs)
   9. [Queues](#queues)
   10. [Middleware](#middleware)
3. [Debugging](#debugging)

# Usage

## Install

Requires node v18.

Note: This package is native [ESM](https://nodejs.org/api/esm.html). See this
guide on how to [convert to or use ESM packages](https://gist.github.com/sindresorhus/a39789f98801d908bbc7ff3ecc99d99c).

Install from npm:

```
npm install integreat
```

You will probably also need some [transporters](#transporters) and
[adapters](#adapters), and the basic transformers in
[`integreat-transformers`](https://github.com/integreat-io/integreat-transformers).

## Basic example

The following is the "hello world" example of Integreat. As most hello world
examples, this is a bit too trivial a use case to demonstrate the real
usefulness of Integreat, but it shows you the simplest setup possible.

Here, we fetch cat facts from the API endpoint
'https://cat-fact.herokuapp.com/facts', which returns data in JSON and requires
no authentication. The returned list of facts are mutated and cast to the
`fact` schema. We only fetch data _from_ the service, and no data is sent _to_
it.

```javascript
import Integreat from 'integreat'
import httpTransporter from 'integreat-transporter-http'
import jsonAdapter from 'integreat-adapter-json'

const schemas = [
  {
    id: 'fact', // The id of the schema
    shape: {
      // The fields of the type
      id: 'string', // An id field will always be included, but we define it here for readability
      text: 'string', // The text of the cat fact
      createdAt: 'date', // The created date (`createdAt` and `updatedAt` will always be dates)
    },
    access: { allow: 'all' }, // No access restrictions
  },
]

const services = [
  {
    id: 'catfact', // The id of the service
    transporter: 'http', // Use the http transporter
    adapters: ['json'], // Run the request and the response through the json adapter
    options: {
      transporter: {
        // Options for the transporter
        uri: 'https://cat-fact.herokuapp.com/facts', // Only the uri is needed here
      },
    },
    endpoints: [
      {
        match: { action: 'GET', type: 'fact' }, // Match to a GET action for type 'fact'
        mutation: {
          $direction: 'from', // We're mutating data _from_ the service
          // Here we're mutating `response.data` and "setting it back" where we found it ...
          'response.data': [
            'response.data[]',
            {
              $iterate: true, // Mutate each item in an array
              id: '_id', // The id is called `_id` the data from the service
              text: 'text', // text is called `text`
              createdAt: 'createdAt', // Creation date is called `createdAt`
            },
          ],
        },
      },
    ],
  },
]

// Create the Integreat instance from our definitions and provide the
// transporters and adapters we require.
const great = Integreat.create(
  { schemas, services },
  { transporters: { http: httpTransporter }, adapters: { json: jsonAdapter } },
)

// Prepare an action to fetch all cat facts from the service `catfact`
const action = { type: 'GET', payload: { type: 'fact', service: 'catfact' } }

// Dispatch the action and get the response
const response = await great.dispatch(action)
```

The `response` object will look like this:

```javascript
{
   status: 'ok',
   data: [
    {
      id: '58e008780aac31001185ed05',
      $type: 'fact',
      text: 'Owning a cat can reduce the risk of stroke and heart attack by a third.',
      createdAt: new Date('2018-03-29T20:20:03.844Z')
    },
    // ...
  ]
}
```

# Integreat concepts

As mentioned in the introduction, the building blocks of Integreat are
services, transporters and adapters, mutation pipelines, and schemas.

## Services

A service is the API, database, FTP server, queue, etc. that you want to get
data from and/or set data to. We pass on a set of service definitions to
Integreat, specifying what transporter, adapters, authentication schemas it
requires, in adition to defining the different endpoints available on the
service, how they should be called, and how data should be mutated in each
case.

We'll get back to the details of all of this in turn, but first we want to
highlight how central the concept of a service is to Integreat. Basically, in
Integreat "everything is a service". A simple REST/JSON API is a service, a
database is a service, and everything external you want to communicate with are
services. Want to set up a queue to handle actions one by one? That's a
service. Want to cache data in a memory store? That's a service. Want to
schedule actions to run on intervals? That's a service to. By simply defining
services and their specifics, you may set up a variety of different types of
configurations with the same few building blocks. This is very powerful as
soon as you get into the right mindset.

Services are configured by service definitions, and tells Integreat how to
fetch data from a service, how to mutate this data to schemas, and how to send
data back to the service.

The service definition object includes the transporter id, adapter ids, any
authentication method, the endpoints for fetching from and sending to the
service, mutations that data to all endpoints will pass through, and options
for transporters, adapters, etc.

```javascript
{
  id: <service id>,
  transporter: <transporter id>,
  adapters: [<adapter id>, <adapter id>, ...],
  auth: <auth config>,
  meta: <type id>,
  options: {...},
  mutation: <mutation pipeline>,
  endpoints: [
    <endpoint definition>,
    ...
  ]
}
```

Service definitions are passed to Integreat on creation through the
`Integreat.create()` function. There is no way to change service defintions
after creation.

See [mutations](#mutations) for a description of how to define the mutation
pipeline for a service.

The `auth` property should normally be set to the id of an
[auth definition](#service-authentication), if the service requires
authentication. In cases where the service is authenticated by other means,
e.g. by including username and password in the uri, set the `auth` property to
`true` to signal that this is an authenticated service. For services accepting
incoming actions, `auth` should be set to an object with
`{ outgoing: <auth id | true>, incoming: <auth id | true>}`. To accept several
incoming actions, provide an array of `<auth id | true>`, and they will be run
from first to last until one of them returns an ident or an error other than
`noaccess`.

> [!NOTE]
> When connecting to a service for listening, the `outgoing` auth is
> used. `incoming` is only used for validating the actions being dispatched
> "back" from the service.

In `options`, you may provide options for transporters and adapters. It is
merged with the `options` object on the endpoint. See
[the `options` object](#options-object) for more on this.

### Endpoints

A service will have at least one endpoint, but often there will be several.
Endpoints are the definitions of the different ways Integreat may interact with
a service. You decide how you want to set up the endpoints and what is the
right "endpoint design" for a service, but there might be one endpoint for each
operation that can be done with a type of data.

For example, let's say you have a simple REST API with blog articles and
authors. There will most likely be an endpoint to fetch all (or some) articles,
one endpoint for fetching one article by id, one endpoint for creating an
article, one for updating an article, and so on. And you'll have similar
endpoints for authors, one endpoint for fetching all, one for fetching one by
id, one endpoint for creating an author, etc. As this is REST, each endpoint
will address a different combination of urls and http verbs (through the
transporter).

As another example, you may be accessing a database of articles and authors
directly. The configuration details will be very different than for a REST API,
but you'll probably have the same endpoints, fetching all articles, fetching
one, creating, updating, and the same all over for users. Instead of urls and
http verbs, as for REST, these endpoints will address different databases and
different database operations (through the transporter).

> [!NOTE]
> This is not to say that Integreat requires you to set up endpoints
> exactly as described in these examples, it might be that you would like to
> set up an endpoint that handles many of these cases. The intention here is
> just to give you an understanding of what an endpoint is in Integreat.

When you dispatch an action, Integreat will figure out what service and what
endpoint to send the action to. The target service is often specified in the
action payload with the `targetService` (or shorthand `service`) property, but
if not, the default service of the schema specified with the payload `type`
property, will be used.

The `targetService` property will be set on the the action payload when it is
sent to the transporter, before it goes through the middleware. There are two
exceptions to this, however. `targetService` will not be set for a queued
action (and it will not be removed if it is already set), and it will not be
set for actions where the `meta.options.doSetTargetService` flag is set to
`false`.

The matching to an endpoint is done by finding the endpoint whose `match`
object matches the action with most accuracy. The rules of the endpoint
matching is describe in more details [below](#match-properties).

Here's the format of an endpoint definition:

```javascript
{
  id: <endpoint id>,
  match: {
    type: <schema id>,
    scope: <'collection'|'member'|'members'|'all'>,
    action: <action type>,
    params: {...},
    incoming: <boolean>,
    conditions: [...]
  },
  validate: [
    {
      condition: <mutation pipeline>,
      failResponse: <response object>
    }
  ],
  mutate: <mutation pipeline>,
  adapters: [<adapter id>, <adapter id>, ...],
  auth: <auth config>,
  allowRawRequest: <boolean>,
  allowRawResponse: <boolean>,
  castWithoutDefaults: <boolean>,
  options: {...},
}
```

All of these properties are optional. An empty endpoint defintion object will
match anything, pass on the action to the transporter untouched, and relay any
response coming back. This might be what you need, but often you'll want to
specify a few things:

- `id`: The endpoint may have an id, which you may use to specify that you want
  an action to go to this particular id. However, most of the time you'll set
  up the `match` object so that Integreat will decide what endpoint to use for
  the action you dispatch.
- `match`: The match object is used to decide the right endpoint for an action.
  More one this in the [Match properties](#match-properties) section.
- `validate`: This is an array of condition that have to be met in order for
  Integreat to proceed with the endpoint. The `condition` is a mutation
  pipeline that should return a truthy value for the validation to pass. Any
  falsy value will cause the validation to fail. If `validate` is missing or
  an empty array, no validation will be done. This may sound similar to `match
  `, but `validate` is only processed after a match is found, and if the
  validation fails, no other endpoint is considered. On a failing validation,
  the `failResponse` is returned as the response from this action, or a
  `badrequest` response if no `failResponse` is provided. There's also a
  shorthand, where you set `failResponse` to a string, which will be the
  `error` message of the `badrequest` response. The response is passed through
  the mutation pipeline.
- `mutate`: A mutation pipeline for the endpoint. The pipeline is run for both
  actions going to a service and the response coming back, so keep this in mind
  when you set up this pipeline. See [Mutation pipelines](#mutations)
  for more on how to define the mutation. `mutation` is an alias for `mutate`.
- `adapters`: An array of adapter ids that will be appended to the array of
  adapters set on the service.
- `auth`: Auth config that will override the `auth` config on the service. See
  [description of `auth` under Services](#services) for more on this. The
  endpoint `auth` will only apply in cases where we have an endpoint, like when
  we're sending a request to a service or receiving an incoming request, but
  when we're e.g. connecting to a service to start listening, the `auth` on the
  service will be used. This also goes for incoming requests where the
  transporter does not provide an action with the auth attempt.
- `allowRawRequest`: When set to `true`, payload `data` sent to this endpoint
  will not by cast automatically nor will an error be returned if the data is
  not typed.
- `allowRawResponse`: When set to `true`, response `data` coming from this
  endpoint will not be cast automatically nor will an error be returned if the
  data is not typed. The default is `false`, expcept for incoming endpoints
  (endpoints where `match` object has `incoming: true`) where the default value
  is `true`.
- `castWithoutDefaults`: Set to `true` when you don't want to set default
  values on casted data. This also means no `id` will be generated and no
  `createdAt` or `updatedAt` will be set – when any of these are missing in
  the data. Default is `false`.
- `options`: This object is merged with the `options` object on the service
  definition, and provide options for transporters and adapters. See
  [the `options` object](#options-object) for more on this.

#### Match properties

An endpoint may specify none or more of the following match properties:

- `type`: When set, the endpoint will only be used for actions with the
  specified schema type (the schema's id). `type` may also be an array of
  types, matching any one of the schemas in the list.
- `scope`: May be `member`, `members`, `collection`, or `all`, to specify that
  the endpoint should be used to request one item (member) by id, several items
  by ids (members), or an entire collection of items. Setting this to `member`
  or `members` will only match actions with a payload `id` property, and the
  `id` should be an array of ids for `members`. Not setting this property, or
  setting it to `all`, signals an endpoint that will work for all scopes.
- `action`: May be set to the type string of an action. The endpoint will match
  only actions of this type. When this is not specified, any action type will
  match. `action` may also be a list of action types, matching any of these.
- `params`: This object should list all params that this endpoint supports. A
  param in this context is any property on the action payload except `type`,
  `id`, or `data`. Use the param name as key on this object and set the value
  to `true` if it is required, and `false` if it is optional. When matching
  endpoints, an action will only match if it has all the required params, and
  in case several match, the endpoint with more specified params will be
  preferred.
- `incoming`: If this is `true`, it will only match incoming actions, if
  `false` only outgoing, and if not set, it will match both.
- `conditions`: An array of mutation pipelines that will be run on the action
  to see if it's a fit for this endpoint. If all pipelines return a truthy
  value, the endpoint is chosen (given that the other match properties also
  match). We rely on JavaScript definition of 'truthy' here, so any value that
  is not `false`, `null`, `undefined`, `0`, `NaN`, or `''` will be considered
  truthy.

> [!NOTE]
> There used to be a `filters` property on the endpoint match object. It is
> still supported, but it's deprecated and will be removed in v1.1. Please use
> `conditions` instead.

> [!NOTE]
> Editor's note: Describe what incoming actions are, and give more details on
> filters.

There might be cases where several endpoints match an action, and in these
cases the endpoint with the highest level of specificity will be used. E.g.,
for a `GET` action asking for resources of type `entry`, an endpoint with both
`action: 'GET'` and `type: 'entry'` is picked over an endpoint matching all
`GET` actions regardless of type. For `params` and `filters` this is decided by
the highest number of properties on these objects.

The order of the endpoints in the `endpoints` list matters only when two
endpoints are equally specified with the same match properties specified. Then
the first one is used.

When no match properties are set, the endpoint will match any actions, as long
as no other endpoints match.

Finally, if an action specifies the endpoint id with the `endpoint`
[payload property](#payload-properties), this overrides all else, and the
endpoint with the id is used regardless of how the match object would apply.

Example service definition with endpoint match object:

```javascript
{
  id: 'entries',
  transporter: 'http',
  endpoints: [
    {
      match: {
        type: 'entry',
        action: 'GET',
        scope: 'collection',
        params: {
          author: true,
          archive: false
        }
      },
      options: {
        transporter: {
          uri: 'https://example.api.com/1.0/{author}/{type}_log?archive={archive}'
        }
      }
    }
  ],
  // ...
}
```

### Options object

A service defintion may have `options` object in two places: Direction on the
service definition and on any of the endpoints. When an action is sent to an
endpoint, the combination of the two `options` are used. Also, there may be
different options for the transporter and for the adapters.

Example of an `options` object set on the service definition:

```javascript
{
  id: 'entries',
  options: {
    uri: 'https://ourapi.com/v1',
    transporter: {
      method: 'POST',
      incoming: { port: 3000 }
    },
    adapters: {
      xml: { namespaces: { ... } },
      // ...
    }
  }
}
```

Any properties set directly on the `options` object or on a `transporter`
property, are treated as options for the transporter. If there are properties
on both the `options` and a `transporter` object, they will be merged, with the
`transporter` object having priority if conflicts. This is a shallow merge, so
objects used in the options will not be merged.

In the example above, the options passed to the transporter will include `uri`,
`method`, and `incoming`.

The `incoming` object on the transporter options is a bit special, as it holds
separate options for transporters that support incoming requests trough the
`listen()` method. If there are `incoming` objects on both the `options` and
`transporter` objects, they will be merged, again with priority to the one on
the `transporter` object.

Note that we recommend setting transporter options on the `transporter` object
for clarity, but both will work.

Adapter options may be given in an `adapters` object, where each adapter may
have its own options, set with the id of the adapter as a key. In the example
above, the `xml` adapter will be given the `namespaces` object. A requirement
for this, is that the adapter actually have an id. Adapters provided directly
on service definition may not have an id, but all adapters that are referenced
by an id, will also be given options set on that id, which is the common behavior.

Finally, when all this sorting have been done on options from both the service
definition and an endpoint, the two options structures are merged before being
used. Here, the endpoint options take priority, so that you may set a general
option on the service, and override it on the endpoint.

Example of endpoint options overriding service options:

```javascript
{
  id: 'entries',
  options: {
    transporter: {
      uri: 'https://ourapi.com/v1',
      method: 'GET',
    }
  },
  endpoints: [
    {
      match: { ... }
    },
    {
      match: { ... },
      options: {
        transporter: {
          method: 'POST'
        }
      }
    }
  ]
}
```

Here, the first enpoint will be given `method: 'GET'`, while the next will get
`method: 'POST'`.

Before actions are passed through mutations and finally passed to the
transporter, the merged transporter options is set on an `options` property in
the `meta` object of the action. This way, you may also mutate these options
before they reach the transporter.

### Service authentication

This definition format is used to authenticate with a service:

```javascript
{
  id: <id>,
  authenticator: <authenticator id>,
  options: {
    // ...
  },
  overrideAuthAsMethod: <auth-as method>,
}
```

- `id`: The id used to reference this authentication, especially from the
  [service definition](#services).
- `authenticator`: The id of an [authenticator](#authenticators) used to
  authenticate requests. Integreat comes with a few basic ones built in, and
  there are others available.
- `options`: An object of values meaningful to the authenticator. See the
  documentation of each authenticator to learn how it should be configured.
- `overrideAuthAsMethod`: Transporters specify a default method for getting an
  auth object that makes sense for authenticating with the service. For
  instance, the HTTP transporter has `asHttpHeaders` as the default, to get the
  relevant auth headers to send with the request. With `overrideAuthAsMethod`,
  you may override this in the service auth definition when relevant. Default
  value is `undefined`, meaning "no override". Note that we say "method" here,
  but the value is a string with the name of the auth-as method to use.

The authenticator is responsible for doing all the heavy-lifting, based on the
options provided in the service authentication definition.

### Configuring service metadata

Integreat supports getting and setting metadata for a service. The most common
use of this is to keep track of when data of a certain type was last synced.

Some services may have support for storing their own metadata, but usually you
set up a dedicated service for storing other services' metadata. A few
different pieces goes into setting up a meta store:

- A meta schema with the fields available as metadata
- A service for storing metadata, with an endpoint suporting the metadata
  schema
- Possible a metadata mutation for the metadata service

When all of this is set up, you activate the metadata on the service the
metadata will be stored for, by setting the `meta` property to the id of the
schema defining the metadata fields. The `service` set on the schema will tell
Integreat what service to get and set the metadata from/to.

The schema will look something like this:

```javascript
{
  id: 'meta', // You may give it any id you'd like and reference it on the `meta` prop on the service
  service: <id of service handling the metadata>,
  shape: {
    <metadataKey>: <type string>,
    // ...
  }
}
```

To get or set metadata, use [`GET_META`](#get_meta) and [`SET_META`](#set_meta)
with the service you are getting metadata from as the `service`. Integreat will
figure out the rest.

## Transporters

A transporter handles all the details of sending and receiving data to and from
a service. When dispatching an action to a service, the action will be handled
in a relevant manner for the type of service the transporter supports, e.g.
sending an http requrest for the HTTP transporter, or doing a query to a
database for the MongoDb transporter. Some transporters may also support
listening to a service, e.g. the HTTP transporter listing for incoming requests
or the MQTT transporter subscribing to events on a topic.

Integreat has transporters for some common cases, and more may come:

- [Bull](https://github.com/integreat-io/integreat-transporter-bull)
- [FTP](https://github.com/integreat-io/integreat-transporter-ftp)
- [HTTP](https://github.com/integreat-io/integreat-transporter-http)
- [MongoDb](https://github.com/integreat-io/integreat-transporter-mongodb)
- [MQTT](https://github.com/integreat-io/integreat-transporter-mqtt)
- [Redis](https://github.com/integreat-io/integreat-transporter-redis)

You may write your own transporters if your case is not covered by any of
these. Documentation on developing transporters are coming.

Integreat will handle the transporters based on you configurations, but there
are some specifics to each transporter, like HTTP needing an `uri` option or
MongoDb needing a `collection` option. See the documentation of each
transporter for more.

## Adapters

Adapters are working together with transporters to prepare the incoming and
outgoing data in accordance with the type of services they support.

As an example, the HTTP transporter will return data from a response as a
string, since there is no common way to treat the response body. So for a JSON
API, you will configure the JSON adapter to make sure the data from the
mutations are sent as a JSON string, and that the JSON comming back from the
service is parsed before mutation starts. For a service using XML, you would
instead set up the XML adapter, and perhaps also the SOAP adapter, to again
stringify and parse the data going back and forth.

The MongoDb transporter, on the other hand, does not require any adapters, as
documents from the database will always come as arrays and object, and may be
fed directly into the mutation pipelines.

Integreat currently have the following adapters:

- [CSV](https://github.com/integreat-io/integreat-adapter-csv)
- [JSON](https://github.com/integreat-io/integreat-adapter-json)
- [SOAP](https://github.com/integreat-io/integreat-adapter-soap)
- [Url encoded form data](https://github.com/integreat-io/integreat-adapter-form)
- [URI templates](https://github.com/integreat-io/integreat-adapter-uri)
- [XML](https://github.com/integreat-io/integreat-adapter-xml)

You may write your own adapters as well, and documentation on this is coming.

## Authenticators

At its simplest, an authenticator will provide necessary credientials to an
outgoing action, or an ident to an incoming action. Some authenticators do this
based only on the options provided, while others will do a more complex dance
with the service or a third-party service, like with OAuth2.

When [setting up a service](#services), you may provide it with an auth id that
refers to a [service authentication definition](#service-authentication), that
again refers to an authenticator by id. The service auth definition also holds
options for the authenticator, so when assigning an auth id to a service,
you're assigning it an authenticator with those specific options. Another
service may use the same authenticator, but with different options, and you
would set this up with a different service authentication definition.

Authentication for outgoing actions are done when sending the action. When
authenticated, an auth object is retrieved with the auth-as method specified on
the transporter (e.g. `asHttpHeaders` for the http transporter), or on the
`overrideAuthAsMethod` in [auth options](#service-authentication) if set. The
auth object is passed to the transporter on the action `meta.auth` prop. It is
applied just before sending it, though, so it will be available to service
middleware, but not to the mutation pipeline. This is done to expose
credentials in as few places as possible. If you however _want_ to have the
auth object in mutations, set `authInData` to `true` on the service or
endpoint options, and authentication will be done in the `preflightAction`
step instead, making it available on `meta.auth` throughout the entire mutation pipeline.

For incoming actions, authentication is done when a listening action calls the
`authenticate()` callback. The `validate()` method on the authenticator is used
here, which will provide the transporter with an authorized ident.

Available authenticators:

- `action`: Will dispatch an action and use the response data to create an
  authentication. The `options` should have the action type as `action`, and
  the entire payload as `payload`. The response data should have an `auth`
  object, that will be used directly, and an optional `expire` that is a
  timestamp on which the auth will expire, in milleseconds since Epoc
  (1970-01-01). If no `expire` is returned, the `expireIn` option will be used
  if present. It is given as milliseconds before the auth should expire, or a
  `ms` string like `'1h'`.
- `http`: Supports http native authentications, like `Basic` and `Bearer`. It's
  included with the
  [HTTP transporter](https://github.com/integreat-io/integreat-transporter-http).
- `ident`: Will always grant access and `validate()` will return an ident with
  the id provided in `identId` on the `options` object, or `'anonymous'` if no
  `identId` is provided. This is built into Integreat.
- `options`: Will pass on the options as authentication, so whatever you
  provide here is the authentication. What options to provide, then, is
  depending on what the relevant transporter requires. For outgoing actions,
  the options are provided as is. Incoming action are validated agains the
  values given in the options (the keys may be dot notation paths in this
  case, and `identId` is excluded). An ident with the `identId` from the
  options as `id`, is returned if the action matches. This is built into
  Integreat.
- `token`: A simple way of authenticating with a given token. For HTTP
  requests, the token will be provided as a `Authorization` header, and a
  configurable prefix like `Basic` or `Bearer`. This is built into Integreat.
- [`jwt`](https://github.com/integreat-io/integreat-authenticator-jwt): Will
  generate and encode a JavaScript Web Token (JWT) based on the options.
- [`oauth2`](https://github.com/integreat-io/integreat-authenticator-oauth2):
  Will run the balett of calling different OAuth2 endpoints and receive a token
  based on the provided options.

## Mutations

Both on the service and on endpoints, you define mutation pipelines. The
service mutation is run before the endpoint mutation for data coming from a
service, and in the oposite order when going to a service.

A nice - but sometimes complicated - thing about mutations, is that they are
run in both directions. They are by default defined for mutating data coming
_from_ a service, and will be run in reverse for data going _to_ a service. In
some cases this reversing of the pipeline will work as expected without
modifications -- you define the mutation pipeline for data coming _from_ the
service, and the reversed pipeline works _to_ as well. But many times you need
to make adjustments and sometimes you'll have to have separate steps based on
the direction. We'll get into more details in the following.

A mutation pipeline consists of one or more steps that the data will go
through, before coming out on the other in the desired shape. It helps
picturing this as an actual pipeline. After each step, data will be in a
different shape, and this is the input to the next step.

You define a pipeline in Integreat with an array, although for a pipeline with
only one step, you may skip the array for simplicity.

Each step may be one of the following:

- [**A dot notation path**](#dot-notation-paths), e.g. `path.to.data`. The data
  at that path will be extracted, and will be provided as the data to the next
  step in the pipeline. When going in reverse, the data will be set on that
  path instead.
- **A mutation object** is an object that basically describes the object you
  want as a result, where the keys are dot notation paths and the values are
  mutation pipelines. Each pipeline on the mutation object will be run on the
  data, and then set on the path, resulting in an object that will be passed on
  to the next step. Setting `$iterate: true` on the object will cause it to
  iterate over items in an array, otherwise it will be applied to the array.
  Setting `$modify: true` will cause any properties on an object in the
  pipeline not set in the mutation, to be included, much like the spread in
  JavaScript. Setting `$modify` to a path works the same, but you will spread
  from the object at the path (`$modify: true` is equal to `$modify: '.'`).
- **A transform object** letting you run a transformer function on the data,
  e.g. `{ $transform: 'number' }` to transform the value into a number, or
  `undefined` if not possible.
- **A filter object** that will run a transformer function on the data and
  filter away any items not resulting in a truthy value. As an example,
  `{ $filter: 'boolean' }` will filter away anything that is not convertable to
  `true` in JS rules. When applied to an array, you'll get an array where items
  are filtered away. For an object or a plain value, filtering away will means
  `undefined` is passed on to the next step in the pipeline.
- **An if object** that runs a `then` pipeline if the provided pipeline returns
  truthy, and the `else` pipeline if it returns falsy.
- **A cast object**, e.g. `{ $cast: 'author' }` that casts the data into a
  schema, removing all properties that is not part of the shape of the schema,
  and transforming all values to the expected types or `undefined` if not
  possible. (Not available until v1.0)

### Dot notation paths

At its most basic, a dot notation path is just a property key, like `content`.
You may dive into a data structure by adding a key from the next level,
separated by a dot, like `content.articles`. With an object like this:

```javascript
{
  content: {
    articles: [{ id: '1' }, { id: '2' }],
    authors: [{ id: 'john' }]
  }
}
```

... the path `content.articles` will give you the array
`[{ id: '1' }, { id: '2' }]`.

You may add brackets to the path to traverse into arrays, e.g.
`content.articles[0]` will give you the object `{ id: '1' }`, and
`content.articles[0].id` will give you `'1'`.

Empty brackets, like `content.articles[]` will ensure that you get an array
back. If the data at the path is an array, this will return the same as
`content.articles`, but if the path returns an object or a plain value, it will
be returned in an array.

When mapping data _to_ a service, the paths are used to reconstruct the data
format the service expects. Only properties included in the paths will be
created.

Arrays are reconstructed with any object or value at the first index, unless a
single, non-negative index is specified in the path.

You may use a carret `^` to go one level up -- to the parent -- in the data
(after going down), so after `content.articles`, the path `^.authors` will
return `[{ id: 'john' }]`. Arrays count as one level, so after
`content.articles[0]` you will need to go up twice like so: `^.^.authors`.

A double carret `^^` takes you to the top -- the root -- so after
`content.articles[0].id`, `^^.content.authors` returns `[{ id: 'john' }]`.

Carret notations -- parents and roots -- does not currently work in reverse,
but they might in a future version.

### Non-values

The behavior of some transformers are based upon certain values being
non-values. E.g. `{ $alt: [<pipeline 1>, <pipeline 2>] }` will use the value
from the first pipeline if it returns a value, otherwise the value from the
second pipeline, meaning it will check for non-values. By default `null`,
`undefined`, and `''` (empty string) are non-values. By setting the `nonvalues`
param to an array of values in the defintions object you pass to
`Integreat.create()`, you may specify your own non-values.

If you don't want empty string to a non-value, for instance, you do this:

```javascript
const great = Integreat.create({
  nonvalues: [null, undefined],
  // ... other definitions
})
```

## Schemas

A central idea to Integreat, is that any integration has two sides; the getting
of data from one service and the sending of data to another. Instead of setting
up an integration directly from A to B, you have a schema in middle, and
configure how data from A will be mutated to a schema, and then have data in
that schema will be mutated and sent to B.

This is a useful abstraction, and if you ever need to change one side, you can
do so without involving the other side. If you need to switch out service B
with service C, you can do so without involving the configuration of service
A, or you can send data to both B and C, using the same setup for service A.

To be clear, you can setup flows without schemas in Integreat, but then you may
loose this flexibility and maintainability.

A schema describe the data you expected to get out of Integreat, or send
through it. You basically define the fields and their types, and may then cast
data to that shape. Note that data on an action for a specified type, will be
automatically cast to that type.

```javascript
{
  id: <schema id>,
  plural: <the id in plural>,
  service: <the default service for this schema>,
  shape: {
    <fieldId>: <field type>,
    <fieldId>: {
      $type: <field type>,
      default: <default value>
      const: <value that will override any other value>
    },
  },
  access: <access def>
}
```

- `id`: The id of the schema, used to reference it in actions (the payload
  `type`), when casting to the schema with `{ $type: '<schema id>' }`, and to
  signal what schema a data object is cast to (the `$type` prop on typed data
  items). The convention is to use singular mode for the `id`, e.g. if your
  defining a schema for articles, you would give it the id `'article'`.
- `plural`: When the plural of `id` is not simply a matter of adding an `'s'`,
  you may specify the plural mode here. E.g. `id: 'entry'` would have
  `plural: 'entries'`. This is not used by Integreat right now, but it may be
  used in the future for error messages, generating APIs from schemas, etc.
- `service`: You may specify a default service for the schema when it makes
  sense. This allows you to dispatch an action for a type without specifying
  the target service, e.g. `{ type: 'GET', payload: { type: 'article' } }`,
  and have Integreat use the default service. This is a way of hiding
  configuration details from the code dispatching the actions, and you may
  also change the default service without changing the dispatching code if
  need be. You may always override this by specifying a `service` on the
  action payload.
- `shape`: This is where you define all the fields, see
  [the section below](#the-shape-of-a-schema).
- `generateId`: Set this to `true` to generate a unique id for the `id` field
  when the data being cast does not provide an `id`. Default is `false`, which
  will just set `id: null`. The id will be 36 chars and consist of A-Z, a-z,
  0-9, underscore `'_'`, and hyphen `'-'`.
- `access`: Integreat lets you define authorization schemes per schema. All use
  of data cast to a schema will then be controlled by the rules you set here.
  See [Access rules](#access-rules) below for details on these rules. Note that
  `access` is optional, but when you get data from a service where any form of
  authentication is used to access the data, you will not be able to do
  anything with the data unless you cast it to a schema with `access` set up
  (or specifically says that you allow raw data from that endpoint).

### The shape of a schema

The shape is defined by an object where each key is the id of a field, which
may contain only alphanumeric characters, and may not start with a digit. A
schema cannot have the same id as a primitive type (see list below).

The values on this object define the types of the fields and a few other
optional features:

```javascript
{
  $type: <field type>,
  default: <default value>
  const: <value that will override any other value>
}
```

The `$type` prop sets the type of the field. The available primitive types, are
`string`, `integer`, `float` (or `number`), `boolean`, and `date`. A field may
also have another schema as its type, in which case the id of the schema is set
in `$type`. An example can be an
`article` schema with an `author` field of type `user`, referring to a schema
with id `user`. When casting the `article`, data on the `author` prop will be
cast with the `user` schema.

The `default` value will be used when the field is `undefined`, `null`, or not
preset in data object being cast to this schema. If `default` is set to a
function, the function will be run with no argument, and the returned value is
used as the default value. When no `default` is given, `undefined` is used.

The `const` value override any value you provide to the field. It may be useful
if you want a field to always have a fixed value. Just as for `default`, you
may set it to a function, in which case the function will be run without
arguments and the returned value will be used.

If both `const` and `default` are set, `const` will be used.

When only setting the field type, you don't need to provide the entire object,
you can just provide the type string.

Example schema:

```javascript
{
  id: 'article',
  shape: {
    id: 'string', // Not needed, as it is always provided, but it's good to include for clarity
    title: { $type: 'string', default: 'Unnamed article' },
    text: 'string',
    readCount: 'integer',
    archived: { $type: 'boolean', default: false },
    rating: 'float',
    createdAt: 'date',
    updatedAt: 'date'
  },
  access: 'all'
}
```

Note that if you provide the `id` field, it should be set to type `'string'` or
Integreat will throw. The same happens if you set `createdAt` or `updatedAt` to
anything else than the type `'date'`. If you don't include these fields,
Integreat will include the `id` for you, but not `createdAt` or `updatedAt`.

### Typed data

When data is cast to a schema, the data will be in the following format:

```
{
  id: <string>,
  $type: <schema>,
  createdAt: <date>,
  updatedAt: <date>,
  <key>: <value>,
  <key>: { id: <string>, $ref: <schema> },
  <key: [{ id: <string>, $type: <schema>, ... }],
  ...
}
```

- `id`: The id is mandatory and created by Integreat when it is not included in
  the schema. If you don't map anything to the id prop, it will be set to
  `null`, unless the schema is set up with `generateId: true`, in which case a
  universally unique id will be generated for you.
- `$type`: Set to the id of the schema by Integreat. This is a signal that the
  data has been cast.
- `createdAt`: This is not mandatory, but has special meaning. When a schema
  has a `createdAt` field, but the date is not set in the data, it will be set
  to the same as `updatedAt` (if provided) or to the current date/time.
- `updatedAt`: Just as `createdAt`, this is not mandatory. When a schema has
  an `updatedAt` field, and the date is not set in the data, it will be set to
  the same as `createdAt` (if provided) or the current date/time.
- `<key>`: Then follows the values of all the fields specified in the schema.
  Any value not provided in the data will be set to their default value, unless
  `castWithoutDefaults` is set to `true` in
  [the endpoint definition](#endpoints). When casting a value results in
  `undefined`, it will not be included on the returned object. Fields that has
  the `id` of other schemas as their type, will be objects. If only the id is
  provided in the data, the `{ id: <string>, $ref: <schema id> }` format will
  be used, with `$ref` being the id of the field type schema. When more data is
  provided, Integreat will cast it to the target schema and provide the entire
  data object, or array of objects, with the relevant `$type`.

### Access rules

Set the `access` property on a schema to enforce permission checking. This
applies to any service that provides data in this schema.

The simplest access rule is `auth`, which means that anyone can do anything
with the data of this schema, as long as they are authenticated. Being
authenticated, in this context, means that the dispatched action has an
`ident` in the `meta` object. See [the section on idents](#idents) for more on
this.

Example of a schema with an access rule:

```javascript
{
  id: 'article',
  shape: {
    // ...
  },
  access: 'auth'
}
```

To signal that the schema really has no need for authorization, use `all`.
This is not the same as not setting the `auth` prop, as `all` will override
Integreat's principle of not letting authorized data out of Integreat without
an access rule. `all` allows anybody to access the data, even the
unauthenticated.

On the other end of the spectrum, `none` will allow no one to access data cast
to this schema, no matter who they are.

For more fine-grained rules, set `access` to an access definition object with
rules telling Integreat which rights to require when performing different
actions with a given schema. These rules apply to the [idents](#idents) set on
the action `meta` object.

The following access rule props are available:

- `allow`: Set to `all`, `auth`, or `none`, to give access to everybody, only
  the authenticated, or no one at all. This is what we describe in short form
  above, where we provided this string instead of a access rule object.
- `role`: Authorize only idents with this `role`. May also be an array.
- `ident`: Authorize only idents with this precise `id`. May also be an array.
- `roleFromField`: Same as `role`, except the role is gotten from a field in
  the schema. When authorizing data cast to this schema, the value of the role
  field needs to be identical to (one of) the role(s) of the ident.
- `identFromField` - The same as `roleFromField`, but for an ident id.

In addition, you may override the general access rules of a schema with
specific rules for a type of action, by setting an `action` object with access
rules for action types. Here's an example of an access definition for allowing
all authorized idents to `GET` data in a certain shema, requiring the role
`admin` for `SET`s, and disallowing all other actions with the general rule
`allow: 'none'`:

```javascript
{
  id: 'article',
  shape: {
    // ...
  },
  access: {
    allow: 'none',
    actions: {
      GET: { allow: 'auth' },
      SET: { role: 'admin' }
    }
  }
}
```

Note that these action specific rules only applies to actions being sent to a
service. Some actions will never reach a service, but will instead trigger
other actions, and access will be granted or rejected only for the actions
that are about to be sent to a service. E.g. when you dispatch a `SYNC`
action, it starts off by dispatching one or more `GET` actions. The `SYNC`
action is not subjected to any access rules, but the `GET` actions are, and so
the `SYNC` will fail if one of the `GET` is rejected.

Another example, intended for