iotagent-node-lib

# IoT Agent API  - [Preface](#preface) - [Topics](#topics) - [Terminology](#terminology) - [IoT Agent information model](#iot-agent-information-model) - [Config groups](#config-groups) - [Devices](#devices) - [Uniqueness of groups and devices](#uniqueness-of-groups-and-devices) - [Special measures and attributes names](#special-measures-and-attributes-names) - [Device to NGSI Mapping](#device-to-ngsi-mapping) - [Device autoprovision and entity creation](#device-autoprovision-and-entity-creation) - [Entity Name expression support](#entity-name-expression-support) - [Multientity support](#multientity-support) - [Metadata support](#metadata-support) - [NGSI LD data and metadata considerations](#ngsi-ld-data-and-metadata-considerations) - [Advice on Attribute definitions](#advice-on-attribute-definitions) - [Reuse of attribute names](#reuse-of-attribute-names) - [Reuse of attribute types](#reuse-of-attribute-types) - [How to specify attribute Units of Measurement](#how-to-specify-attribute-units-of-measurement) - [Measurement persistence options](#measurement-persistence-options) - [Autoprovision configuration (autoprovision)](#autoprovision-configuration-autoprovision) - [Explicitly defined attributes (explicitAttrs)](#explicitly-defined-attributes-explicitattrs) - [Differences between `autoprovision`, `explicitAttrs`](#differences-between-autoprovision-explicitattrs) - [Expression language support](#expression-language-support) - [Examples of JEXL expressions](#examples-of-jexl-expressions) - [Available functions](#available-functions) - [Expressions with multiple transformations](#expressions-with-multiple-transformations) - [Expression support in metadata](#expression-support-in-metadata) - [Measurement transformation](#measurement-transformation) - [Measurement transformation definition](#measurement-transformation-definition) - [Measurement transformation execution](#measurement-transformation-execution) - [Measurement transformation order](#measurement-transformation-order) - [Multientity measurement transformation support (`object_id`)](#multientity-measurement-transformation-support-object_id) - [Command execution](#command-execution) - [Triggering commands](#triggering-commands) - [Command reception](#command-reception) - [Command confirmation](#command-confirmation) - [TimeInstant and Timestamp flag](#timeinstant-and-timestamp-flag) - [Multimeasure support](#multimeasure-support) - [Overriding global Context Broker host](#overriding-global-context-broker-host) - [Multitenancy, FIWARE Service and FIWARE ServicePath](#multitenancy-fiware-service-and-fiware-servicepath) - [Secured access to the Context Broker](#secured-access-to-the-context-broker) - [NGSI-LD support](#ngsi-ld-support) - [NGSI-LD `GeoProperty` support](#ngsi-ld-geoproperty-support) - [NGSI-LD Linked Data support](#ngsi-ld-linked-data-support) - [NGSI-LD `datasetId` support](#ngsi-ld-datasetid-support) - [API Routes](#api-routes) - [Config group API](#config-group-api) - [Config group datamodel](#config-group-datamodel) - [Config group operations](#config-group-operations) - [Retrieve config groups `GET /iot/groups`](#retrieve-config-groups-get-iotgroups) - [Create config group `POST /iot/groups`](#create-config-group-post-iotgroups) - [Modify config group `PUT /iot/groups`](#modify-config-group-put-iotgroups) - [Remove config group `DELETE /iot/groups`](#remove-config-group-delete-iotgroups) - [Device API](#device-api) - [Device datamodel](#device-datamodel) - [Device operations](#device-operations) - [Retrieve devices /iot/devices `GET /iot/devices`](#retrieve-devices-iotdevices-get-iotdevices) - [Create device `POST /iot/devices`](#create-device-post-iotdevices) - [Get device details `GET /iot/devices/:deviceId`](#get-device-details-get-iotdevicesdeviceid) - [Modify device `PUT /iot/devices/:deviceId`](#modify-device-put-iotdevicesdeviceid) - [Remove device `DELETE /iot/devices/:deviceId`](#remove-device-delete-iotdevicesdeviceid) - [Batch Operations](#batch-operations) - [Remove devices `POST /iot/op/delete`](#remove-devices-post-iotopdelete) - [Miscellaneous API](#miscellaneous-api) - [Log operations](#log-operations) - [Modify Loglevel `PUT /admin/log`](#modify-loglevel-put-adminlog) - [Retrieve log level `GET /admin/log`](#retrieve-log-level-get-adminlog) - [About operations](#about-operations) - [List IoTA Information `GET /iot/about`](#list-iota-information-get-iotabout) - [Metrics](#metrics) - [Retrieve metrics `GET /metrics`](#retrieve-metrics-get-metrics)  # Preface The IoT Agent mission is to provide a common abstraction layer between the devices and the NGSI entities stored in Context Broker. In order to achieve this, the IoT Agent sits between the Context Broker and a set of devices. It is in charge of translating the information coming from the devices into NGSI requests and viceversa. The **IoT Agent node library** is a Node.js module that can be used to implement IoT Agents. It provides a set of common functionality that can be used to implement the different IoT Agents, offering a simple REST API which provides common functionality to access, provision and decommission devices and config groups of devices. This document describes the API provided by the IoT Agent node library. # Topics ## Terminology - **Devices**: A resource that match physical devices that are connected to the IoT Agent. Each device has a set of attributes that can be read or written and a set of commands that can be invoked. The device is identified by a `device_id` and points particular entity in the context broker. - **Config Groups**: Also known as `provisioning groups` or `service groups`. A logical group of devices. Each Config Group has a set of attributes that can be read or written. The config group is identified by a an `apikey`, used to authenticate the requests coming from the devices. - **Measurements**: A set of values that are sent by a device to the IoT Agent. - **Service**: It is the `FIWARE-Service` that the device or config group belongs to. - **Subservice**: It is the specific `FIWARE-ServicePath` that the device or config group belongs to. - **provision**: The process of creating a new device. A device provisioned means that the device has been already created in the IoT Agent. It can also refer to the group creation process. - **autoprovision**: The process of creating a new device when a measure arrives to the IoT Agent and the device is not provisioned yet. The attributes, entity type and other information is taken from the config group, and entity name is generated according to the entity type and device ID or as a result of an expression if `entityNameExp` is defined. ## IoT Agent information model IoT Agents models 2 different kinds of resources: devices and config groups. Devices are the physical devices that send measurements to the IoT Agent. Config groups are logical groups of devices that share the same configuration. A config group contains zero or more devices. ```mermaid erDiagram "Config Group" ||--o{ Devices : contains ``` ### Config groups Config groups provides a template configuration for the all devices that belong to them. This allows to provision a set of devices with a single operation. They are identified by an `apikey` and a `resource` and mapped to a particular entity type. Once a measure is received by the IoT Agent, the `apikey` and `resource` are used to identify the config group to which the device belongs. The config group is used to map the measure to a particular entity type and to provide the information needed to interact with the Context Broker. If the device already exists in the Context Broker, the IoT Agent will update the entity with the new values. If the device does not exist, the IoT Agent will create it with the information provided by the config group and eventually will also create the entity in the Context Broker. This last operation is only possible if the IoT Agent is configured to use autoprovisioning. For every config group, the pair (resource, apikey) _must_ be unique (as it is used to identify which group to assign to which device). Those operations of the API targeting specific resources will need the use of the `resource` and `apikey` parameters to select the appropriate instance. Config groups can be created with preconfigured sets of attributes, service and subservice information, security information and other parameters. The specific parameters that can be configured for a given config group are described in the [Config group datamodel](#config-group-datamodel) section. ### Devices A device contains the information that connects a physical device to a particular entity in the Context Broker. Devices are identified by a `device_id`, and they are associated to an existing config group based in `apikey` matching. For instance, let's consider a situation in which a config group has been provisioned with `type=X`/`apikey=111` and no other config group has been provisioned. The IoT Agents offer a provisioning API where devices can be preregistered, so all the information about service and subservice mapping, security information and attribute configuration can be specified in a per device way instead of relaying on the config group configuration. The specific parameters that can be configured for a given device are described in the [Device datamodel](#device-datamodel) section. If devices are not pre-registered, they will be automatically created when a measure arrives to the IoT Agent - this process is known as autoprovisioning. The IoT Agent will create an empty device with the group `apikey` and `type` - the associated document created in database doesn't include config group parameters (in particular, `timestamp`, `explicitAttrs`, `active` or `attributes`, `static` and `lazy` attributes and commands). The IoT Agent will also create the entity in the Context Broker if it does not exist yet. This behavior allows that autoprovisioned parameters can freely established modifying the device information after creation using the provisioning API. However, note that if a device (autoprovisioned or not) doesn't have these parameters defined at device level in database, the parameters are inherit from config group parameters. ### Uniqueness of groups and devices Group uniqueness is defined by the combination of: resource and apikey. This is so because given a measure (identified by an apikey) there is no way to identify group related if apikey is not unique for all services and subservices. Device uniqueness is defined by the combination of: service, subservice, device_id and apikey. Note that several devices with the same device_id are allowed in the same service and subservice as long as their apikeys are different. ## Special measures and attributes names In case of arriving measures with name `id` or `type`, they are automatically transformed to `measure_id` and `measure_type` attributes at Context Broker update. The reason behind this is to avoid to collide with the original entity ID and type, as mechanism that enable store measure values from parameters called with the same name. It only applies to autoprovisioned attributes and is also available at JEXL context with the same name (`measure_id` or `measure_type`). In case of provisioning attributes using `id` or `type` as names (please don't do that ;), they are ignored. ## Device to NGSI Mapping The way to map the information coming or going to the device to the NGSI attributes is defined in the group or device. It is possible to define the entity type and the entity ID that a device will use in the Context Broker. It can be configured for a single device in the device provisioning, or it can be defined for all the devices in a group. The entity type should be defined both in the group and in the device, but the entity name (entity ID) is not defined in the group. In that case, if there is no a existing device the same device ID, the entity name of the device generated will be a concatenation of the entity type and the device ID (I.E: `entityType:device_id`). If you need to generate the entity name differently, it is possible to define an expression to generate it, using the parameter `entityNameExp` in the group as described in the [Entity Name expression support](#entity-name-expression-support) section. It is also possible to configure how each of the measures obtained from the device is mapped to different attributes. The name and type of the attribute is configured by the user (globally for all the devices in the group or in a per device preprovisioning). Device measures can have four different behaviors: - **`attributes`**: Are measures that are pushed from the device to the IoT agent. This measure changes will be sent to the Context Broker as updateContext requests over the device entity. NGSI queries to the context broker will be resolved in the Broker database. For each attribute, its `name` and `type` must be provided. Additional `metadata` is optional. They are called internally as _active attributes_. - **`lazy`**: Passive measures that are pulled from the device to the IoT agent. When a request for data from a lazy attribute arrives to the Context Broker, it forwards the request to the IoT Agent (that behaves as NGSI Context Provider for all the lazy attributes or commands). The IoT Agent will then ask the device for the information needed, transform that information to a NGSI format and return it to the Context Broker. This operation will be synchronous from the customer perspective: the Context Broker won't return a response until the device has returned its response to the IoT Agent. For each attribute, its `name` and `type` must be provided. They are called internally as _lazy attributes_. - **`static`**: It is static attributes that are persisted in the Context Broker. They are not updated by the device, but they can be modified by the user. They are useful to store information about the device that is not updated by the device itself. For instance, a `location` static attribute is can be used to store the location of a fixed device. - **`commands`**: Commands are actions that can be invoked in the device. They are entity attributes, but they are not updated by the device, they are updated by the Context Broker. In this case, the interaction will begin by setting an attribute in the device's entity, for which the IoT Agent will be regitered as Context Provider. The IoT Agent will return an immediate response to the Context Broker, and will be held responsible of contacting the device to perform the command itself using the device specific protocol. Special `status` and `info` attributes should be update. For each command, its `name` and `type` must be provided. For further information, please refer to [Command execution](#command-execution) section. All of them have the same syntax, a list of objects with the following attributes: - **object_id** (optional): name of the attribute as coming from the device. - **name** (mandatory): ID of the attribute in the target entity in the Context Broker. Note that `id` and `type` are not valid attribute names at Context Broker. Thus, although a measure named `id` or `type` will not break the IoT Agent, they are silently ignored and never progress toward Context Broker entities. - **type** (mandatory): name of the type of the attribute in the target entity. - **metadata** (optional): additional static metadata for the attribute in the target entity. (e.g. `unitCode`) Some advanced features also allow the use of the following optional fields: - **expression**: indicates that the value of the target attribute will not be the plain value or the measurement, but an expression based on a combination of the reported values. See the [Expression Language definition](#expression-language-support) for details - **skipValue**: indicates that if the result of applying `expression` to a measure is equal to the value of `skipValue` then the attribute corresponding to the measure is not sent to CB. In other words, this field **is not an expression**, it is a value that is compared with the result of applying `expression` to a measure. By default if `skipValue` is not defined then is considered as `null` (i.e. if the result of apply `expression` results in `null` then corresponding attribute is not sent to CB). It is only used if `expression` is provided (otherwise is ignored). - **entity_name**: the presence of this attribute indicates that the value will not be stored in the original device entity but in a new entity with an ID given by this attribute. The type of this additional entity can be configured with the `entity_type` attribute. If no type is configured, the device entity type is used instead. Entity names can be defined as expressions, using the [Expression Language definition](#expression-language-support). - **entity_type**: configures the type of an alternative entity. Additionally for commands (which are attributes of type `command`) the following fields are optional: - **expression** indicates that the value of the target command will not be the plain value or the command, but an expression based on a combination of the returned values. See the [Expression Language definition](#expression-language-support) for details - **payloadType**: indicates how command payload will be transformed before be sent to device. Please have a look to particular IOTAs documentation for allowed values of this field in each case. - **contentType**: `content-type` header used when send command by HTTP transport (ignored in other kinds of transports) - **headers**: extra customer headers used when send command by HTTP transport (ignored in other kinds of transports) Check full detail of these fields in [comand-transformations](https://github.com/telefonicaid/iotagent-json/blob/master/docs/usermanual.md#commands-transformations) Note that, when information coming from devices, this means measures, are not defined neither in the group, nor in the device, the IoT agent will store that information into the destination entity using the same attribute name than the measure name, unless `explicitAttrs` is defined. Measures `id` or `type` names are invalid, and will be ignored. ## Device autoprovision and entity creation For those agents that uses IoTA Node LIB version 3.4.0 or higher, you should consider that the entity is not created automatically when a device is created. This means that all entities into the Context Broker are created when data arrives from a device, no matter if the device is explicitly provisioned (via [device provisioning API](#create-device-post-iotdevices)) or autoprovisioned. If for any reason you need the entity at CB before the first measure of the corresponding device arrives to the IOTAgent, you can create it in advance using the Context Broker [NGSI v2 API](https://github.com/telefonicaid/fiware-orion/blob/master/doc/manuals/orion-api.md). ## Entity Name expression support By default, the entity name used to persist the device measures in the Context Broker can be defined in the device provisioning or, if the device is autoprovisioned, it is generated by the IoT Agent as a concatenation of the entity type and the device ID. If you need to generate the entity name differently, it is possible to define an expression to generate it, using the [Expression Language](#expression-language-support) through the `entityNameExp` field in the group. With this feature, the entity name can be generated dynamically based not only on the device ID and entity type, but also on the measures reported by the device or any other context information. The `entityNameExp` field is only available at the group level. **Important**: when using `entityNameExp`, the `entity_name` field in the device provisioning is ignored. This means that the entity name used to store the device information in the Context Broker is always generated by the `entityNameExp` expression. If you need to explicitly define the entity name for a particular device, you can include a particular condition in the `entityNameExp` expression to handle that case (e.g. `id == 'myDevice' ? 'myEntity' : entityType + ':' + id`). The following example shows how to define an entity name expression: ```json { "services": [ { "resource": "/json", "apikey": "801230BJKL23Y9090DSFL123HJK09H324HV8732", "entity_type": "TemperatureSensor", "entityNameExp": "id + '__' + sn", "attributes": [ { "object_id": "t", "name": "temperature", "type": "Number" }, { "object_id": "sn", "name": "serialNumber", "type": "Text" } ] } ] } ``` As defined above, the `entityNameExp` is `id + '__' + sn` and it will generate the entity name by concatenating the device ID and the serial number reported by the device. For example, for a given measure with `id` equal to `dev123` and `sn` equal to `ABCDEF`, the resulting entity name will be `dev123__ABCDEF`. Note that, when using `entityNameExp`, the `entity_name` of the device provisioning is set to the result of the expression the first time the device is created. If the expression is modified later, the `entity_name` of the device provisioning will not be updated, but the value used to persist the device measures in the Context Broker will be the result of the new expression. This can lead to a situation where the `entity_name` of the device provisioning and the entity name used in the Context Broker are different. ## Multientity support The IOTA is able to persists measures coming from a single device to more than one entity, declaring the target entities through the config group or device provision APIs. ```json { "devices": [ { "protocol": "IoTA-UL", "entity_name": "urn:ngsi-ld:Device:contador12", "entity_type": "multientity", "attributes": [ { "object_id": "cont1", "name": "vol", "type": "Text", "entity_name": "urn:ngsi-ld:Device:WaterMeterSoria01", "entity_type": "WaterMeter" }, { "object_id": "cont2", "name": "vol", "type": "Text", "entity_name": "urn:ngsi-ld:Device:WaterMeterSoria02", "entity_type": "WaterMeter" }, { "object_id": "cont3", "name": "vol", "type": "Text", "entity_name": "urn:ngsi-ld:Device:WaterMeterSoria03", "entity_type": "WaterMeter" } ], "device_id": "contador12" } ] } ``` ## Metadata support Both `attributes` and `static_attributes` may be supplied with metadata when creating a config group, so that the units of measurement can be placed into the resultant entity. e.g.: ```json { "entity_type": "Lamp", "resource": "/iot/d", "protocol": "PDI-IoTA-UltraLight", "commands": [ { "name": "on", "type": "command" }, { "name": "off", "type": "command" } ], "attributes": [ { "object_id": "s", "name": "state", "type": "Text" }, { "object_id": "l", "name": "luminosity", "type": "Integer", "metadata": { "unitCode": { "type": "Text", "value": "CAL" } } } ], "static_attributes": [ { "name": "category", "type": "Text", "value": ["actuator", "sensor"] }, { "name": "controlledProperty", "type": "Text", "value": ["light"], "metadata": { "includes": { "type": "Text", "value": ["state", "luminosity"] }, "alias": { "type": "Text", "value": "lamp" } } } ] } ``` ### NGSI-LD data and metadata considerations When provisioning devices for an NGSI-LD Context Broker, `type` values should typically correspond to one of the following: - `Property`, `Relationship`, `GeoProperty`, `LanguageProperty` - Native JSON types (e.g. `String`, `Boolean`, `Float` , `Integer` `Number`) - Temporal Properties (e.g. `Datetime`, `Date` , `Time`) - GeoJSON types (e.g `Point`, `LineString`, `Polygon`, `MultiPoint`, `MultiLineString`, `MultiPolygon`) Most NGSI-LD attributes are sent to the Context Broker as _properties_. If a GeoJSON type or native JSON type is defined, the data will be converted to the appropriate type. Temporal properties should always be expressed in UTC, using ISO 8601. This ISO 8601 conversion is applied automatically for the `observedAt` _property-of-a-property_ metadata where present. Data for any attribute defined as a _relationship_ must be a valid URN. Note that when the `unitCode` metadata attribute is supplied in the provisioning data under NGSI-LD, the standard `unitCode` _property-of-a-property_ `String` attribute is created. Other unrecognised `type` attributes will be passed as NGSI-LD data using the following JSON-LD format: ```json "<property_name>": { "type" : "Property", "value": { "@type": "<property_type>", "@value": { string or object} } } ``` `null` values will be passed in the following format: ```json "<property_name>": { "type" : "Property", "value": { "@type": "Intangible", "@value": null } } ``` ## Advice on Attribute definitions ### Reuse of attribute names Check for the existence of the same Attribute on any of the other models and reuse it, if pertinent. Have a look at schema.org trying to find a similar term with the same semantics. Try to find common used ontologies or existing standards well accepted by the Community, or by goverments, agencies, etc. For instance, Open311 for civic issue tracking or Datex II for transport systems. ### Reuse of attribute types When possible reuse [schema.org](http://schema.org/) data types (`Text`, `Number`, `DateTime`, `StructuredValue`, etc.). Remember that `null` is not allowed in NGSI-LD and therefore should be avoided as a value. ### How to specify attribute Units of Measurement If your data use the default unit defined in the Data Model, you don't need to specify any. It is implied. Unless explicitly stated otherwise, all FIWARE data models use the metric system of measurements by default. Regardless the model specification include explicit reference to the scale adopted. If your data use a different unit, you will need to use the `unitCode` metadata annotation in your data (and you will need to adopt the normalised representation). The code used should be taken from those defined by [UN/CEFACT](https://www.unece.org/fileadmin/DAM/cefact/recommendations/rec20/rec20_rev3_Annex3e.pdf). E.g.: ```json { "object_id": "l", "name": "length", "type": "Integer", "metadata": { "unitCode": { "type": "Text", "value": "FOT" } } } ``` ## Measurement persistence options There are 2 different options to configure how the IoTAgent stores the measures received from the devices, depending on the following parameters: - `autoprovision`: If the device is not provisioned, the IoTAgent will create a new device and entity for it. - `explicitAttrs`: If the measure element (object_id) is not defined in the mappings of the device or config group provision, the measure is stored in the Context Broker by adding a new attribute to the entity with the same name of the undefined measure element. ### Autoprovision configuration (autoprovision) By default, when a measure arrives to the IoTAgent, if the `device_id` does not match with an existing one, then, the IoTA creates a new device and a new entity according to the config group. Defining the field `autoprovision` to `false` when provisioning the config group, the IoTA to reject the measure at the southbound, allowing only to persist the data to devices that are already provisioned. It makes no sense to use this field in device provisioning since it is intended to avoid provisioning devices (and for it to be effective, it would have to be provisional). Further information can be found in section [Devices](#devices) ### Explicitly defined attributes (explicitAttrs) If a given measure element (object_id) is not defined in the mappings of the device or config group, the measure is stored in the Context Broker by adding a new attribute to the entity with the same name of the undefined measure element. By adding the field `explicitAttrs` with `true` value to device or config group, the IoTAgent rejects the measure elements that are not defined in the mappings of device or config group, persisting only the one defined in the mappings of the provision. If `explicitAttrs` is provided both at device and config group level, the device level takes precedence. Additionally `explicitAttrs` can be used to define which measures (identified by their attribute names, not by their object_id) defined in JSON/JEXL array will be propagated to NGSI interface. Note that when `explicitAttrs` is an array or a JEXL expression resulting in to Array, if this array is empty then `TimeInstant` is not propaged to CB. The different possibilities are summarized below: Case 1 (default): ``` "explicitAttrs": false ``` every measure will be propagated to NGSI interface, including all static attributes. Case 2: ``` "explicitAttrs": true ``` In this case, should only progress active and static attributes defined in the device or group provision (`TimeInstant` attribute will be also included if enabled), including also all static attributes. In other words, having `"explicitAttrs":true` would prevent the IoTA creating attributes into the related entity within the context broker from measures that are not explicitly defined in the device or group provision. Note that attributes defined in the provision that are not receiving a measure (or having a expression defined that is resulting `null`) will not progress (this means, the NGSI request to update the entity in the context broker is not going to include that attribute) unless `skipValue` is defined to other value than `null` Case 3: ``` "explicitAttrs": "['attr1','atrr2']" ``` just NGSI attributes defined in the array (identified by their attribute names, not by their object_id, plus conditionally TimeInstant) will be propagated to NGSI interface (note that in this case the value of `explicitAttrs` is not a JSON but a JEXL Array that looks likes a JSON). Only static attributes included in that array will be propagated to NGSI interface. All attributes contained in the array must be defined as `attributes` or `static_attributes`. Not defined measures (`object_id`) will be dropped, even if they are defined in the `explicitAttrs` array. Case 4: ``` "explicitAttrs": "['attr1','atrr2',{object_id:'active_id'}]" ``` just NGSI attributes defined in the array (identified by their attribute names and/or by their object_id) will be propagated to NGSI interface (note that in this case the value of `explicitAttrs` is not a JSON but a JEXL Array/Object that looks likes a JSON). This is necessary when same attribute names are used within multiple entities. Only static attributes included in that array will be propagated to NGSI interface. Note that in the previous case show above, when selecting the object_id (with `{object_id:'active_id'}`), the attribute must be defined. In other words, it would not work if the attribute with the corresponding `object_id`, is not defined. Case 5: ``` "explicitAtttr": "<JEXL expression resulting in bool or array>" ``` depending on the JEXL expression evaluation: - If it evaluates to `false` every measure will be propagated to NGSI interface (as in case 1) - If it evaluates to `true` just measures defined in active, static (plus conditionally TimeInstant) will be propagated to NGSI interface (as in case 2) - If it evaluates to an array just measures defined in the array (identified by their attribute names, not by their object_id) will be will be propagated to NGSI interface (as in case 3) ### Differences between `autoprovision`, `explicitAttrs` Since those configuration parameters are quite similar, this section is intended to clarify the relation between them. If `autoprovision` is set to `true` (default case), the agent will perform an initial request creating a new entity into the Context Broker with **only** the static and active attributes provisioned in the config group, and also a new Device in the agent, every time a measure arrives with a new `device_id`. Otherwise, this measure is ignored. This is something related to the **southbound**. What `explicitAttrs` does is to filter from the southbound the parameters that are not explicitly defined in the device provision or config group. That also would avoid propagating the measures to the Context Broker. ## Expression language support The IoTAgent Library provides an expression language for measurement transformation and other purposes. This expression language is based on the [TomFrost/JEXL](https://github.com/TomFrost/Jexl) library. The common usage of this feature is to adapt the information coming from the South Bound APIs to the information reported to the Context Broker. This is really useful when you need to adapt measure (for example, to change the units, or to apply a formula to). All the usage of expression in the IoT Agent are: - [Measurement transformation](#measurement-transformation). - [Metadata](#expression-support-in-metadata) - Commands payload transformation (push and pull). - Auto provisioned devices entity name. It is configured at config Group level by setting the `entityNameExp` parameter. It defines an expression to generate the Entity Name for autoprovisioned devices. More information in the [Entity Name expression support](#entity-name-expression-support) section. - Dynamic `endpoint` definition. Configured at device level, it defines where the device listen for push http commands. It can be either a static value or an expression. The agent provides additional information, hereafter referred as **context**, in order to be used to evaluate the expression. In all cases the following data is available to all expressions: - `id`: device ID - `entity_name`: NGSI entity Name (principal) - `type`: NGSI entity type (principal) - `service`: device service (`Fiware-Service`) - `subservice`: device subservice (`Fiware-ServicePath`) - `staticAttributes`: static attributes defined in the device or config group - `oldCtxt`: previous JEXL context (related to last processed measure) Additionally, for attribute expressions (`expression`, `entity_name`), `entityNameExp` and metadata expressions (`expression`) the following is available in the **context** used to evalute: - measures, as `<AttributeName>` - metadata (both for attribute measurement in the case of NGSI-v2 measurements and static attribute) are available in the **context** under the following convention: `metadata.<AttributeName>.<MetadataName>` or `metadata.<StaticAttributeName>.<MetadataName>` in a similar way of defined for [Context Broker](https://github.com/telefonicaid/fiware-orion/blob/master/doc/manuals/orion-api.md#metadata-support) ### Examples of JEXL expressions The following table shows expressions and their expected outcomes taking into account the following measures at southbound interface: - `value` with value 6 (number) - `ts` with value 1637245214901 (unix timestamp) - `name` with value `"DevId629"` (string) - `object` with value `{name: "John", surname: "Doe"}` (JSON object) - `array` with value `[1, 3]` (JSON Array) | Expression | Expected outcome | Format | Playground | | :-------------------------------------------- | :---------------------------------------- | ------------------- | ------------- | | `5 * value` | `30` | Integer | [Example][6] | | `(6 + value) * 3` | `36` | Integer | [Example][7] | | `value / 12 + 1` | `1.5` | Float | [Example][8] | | `(5 + 2) * (value + 7)` | `91` | Integer | [Example][9] | | `value * 5.2` | `31.2` | Float | [Example][10] | | `"Pruebas " + "De Strings"` | `"Pruebas De Strings"` | String | [Example][11] | | `name + "value is " +value` | `"DevId629 value is 6"` | String | [Example][12] | | `{coordinates: [value,value], type: 'Point'}` | `{"coordinates": [6,6], "type": "Point"}` | GeoJSON `Object` | [Example][13] | | <code>ts|toisodate</code> | `2021-11-18T14:20:14.901Z` | ISO 8601 `DateTime` | [Example][14] | Support for `trim`, `length`, `substr` and `indexOf` transformations was added. | Expression | Expected outcome | Playground | | :-------------------------------------------------------- | :--------------- | ------------- | | <code>" a "| trim</code> | `a` | [Example][15] | | <code>name|length</code> | `8` | [Example][16] | | <code>name|indexOf("e")</code> | `1` | [Example][17] | | <code>name|substr(0,name|indexOf("e")+1)</code> | `"De"` | [Example][18] | ### Available functions There are several predefined JEXL transformations available to be used at any JEXL expression. The definition of those transformations and their JavaScript implementation can be found at jexlTransformsMap.js. The library module also exports a method `iotAgentLib.dataPlugins.expressionTransformation.setJEXLTransforms(Map)` to be used by specific IoT Agent implementations in order to incorporate extra transformations to this set. It is important to remark that the lib `jexlTransformsMap` cannot be overwritten by the API additions. The idea behind this is to be able to incorporate new transformations from the IoT Agent configuration file in a fast and tactical way. Current common transformation set: | JEXL Transformation | Equivalent JavaScript Function | | ------------------------------------ | ------------------------------------------------------------------------------------------------------------------------ | | jsonparse: (str) | `JSON.parse(str);` | | jsonstringify: (obj) | `JSON.stringify(obj);` | | indexOf: (val, char) | `String(val).indexOf(char);` | | length: (val) | `String(val).length;` | | trim: (val) | `String(val).trim();` | | substr: (val, int1, int2) | `String(val).substr(int1, int2);` | | addreduce: (arr) | <code>arr.reduce((i, v) | i + v));</code> | | lengtharray: (arr) | `arr.length;` | | typeof: (val) | `typeof val;` | | isarray: (arr) | `Array.isArray(arr);` | | isnan: (val) | `isNaN(val);` | | parseint: (val) | `parseInt(val);` | | parsefloat: (val) | `parseFloat(val);` | | toisodate: (val) | `new Date(val).toISOString();` | | timeoffset:(isostr) | `new Date(isostr).getTimezoneOffset();` | | tostring: (val) | `val.toString();` | | urlencode: (val) | `encodeURI(val);` | | urldecode: (val) | `decodeURI(val);` | | replacestr: (str, from, to) | `str.replace(from, to);` | | replaceregexp: (str, reg, to) | `str.replace(new RegExp(reg), to);` | | replaceallstr: (str, from, to) | `str.replaceAll(from, to);` | | replaceallregexp: (str, reg, to) | `str.replaceAll(new RegExp(reg,"g"), to);` | | split: (str, ch) | `str.split(ch);` | | joinarrtostr: (arr, ch) | `arr.join(ch);` | | concatarr: (arr, arr2) | `arr.concat(arr2);` | | mapper: (val, values, choices) | <code>choices[values.findIndex((target) | target == val)]);</code> | | thmapper: (val, values, choices) | <code>choices[values.reduce((acc,curr,i,arr) | (acc==0)||acc?acc:val<=curr?acc=i:acc=null,null)];</code> | | bitwisemask: (i,mask,op,shf) | <code>(op==="&"?parseInt(i)&mask: op==="|"?parseInt(i)|mask: op==="^"?parseInt(i)^mask:i)>>shf;</code> | | slice: (arr, init, end) | `arr.slice(init,end);` | | addset: (arr, x) | <code>{ return Array.from((new Set(arr)).add(x)) }</code> | | removeset: (arr, x) | <code>{ let s = new Set(arr); s.delete(x); return Array.from(s) }</code> | | touppercase: (val) | `String(val).toUpperCase()` | | tolowercase: (val) | `String(val).toLowerCase()` | | round: (val) | `Math.round(val)` | | floor: (val) | `Math.floor(val)` | | ceil: (val) | `Math.ceil(val)` | | tofixed: (val, decimals) | `Number.parseFloat(val).toFixed(decimals)` | | gettime: (d) | `new Date(d).getTime()` | | toisostring: (d) | `new Date(d).toISOString()` | | localestring: (d, timezone, options) | `new Date(d).toLocaleString(timezone, options)` | | now: () | `Date.now()` | | hextostring: (val) | `new TextDecoder().decode(new Uint8Array(val.match(/.{1,2}/g).map(byte => parseInt(byte, 16))))` | | valuePicker: (val,pick) | <code>valuePicker: (val,pick) => Object.entries(val).filter(([_, v]) => v === pick).map(([k, _]) => k)</code> | | valuePickerMulti: (val,pick) | <code>valuePickerMulti: (val,pick) => Object.entries(val).filter(([_, v]) => pick.includes(v)).map(([k, _]) => k)</code> | You have available this [JEXL interactive playground][99] with all the transformations already loaded, in which you can test all the functions described above. ### Expressions with multiple transformations When you need to apply multiples transformations to the same value, you can use the pipe character `|` to separate the transformations. For example, if you want to apply the `trim` and `replacestr` transformations to a value, you can use the following expression: ```json { "expression": "variable | trim | replacestr('hello','hi')" } ``` If variable takes value `hello world`, the result of the previous expression will be `hi world`. Another example using functions that return more than one value is the following: ```json { "expression": "location | split(', ')[1] | parsefloat()" } ``` For a location value `"40.4165, -3.70256"`, the result of the previous expression will be `-3.70256`. ### Expression support in metadata Metadata could also has `expression` like attributes in order to expand it: e.g.: ```json { "entity_type": "Lamp", "resource": "/iot/d", "protocol": "PDI-IoTA-UltraLight", "commands": [ { "name": "on", "type": "command" }, { "name": "off", "type": "command" } ], "attributes": [ { "object_id": "s", "name": "state", "type": "Text" }, { "object_id": "l", "name": "luminosity", "type": "Integer", "metadata": { "unitCode": { "type": "Text", "value": "CAL" } } } ], "static_attributes": [ { "name": "category", "type": "Text", "value": ["actuator", "sensor"] }, { "name": "controlledProperty", "type": "Text", "value": ["light"], "metadata": { "includes": { "type": "Text", "value": ["state", "luminosity"], "expression": "level / 100" }, "alias": { "type": "Text", "value": "lamp" } } } ] } ``` Note that there is no order into metadata structure and there is no warranty about which metadata attribute expression will be evaluated first. ## Measurement transformation The IoTAgent Library provides support for measurement transformation using a [Expression Language](#expression-language-support) This feature can be used to adapt the information coming from the South Bound APIs to the information reported to the Context Broker. This is really useful when you need to adapt measure (for example, to change the units, or to apply a formula to). ### Mea