@itentialopensource/adapter-kafka

## Adapter Properties and Descriptions This section defines **all** the properties that are available for the adapter, including detailed information on what each property is for. If you are not using certain capabilities with this adapter, you do not need to define all of the properties. An example of how the properties for this adapter can be used with tests or IAP are provided in the **Installation** section. To view all client, consumer and producer options please follow [this](https://www.npmjs.com/package/kafka-node?activeTab=readme) link. ```json { "host": "localhost", "port": 9092, "interval_time": 5000, "stub": false, "client": { "connectTimeout": 10000, "requestTimeout": 30000, "autoConnect": true, "connectRetryOptions": {}, "idleConnection": 300000, "reconnectOnIdle": true, "maxAsyncRequests": 10, "sslOptions": {}, "sasl": {} }, "producer": { "requireAcks": 1, "ackTimeoutMs": 100, "partitionerType": 0 }, "consumer": { "groupId": "kafka-node-group", "autoCommit": false, "autoCommitIntervalMs": 5000, "fetchMaxWaitMs": 100, "fetchMinBytes": 1, "fetchMaxBytes": 1048576, "fromOffset": true, "encoding": "utf8", "keyEncoding": "utf8" } } ``` ## IMPORTANT When adapter works in consumer configuration: `autoCommit:false` `fromOffset:true` current offset setting on adapter startup is loaded from .topic.json file for each (topic:partition). Setting of property `interval_time` affects adapter's behaviour after adapter restart. If adapter is restarted before .topics.json is updated with latest offset for given (topic:partition), then after adapter goes up, consumer offsets are set based on .topic.json content or outOfRange resolution. Depending on the offset being set all messages present on kafka server with offset=(consumer offset + 1) will be read and emitted to subscriber (e.g. OperationManager) after restart. This can trigger duplicated jobs run by OperationManager. To avoid that, operator shall wait `interval_time` after last message read by consumer before restarting adapter to avoid duplicated jobs. ### Topic Properties Subscribe to kafka topics by providing topic properties in the adapter service config. Operator can configure message filtering per topic by providing filters in `topics[*].subscriberInfo[*].filters` property. Regular expressions are accepted, guide on how to build regular expressions: https://developer.mozilla.org/en-US/docs/Web/JavaScript/Guide/Regular_Expressions Operator shall stick to simple regular expressions as complex ones could cause catastrophic backtracking problem https://www.regular-expressions.info/catastrophic.html leading to adapter being unresponsive. | Option | Description | | ------- | ------- | | name | Topic name you are subscribing to | | always | Always subsribe to this topic if set to true | | subscriberInfo | subscriber information | | subname | subscriber name | | filters | regex to filter the incoming messages | | rabbit | IAP rabbit queue name where the messages will be published. If set to `kafka`, events will be published to a topic with the same name as the Kafka topic. | | throttle | not being used by the adapter currently. set to empty object | #### Example of adapter's configuration with filtering applied: ```json "topics": [ { "name": "t1", "always": true, "subscriberInfo": [ { "subname": "default", "filters": [ "PIXI", "DRED", "[abc]", "F: (\\w+), L: (\\w+)", "\\d{3,4}" ], "rabbit": "kafka", "throttle": {} } ] } ], ``` When operator skips to provide `filters` property or the property contains no actual filters set, then all messages are passed. #### Example of adapter's configuration with multiple paritions and subscribers per topic: ```json "topics": [ { "name": "topic1", "always": true, "partition": 0, "subscriberInfo": [ { "subname": "default", "filters": [ "PIXI", "DRED", "F: (\\w+), L: (\\w+)", "\\d{3,4}" ], "rabbit": "topic1", "throttle": {} }, { "subname": "sub2", "filters": [ "[abc]", "F: (\\w+), L: (\\w+)", "\\d{3,4}" ], "rabbit": "topic1-s2", "throttle": {} } ] }, { "name": "topic1", "always": true, "partition": 1, "subscriberInfo": [ { "subname": "default", "filters": [ "PIXI", "DRED", "[abc]", "F: (\\w+), L: (\\w+)", "\\d{3,4}" ], "rabbit": "topic1-p1", "throttle": {} } ] }, { "name": "test-6", "always": true } ], ``` **Note**: If no parition is supplied, only messages on partition 0 will be consumed. Additionally, if the rabbit topic is `kafka`, events will be published to a topic with the same name as the Kafka topic. For example, topic `test-6` above will be published to the `test-6` rabbit topic. ### Connection Properties These base properties are used to connect to Kafka upon the adapter initially coming up. It is important to set these properties appropriately. Kafka server location is set either by providing (host, port) tuple or by providing hostList property | Property | Description | | ------- | ------- | | host | Optional. A fully qualified domain name or IP address.| | port | Required if `host` set. Used to connect to the server.| | hostList | Optional. A string of kafka broker/host combination delimited by comma| | interval_time | Optional. The Kafka adapter keeps information about topics and offsets in memory in order to be more efficient. In order to work across restarts the adapter must persist the data. So the data is written into the .topics.json file. This write time defines how often to write the file ### Client Properties The following properties are used to define the Kafka Client. These properties all have default values in the adapter and in Kafka. Definitions are taken from kafka-node page as these properties are directly passed to Kafka. | Property | Description | | ------- | ------- | | connectTimeout | ms it takes to wait for a successful connection before moving to the next host.| | requestTimeout | ms for a kafka request to timeout .| | autoConnect | automatically connect when KafkaClient is instantiated otherwise you need to manually call connect.| | connectRetryOptions | object hash that applies to the initial connection.| | idleConnection | ms before allowing the broker to disconnect an idle connection from a client. | | reconnectOnIdle | when the connection is closed due to client idling, client will attempt to auto-reconnect.| | maxAsyncRequests | maximum async operations at a time toward the kafka cluster..| | sslOptions | Object, options to be passed to the tls broker sockets, ex. { rejectUnauthorized: false }.| | sasl | Object, SASL authentication configuration (only SASL/PLAIN is currently supported), ex. { mechanism: 'plain', username: 'foo', password: 'bar' }.| An example client properties is as follows: ```json "client": { "connectTimeout": 10000, "requestTimeout": 3000, "autoConnect": true, "connectRetryOptions": {}, "idleConnection": 300000, "reconnectOnIdle": true, "maxAsyncRequests": 10, "sslOptions": { "enableTrace": false, "rejectUnauthorized": false }, "sasl": { "mechanism": "SCRAM-SHA-512", "username": "foo", "password": "bar" } } ``` ## Support for different users for producer and consumer Add two different sasl objects in client. See example below- ```json "client": { "connectTimeout": 10000, "requestTimeout": 3000, "autoConnect": true, "connectRetryOptions": {}, "idleConnection": 300000, "reconnectOnIdle": true, "maxAsyncRequests": 10, "sslOptions": { "enableTrace": false, "rejectUnauthorized": false }, "consumersasl": { "mechanism": "SCRAM-SHA-512", "username": "foo-consumer", "password": "bar" }, "producersasl": { "mechanism": "SCRAM-SHA-512", "username": "foo-producer", "password": "bar" } } ``` #### Authentication Options In this example we are using SASL/SCRAM authentication. The adapter also supports using plaintext authentication. To do this we would remove the `sslOptions` object and provide our authentication details in the `sasl` object. Finally, to use no authentication, remove both the `sslOptions` and `sasl` properties. To view all client options and authentication methods please follow [this](https://www.npmjs.com/package/kafka-node#options) link. ### Producer Properties The following properties are used to define the Kafka Producer. These properties all have default values in Kafka. Definitions are taken from kafka-node page as these properties are directly passed to Kafka. An example producer properties object can be viewed in the [sampleProperties](sampleProperties.json) file. | Property | Description | | ------- | ------- | | requireAcks | Configuration for when to consider a message as acknowledged, default 1. | | ackTimeoutMs | The amount of time in milliseconds to wait for all acks before considered, default 100ms.| | partitionerType | Partitioner type (default = 0, random = 1, cyclic = 2, keyed = 3, custom = 4), default 0.| ### Consumer Properties The following properties are used to define the Kafka Consumer. These properties all have default values in Kafka. Definitions are taken from kafka-node page as these properties are directly passed to Kafka. An example consumer properties object can be viewed in the [sampleProperties](sampleProperties.json) file. | Property | Description | | ------- | ------- | | groupId | consumer group id, default `kafka-node-group`.| | autoCommit | Auto commit config.| | autoCommitIntervalMs | | | fetchMaxWaitMs | max wait time is the maximum amount of time in milliseconds to block waiting if insufficient data is available at the time the request is issued, default 100ms| | fetchMinBytes | minimum number of bytes of messages that must be available to give a response, default 1 byte.| | fetchMaxBytes | maximum bytes to include in the message set for this partition. This helps bound the size of the response.| | fromOffset | If set true, consumer will fetch message from the given offset in the payloads.| | encoding | If set to 'buffer', values will be returned as raw buffer objects..| | keyEncoding | |