@oriolrius/node-red-contrib-kafka
Version:
Node-RED Kafka nodes: Send, Receive, and Schema validation with modern KafkaJS
225 lines (193 loc) • 10.4 kB
HTML
<script type="text/html" data-help-name="hm-kafka-history-reader">
<p>Retrieves historical messages from a Kafka topic starting from a specific offset.</p>
<h3>Details</h3>
<p>This node reads historical messages from a Kafka topic starting from a specified offset position.
You can read messages either forward (after the offset) or backward (before the offset).
The node creates a temporary consumer with a unique group ID to read messages without affecting
your main consumer offsets.</p>
<p><strong>Direction Modes:</strong></p>
<ul>
<li><strong>Forward:</strong> Reads N messages starting AFTER the specified offset</li>
<li><strong>Backward:</strong> Reads the last N messages that come BEFORE the specified offset</li>
</ul>
<p><strong>Examples:</strong></p>
<ul>
<li><strong>Forward from offset 1300, max 3 messages:</strong><br/>
Result: messages at offsets 1301, 1302, 1303</li>
<li><strong>Backward from offset 1300, max 3 messages:</strong><br/>
Result: messages at offsets 1297, 1298, 1299</li>
</ul>
<p><strong>Note:</strong> The offset parameter is required. The node will error if no offset is specified.</p>
<h3>Inputs</h3>
<dl class="message-properties">
<dt>maxMessages <span class="property-type">number</span></dt>
<dd>Maximum number of messages to retrieve (overrides config)</dd>
<dt>offset <span class="property-type">number</span></dt>
<dd>Starting offset position for reading messages (overrides config)</dd>
<dt>direction <span class="property-type">string</span></dt>
<dd>Reading direction: "forward" or "backward" (overrides config)</dd>
<dt>timeoutMs <span class="property-type">number</span></dt>
<dd>Maximum time to spend reading in milliseconds (default: 30000)</dd>
</dl>
<h3>Outputs</h3>
<dl class="message-properties">
<dt>payload.messages <span class="property-type">array</span></dt>
<dd>Array of historical messages with payload, topic, offset, partition, key, timestamp, and headers</dd>
<dt>payload.summary <span class="property-type">object</span></dt>
<dd>Summary information including message count, direction, and starting offset</dd>
<dt>payload.request <span class="property-type">object</span></dt>
<dd>Details about the request parameters including maxMessages, topic, direction, offset, and execution time</dd>
</dl>
<h3>Configuration</h3>
<ul>
<li><strong>Broker</strong>: Kafka broker configuration node</li>
<li><strong>Topic</strong>: Kafka topic to read from</li>
<li><strong>Max Messages</strong>: How many messages to retrieve</li>
<li><strong>Offset</strong>: Starting offset position (required)</li>
<li><strong>Direction</strong>: Forward (after offset) or Backward (before offset)</li>
<li><strong>Schema Validation</strong>: Enable Confluent Schema Registry integration</li>
</ul>
<h3>Example Usage</h3>
<p><strong>Basic Forward Reading:</strong><br/>
Configure the node with offset=1000, direction=forward, maxMessages=5. When triggered, it will return
messages at offsets 1001, 1002, 1003, 1004, 1005.</p>
<p><strong>Basic Backward Reading:</strong><br/>
Configure the node with offset=1000, direction=backward, maxMessages=5. When triggered, it will return
the last 5 messages before offset 1000 (e.g., offsets 995, 996, 997, 998, 999).</p>
<p><strong>Dynamic Configuration:</strong><br/>
Send a message with <code>msg.offset = 2000</code>, <code>msg.direction = "forward"</code>, and
<code>msg.maxMessages = 3</code> to override the node configuration dynamically.</p>
<p><strong>With Schema Registry:</strong><br/>
Enable schema validation to automatically decode Avro messages using the configured schema registry.</p>
<p>This node is perfect for getting historical context before starting real-time consumption or for
debugging specific message ranges in your Kafka topics.</p>
</script>
<script type="text/html" data-template-name="hm-kafka-history-reader">
<div class="form-row">
<label for="node-input-name"><i class="fa fa-tag"></i> Name</label>
<input type="text" id="node-input-name" placeholder="Kafka History Reader">
</div>
<div class="form-row">
<label for="node-input-broker"><i class="fa fa-random"></i> Broker</label>
<input type="text" id="node-input-broker">
</div>
<div class="form-row">
<label for="node-input-topic"><i class="fa fa-tag"></i> Topic</label>
<input type="text" id="node-input-topic" placeholder="my-topic">
</div>
<div class="form-row">
<label for="node-input-maxMessages"><i class="fa fa-hashtag"></i> Max Messages</label>
<input type="number" id="node-input-maxMessages" value="10" min="1" max="1000">
<div class="form-tips">Maximum number of messages to retrieve</div>
</div>
<div class="form-row">
<label for="node-input-offset"><i class="fa fa-location-arrow"></i> Offset</label>
<input type="number" id="node-input-offset" placeholder="Start offset (required)" min="0">
<div class="form-tips">Offset position to start reading from</div>
</div>
<div class="form-row">
<label for="node-input-direction"><i class="fa fa-arrows-h"></i> Direction</label>
<select id="node-input-direction">
<option value="forward">Forward (read N messages after offset)</option>
<option value="backward">Backward (read N messages before offset)</option>
</select>
<div class="form-tips">Forward: reads from offset+1 onwards. Backward: reads the last N messages before the offset.</div>
</div>
<div class="form-row">
<label for="node-input-encoding"><i class="fa fa-code"></i> Encoding</label>
<select id="node-input-encoding">
<option value="utf8">UTF-8</option>
<option value="ascii">ASCII</option>
<option value="base64">Base64</option>
<option value="hex">Hex</option>
</select>
</div>
<!-- Schema Validation Section -->
<hr style="margin: 20px 0;"/>
<div class="form-row">
<input type="checkbox" id="node-input-useSchemaValidation" style="display: inline-block; width: auto; vertical-align: top;">
<label for="node-input-useSchemaValidation" style="width: auto; margin-left: 5px;">Enable Schema Validation</label>
</div>
<div id="node-config-schema" style="display: none; padding-left: 20px; border-left: 3px solid #ccc; margin-left: 10px;">
<div class="form-row">
<label for="node-input-registryUrl"><i class="fa fa-globe"></i> Schema Registry URL</label>
<input type="text" id="node-input-registryUrl" placeholder="http://localhost:8081">
</div>
<div class="form-row">
<input type="checkbox" id="node-input-useRegistryAuth" style="display: inline-block; width: auto; vertical-align: top;">
<label for="node-input-useRegistryAuth" style="width: auto; margin-left: 5px;">Use Authentication</label>
</div>
<div id="node-config-registry-auth" style="display: none; padding-left: 20px; border-left: 3px solid #ddd; margin-left: 10px;">
<div class="form-row">
<label for="node-input-registryUsername"><i class="fa fa-user"></i> Username</label>
<input type="text" id="node-input-registryUsername">
</div>
<div class="form-row">
<label for="node-input-registryPassword"><i class="fa fa-lock"></i> Password</label>
<input type="password" id="node-input-registryPassword">
</div>
</div>
</div>
</script>
<script type="text/javascript">
RED.nodes.registerType('hm-kafka-history-reader', {
category: 'IOT',
paletteLabel: "kafka history",
color: '#673AB7',
defaults: {
name: { required: false },
broker: { type: "hm-kafka-broker", required: true },
topic: { required: true },
maxMessages: { value: 10, required: true, validate: function(v) {
return v > 0 && v <= 1000;
}},
offset: { required: false },
direction: { value: "forward" },
encoding: { value: "utf8" },
useSchemaValidation: { value: false },
registryUrl: { required: false, value: "http://localhost:8081" },
useRegistryAuth: { value: false },
registryUsername: { required: false },
registryPassword: { required: false }
},
inputs: 1,
outputs: 1,
icon: "db.png",
label: function () {
if (this.name) {
return this.name;
}
const dir = this.direction === 'backward' ? '←' : '→';
const offset = this.offset || '?';
return `${dir} ${this.maxMessages || 10} msgs from ${offset}`;
},
labelStyle: function() {
return this.name ? "node_label_italic" : "";
},
oneditprepare: function() {
// Handle schema validation checkbox
$("#node-input-useSchemaValidation").change(function() {
if ($(this).is(":checked")) {
$("#node-config-schema").slideDown();
} else {
$("#node-config-schema").slideUp();
}
});
// Handle registry auth checkbox
$("#node-input-useRegistryAuth").change(function() {
if ($(this).is(":checked")) {
$("#node-config-registry-auth").slideDown();
} else {
$("#node-config-registry-auth").slideUp();
}
});
// Show/hide sections based on current values
if (this.useSchemaValidation) {
$("#node-config-schema").show();
}
if (this.useRegistryAuth) {
$("#node-config-registry-auth").show();
}
}
});
</script>