node-red-contrib-prib-functions
Version:
Node-RED added node functions.
258 lines (239 loc) • 10.9 kB
HTML
<script type="text/javascript">
RED.nodes.registerType('columnar', {
category: 'storage',
color: '#8BC34A',
defaults: {
name: {value:"", required:false},
action: {value:"readFile", required:true},
filePath: {value:"", required:false},
sqlQuery: {value:"", required:false},
outputProperty: {value:"result", required:false},
parameterMappings: {value:[], required:false}
},
inputs: 1,
inputLabels: "in",
outputs:1,
outputLabels: function() { return [this.outputProperty || "result"]; },
paletteLabel: "Columnar Store",
icon: "columnar.svg",
label: function() {
const actionLabels = {
readFile: "Read",
writeFile: "Write",
appendFile: "Append",
queryFile: "Query",
sqlQuery: "SQL Query",
getSchema: "Get Schema",
getMetadata: "Get Metadata"
};
const actionLabel = actionLabels[this.action] || this.action;
if (this.name) {
return this.name;
}
if (this.action === 'sqlQuery' && this.sqlQuery) {
// Show truncated SQL query in label
const sqlPreview = this.sqlQuery.length > 30 ?
this.sqlQuery.substring(0, 27) + '...' :
this.sqlQuery;
return sqlPreview;
}
return actionLabel + " Columnar Store";
},
labelStyle: function() {
return "node_label_italic";
},
oneditprepare: function() {
// Show/hide SQL query field based on action
function updateSqlField() {
const action = $('#node-input-action').val();
if (action === 'sqlQuery') {
$('#sql-query-row').show();
} else {
$('#sql-query-row').hide();
}
}
$('#node-input-action').change(updateSqlField);
updateSqlField(); // Initial check
},
oneditsave: function() {
// No parameter mappings needed; explicit :msg., :flow., :global. markers only
}
});
</script>
<script type="text/x-red" data-template-name="columnar">
<div class="form-row">
<label for="node-input-name"><i class="fa fa-tag"></i> Name</label>
<input type="text" id="node-input-name" placeholder="Name">
</div>
<div class="form-row">
<label for="node-input-action"><i class="fa fa-cogs"></i> Action</label>
<select type="text" id="node-input-action">
<option value="readFile">Read File</option>
<option value="writeFile">Write File</option>
<option value="appendFile">Append File</option>
<option value="queryFile">Query File</option>
<option value="sqlQuery">SQL Query</option>
<option value="getSchema">Get Schema</option>
<option value="getMetadata">Get Metadata</option>
</select>
</div>
<div class="form-row">
<label for="node-input-filePath">File Path</label>
<input type="text" id="node-input-filePath" placeholder="/path/to/file.columnar">
</div>
<div class="form-row">
<label for="node-input-outputProperty"><i class="fa fa-sign-out"></i> Output Property</label>
<input type="text" id="node-input-outputProperty" placeholder="result">
</div>
<div class="form-row" id="sql-query-row" style="display:none;">
<label for="node-input-sqlQuery"><i class="fa fa-code"></i> SQL Query</label>
<textarea type="text" id="node-input-sqlQuery" placeholder="SELECT * FROM ? WHERE ..." rows="3" style="width:100%; resize:vertical;"></textarea>
</div>
</script>
<script type="text/x-red" data-help-name="columnar">
<p>Columnar Store node for reading and writing columnar data files. This implementation is based on <a href="https://github.com/apache/parquet-format/" target="_blank">Apache Parquet specification</a> concepts including columnar storage, metadata structures, and compression principles, but provides a simplified JavaScript implementation for basic use cases.</p>
<h3>Actions</h3>
<ul>
<li><b>Read File:</b> Reads all records from a Columnar Store file</li>
<li><b>Write File:</b> Writes records to a Columnar Store file (overwrites existing file)</li>
<li><b>Append File:</b> Appends records to an existing Columnar Store file (creates file if it doesn't exist)</li>
<li><b>Query File:</b> Reads and filters records from a Columnar Store file</li>
<li><b>SQL Query:</b> Execute SQL queries with aggregates, grouping, sorting, JOINs, and filtering</li>
<li><b>Get Schema:</b> Retrieves the schema information from a Columnar Store file</li>
<li><b>Get Metadata:</b> Retrieves metadata information from a Columnar Store file</li>
</ul>
<h3>Configuration</h3>
<ul>
<li><b>File Path:</b> Path to the Columnar file (.columnar extension). Can be overridden in msg.payload.filePath</li>
<li><b>Output Property:</b> Property name on msg object where results will be stored (default: "result")</li>
<li><b>SQL Query:</b> (SQL Query action only) Hardcoded SQL query to execute. Can be overridden by msg.payload.sql. Hardcoded queries are parsed once and cached for better performance.</li>
<li><b>Parameters:</b> <b>Only</b> <code>:msg.path</code>, <code>:flow.path</code>, or <code>:global.path</code> markers are allowed in SQL. Example: <code>WHERE a.name = :msg.payload.name AND a.age > :flow.minAge</code>. No other parameter markers are supported.</li>
</ul>
<h3>Input/Output Formats</h3>
<h4>Read File</h4>
<p><b>Input:</b> Optional filePath in msg.payload</p>
<p><b>Output:</b></p>
<pre>{
"result": {
"records": [...],
"count": 100,
"schema": {...},
"filePath": "/path/to/file.columnar"
}
}</pre>
<h4>Write File</h4>
<p><b>Input:</b></p>
<pre>{
"records": [
{"name": "John", "age": 30, "city": "NYC"},
{"name": "Jane", "age": 25, "city": "LA"}
],
"filePath": "/path/to/output.columnar",
"schema": {...} // optional, auto-inferred if not provided
}</pre>
<p><b>Output:</b></p>
<pre>{
"result": {
"filePath": "/path/to/output.columnar",
"recordsWritten": 2,
"schema": {...}
}
}</pre>
<h4>Query File</h4>
<p><b>Input:</b></p>
<pre>{
"filePath": "/path/to/file.columnar",
"limit": 100, // optional
// either provide a javascript filter function or a ridMap to perform
// a column‑based intersection. The columnar layout stores each field
// as its own buffer along with a list of row‑ids (rids). A ridMap is an
// object whose keys are column names and whose values are arrays or
// Sets of rids; only rows present in *every* set will be returned.
// Example using a filter function:
"filter": "function(record) { return record.age > 25; }" // optional
// Example using a ridMap (select rows 1 and 2 from the 'age' column):
// "ridMap": {"age": [1,2]}
}</pre>
<p><b>Output:</b></p>
<pre>{
"result": {
"records": [...],
"count": 50,
"totalScanned": 100,
"schema": {...},
"filtered": true
}
}</pre>
<h4>SQL Query</h4>
<p><b>Input:</b> SQL query can be configured in the node or provided in msg.payload.sql (msg.payload.sql takes precedence)</p>
<pre>{
"filePath": "/path/to/file.columnar",
"sql": "SELECT * FROM ? WHERE a.name = :msg.payload.name AND a.age > :flow.minAge"
}</pre>
<p><b>Parameters:</b> Use <code>:msg.path</code>, <code>:flow.path</code>, or <code>:global.path</code> in your SQL to reference values from <code>msg</code>, <code>flow</code>, or <code>global</code> context. <b>Other parameter markers are not supported and will cause an error.</b></p>
<p><b>Supported Features:</b></p>
<ul>
<li><b>SELECT:</b> Column names, <code>*</code>, or aggregates (COUNT, SUM, AVG, MIN, MAX, COUNT DISTINCT)</li>
<li><b>FROM/JOIN:</b> Main table (<code>?</code>) with optional INNER/LEFT/RIGHT/FULL OUTER JOIN on other .columnar files</li>
<li><b>WHERE:</b> Filter records using JavaScript expressions</li>
<li><b>GROUP BY:</b> Group results by one or more columns with aggregate functions</li>
<li><b>HAVING:</b> Filter grouped results</li>
<li><b>ORDER BY:</b> Sort by column(s) with ASC/DESC</li>
<li><b>LIMIT:</b> Restrict result count</li>
<li><b>Parameters:</b> Use <code>:msg.path</code>, <code>:flow.path</code>, or <code>:global.path</code> in queries. No other parameter markers are allowed.</li>
</ul>
<p><b>JOIN Syntax:</b></p>
<pre>[INNER|LEFT|RIGHT|FULL OUTER] JOIN 'filepath.columnar' ON table1.col = table2.col</pre>
<p><b>Output:</b></p>
<pre>{
"result": [ /* array of result rows */ ]
}</pre>
<h4>Get Schema</h4>
<p><b>Input:</b> Optional filePath in msg.payload</p>
<p><b>Output:</b></p>
<pre>{
"result": {
"schema": {...},
"fields": ["name", "age", "city"],
"filePath": "/path/to/file.columnar"
}
}</pre>
<h4>Get Metadata</h4>
<p><b>Input:</b> Optional filePath in msg.payload</p>
<p><b>Output:</b></p>
<pre>{
"result": {
"metadata": {...},
"filePath": "/path/to/file.columnar",
"numRows": 1000,
"numRowGroups": 5
}
}</pre>
<h3>Data Types</h3>
<p>Columnar supports various data types. The node automatically infers types when writing:</p>
<ul>
<li>String → UTF8</li>
<li>Number → DOUBLE</li>
<li>Boolean → BOOLEAN</li>
<li>Date → TIMESTAMP</li>
</ul>
<h3>Implementation Notes</h3>
<ul>
<li>Based on <a href="https://github.com/apache/parquet-format/" target="_blank">Apache Parquet specification</a> concepts</li>
<li>Implements columnar storage for efficient data organization</li>
<li>Uses metadata structures inspired by Parquet format specification</li>
<li>Applies compression principles from the Parquet specification</li>
<li>Custom JavaScript implementation, not a full Apache Parquet compatible format</li>
<li>Best suited for basic tabular data storage and retrieval</li>
<li>Supports filtering via row‑id (rid) maps; query operations on
multiple columns efficiently intersect the provided rid sets</li>
</ul>
<h3>Error Handling</h3>
<p>Errors are reported in msg.error. Common issues:</p>
<ul>
<li>File not found</li>
<li>Invalid Columnar format</li>
<li>Schema mismatches</li>
<li>Permission issues</li>
</ul>
</script>