node-red-contrib-dynamorse-file-io/nodes/12-cloud-store-in.html

Dynamorse File IO nodes for Node-RED

<!--
  Copyright 2018 Streampunk Media Ltd.

  Licensed under the Apache License, Version 2.0 (the "License");
  you may not use this file except in compliance with the License.
  You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

  Unless required by applicable law or agreed to in writing, software
  distributed under the License is distributed on an "AS IS" BASIS,
  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  See the License for the specific language governing permissions and
  limitations under the License.
-->

<script type="text/javascript">
  RED.nodes.registerType('cloud-store-in',{
    category: 'funnel',
    color: '#FDE8C3',
    defaults: {
      name: { value : "" },
      description: { value: "" },
      maxBuffer : { value : 10, validate : RED.validators.number() },
      region: { value : "eu-west-1" },
      bucket: { value : "" },
      key: { value : "" },
      queueSize: { value : 4, validate : RED.validators.number() },
      loop : { value : false },
      regenerate : { value : false }
    },
    inputs:0,
    outputs:1,
    icon: "funnel.png",
    label: function() {
      return this.name || "cloud-store-in";
    }
  });
</script>

<style>
.param-table table, th, td {
  vertical-align: top;
  padding: 3px;
}
</style>

<script type="text/x-red" data-template-name="cloud-store-in">
  <!-- Common to all funnel nodes -->
  <div class="form-row">
    <label for="node-input-name"><i class="fa fa-tag"></i> name</label>
    <input type="text" id="node-input-name" placeholder="name">
  </div>
  <div class="form-row">
    <label for="node-input-description"><i class="fa fa-info-circle"></i> description</label>
    <input type="text" id="node-input-description" placeholder="source and flow description">
  </div>
  <div class="form-row">
    <label for="node-input-maxBuffer"><i class="fa fa-suitcase"></i> max buffer</label>
    <input type="number" id="node-input-maxBuffer" placeholder="10" style="width: 100px">
  </div>
  <div class="form-row"> <!-- TODO Consider changing to a drop down choice list. -->
   <label for="node-input-region"><i class="fa fa-globe" aria-hidden="true"></i> region</label>
   <select id="node-input-sampling" style="width: 120px">
     <option value="us-east-2">US East (Ohio)</option>
     <option value="us-east-1">US East (N. Virginia)</option>
     <option value="us-west-1">US West (N. California)</option>
     <option value="us-west-2">US West (Oregon)</option>
     <option value="ca-central-1">Canada (Central)</option>
     <option value="ap-south-1">Asia Pacific (Mumbai)</option>
     <option value="ap-northeast-2">Asia Pacific (Seoul)</option>
     <option value="ap-southeast-1">Asia Pacific (Singapore)</option>
     <option value="ap-southeast-2">Asia Pacific (Sydney)</option>
     <option value="ap-northeast-1">Asia Pacific (Tokyo)</option>
     <option value="eu-central-1">EU (Frankfurt)</option>
     <option value="eu-west-1" selected>EU (Ireland)</option>
     <option value="eu-west-2">EU (London)</option>
     <option value="sa-east-1">South America (São Paulo)</option>
   </select>
  </div>
  <div class="form-row">
   <label for="node-input-bucket"><i class="fa fa-archive" aria-hidden="true"></i> bucket</label>
   <input type="text" id="node-input-bucket" placeholder="bucket">
  </div>
  <div class="form-row">
   <label for="node-input-key"><i class="fa fa-key" aria-hidden="true"></i> key</label>
   <input type="text" id="node-input-key" placeholder="flowKey">
  </div>
  <div class="form-row">
    <label for="node-input-queueSize"><i class="fa fa-fighter-jet" aria-hidden="true"></i> queue size</label>
    <input type="number" min="1" id="node-input-queueSize" placeholder="4" style="width: 100px">
  </div>
  <div class="form-row">
    <label for="node-input-loop"><i class="fa fa-repeat"></i> loop?</label>
    <input type="checkbox" id="node-input-loop" style="width: 100px">
    <label for="node-input-regenerate"><i class="fa fa-clock-o"></i> regenerate?</label>
    <input type="checkbox" id="node-input-regenerate" style="width: 100px">
  </div>
</script>

<script type="text/x-red" data-help-name="cloud-store-in">

  <h1>Cloud store input</h1>
  <p>Read an object or objects that are sequence of concatenated payloads for
  grains from an AWS S3-style cloud store. Metdata stored with the objects
  allows for a stream to be recreated as it was stored.</p>

  <p>Prior to using this capability, security credentials must have been
  established so that Node.JS, and hence Node-RED, can access the buckets of an
  assocatied AWS account. This could be by creating a local file called
  <code>~/.aws/credentials</code> or by assigning an IAM role with S3 write
  permissions. See the AWS documentation on <a href="http://docs.aws.amazon.com/sdk-for-javascript/v2/developer-guide/setting-credentials-node.html">
  Setting Credntials in Node.js</a>.</p>

  <p>The bucket and objects that are to be read must already have been created.</p>

  <p>If the chunk size for the stream is sufficiently large (greater than 100000
  bytes) then the stream will be read in chunks in parallel. Configure this using
  the <em>queue size</em> parameter.</p>

  <h3>Parameters</h3>
  <div class="param-table">
  <table>
   <tr>
    <th>parameter</th><th>description</th>
   </tr>
   <tr>
    <td><i class="fa fa-tag" aria-hidden="true"></i> name</td>
    <td>Name of the node.</td>
   </tr>
   <tr>
    <td><i class="fa fa-info-circle"></i> description</td>
    <td>Optional description of the output NMOS source and flow.</td>
   </tr>
   <tr>
    <td><i class="fa fa-suitcase"></i> max buffer</td>
    <td>Maximum number of grains to buffer before overflowing.</td>
   </tr>
   <tr>
    <td><i class="fa fa-globe" aria-hidden="true"></i> region</td>
    <td>Region in S3 to use.</td>
   </tr>
   <tr>
    <td><i class="fa fa-archive" aria-hidden="true"></i> bucket</td>
    <td>Name of the S3 bucket where the grain payloads are stored.</td>
   </tr>
   <tr>
    <td><i class="fa fa-key" aria-hidden="true"></i> key</i></td>
    <td>Object key used to name the object or sequence of objects.</td>
   </tr>
   <tr>
    <td><i class="fa fa-fighter-jet" aria-hidden="true"></i> queue size</td>
    <td>Number of concurrent streams to use for stream reading. Set to 1 for synchronous.</td>
   </tr>
   <tr>
    <td><i class="fa fa-repeat"></i> loop?</td>
    <td>Should the file be repeated indefinitely?</td>
   </tr>
   <tr>
    <td><i class="fa fa-clock-o"></i> regenerate?</td>
    <td>Should the grain metadata be regenerated locally?</td>
   </tr>
  </table>
  </div>

  <p>Note that this interface does not use the S3 HTTP/S interface directly,
  relying instead on the official <a href="https://www.npmjs.com/package/aws-sdk">
  AWS Javascript SDK module <code>aws-sdk</code></a>.</p>

  <p>Streams of variable bitrate are not yet supported.</p>

  <h3>Maturity</h3>
  <p>Implementation is progress -
  <a href="https://github.com/Streampunk/dynamorse/issues/7">status</a>.
</script>