stream-flow-control
Version:
Stream Flow Control
1,309 lines (989 loc) • 54.3 kB
Markdown
# sfc: Streams Flow Control
[](https://gitlab.com/agustin.moyano/sfc/-/commits/master) [](https://gitlab.com/agustin.moyano/sfc/-/commits/master)
Node streams are great, I love them, but always felt that something was missing.
When you work with streams it's always linear..
I mean, you read something, then throw a bunch of transforms and then write that info somewhere.
_pipeline_ and _finish_ tools from _stream_ module are great, but still you don't have much control on how data flows,
specially if you need parallel execution, or flow to different streams on certain conditons, etc.
That's why I've created _sfc_. You can think of _sfc_ as a stateless workflow. You can flow a message contitionally through
one or multiple streams, transform them, apply a bunch of rules, chain many rules together and flow data to them when
specific events are fired, join multiple sources on a single stream, or just flow the first one that arrived, and then
you can wrap everything in a goal, for ease of processing.
Even better, you can define all your streams in a YAML or JSON file!
Basically this tool lets you define _pipeflows_ instead of _pipelines_.
* In a _pipeline_ you connect one pipe after another, forming a straight line.
* In _pipeflows_ you orchestrate how pipes are connected to each other.
## What's next
Add streams and custom types from npm modules
# Stream Flow Editor
There is a vscode extension available for visually editing stream flows: [Stream Flow Editor](https://marketplace.visualstudio.com/items?itemName=agmoyano.stream-flow-editor)
# Installation
```bash
npm install --save stream-flow-control
```
# How to use
```javascript
const {sfc} = require('stream-flow-control');
sfc.parseFile('./path/to/your/streams/definition');
const MyGoal = sfc.get('Goal', 'MyGoal');
```
## What's new
* __1.1.0__: Add support for building custom types. Now you can create a template of your stream, and reuse it within your yaml. Much as *_require* key in the top level, or inside a Goal, you can use *_customTypes* key to declare them.
* __1.2.0__: Add pause/resume features into manager.
## Tools
### sfc
This singleton is what you'd mostly interact. It has two crucial jobs.
The first is to manage instances of different kinds of streams.
Use _get_ method to retrieve an instance, and _set_ method to register an instance. Classes in this project autoregisters to sfc when you instance them, provided you gave it a name.
The second crucial job is to parse YAML or JSON files, or direcories with files and build _pipeflows_. Anything built through files is retrievable by sfc.
## Classes
### Flow
The following schemas explain, in part, how different flow classes work
* <a href="#FlowAll">FlowAll</a>
This schema shows the "when" condition. You can attach functions that evaluate the message. If the function returns true, the message is passed to every stream attached to the function, else no message is sent. You can attach a stream to a special condition called "none". If no "when" condition is met, then the message will be sent to the "none" stream.
* <a href="#FlowEach">FlowEach</a>
Sends a message for each element of the message. Tipically used when the message is an array and you need to process each item individually.
* <a href="#FlowFirst">FlowFirst</a>
Lets only the first message that arrives to pass through. By default, it groups the messages from different sources by the order in which they arrive, to determine which is the first, and which gets discarded, but the condition can be overwritten.
* <a href="#FlowHold">FlowHold</a>
Hold flow from multiple source streams until a condition is met. If no method is overwitten, then holds flow until end is signaled on all sources.
* <a href="#FlowJoin">FlowJoin</a>
Join messages from multiple sources into a single messages. If no method is overwitten, then releases a message when there is exactly one message from each source.
* <a href="#FlowOne">FlowOne</a>
Check all piped streams, and flow data to the first that matches criteria.
* <a href="#FlowWait">FlowWait</a>
Wait untill every source ended to emit end event.
### <a href="#Goal">Goal</a>
Goal is a _'black box'_ kind of class. You can hide complex piping and rules inside one of these.
Goal is a _Writable_, so it's got one input, but has two outputs: resolve and reject.
```javascript
// With Goal you can:
//Pipe to another stream
goal.resolve(successStream);
goal.reject(failStream);
reader.pipe(goal);
//Use node style callback
reader.pipe(goal).callback((err, data) => {
//... do something
});
//Use Promises
reader.pipe(goal)
.then(data => {})
.catch(err => {});
//Listen to events
goal.on('resolve', (data) => {});
goal.on('reject', (err) => {});
reader.pipe(goal);
```
Goal hooks to _error_ event from all it's children and emits a reject (on all it's variants: event, promise, callback, pipe), so make sure to channel all your errors through the callback if you are using a Transform or Writable within your goal.
### <a href="#Rule">Rule</a>
It's a class designed to model complex business logic, but it could be used for any purpose. You must override the method _rule_. Within that method you can emit as many events as you like, or return any value.
```javascript
cont rule = new Rule({
rule: (payload) => {
if(payload.data.x) {
// Every stream with someEvent will be sent streamed with data
this.emit('someEvent', payload.data);
// Every stream with someEvent will be sent streamed with data
this.emit('otherEvent', 'arbitrary message');
// If we don't return anything,
// nothing will be piped to resolve nor reject streams
} else if (payload.data.y) {
// We can modify payload
payload.data = {new: object};
// If we return true, modified payload will be sent to resolveStream
return true;
} else if(payload.data.z) {
// If false is returned, payload will be sent to rejectStream
return false;
} else if(payload.data.a) {
// If we return anything but 'true' or 'false' it will be sent to rejectStream
return 'An error message';
} else if(payload.data.b) {
//If we throw something, it will be sent to rejectStream
throw new Error('This is an exception');
}
}
});
//When 'someEvent' is fired, data will be piped to someStream
rule.chain('someEvent', someStream);
//When 'otherEvent' is fired, data will be piped to otherStream
rule.chain('otherEvent', otherStream);
//When rule is resolved, data will be piped to resolveStream
rule.resolve(resolveStream);
//When rule is rejected, data will be piped to rejectStream
rule.reject(rejectStream);
```
## YAML/JSON
You can configure your pipeflow through a (not so) simple yaml/json file.
This is a standard pipeflow definition:
```yaml
# Pipeflow definition
_require: #1
'{Router}': express #2
path: path #3
_customTypes: #4
typeName: #5
constructor: |
code of constructor that must return
a new stream
nameOfStream1: #6
# Stream definition
nameOfStream2: #6
# Stream definition
# .. define as many streams as you need
```
As you can see, to define multiple streams, you declare them one under the other, and then, in the stream definition, you can reference them by type and name to pipe it with other streams.
This is true for all streams, except for Goal, that it's the only one that has the special method _build_ where you can declare child streams, but we'll se that in detail later.
Now this is a standard stream definition:
```yaml
nameOfStrem: #6
type: #7
options: #8
someOption: someValue #9
someMethod: #10
_type: method
params:
- array
- with
- param
- names
code:
- console.log('lines of code')
- console.log('multiple lines if array')
constructor: | #11
return new MyStream(options)
methods: #12
methodName:
params:
- method
- param
code: |
code of
param
on: #13
eventName:
params:
- data
code:
- do something
- with data
once: #14
eventName:
params:
- data
code:
- do something
- with data
pon: #15
eventName:
params:
- data
code:
- do something
- with data
ponce: #16
eventName:
params:
- data
code:
- do something
- with data
when: #17
- cond:
- body of function
dst:
- type: dstType
name: dstName
pipe: #18
- type: dstType
name: dstName
none: #19
- type: dstType
name: dstName
resolve: #20
- type: dstType
name: dstName
reject: #21
- type: dstType
name: dstName
chain: #22
eventName:
- type: dstType
name: dstName
build: #23
# Pipeflow definition
```
Not all keys apply for every stream definition, but here it's placed all posibilities. Now I'll explain each part of the definition:
1. *_require*: There are certain definitions that later gets compiled to functions. If inside that function you need to call outside modules like _crypto_, _path_, etc, you can define them here, and they will be placed inside every compiled function's scope, along with _sfc_ and every class defined here, _Process_ and _Global_.
1. If inside *_require* you defined something like `'{MyClass, MySecondClass}': mymodule`, it's the equivalent as declaring `const {MyClass, MySecondClass} = require('mymodule')``
1. If inside *_require* you defined something like `name: mymodule`, it's the equivalent as declaring `const name = require('mymodule')`
1. *_customTypes*: If you got a kind of stream that you plan to use multiple times within your pipeflow, you can define that type here. Once you define a type, inside your stream definition you can use it as a value for the type key.
1. A type definition requires a _constructor_ key, where the stream get's built. You **must** return a stream instance.
1. Stream name. It's how this stream will be identified (along with type) by `sfc.get(...)`. It's the same as `new StreamType({name: 'nameOfStrem'})` for classes defined here.
1. Type of stream we are building. It's how this stream will be identified (along with type) by `sfc.get(...)`. It's the only mandatory parameter a stream definition needs. If you are building any of the classes defined in this project, then the value should be the class name, for example _FlowAll_, _Goal_, _Transform_, etc. But really, the value can be anything you imagine. Just rememeber to place the type name correctly when you reference them.
1. Options to be passed to the stream constructor.
1. If inside *options* you defined something like `someOption: someValue`, it's the same as `new MyStream({someOption: 'someValue'})`. Of course, this can be anything that yaml supports, as objects, arrays, etc.
1. If the option is an object, and the object contains `_type: method`, this option will be compiled to a function. So, this would be the equivalent as declaring `new MyStream({someMethod: function(array, with, param, names) { ... }})``
1. If you are using a class (that must be declared in context through _require, or be part of this project) and you don't want the _type_ to match the class name, or maybe the stream is created by calling a method like `fs.createReadStream()`, you'll need to define a constructor. The constructor function will always receive an _options_ parameter, so there's no need to define params key. In this definition you should just place the function body that returns the stream.
1. If you need to overwrite some method, you can declare it in the _methods_ section. A method definition must consist of a method name, params and function code. For classes defined in this project here are the possible methods you could overwrite:
* Writable
* write
* writev
* destroy
* final
* Readable
* read
* destroy
* Duplex
* write
* writev
* read
* destroy
* final
* Transform
* transform
* flush
* FlowHold
* hold
* release
* check
* FlowJoin
* join
* Rule
* rule
* FlowFirst
* identify
* match
* criteria
1. Register a listener for an event. Equivalent to declare `MyStream.on('eventName', function(data) {})`
1. Register a listener for an event. Equivalent to declare `MyStream.once('eventName', function(data) {})`
1. Register a listener for an event. Equivalent to declare `MyStream.prependListener('eventName', function(data) {})`
1. Register a listener for an event. Equivalent to declare `MyStream.prependOnceListener('eventName', function(data) {})`
1. Specific definition for _FlowAll_ and _FlowOne_ classes. It registers a conditional piping to a stream. It needs 2 parameters:
* _cond_: Condition to wich a payload is evaluated. This will be compiled to a function, and will always receive a 'payload' as a parameter. You should return _true_ or _false_ if you want to pipe data to _dst_.
* _dst_: Stream identifier if _cond_ returns true. It must consist of an object defining a _type_ and _name_.
1. It can be used with any class that defines a _pipe_ method. It can also be defined as an array of detinations.
1. It can be used with any class that defines a _none_ method. It depends on each class that implements this method on how it's used, but the standard way would be that data is sent through _none_ when no suitable stream could be found previously.
1. It can be used with any class that defines a _resolve_ method for piping to another stream.
1. It can be used with any class that defines a _reject_ method for piping to another stream.
1. Specific to _Rule_ class. You should define an event name, and streams to pipe to when this event is fired.
1. Specific to _Goal_ class. Here you define the 'pipeflow' messages will travel since they enter this goal. In _build_ you should place the exact same definition as a general pipeflow. You can define anything you like, even child goals, but there are several considerations you should keep in mind:
* If inside a _build_ you define a *_require*, it will be merged with the one defined in parent scope (if defined). So there's no need to declare twice something in child *_require*'s.
* Inside a _build_ definition, you must declare a `__goal__` key. This marks the starting point where child streams will be piped.
* Inside a _build_ definition, you can declare the strings `__resolve__` or `__reject__`, and messages will be sent to anything that was piped to the corresponding goal's methods.
```yaml
myGoal:
type: Goal
build:
__goal__:
- type: Transform
name: myTransform
myTransform:
type: Transform
methods:
transform:
params:
- payload
- encoding
- callback
code: |
try {
this.push(JSON.stringify(payload));
callback();
} catch(e) {
callback(e);
}
pipe: __resolve__
```
Remember that a goal hooks to the 'error' event of all it's children, so in this case, if payload is an object that can be strinifyed, then the message will be piped to any stream that hooked to the _resovle_ method of the goal. If payload cannot be stringifyed and an exception is thrown, the exception will be piped to any stream that hooked to the _reject_ method of the goal
### A note on compiling functions
There are some definitions in yaml/json that are compiled to functions. In every definition (except when specifically stated) you'll need 2 keys
* _params_: It's an array of strings that defines how parameters will be named and accessed in the function's code
* _code_: The body of the function. It can be stated as a string, or an array of strings that later will be joined with a new line.
so, for example the following definitions are equivalent:
```yaml
methods:
myMethod:
params:
- data
- callback
code: |
console.log(data)
callback()
---
methods:
myMethod:
params:
- data
- callback
code:
- console.log(data)
- callback()
```
both of this definitinons compile to the following function:
```javascript
function myMethod(data, callback) {
console.log(data)
callback()
}
```
### A note on referencing
The definitions that are used for piping data to other streams are _when_, _pipe_, _none_, _resolve_, _reject_ and _chain_. In everyone of them you must define a reference to one or more streams.
This means that you can, for example define de following
```yaml
pipe:
type: aType
name: aName
```
That it would be translated as
```javascript
myStream.pipe(sfc.get('aType', 'aName'))
```
But you could also pipe to multiple streams like
```yaml
pipe:
- type: aType
name: aName
- type: bType
name: bName
```
That it would be translated as
```javascript
myStream.pipe(sfc.get('aType', 'aName'))
myStream.pipe(sfc.get('bType', 'bName'))
```
When a message arrives to _myStream_ it will be delivered to both piped streams.
### A note on scopes
The only element that creates a diferent _scope_ is a _Goal_ instance. Because elements are only build when they are accessed, elements that where declared within an external or child goal may not yet exist.
So as a rule, it's safe to reference streams that exist on the same scope, or in the parent scope, but __do not__ try to reference a stream that's in a child or sibling scope.
<a name="api"></a>
# API
## Classes
<dl>
<dt><a href="#Goal">Goal</a> ⇐ <code>Writable</code></dt>
<dd><p>Goal wrapps many streams within, hidding complex business logic. Represents a goal to achieve.</p>
</dd>
<dt><a href="#DataWrapper">DataWrapper</a></dt>
<dd><p>Helper class that wrapps streaming payloads inside a <a href="#Goal"><code>Goal</code></a>. It holds history of in which streams it was flown through.</p>
</dd>
<dt><a href="#ReadableWrapper">ReadableWrapper</a> ⇐ <code>Readable</code></dt>
<dd><p>Wrapper for <em>Readable</em> streams that knows what to do with <a href="#DataWrapper">DataWrapper</a> messages.</p>
</dd>
<dt><a href="#WritableWrapper">WritableWrapper</a> ⇐ <code>Writable</code></dt>
<dd><p>Wrapper for <em>Writable</em> streams that knows what to do with <a href="#DataWrapper">DataWrapper</a> messages.</p>
</dd>
<dt><a href="#TransformWrapper">TransformWrapper</a> ⇐ <code>Transform</code></dt>
<dd><p>Wrapper for <em>Transform</em> streams that knows what to do with <a href="#DataWrapper">DataWrapper</a> messages.</p>
</dd>
<dt><a href="#DuplexWrapper">DuplexWrapper</a> ⇐ <code>Duplex</code></dt>
<dd><p>Wrapper for <em>Duplex</em> streams that knows what to do with <a href="#DataWrapper">DataWrapper</a> messages.</p>
</dd>
<dt><a href="#Manager">Manager</a> ⇐ <code>Writable</code></dt>
<dd><p>Mannager is your "toolbelt" class. It's responsible to regiter classes, fetch instances, parse yaml files and building stream chains.</p>
</dd>
<dt><a href="#Rule">Rule</a> ⇐ <code>Writable</code></dt>
<dd><p>A rule represents a piece of business logic. Streams can be chained by an arbitrary event name, or by resolve/reject functions.</p>
</dd>
<dt><a href="#FlowWait">FlowWait</a> ⇐ <code>Readable</code></dt>
<dd><p>Wait untill every source ended to emit end event.</p>
</dd>
<dt><a href="#FlowJoin">FlowJoin</a> ⇐ <code>Readable</code></dt>
<dd><p>Join messages from multiple sources into a single messages. If no method is overwitten, then releases a message when there is exactly one message from each source.</p>
</dd>
<dt><a href="#FlowAll">FlowAll</a> ⇐ <code>Writable</code></dt>
<dd><p>Check all piped streams, and flow data to all which matches criteria.</p>
</dd>
<dt><a href="#FlowHold">FlowHold</a> ⇐ <code>Readable</code></dt>
<dd><p>Hold flow from multiple source streams until a condition is met. If no method is overwitten, then holds flow until end is signaled on all sources.</p>
</dd>
<dt><a href="#FlowOne">FlowOne</a> ⇐ <code>Writable</code></dt>
<dd><p>Check all piped streams, and flow data to the first that matches criteria.</p>
</dd>
<dt><a href="#FlowEach">FlowEach</a> ⇐ <code>Writable</code></dt>
<dd><p>Stream every element of an array individually.</p>
</dd>
<dt><a href="#FlowFirst">FlowFirst</a> ⇐ <code>Readable</code></dt>
<dd><p>Flow the first message from multiple source streams and discard the rest.</p>
</dd>
</dl>
## Typedefs
<dl>
<dt><a href="#Condition">Condition</a> ⇒ <code>boolean</code></dt>
<dd><p>Condition function for flowing to a stream</p>
</dd>
<dt><a href="#Process">Process</a> : <code>function</code></dt>
<dd><p>Payload processing method</p>
</dd>
<dt><a href="#Then">Then</a> ⇒ <code>Promise</code></dt>
<dd><p>Thenable function</p>
</dd>
<dt><a href="#Catch">Catch</a> ⇒ <code>Promise</code></dt>
<dd><p>Thenable function</p>
</dd>
<dt><a href="#Identify">Identify</a> ⇒ <code>any</code></dt>
<dd><p>Returns the identity of a message. It could be an internal id, or an id of a wrapped message.</p>
</dd>
<dt><a href="#NodeCallback">NodeCallback</a> : <code>function</code></dt>
<dd><p>Node style callback</p>
</dd>
<dt><a href="#RuleProcess">RuleProcess</a> ⇒ <code>boolean</code> | <code>any</code> | <code>undefined</code></dt>
<dd><p>Process rules. You may emit events to send data to streams.</p>
</dd>
</dl>
<a name="Goal"></a>
## Goal ⇐ <code>Writable</code>
Goal wrapps many streams within, hidding complex business logic. Represents a goal to achieve.
**Kind**: global class
**Extends**: <code>Writable</code>
* [Goal](#Goal) ⇐ <code>Writable</code>
* [new Goal([options])](#new_Goal_new)
* [.resolve(dst)](#Goal+resolve)
* [.callback(cb)](#Goal+callback)
* [.reject(dst)](#Goal+reject)
* [.then(thenCallback, [catchCallback])](#Goal+then) ⇒ <code>Promise</code>
* [.catch(catchCallback)](#Goal+catch) ⇒ <code>Promise</code>
<a name="new_Goal_new"></a>
### new Goal([options])
Create a Goal stream
| Param | Type | Description |
| --- | --- | --- |
| [options] | <code>object</code> | Global options |
| [options.name] | <code>string</code> | Name for this stream |
<a name="Goal+resolve"></a>
### goal.resolve(dst)
Pipe to a destination stream when goal was achieved
**Kind**: instance method of [<code>Goal</code>](#Goal)
| Param | Type |
| --- | --- |
| dst | <code>Writable</code> |
<a name="Goal+callback"></a>
### goal.callback(cb)
Register a node style callback, called when goal is resolved or rejected
**Kind**: instance method of [<code>Goal</code>](#Goal)
| Param | Type |
| --- | --- |
| cb | [<code>NodeCallback</code>](#NodeCallback) |
<a name="Goal+reject"></a>
### goal.reject(dst)
Pipe to a destination stream when goal was not achieved
**Kind**: instance method of [<code>Goal</code>](#Goal)
| Param | Type |
| --- | --- |
| dst | <code>Writable</code> |
<a name="Goal+then"></a>
### goal.then(thenCallback, [catchCallback]) ⇒ <code>Promise</code>
Thenable function to work with promises.
**Kind**: instance method of [<code>Goal</code>](#Goal)
| Param | Type | Description |
| --- | --- | --- |
| thenCallback | <code>ThenCallback</code> | called when goal is achieved |
| [catchCallback] | <code>CatchCallback</code> | called when goal is not achieved |
<a name="Goal+catch"></a>
### goal.catch(catchCallback) ⇒ <code>Promise</code>
Thenable function to work with promises.
**Kind**: instance method of [<code>Goal</code>](#Goal)
| Param | Type | Description |
| --- | --- | --- |
| catchCallback | <code>CatchCallback</code> | called when goal is not achieved |
<a name="DataWrapper"></a>
## DataWrapper
Helper class that wrapps streaming payloads inside a [<code>Goal</code>](#Goal). It holds history of in which streams it was flown through.
**Kind**: global class
<a name="ReadableWrapper"></a>
## ReadableWrapper ⇐ <code>Readable</code>
Wrapper for _Readable_ streams that knows what to do with [DataWrapper](#DataWrapper) messages.
**Kind**: global class
**Extends**: <code>Readable</code>
<a name="WritableWrapper"></a>
## WritableWrapper ⇐ <code>Writable</code>
Wrapper for _Writable_ streams that knows what to do with [DataWrapper](#DataWrapper) messages.
**Kind**: global class
**Extends**: <code>Writable</code>
<a name="TransformWrapper"></a>
## TransformWrapper ⇐ <code>Transform</code>
Wrapper for _Transform_ streams that knows what to do with [DataWrapper](#DataWrapper) messages.
**Kind**: global class
**Extends**: <code>Transform</code>
<a name="DuplexWrapper"></a>
## DuplexWrapper ⇐ <code>Duplex</code>
Wrapper for _Duplex_ streams that knows what to do with [DataWrapper](#DataWrapper) messages.
**Kind**: global class
**Extends**: <code>Duplex</code>
<a name="Manager"></a>
## Manager ⇐ <code>Writable</code>
Mannager is your "toolbelt" class. It's responsible to regiter classes, fetch instances, parse yaml files and building stream chains.
**Kind**: global class
**Extends**: <code>Writable</code>
* [Manager](#Manager) ⇐ <code>Writable</code>
* [.clean()](#Manager+clean) ⇒ [<code>Manager</code>](#Manager)
* [.pause()](#Manager+pause) ⇒ [<code>Manager</code>](#Manager)
* [.resume()](#Manager+resume) ⇒ [<code>Manager</code>](#Manager)
* [.set(type, [name], elem)](#Manager+set)
* [.get(type, name)](#Manager+get) ⇒ <code>Readable</code> \| <code>Writable</code> \| <code>null</code>
* [.parse(confStr, cb)](#Manager+parse)
* [.parseFiles([filePaths])](#Manager+parseFiles) ⇒ [<code>Manager</code>](#Manager)
<a name="Manager+clean"></a>
### sfc.clean() ⇒ [<code>Manager</code>](#Manager)
Clean namager state
Useful when you need to start again
**Kind**: instance method of [<code>Manager</code>](#Manager)
**Returns**: [<code>Manager</code>](#Manager) - 'this' element for chainable purpose.
<a name="Manager+pause"></a>
### sfc.pause() ⇒ [<code>Manager</code>](#Manager)
Set manager in paused mode.
Use it before parsing files like `sfc.pause().parseFiles(...)`
Useful when you need all streams to be created before piping
**Kind**: instance method of [<code>Manager</code>](#Manager)
**Returns**: [<code>Manager</code>](#Manager) - 'this' element for chainable purpose.
<a name="Manager+resume"></a>
### sfc.resume() ⇒ [<code>Manager</code>](#Manager)
Set inner streams in fowing mode.
Use it after parsing files like `sfc.pause().parseFiles(...).resume()`
Useful when you need all streams to be created before piping
**Kind**: instance method of [<code>Manager</code>](#Manager)
**Returns**: [<code>Manager</code>](#Manager) - 'this' element for chainable purpose.
<a name="Manager+set"></a>
### sfc.set(type, [name], elem)
Register a stream
**Kind**: instance method of [<code>Manager</code>](#Manager)
| Param | Type | Description |
| --- | --- | --- |
| type | <code>string</code> | type of stream |
| [name] | <code>string</code> | name for this stream |
| elem | <code>Writable</code> \| <code>Readable</code> | the stream to register |
<a name="Manager+get"></a>
### sfc.get(type, name) ⇒ <code>Readable</code> \| <code>Writable</code> \| <code>null</code>
Retrieve a stream by type and name. If stream has ended, creates a new instance
**Kind**: instance method of [<code>Manager</code>](#Manager)
**Returns**: <code>Readable</code> \| <code>Writable</code> \| <code>null</code> - returns the indicated stream or null if not found
| Param | Type | Description |
| --- | --- | --- |
| type | <code>string</code> | type of stream |
| name | <code>string</code> | the name of the stream to retrieve |
<a name="Manager+parse"></a>
### sfc.parse(confStr, cb)
Parse a JSON or YAML string and create streams
**Kind**: instance method of [<code>Manager</code>](#Manager)
| Param | Type | Description |
| --- | --- | --- |
| confStr | <code>string</code> | JSON or YAML string. |
| cb | [<code>NodeCallback</code>](#NodeCallback) | Callback returning an error if occurred. |
<a name="Manager+parseFiles"></a>
### sfc.parseFiles([filePaths]) ⇒ [<code>Manager</code>](#Manager)
Parse a file or directory and build streams. If no parameter is passed, parseFiles will automatically search for _sfc_ directory, and files that ends with _.sfc_, _.sfc.yaml_, _.sfc.yml_ or _.sfc.json_.
**Kind**: instance method of [<code>Manager</code>](#Manager)
**Returns**: [<code>Manager</code>](#Manager) - returns 'this' for chainable purposes
| Param | Type | Description |
| --- | --- | --- |
| [filePaths] | <code>string</code> \| <code>array</code> | path of yaml/json file or a directory containing those files. |
<a name="Rule"></a>
## Rule ⇐ <code>Writable</code>
A rule represents a piece of business logic. Streams can be chained by an arbitrary event name, or by resolve/reject functions.
**Kind**: global class
**Extends**: <code>Writable</code>
* [Rule](#Rule) ⇐ <code>Writable</code>
* [new Rule([options])](#new_Rule_new)
* [.chain(event, dst)](#Rule+chain)
* [.resolve(dst)](#Rule+resolve)
* [.reject(dst)](#Rule+reject)
<a name="new_Rule_new"></a>
### new Rule([options])
Create a Rule stream
| Param | Type | Description |
| --- | --- | --- |
| [options] | <code>object</code> | Global options |
| [options.name] | <code>string</code> | Name for this stream |
| options.rule | [<code>RuleProcess</code>](#RuleProcess) | Method that defines the rule logic. If inside the rule you emit an event with data, it will be sent to streams attached to the corresponding event on chain method. If this function returns exactly _true_, incoming payload will be sent to streams attached in _resolve_ function. If this function returns exactly _false_, incoming payload will be sent to streams attached in _reject_ function. If this function returns _undefined_, no message will be sent to either streams attached to _resolve_ or _reject_. If anything else is returned, or an exception is thrown, that will be sent to streams attached in _reject_ function. |
<a name="Rule+chain"></a>
### rule.chain(event, dst)
Register streams to pipe to when _event_ is fired.
**Kind**: instance method of [<code>Rule</code>](#Rule)
| Param | Type | Description |
| --- | --- | --- |
| event | <code>string</code> | event name. |
| dst | <code>Writable</code> | stream to pipe to. |
<a name="Rule+resolve"></a>
### rule.resolve(dst)
Register streams to pipe to when _options.rule_ returns _true_
**Kind**: instance method of [<code>Rule</code>](#Rule)
| Param | Type | Description |
| --- | --- | --- |
| dst | <code>Writable</code> | stream to pipe to |
<a name="Rule+reject"></a>
### rule.reject(dst)
Register streams to pipe to when _options.rule_ returns anything else __but__ _true_, or when an exception is thrown.
**Kind**: instance method of [<code>Rule</code>](#Rule)
| Param | Type | Description |
| --- | --- | --- |
| dst | <code>Writable</code> | stream to pipe to |
<a name="FlowWait"></a>
## FlowWait ⇐ <code>Readable</code>
Wait untill every source ended to emit end event.
**Kind**: global class
**Extends**: <code>Readable</code>
<a name="new_FlowWait_new"></a>
### new FlowWait(options)
Create a FlowWait stream
| Param | Type | Description |
| --- | --- | --- |
| options | <code>object</code> | Global options. |
| [options.name] | <code>string</code> | Name for this stream. |
<a name="FlowJoin"></a>
## FlowJoin ⇐ <code>Readable</code>
Join messages from multiple sources into a single messages. If no method is overwitten, then releases a message when there is exactly one message from each source.
**Kind**: global class
**Extends**: <code>Readable</code>
<a name="new_FlowJoin_new"></a>
### new FlowJoin(options)
Create a FlowJoin stream
| Param | Type | Description |
| --- | --- | --- |
| options | <code>object</code> | Global options. |
| [options.name] | <code>string</code> | Name for this stream. |
| [options.join] | [<code>Process</code>](#Process) | How messages are joined. |
<a name="FlowAll"></a>
## FlowAll ⇐ <code>Writable</code>
Check all piped streams, and flow data to all which matches criteria.
**Kind**: global class
**Extends**: <code>Writable</code>
* [FlowAll](#FlowAll) ⇐ <code>Writable</code>
* [new FlowAll([options])](#new_FlowAll_new)
* [.when(cond, dst)](#FlowAll+when) ⇒ <code>Writable</code> \| [<code>FlowAll</code>](#FlowAll)
* [.pipe(dst)](#FlowAll+pipe) ⇒ <code>Writable</code> \| [<code>FlowAll</code>](#FlowAll)
* [.none(dst)](#FlowAll+none) ⇒ <code>Writable</code> \| [<code>FlowAll</code>](#FlowAll)
* [._write(payload, encoding, cb)](#FlowAll+_write)
<a name="new_FlowAll_new"></a>
### new FlowAll([options])
Create a FlowAll stream
| Param | Type | Description |
| --- | --- | --- |
| [options] | <code>object</code> | Global options |
| [options.name] | <code>string</code> | Name for this stream |
<a name="FlowAll+when"></a>
### flowAll.when(cond, dst) ⇒ <code>Writable</code> \| [<code>FlowAll</code>](#FlowAll)
Set stream(s) to pipe if criteria is match
**Kind**: instance method of [<code>FlowAll</code>](#FlowAll)
**Returns**: <code>Writable</code> \| [<code>FlowAll</code>](#FlowAll) - if dst is an array, then returns "this", else dst is returned
| Param | Type |
| --- | --- |
| cond | [<code>Condition</code>](#Condition) |
| dst | <code>Writable</code> \| <code>Array.<Writable></code> |
<a name="FlowAll+pipe"></a>
### flowAll.pipe(dst) ⇒ <code>Writable</code> \| [<code>FlowAll</code>](#FlowAll)
Set stream(s) to pipe unconditionally
**Kind**: instance method of [<code>FlowAll</code>](#FlowAll)
**Returns**: <code>Writable</code> \| [<code>FlowAll</code>](#FlowAll) - if dst is an array, then returns "this", else dst is returned
| Param | Type |
| --- | --- |
| dst | <code>Writable</code> \| <code>Array.<Writable></code> |
<a name="FlowAll+none"></a>
### flowAll.none(dst) ⇒ <code>Writable</code> \| [<code>FlowAll</code>](#FlowAll)
Set stream(s) to pipe when no other stream matched criteria. If unconditional piping was set, this will never be used.
**Kind**: instance method of [<code>FlowAll</code>](#FlowAll)
**Returns**: <code>Writable</code> \| [<code>FlowAll</code>](#FlowAll) - if dst is an array, then returns "this", else dst is returned
| Param | Type | Description |
| --- | --- | --- |
| dst | <code>Writable</code> \| <code>Array.<Writable></code> | Destination stream |
<a name="FlowAll+_write"></a>
### flowAll.\_write(payload, encoding, cb)
Internal _write method. Do not call directly.
Do not override unless you are sure of what you are doing
**Kind**: instance method of [<code>FlowAll</code>](#FlowAll)
| Param | Type | Description |
| --- | --- | --- |
| payload | [<code>DataWrapper</code>](#DataWrapper) \| <code>\*</code> | chunk of data to be written |
| encoding | <code>string</code> | encoding of data when objectMode=false. Never used. |
| cb | <code>WriteCallback</code> | Internal Writable callback |
<a name="FlowHold"></a>
## FlowHold ⇐ <code>Readable</code>
Hold flow from multiple source streams until a condition is met. If no method is overwitten, then holds flow until end is signaled on all sources.
**Kind**: global class
**Extends**: <code>Readable</code>
* [FlowHold](#FlowHold) ⇐ <code>Readable</code>
* [new FlowHold(options)](#new_FlowHold_new)
* [._sources](#FlowHold+_sources)
* [._payloads](#FlowHold+_payloads)
<a name="new_FlowHold_new"></a>
### new FlowHold(options)
Create a FlowHold stream
| Param | Type | Description |
| --- | --- | --- |
| options | <code>object</code> | Global options. |
| [options.name] | <code>string</code> | Name for this stream. |
| [options.hold] | [<code>Process</code>](#Process) | How messages are stored. |
| [options.check] | <code>function</code> | Checks a certain condition is met and call _options.release_. |
| [options.release] | <code>function</code> | Controls how messages are released. |
<a name="FlowHold+_sources"></a>
### flowHold.\_sources
**Kind**: instance property of [<code>FlowHold</code>](#FlowHold)
**Properties**
| Name | Description |
| --- | --- |
| Source | streams |
<a name="FlowHold+_payloads"></a>
### flowHold.\_payloads
**Kind**: instance property of [<code>FlowHold</code>](#FlowHold)
**Properties**
| Name | Description |
| --- | --- |
| Stored | messages. There is an array for each source |
<a name="FlowOne"></a>
## FlowOne ⇐ <code>Writable</code>
Check all piped streams, and flow data to the first that matches criteria.
**Kind**: global class
**Extends**: <code>Writable</code>
* [FlowOne](#FlowOne) ⇐ <code>Writable</code>
* [new FlowOne([options])](#new_FlowOne_new)
* [.when(cond, dst)](#FlowOne+when) ⇒ <code>Writable</code> \| [<code>FlowAll</code>](#FlowAll)
* [.pipe(dst)](#FlowOne+pipe) ⇒ <code>Writable</code> \| [<code>FlowAll</code>](#FlowAll)
* [.none(dst)](#FlowOne+none) ⇒ <code>Writable</code> \| [<code>FlowAll</code>](#FlowAll)
* [._write(payload, encoding, cb)](#FlowOne+_write)
<a name="new_FlowOne_new"></a>
### new FlowOne([options])
Create a FlowOne stream
| Param | Type | Description |
| --- | --- | --- |
| [options] | <code>object</code> | Global options |
| [options.name] | <code>string</code> | Name for this stream |
<a name="FlowOne+when"></a>
### flowOne.when(cond, dst) ⇒ <code>Writable</code> \| [<code>FlowAll</code>](#FlowAll)
Set stream(s) to pipe if criteria is match
**Kind**: instance method of [<code>FlowOne</code>](#FlowOne)
**Returns**: <code>Writable</code> \| [<code>FlowAll</code>](#FlowAll) - if dst is an array, then returns "this", else dst is returned
| Param | Type |
| --- | --- |
| cond | [<code>Condition</code>](#Condition) |
| dst | <code>Writable</code> \| <code>Array.<Writable></code> |
<a name="FlowOne+pipe"></a>
### flowOne.pipe(dst) ⇒ <code>Writable</code> \| [<code>FlowAll</code>](#FlowAll)
Set stream(s) to pipe unconditionally
**Kind**: instance method of [<code>FlowOne</code>](#FlowOne)
**Returns**: <code>Writable</code> \| [<code>FlowAll</code>](#FlowAll) - if dst is an array, then returns "this", else dst is returned
| Param | Type |
| --- | --- |
| dst | <code>Writable</code> \| <code>Array.<Writable></code> |
<a name="FlowOne+none"></a>
### flowOne.none(dst) ⇒ <code>Writable</code> \| [<code>FlowAll</code>](#FlowAll)
Set stream(s) to pipe when no other stream matched criteria. If unconditional piping was set, this will never be used.
**Kind**: instance method of [<code>FlowOne</code>](#FlowOne)
**Returns**: <code>Writable</code> \| [<code>FlowAll</code>](#FlowAll) - if dst is an array, then returns "this", else dst is returned
| Param | Type | Description |
| --- | --- | --- |
| dst | <code>Writable</code> \| <code>Array.<Writable></code> | Destination stream |
<a name="FlowOne+_write"></a>
### flowOne.\_write(payload, encoding, cb)
Internal _write method. Do not call directly.
Do not override unless you are sure of what you are doing
**Kind**: instance method of [<code>FlowOne</code>](#FlowOne)
| Param | Type | Description |
| --- | --- | --- |
| payload | [<code>DataWrapper</code>](#DataWrapper) \| <code>\*</code> | chunk of data to be written |
| encoding | <code>string</code> | encoding of data when objectMode=false. Never used. |
| cb | <code>WriteCallback</code> | Internal Writable callback |
<a name="FlowEach"></a>
## FlowEach ⇐ <code>Writable</code>
Stream every element of an array individually.
**Kind**: global class
**Extends**: <code>Writable</code>
* [FlowEach](#FlowEach) ⇐ <code>Writable</code>
* [new FlowEach([options])](#new_FlowEach_new)
* [.pipe(dst)](#FlowEach+pipe) ⇒ <code>Writable</code> \| [<code>FlowAll</code>](#FlowAll)
* [.none(dst)](#FlowEach+none) ⇒ <code>Writable</code> \| [<code>FlowAll</code>](#FlowAll)
<a name="new_FlowEach_new"></a>
### new FlowEach([options])
Create a FlowEach stream
| Param | Type | Description |
| --- | --- | --- |
| [options] | <code>object</code> | Global options |
| [options.name] | <code>string</code> | Name for this stream |
<a name="FlowEach+pipe"></a>
### flowEach.pipe(dst) ⇒ <code>Writable</code> \| [<code>FlowAll</code>](#FlowAll)
Set stream(s) to pipe to
**Kind**: instance method of [<code>FlowEach</code>](#FlowEach)
**Returns**: <code>Writable</code> \| [<code>FlowAll</code>](#FlowAll) - if dst is an array, then returns "this", else dst is returned
| Param | Type |
| --- | --- |
| dst | <code>Writable</code> \| <code>Array.<Writable></code> |
<a name="FlowEach+none"></a>
### flowEach.none(dst) ⇒ <code>Writable</code> \| [<code>FlowAll</code>](#FlowAll)
Set stream(s) to pipe to when no other stream was piped
**Kind**: instance method of [<code>FlowEach</code>](#FlowEach)
**Returns**: <code>Writable</code> \| [<code>FlowAll</code>](#FlowAll) - if dst is an array, then returns "this", else dst is returned
| Param | Type | Description |
| --- | --- | --- |
| dst | <code>Writable</code> \| <code>Array.<Writable></code> | Destination stream |
<a name="FlowFirst"></a>
## FlowFirst ⇐ <code>Readable</code>
Flow the first message from multiple source streams and discard the rest.
**Kind**: global class
**Extends**: <code>Readable</code>
* [FlowFirst](#FlowFirst) ⇐ <code>Readable</code>
* [new FlowFirst(options)](#new_FlowFirst_new)
* [._sources](#FlowFirst+_sources)
<a name="new_FlowFirst_new"></a>
### new FlowFirst(options)
Create a FlowFirst stream
| Param | Type | Description |
| --- | --- | --- |
| options | <code>object</code> | Global options |
| [options.name] | <code>string</code> | Name for this stream |
| options.identify | [<code>Identify</code>](#Identify) | Function that returns a message id to match |
| [options.criteria] | [<code>Condition</code>](#Condition) | Add an extra criteria for message to match. Return false to discard the message. |
| [options.match] | [<code>Process</code>](#Process) | How messages are matched according to _options.identify_ function. |
<a name="FlowFirst+_sources"></a>
### flowFirst.\_sources
**Kind**: instance property of [<code>FlowFirst</code>](#FlowFirst)
**Properties**
| Name | Description |
| --- | --- |
| Source | streams |
<a name="Condition"></a>
## Condition ⇒ <code>boolean</code>
Condition function for flowing to a stream
**Kind**: global typedef
**Returns**: <code>boolean</code> - if true, condition is accepted and data is flown to stream
| Param | Type | Description |
| --- | --- | --- |
| payload | [<code>DataWrapper</code>](#DataWrapper) \| <code>any</code> | chunk that's flown through the stream |
<a name="Process"></a>
## Process : <code>function</code>
Payload processing method
**Kind**: global typedef
| Param | Type | Description |
| --- | --- | --- |
| payload | [<code>DataWrapper</code>](#DataWrapper) \| <code>any</code> | chunk that's going to be processed somehow |
<a name="Then"></a>
## Then ⇒ <code>Promise</code>
Thenable function
**Kind**: global typedef
| Param | Type | Description |
| --- | --- | --- |
| payload | [<code>DataWrapper</code>](#DataWrapper) \| <code>any</code> | chunk that's going to be processed somehow |
<a name="Catch"></a>
## Catch ⇒ <code>Promise</code>
Thenable function
**Kind**: global typedef
| Param | Type | Description |
| --- | --- | --- |
| error | [<code>DataWrapper</code>](#DataWrapper) \| <code>any</code> | error message |
<a name="Identify"></a>
## Identify ⇒ <code>any</code>
Returns the identity of a message. It could be an internal id, or an id of a wrapped message.
**Kind**: global typedef
**Returns**: <code>any</code> - identity of the message
| Param | Type | Description |
| --- | --- | --- |
| payload | [<code>DataWrapper</code>](#DataWrapper) \| <code>any</code> | chunk that's flown through the stream |
<a name="NodeCallback"></a>
## NodeCallback : <code>function</code>
Node style callback
**Kind**: global typedef
| Param | Type |
| --- | --- |
| error | <code>any</code> |
| data | <code>any</code> |
<a name="RuleProcess"></a>
## RuleProcess ⇒ <code>boolean</code> \| <code>any</code> \| <code>undefined</code>
Process rules. You may emit events to send data to streams.
**Kind**: global typedef
| Param | Description |
| --- | --- |
| (DataWrapper|any)} | payload chunk that's flown through the stream |
# Example
Here's a rather complex YAML configuration example. You can find how all piece together [here](https://gitlab.com/agustin.moyano/sfc/-/tree/master/test/test2)
```yaml
_require:
crypto: crypto
fs: fs
authenticate:
type: Goal
build:
__goal__:
type: FlowAll
name: choose_method
choose_method:
options:
none_reason:
code: 404
message: 'No valid authentication method provided'
type: FlowAll
when:
- cond: |
const req = payload.data
return req.headers && req.headers.authorization && /^Basic /.test(req.headers.authorization)
dst:
type: Transform
name: parse_basic
- cond: |
const req = payload.data
if(!(req.query && req.query.user && req.query.pass) && !(req.body && req.body.user && req.body.pass)) return false
req.auth = {user: req.query.user||req.body.user, pass: req.query.pass||req.body.pass}
return true
dst:
type: FlowFirst
name: first_auth
- cond: |
const req = payload.data
return (req.query && req.query.token) || (req.headers && req.headers.authorization && /^Bearer /.test(req.headers.authorization))
dst:
type: Transform
name: parse_token
none: __reject__
once:
prefinish:
code: |
const payload = new DataWrapper(this.goal._src, this);
this._write(payload, null, ()=>{});
parse_basic:
type: Transform
methods:
transform:
code: |
let basic = payload.data.headers.authorization.replace(/^Basic (.*)$/, (str, match)=>match);
let auth_array = Buffer.from(basic, 'base64').toString('utf8').split(':');
payload.data.auth = {user: auth_array[0], pass: auth_array[1]};
this.push(payload);
cb()
params:
- payload
- encoding
- cb
pipe:
type: FlowFirst
name: first_auth
first_auth:
type: FlowFirst
methods:
identify:
code: |
return payload.map.FlowAll.choose_method
params:
- payload
pipe:
type: Transform
name: validate_user
parse_token:
type: Transform
methods:
transform:
code: |
try {
const tokens = JSON.parse(fs.readFileSync('./tokens.json')).tokens
const token = (payload.data.query && payload.data.query.token) || payload.data.headers.authorization.replace(/^Bearer (.*)$/, (str, match)=>match)
if(!tokens.hasOwnProperty(token)) return callback(payload.getChild(this).setData({code: 401, message: 'Invalid token'}))
payload.data.userid = tokens[token]
this.push(payload)
callback()
} catch(e) {
callback(payload.getChild(this).setData({code: 500, message: e.message}))