streamer

<!DOCTYPE html> <html> <head> <title>readme.js</title> <meta http-equiv="content-type" content="text/html; charset=UTF-8"> <link rel="stylesheet" media="all" href="docco.css" /> </head> <body> <div id="container"> <div id="background"></div> <table cellpadding="0" cellspacing="0"> <thead> <tr> <th class="docs"> <h1> readme.js </h1> </th> <th class="code"> </th> </tr> </thead> <tbody> <tr id="section-1"> <td class="docs"> <div class="pilwrap"> <a class="pilcrow" href="#section-1">¶</a> </div> </td> <td class="code"> <div class="highlight"><pre>/* vim:set ts=2 sw=2 sts=2 expandtab */ /*jshint asi: true undef: true es5: true node: true devel: true forin: true latedef: false globalstrict: true */ /*global define: true setInterval: true */ /* html version: http://jeditoolkit.com/streamer/docs/readme.html */ 'use strict';</pre></div> </td> </tr> <tr id="section-2"> <td class="docs"> <div class="pilwrap"> <a class="pilcrow" href="#section-2">¶</a> </div> In computing, the term stream is used in a number of ways, in all cases referring to a sequence of elements made available over time. </td> <td class="code"> <div class="highlight"><pre></pre></div> </td> </tr> <tr id="section-3"> <td class="docs"> <div class="pilwrap"> <a class="pilcrow" href="#section-3">¶</a> </div> Let's create a very basic stream representing a sequence of elements from 1 to 3. </td> <td class="code"> <div class="highlight"><pre>function stream(next) { [ 1, 2, 3 ].forEach(function(element) { next(element) }) }</pre></div> </td> </tr> <tr id="section-4"> <td class="docs"> <div class="pilwrap"> <a class="pilcrow" href="#section-4">¶</a> </div> From this example we can define a stream as: a function representing a sequence of elements. It can be read by calling with one function argument, which will be called back with each element of the sequence. </td> <td class="code"> <div class="highlight"><pre></pre></div> </td> </tr> <tr id="section-5"> <td class="docs"> <div class="pilwrap"> <a class="pilcrow" href="#section-5">¶</a> </div> So we can print our stream like this: </td> <td class="code"> <div class="highlight"><pre>stream(function onEach(element) { console.log(element) })</pre></div> </td> </tr> <tr id="section-6"> <td class="docs"> <div class="pilwrap"> <a class="pilcrow" href="#section-6">¶</a> </div> <pre><code> 1 2 3 </code></pre> </td> <td class="code"> <div class="highlight"><pre></pre></div> </td> </tr> <tr id="section-7"> <td class="docs"> <div class="pilwrap"> <a class="pilcrow" href="#section-7">¶</a> </div> Or, we can create a convenience <a href="http://en.wikipedia.org/wiki/Higher-order_function">high-order function</a> for printing streams. </td> <td class="code"> <div class="highlight"><pre>function print(stream) { stream(function onEach(element) { console.log(element) // Print each element of the sequence. }) }</pre></div> </td> </tr> <tr id="section-8"> <td class="docs"> <div class="pilwrap"> <a class="pilcrow" href="#section-8">¶</a> </div> And, print stream with it: </td> <td class="code"> <div class="highlight"><pre>print(stream)</pre></div> </td> </tr> <tr id="section-9"> <td class="docs"> <div class="pilwrap"> <a class="pilcrow" href="#section-9">¶</a> </div> <pre><code> 1 2 3 </code></pre> </td> <td class="code"> <div class="highlight"><pre></pre></div> </td> </tr> <tr id="section-10"> <td class="docs"> <div class="pilwrap"> <a class="pilcrow" href="#section-10">¶</a> </div> Good, but! Stream is a sequence of elements made available over time. In other words a sequence may be lazy, and our stream definition needs refinement: Stream is a function representing a sequence of elements. It MAY be read by calling it with one function argument, that will be called back with each element when it becomes available. </td> <td class="code"> <div class="highlight"><pre></pre></div> </td> </tr> <tr id="section-11"> <td class="docs"> <div class="pilwrap"> <a class="pilcrow" href="#section-11">¶</a> </div> Let's create a function <code>numbers</code>, that takes <code>min</code> and <code>max</code> numbers and returns a lazy stream of random numbers in a given range. To make the stream lazy, we will make its new elements available every 20ms. </td> <td class="code"> <div class="highlight"><pre>function numbers(min, max) { // Another high-order function to make streams var delta = max - min return function stream(next) { // Actual stream that generates setInterval(function generate() {</pre></div> </td> </tr> <tr id="section-12"> <td class="docs"> <div class="pilwrap"> <a class="pilcrow" href="#section-12">¶</a> </div> We yield a random number in given range every 20ms. </td> <td class="code"> <div class="highlight"><pre> next(min + Math.round(Math.random() * delta)) }, 20) } }</pre></div> </td> </tr> <tr id="section-13"> <td class="docs"> <div class="pilwrap"> <a class="pilcrow" href="#section-13">¶</a> </div> Make a stream of random numbers in the 0 - 100 range. </td> <td class="code"> <div class="highlight"><pre>var numberStream = numbers(0, 100)</pre></div> </td> </tr> <tr id="section-14"> <td class="docs"> <div class="pilwrap"> <a class="pilcrow" href="#section-14">¶</a> </div> And we print it!! </td> <td class="code"> <div class="highlight"><pre>print(numberStream)</pre></div> </td> </tr> <tr id="section-15"> <td class="docs"> <div class="pilwrap"> <a class="pilcrow" href="#section-15">¶</a> </div> <pre><code> 29 33 45 .... </code></pre> </td> <td class="code"> <div class="highlight"><pre></pre></div> </td> </tr> <tr id="section-16"> <td class="docs"> <div class="pilwrap"> <a class="pilcrow" href="#section-16">¶</a> </div> Oops! The stream keeps printing these numbers infinitely. Right, that's because a stream is infinite! So we may have finite and infinite streams, the difference being that finite streams end / stop at some point. And if a stream stops we need to know when that happens. To do that we will add a second, optional <code>stop</code> callback argument that MUST be called once stream reaches its end. Let's redefine our <code>print</code> function with this in mind: </td> <td class="code"> <div class="highlight"><pre>function print(stream) { console.log(">>>") // Opening stream for reading stream(function onElement(element) { console.log(element) // Print each element of stream. }, function onStop() { console.log("<<<") // Stream is stopped. }) }</pre></div> </td> </tr> <tr id="section-17"> <td class="docs"> <div class="pilwrap"> <a class="pilcrow" href="#section-17">¶</a> </div> Now we need a stream to print. Instead of creating another basic stream, this time we will take more generic approach by defining a function that takes an array as an argument and returns a stream of its elements: </td> <td class="code"> <div class="highlight"><pre>function list(array) { return function stream(next, stop) { array.forEach(function(element) { next(element) }) stop() } }</pre></div> </td> </tr> <tr id="section-18"> <td class="docs"> <div class="pilwrap"> <a class="pilcrow" href="#section-18">¶</a> </div> Great let's print something now! </td> <td class="code"> <div class="highlight"><pre>print(list(1, 2, 3))</pre></div> </td> </tr> <tr id="section-19"> <td class="docs"> <div class="pilwrap"> <a class="pilcrow" href="#section-19">¶</a> </div> Right, we should have passed an array to the list. Yeah, so shit happens! And when it happens to the stream, it needs to do something about it. The only reasonable thing is to recover, and if that is not possible then stop and report the reason of failure. This means that the <code>stop</code> callback MAY be called with an error argument, indicating a reason of failure! </td> <td class="code"> <div class="highlight"><pre></pre></div> </td> </tr> <tr id="section-20"> <td class="docs"> <div class="pilwrap"> <a class="pilcrow" href="#section-20">¶</a> </div> Let's adjust our print and streams to do that: </td> <td class="code"> <div class="highlight"><pre>function print(stream) { console.log(">>>") // Opening stream for reading stream(function onElement(element) { console.log(element) // Print each element of stream. }, function onStop(error) { if (!error) return console.log('<<<') // If no error is passed, stream ended</pre></div> </td> </tr> <tr id="section-21"> <td class="docs"> <div class="pilwrap"> <a class="pilcrow" href="#section-21">¶</a> </div> If there is an error print it out as well. </td> <td class="code"> <div class="highlight"><pre> console.log('!!!') console.error(error) }) }</pre></div> </td> </tr> <tr id="section-22"> <td class="docs"> <div class="pilwrap"> <a class="pilcrow" href="#section-22">¶</a> </div> Let's make another version of a function that returns a stream of given elements, in this case though we will use arguments instead of requiring an array argument. </td> <td class="code"> <div class="highlight"><pre>function list() {</pre></div> </td> </tr> <tr id="section-23"> <td class="docs"> <div class="pilwrap"> <a class="pilcrow" href="#section-23">¶</a> </div> Capture arguments as an array. </td> <td class="code"> <div class="highlight"><pre> var elements = Array.prototype.slice.call(arguments, 0)</pre></div> </td> </tr> <tr id="section-24"> <td class="docs"> <div class="pilwrap"> <a class="pilcrow" href="#section-24">¶</a> </div> Stream takes two callback arguments, the first is called with each element when it becomes available, and the second after calling first with all the elements of the stream. </td> <td class="code"> <div class="highlight"><pre> return function stream(next, stop) {</pre></div> </td> </tr> <tr id="section-25"> <td class="docs"> <div class="pilwrap"> <a class="pilcrow" href="#section-25">¶</a> </div> Yield each element of the stream by calling the <code>next</code>. callback. </td> <td class="code"> <div class="highlight"><pre> elements.forEach(function(element) { next(element) })</pre></div> </td> </tr> <tr id="section-26"> <td class="docs"> <div class="pilwrap"> <a class="pilcrow" href="#section-26">¶</a> </div> When we reach the end we stop a stream by calling the <code>stop</code> callback if it's passed. </td> <td class="code"> <div class="highlight"><pre> if (stop) stop() } }</pre></div> </td> </tr> <tr id="section-27"> <td class="docs"> <div class="pilwrap"> <a class="pilcrow" href="#section-27">¶</a> </div> Another attempt to print: </td> <td class="code"> <div class="highlight"><pre>print(list(1, 2, 3))</pre></div> </td> </tr> <tr id="section-28"> <td class="docs"> <div class="pilwrap"> <a class="pilcrow" href="#section-28">¶</a> </div> <pre><code> >>> 1 2 3 <<< </code></pre> </td> <td class="code"> <div class="highlight"><pre></pre></div> </td> </tr> <tr id="section-29"> <td class="docs"> <div class="pilwrap"> <a class="pilcrow" href="#section-29">¶</a> </div> Let's refine our stream definition again: </td> <td class="code"> <div class="highlight"><pre></pre></div> </td> </tr> <tr id="section-30"> <td class="docs"> <div class="pilwrap"> <a class="pilcrow" href="#section-30">¶</a> </div> Stream is a function representing a sequence of elements. It MAY be read by calling it with one function argument, that will be called every time an element becomes available. Stream takes a second optional function argument which is called once the stream is stopped, either without arguments when stream runs out of elements or with an error indicating the failure reason indicating why stream was stopped. </td> <td class="code"> <div class="highlight"><pre></pre></div> </td> </tr> <tr id="section-31"> <td class="docs"> <div class="pilwrap"> <a class="pilcrow" href="#section-31">¶</a> </div> Let's do something interesting from real life, like create a stream of all directory entries including entries from all nested directories (lstree). First we will have to create a few stream based wrappers around node's fs functions. We will start with a function that takes the path for a directory and returns a lazy stream of its entries. If reading a directory fails we will stop the stream with an error: </td> <td class="code"> <div class="highlight"><pre>var fs = require("fs") function ls(path) { return function stream(next, stop) {</pre></div> </td> </tr> <tr id="section-32"> <td class="docs"> <div class="pilwrap"> <a class="pilcrow" href="#section-32">¶</a> </div> see: <a href="http://nodejs.org/docs/v0.4.8/api/fs.html#fs.readdir">http://nodejs.org/docs/v0.4.8/api/fs.html#fs.readdir</a> </td> <td class="code"> <div class="highlight"><pre> fs.readdir(path, function onEntries(error, entries) { var entry</pre></div> </td> </tr> <tr id="section-33"> <td class="docs"> <div class="pilwrap"> <a class="pilcrow" href="#section-33">¶</a> </div> On error we stop a stream with that error. </td> <td class="code"> <div class="highlight"><pre> if (error) return stop(error)</pre></div> </td> </tr> <tr id="section-34"> <td class="docs"> <div class="pilwrap"> <a class="pilcrow" href="#section-34">¶</a> </div> Otherwise we yield each entry. </td> <td class="code"> <div class="highlight"><pre> while ((entry = entries.shift())) next(entry)</pre></div> </td> </tr> <tr id="section-35"> <td class="docs"> <div class="pilwrap"> <a class="pilcrow" href="#section-35">¶</a> </div> Once we yielded all entries we stop a stream. </td> <td class="code"> <div class="highlight"><pre> stop() }) } }</pre></div> </td> </tr> <tr id="section-36"> <td class="docs"> <div class="pilwrap"> <a class="pilcrow" href="#section-36">¶</a> </div> Try it out for the current working directory: </td> <td class="code"> <div class="highlight"><pre>print(ls('./'))</pre></div> </td> </tr> <tr id="section-37"> <td class="docs"> <div class="pilwrap"> <a class="pilcrow" href="#section-37">¶</a> </div> <pre><code> >>> .gitignore History.md package.json readme.js Readme.md streamer.js tests <<< </code></pre> </td> <td class="code"> <div class="highlight"><pre></pre></div> </td> </tr> <tr id="section-38"> <td class="docs"> <div class="pilwrap"> <a class="pilcrow" href="#section-38">¶</a> </div> The next wrapper we will need is <code>fs.stat</code>. We define a function that <code>takes</code> a path and returns a lazy stream with only an element representing <code>stat</code> of the given <code>path</code>. A lazy steam with one element can been seen as a promise or deferred, but don't worry if you are not familiar with that pattern. </td> <td class="code"> <div class="highlight"><pre>function stat(path) { return function stream(next, stop) {</pre></div> </td> </tr> <tr id="section-39"> <td class="docs"> <div class="pilwrap"> <a class="pilcrow" href="#section-39">¶</a> </div> see: <a href="http://nodejs.org/docs/v0.4.8/api/fs.html#fs.stat">http://nodejs.org/docs/v0.4.8/api/fs.html#fs.stat</a> </td> <td class="code"> <div class="highlight"><pre> fs.stat(path, function onStat(error, stats) {</pre></div> </td> </tr> <tr id="section-40"> <td class="docs"> <div class="pilwrap"> <a class="pilcrow" href="#section-40">¶</a> </div> On error we stop the stream with that error. </td> <td class="code"> <div class="highlight"><pre> if (error) return stop(error)</pre></div> </td> </tr> <tr id="section-41"> <td class="docs"> <div class="pilwrap"> <a class="pilcrow" href="#section-41">¶</a> </div> We add the path to the stat itself as it will be very convenient. </td> <td class="code"> <div class="highlight"><pre> stats.path = path</pre></div> </td> </tr> <tr id="section-42"> <td class="docs"> <div class="pilwrap"> <a class="pilcrow" href="#section-42">¶</a> </div> We yield <code>stats</code> and stop the stream. </td> <td class="code"> <div class="highlight"><pre> next(stats) stop() }) } }</pre></div> </td> </tr> <tr id="section-43"> <td class="docs"> <div class="pilwrap"> <a class="pilcrow" href="#section-43">¶</a> </div> Try it out for the current working directory: </td> <td class="code"> <div class="highlight"><pre>print(stat('./'))</pre></div> </td> </tr> <tr id="section-44"> <td class="docs"> <div class="pilwrap"> <a class="pilcrow" href="#section-44">¶</a> </div> <pre><code> >>> { dev: 234881026, ino: 19933437, mode: 16877, nlink: 17, uid: 502, gid: 20, rdev: 0, size: 578, blksize: 4096, blocks: 0, atime: Thu, 09 Jun 2011 10:51:25 GMT, mtime: Thu, 09 Jun 2011 12:48:32 GMT, ctime: Thu, 09 Jun 2011 12:48:32 GMT, path: './' } <<< </code></pre> </td> <td class="code"> <div class="highlight"><pre></pre></div> </td> </tr> <tr id="section-45"> <td class="docs"> <div class="pilwrap"> <a class="pilcrow" href="#section-45">¶</a> </div> Great we are done with the wrappers. Now we can list entries of the directory, but in order to list nested entries we need to distinguish directories from files. To do that we will create a function that takes a directory entries stream and returns a filtered stream containing only entries that are directories. We already can get stats from paths, so we just need to map entry paths to stats. Let's make a generic map function that takes a stream and a mapper function and returns a stream of mapped elements. </td> <td class="code"> <div class="highlight"><pre>function map(lambda, source) { return function stream(next, stop) { source(function onElement(element) { next(lambda(element)) }, stop) } }</pre></div> </td> </tr> <tr id="section-46"> <td class="docs"> <div class="pilwrap"> <a class="pilcrow" href="#section-46">¶</a> </div> Let's try to map numbers into doubled values: </td> <td class="code"> <div class="highlight"><pre>print(map(function(x) { return x * 2 }, list(1, 2, 3)))</pre></div> </td> </tr> <tr id="section-47"> <td class="docs"> <div class="pilwrap"> <a class="pilcrow" href="#section-47">¶</a> </div> <pre><code> >>> 2 4 6 <<< </code></pre> </td> <td class="code"> <div class="highlight"><pre></pre></div> </td> </tr> <tr id="section-48"> <td class="docs"> <div class="pilwrap"> <a class="pilcrow" href="#section-48">¶</a> </div> Now we can implement a function that is the equivalent of <code>ls</code> with the difference that it returns a stream of paths instead of entry filenames. </td> <td class="code"> <div class="highlight"><pre>var join = require("path").join function paths(path) { return map(join.bind(null, path), ls(path)) }</pre></div> </td> </tr> <tr id="section-49"> <td class="docs"> <div class="pilwrap"> <a class="pilcrow" href="#section-49">¶</a> </div> Test drive: </td> <td class="code"> <div class="highlight"><pre>print(paths(process.cwd()))</pre></div> </td> </tr> <tr id="section-50"> <td class="docs"> <div class="pilwrap"> <a class="pilcrow" href="#section-50">¶</a> </div> <pre><code> >>> /Users/gozala/Projects/streamer/History.md /Users/gozala/Projects/streamer/package.json ... <<< </code></pre> </td> <td class="code"> <div class="highlight"><pre></pre></div> </td> </tr> <tr id="section-51"> <td class="docs"> <div class="pilwrap"> <a class="pilcrow" href="#section-51">¶</a> </div> Now we need another equivalent of <code>paths</code> that returns a stream of directory paths only. To do that we need to filter out directories. So let's implement a generic filter function that takes a stream of elements and a filter function and returns the steam of elements for which the filterer returned true. </td> <td class="code"> <div class="highlight"><pre>function filter(lambda, source) { return function stream(next, stop) { source(function onElement(element) { if (lambda(element)) next(element) }, stop) } }</pre></div> </td> </tr> <tr id="section-52"> <td class="docs"> <div class="pilwrap"> <a class="pilcrow" href="#section-52">¶</a> </div> Simple example for filtering out odd numbers from a number stream. </td> <td class="code"> <div class="highlight"><pre>print(filter(function(x) { return x % 2 }, list(1, 2, 3, 4)))</pre></div> </td> </tr> <tr id="section-53"> <td class="docs"> <div class="pilwrap"> <a class="pilcrow" href="#section-53">¶</a> </div> <pre><code> >>> 1 3 <<< </code></pre> </td> <td class="code"> <div class="highlight"><pre></pre></div> </td> </tr> <tr id="section-54"> <td class="docs"> <div class="pilwrap"> <a class="pilcrow" href="#section-54">¶</a> </div> Awesome, going back to our problem, to figure out weather we have a file path or directory path we need to map paths to stats and then filter out only ones from there that are directories: </td> <td class="code"> <div class="highlight"><pre>function dirs(paths) { var stats = map(stat, paths) var dirStats = filter(function(stat) { return stat.isDirectory() }, stats) return map(function(stat) { return stat.path }, dirStats) }</pre></div> </td> </tr> <tr id="section-55"> <td class="docs"> <div class="pilwrap"> <a class="pilcrow" href="#section-55">¶</a> </div> Unfortunately dirs not going to work, because the <code>stats</code> stream is not a stream of <code>stat</code> elements, it is a stream of streams that are streams of <code>stat</code> elements. So what we need is sort of a flattened version of that stream. This is easy to do with another core <code>merge</code> function: </td> <td class="code"> <div class="highlight"><pre>function merge(source) { return function stream(next, stop) { var open = 1 function onStop(error) { if (!open) return false if (error) open = 0 else open -- if (!open) stop(error) } source(function onStream(stream) { open ++ stream(function onNext(value) { if (open) next(value) }, onStop) }, onStop) } }</pre></div> </td> </tr> <tr id="section-56"> <td class="docs"> <div class="pilwrap"> <a class="pilcrow" href="#section-56">¶</a> </div> Let's try a simple example: </td> <td class="code"> <div class="highlight"><pre>print(merge(list(