superchild

<!DOCTYPE html> <html> <head> <title>superchild.js</title> <meta http-equiv="content-type" content="text/html; charset=UTF-8"> <link rel="stylesheet" media="all" href="public/stylesheets/normalize.css" /> <link rel="stylesheet" media="all" href="docco.css" /> </head> <body> <div class="container"> <div class="page"> <div class="header"> <h1>superchild.js</h1> <div class="toc"> <h3>Table of Contents</h3> <ol> <li> <a class="source" href="json-sieve.html"> json-sieve.js </a> </li> <li> <a class="source" href="superchild.html"> superchild.js </a> </li> <li> <a class="source" href="unlogger.html"> unlogger.js </a> </li> </ol> </div> </div> <a href="https://circleci.com/gh/mayanklahiri/node-superchild"><img src="https://circleci.com/gh/mayanklahiri/node-superchild.svg?style=svg" alt="Circle CI"></a> <a href="https://github.com/mayanklahiri/node-superchild/issues"><img src="https://img.shields.io/github/issues/mayanklahiri/node-superchild.svg" alt="GitHub issues"></a> <a href="https://github.com/mayanklahiri/node-superchild/stargazers"><img src="https://img.shields.io/github/stars/mayanklahiri/node-superchild.svg" alt="GitHub stars"></a> <a href="https://raw.githubusercontent.com/mayanklahiri/node-superchild/master/LICENSE"><img src="https://img.shields.io/badge/license-MIT-blue.svg" alt="GitHub license"></a> <a href="https://badge.fury.io/js/superchild"><img src="https://badge.fury.io/js/superchild.svg" alt="npm version"></a> <a href="https://david-dm.org/mayanklahiri/node-superchild.svg"><img src="https://david-dm.org/mayanklahiri/node-superchild.svg" alt="dependencies"></a> <a href="https://www.bithound.io/github/mayanklahiri/node-superchild"><img src="https://www.bithound.io/github/mayanklahiri/node-superchild/badges/score.svg" alt="bitHound Overall Score"></a> <a href="https://www.bithound.io/github/mayanklahiri/node-superchild"><img src="https://www.bithound.io/github/mayanklahiri/node-superchild/badges/code.svg" alt="bitHound Code"></a> <a href="https://twitter.com/intent/tweet?text=Wow:&url=%5Bobject%20Object%5D"><img src="https://img.shields.io/twitter/url/https/github.com/mayanklahiri/node-superchild.svg?style=social" alt="Twitter"></a> Superchild is a POSIX-only (e.g., Linux, Mac OS X) wrapper around node.js’s built-in <code>child_process</code> module, which requires a lot of care and attention to use correctly. Links: <ul> <li><a href="http://mayanklahiri.github.io/node-superchild/superchild.html">Annotated source code</a></li> <li><a href="http://mayanklahiri.github.io/node-superchild/coverage/lib/index.html">Current test coverage</a></li> <li><a href="https://github.com/mayanklahiri/node-superchild">Github page</a></li> <li><a href="https://www.npmjs.com/package/superchild">NPM page</a></li> </ul> The purpose of Superchild is to allow large node.js programs to be split into independent processes (and sub-processes, resulting in process trees), while handling the tedious parts of process management and communication. Superchild aims to be compatible with any program that reads from and writes to their <code>stdin</code>, <code>stdout</code>, and <code>stderr</code> streams, regardless of what language the program is written in. This allows interesting hacks like using <code>ssh</code> to execute a module on a remote host, written in another language, over an encrypted link, while using the near-universal format of line-delimited JSON messages over <code>stdio</code>. Features that make Superchild different from node’s built-in <code>child_process</code> module include the following (many of these are currently possible only due to restricting focus to POSIX platforms, i.e., not Windows): <ol> <li>A single function to replace <code>fork()</code>, <code>exec()</code>, and <code>spawn()</code> from the built-in <code>child_process</code> module. </li> <li>Waits for <code>stdout</code> and <code>stderr</code> streams to end before emitting an <code>exit</code> event, unlike <code>child_process.ChildProcess</code>. </li> <li>Handles isolating child process and its children in a separate, detached process group that can be terminated as a subtree using the POSIX <code>kill</code> command. This means that calling <code>close()</code> on a Superchild instance will kill not just the child process, but all its child processes and so on (i.e., the entire process group lead by the child). Note that if any processes in the sub-tree detach themselves into a new process group, they will not be part of our child’s process group, and will not be killed. </li> <li>Handles graceful termination of child’s entire process group using <code>SIGTERM</code> -> <code>SIGKILL</code> signals with a configurable timeout. </li> <li>Handles unexpected termination of the current process by killing the child’s entire process group immediately with <code>SIGKILL</code>. </li> <li>Automatically serializes and deserializes line-delimited JSON values (LD-JSON) sent to and received from child, intermixed with <code>stdout</code>. <code>stderr</code> is passed through unbuffered. Effectively, this means that the child’s <code>stdout</code> stream is demultiplexed into the child streams <code>stdout_line</code> (parsed raw text lines), <code>json_object</code> (parsed JSON objects), and <code>json_array</code> (parsed JSON arrays). Regular processes have 3 I/O streams (stdin, stdout, stderr); Superchildren have 6 streams (stdin, stdout, stderr, stdout_line, json_object, json_array). </li> </ol> <h2 id="install">Install</h2> <pre><code> npm install superchild </code></pre><h2 id="usage">Usage</h2> <h5 id="run-a-shell-command">Run a shell command</h5> Get a directory listing line-by-line using <code>ls</code>. <pre><code> var superchild = require('superchild'); var child = superchild('ls -lh'); child.on('stdout_line', function(line) { console.log('[stdout]: ', line); }); </code></pre><h5 id="spawn-and-communicate-with-a-module">Spawn and communicate with a module</h5> Spawn a node.js module in another process and communicate with it. Note that the child uses the <code>superchild.unlogger</code> helper function to parse its standard input for LD-JSON arrays and objects. master.js <pre><code> var assert = require('assert'); var superchild = require('superchild'); var child = superchild('node echo.js'); child.send({ some: 'data', }); child.on('json_object', function(jsonObj) { assert.equal(jsonObj.some, 'data'); }); </code></pre>echo.js <pre><code> var unlogger = require('superchild').unlogger; unlogger().on('json_object', function(jsonObj) { // Echo JSON object from parent back to parent. console.log(JSON.stringify(jsonObj)); }); </code></pre><h2 id="events-emitted">Events emitted</h2> Superchild is an EventEmitter. The following events can be listened for using <code>child.on()</code> and <code>child.once()</code> functions. <table> <thead> <tr> <th>Event</th> <th>Arguments</th> <th>Description</th> </tr> </thead> <tbody> <tr> <td><code>exit</code></td> <td><code>code</code>, <code>signal</code></td> <td>Child process exited, identical to <code>child_process.exit</code></td> </tr> <tr> <td><code>stderr_data</code></td> <td><code>dataStr</code></td> <td>Received unbuffered data on child’s <code>stderr</code> stream.</td> </tr> <tr> <td><code>stdout_line</code></td> <td><code>lineStr</code></td> <td>Received a full line of text from the child process.</td> </tr> <tr> <td><code>json_object</code></td> <td><code>jsonObj</code></td> <td>Parsed a line-delimited JSON object from child’s <code>stdout</code> stream.</td> </tr> <tr> <td><code>json_array</code></td> <td><code>jsonArr</code></td> <td>Parsed a line-delimited JSON array from child’s <code>stdout</code> stream.</td> </tr> </tbody> </table> <h2 id="methods">Methods</h2> <table> <thead> <tr> <th>Method</th> <th>Description</th> </tr> </thead> <tbody> <tr> <td><code>send(jsonVal)</code></td> <td>Serialize and send a JSON-serializable object or array to the child as LD-JSON.</td> </tr> <tr> <td><code>close(cb)</code></td> <td>Gracefully terminate the child, invoking the callback when the child has died.</td> </tr> </tbody> </table> <h2 id="requirements">Requirements</h2> <ul> <li><code>node.js</code> version 0.11.13 or higher, due to the use of <code>spawnSync</code>.</li> <li>POSIX-compliant platform, such as Linux or Mac OS.</li> </ul> <h2 id="source-code">Source Code</h2> The full annotated source code of <code>superchild.js</code> follows, generated automatically by Docco. <div class='highlight'><pre>/** @license The MIT License, Copyright (c) 2016 Mayank Lahiri <mayank@iceroad.io> */ var events = require('events') , fmt = require('util').format , fs = require('fs') , spawn = require('./respawn').spawn , spawnSync = require('./respawn').spawnSync , JSONSieve = require('./json-sieve') ; var NODE_MAJOR = parseInt(process.version.replace(/\..+/, '').substr(1), 10); var LEGACY_NODE = NODE_MAJOR < 5; var CLEANUP_PROCESSES = {}; function superchild(commandLine, options) { if (!commandLine) throw new Error('Superchild: empty commandLine provided.'); options = (typeof options == 'object' ? options : {}); var opt = {}, setDefault = function(key, defaultVal) { opt[key] = (key in options) ? options[key] : defaultVal; };</pre></div> <h4 id="default-options">Default options</h4> <ul> <li><code>stringEncoding</code>: character encoding to use while communicating with the child. This is required to make node.js return strings instead of raw buffers. Defaults to utf-8.</li> </ul> <div class='highlight'><pre> setDefault('stringEncoding', 'utf-8');</pre></div> <ul> <li><code>killOnExit</code>: kill child process group when our own process exits, if the child is still running. Defaults to true.</li> </ul> <div class='highlight'><pre> setDefault('killOnExit', true);</pre></div> <ul> <li><code>cleanupTimeoutMs</code>: amount of walltime to wait for child process to exit after <code>close()</code> is called, before sending it SIGTERM. Defaults to 500ms.</li> </ul> <div class='highlight'><pre> setDefault('cleanupTimeoutMs', 500);</pre></div> <ul> <li><code>shell</code>: run the command line through a system shell, defaults to true.</li> </ul> <div class='highlight'><pre> setDefault('shell', true);</pre></div> <ul> <li><code>cwd</code>: working directory to execute child process in, defaults to process.cwd().</li> </ul> <div class='highlight'><pre> setDefault('cwd', process.cwd());</pre></div> <ul> <li><code>detached</code>: makes the process the leader of its own process group.</li> </ul> <div class='highlight'><pre> setDefault('detached', true);</pre></div> <ul> <li><code>env</code>: environment variables to pass to child.</li> </ul> <div class='highlight'><pre> setDefault('env', undefined);</pre></div> Construct an <code>EventEmitter</code> instance to return to the caller. This is a proxy emitter for the child process so that we can re-order events received from the child process if necessary. <div class='highlight'><pre> var emitter = new events.EventEmitter();</pre></div> <h4 id="spawning-the-child-process">Spawning the child process</h4> Spawn a child process with the <code>detached</code> option makes it the leader of its own process group. This allows us to terminate all its descendant processes (if we need to) by running <code>kill -$PGID</code> (negation of the child’s process group id). <div class='highlight'><pre> var child; var hasExited = false, stdoutEnded = false, stderrEnded = false; try { var spawnOpt = { detached: opt.detached, shell: opt.shell, cwd: opt.cwd, stdio: 'pipe', env: opt.env, };</pre></div> Use <code>respawn.spawn()</code> to spawn the child process. <div class='highlight'><pre> child = spawn(commandLine, spawnOpt); child.stdout.setEncoding(opt.stringEncoding); child.stderr.setEncoding(opt.stringEncoding);</pre></div> Node 4 (LTS) and below: no setDefaultEncoding() function. <div class='highlight'><pre> if (child.stdin.setDefaultEncoding) { child.stdin.setDefaultEncoding(opt.stringEncoding); } if (LEGACY_NODE) {</pre></div> Node 4 (LTS) and below: <code>child.pid</code> is not immediately populated, and <code>child_process.spawnSync</code> does not support the <code>detached</code> option. So the best we can do if <code>child.pid</code> is undefined is some exponential backoff polling that is not guaranteed to succeed. <div class='highlight'><pre> var pollIntervalMs = 1; (function pollFn() { if (child.pid) {</pre></div> Successfully have child’s pid <div class='highlight'><pre> emitter.pid = child.pid; if (opt.killOnExit) { CLEANUP_PROCESSES[child.pid] = true; } } else { if (pollIntervalMs < 100) { setTimeout(pollFn, pollIntervalMs); pollIntervalMs *= 2; } } })(); } else {</pre></div> Node 5+ natively returns child.pid set. <div class='highlight'><pre> emitter.pid = child.pid; if (opt.killOnExit) { CLEANUP_PROCESSES[child.pid] = true; } } } catch(e) { throw new Error(fmt( 'superchild: cannot spawn() child process, reason=%s, stack=%s', e.message, e.stack)); }</pre></div> <h4 id="reading-from-the-child">Reading from the child</h4> Read from the child process by parsing out complete lines and line-delimited JSON objects and arrays from its <code>stdout</code> stream using <code>JSONSieve</code>. <div class='highlight'><pre> var sieve = new JSONSieve(); var sieveClosed = false; child.stdout.on('data', function(dataStr) { sieve.observe(dataStr); }); sieve.on('json_object', emitter.emit.bind(emitter, 'json_object')); sieve.on('json_array', emitter.emit.bind(emitter, 'json_array')); sieve.on('stdout_line', emitter.emit.bind(emitter, 'stdout_line')); child.stderr.on('data', emitter.emit.bind(emitter, 'stderr_data')); child.stdout.on('data', emitter.emit.bind(emitter, 'stdout'));</pre></div> <h4 id="writing-to-the-child-send-">Writing to the child: <code>send()</code></h4> <div class='highlight'><pre> emitter.send = function(jsonVal) { try { var serData = JSON.stringify(jsonVal) + '\n'; child.stdin.write(serData, opt.stringEncoding); return { bytesWritten: serData.length, prefix: serData.substr(0, 100), }; } catch(e) { throw new Error('Unable to write to process: ' + e.message); } };</pre></div> <h4 id="terminating-the-child-close-">Terminating the child: <code>close()</code></h4> Kill the child’s process group gracefully. This involves the following steps: <ol> <li>Send a <code>SIGTERM</code> signal to the child process group, allowing all processes in the group to die gracefully.</li> <li>Wait for <code>opt.cleanupTimeoutMs</code> milliseconds.</li> <li>If the child process has not exited, send a <code>SIGKILL</code> to the process group.</li> </ol> This method is a public method that is offered via the return value of the <code>superchild()</code> call. Note that we wait for the proxy emitter’s <code>exit</code> event before invoking the callback rather than the underlying node.js child process’s <code>exit</code> event. This is in order to wait for all output streams to flush before actually emitting the <code>exit</code> event, which is not guaranteed with node’s underlying child_process implementation. <div class='highlight'><pre> emitter.close = function(cb) { cb = cb || function() {}; if (!child || !child.pid) throw new Error('superchild: child is not running.'); var termGroupCmdLine = fmt('/bin/kill -s SIGTERM -- -%d', child.pid); var killGroupCmdLine = fmt('/bin/kill -s SIGKILL -- -%d', child.pid); spawnSync(termGroupCmdLine, {shell: true, stdio: 'inherit'}); setTimeout(function() { if (!sieveClosed) { spawnSync(killGroupCmdLine, {shell: true, stdio: 'inherit'}); } }, opt.cleanupTimeoutMs); emitter.once('exit', cb); };</pre></div> <h4 id="cleaning-up-the-child-process">Cleaning up the child process</h4> If the <code>killOnExit</code> option is specified, then when our own current process exits, kill the child’s process group using the POSIX <code>kill</code> command with the child’s process group ID, which is the negation of its process id, since it is the leader of its own process group. If the child exits, then we need to wait for its streams to close as well, and then flush the JSON sieve before emitting the ‘exit’ event through the proxy emitter. <div class='highlight'><pre> var exitCode, exitSignal; var cleanupFn = function() { if (hasExited && stdoutEnded && stderrEnded && !sieveClosed) { sieve.close(function() { emitter.emit('exit', exitCode, exitSignal); emitter.removeAllListeners(); }); sieveClosed = true; } }; var onStdoutEnd = function() { stdoutEnded = true; cleanupFn(); }; var onStderrEnd = function() { stderrEnded = true; cleanupFn(); }; child.stdout.once('close', onStdoutEnd); child.stderr.once('close', onStderrEnd); child.once('exit', function(code, signal) { hasExited = true; exitCode = code; exitSignal = signal; delete CLEANUP_PROCESSES[child.pid]; cleanupFn(); });</pre></div> <h4 id="return-value">Return value</h4> The <code>superchild()</code> call returns a proxy <code>EventEmitter</code> with two additional methods that are used to control the child process: <code>close()</code> and <code>send()</code>. <div class='highlight'><pre> return emitter; } process.once('exit', function() { Object.keys(CLEANUP_PROCESSES).forEach(function(pid) { var killCmdLine = fmt('/bin/kill -s SIGTERM -- -%d', pid); spawnSync(killCmdLine, {stdio: 'inherit'}); }); }); module.exports = superchild;</pre></div> <h4 id="helper-utilities">Helper utilities</h4> <ul> <li><a href="http://mayanklahiri.github.io/node-superchild/json-sieve.html"><code>Class JSONSieve</code></a>, parses a readable stream into JSON objects and arrays, and stdout lines.</li> <li><a href="http://mayanklahiri.github.io/node-superchild/unlogger.html"><code>function unlogger()</code></a>, an example of establishing bi-directional communication between a parent and child that can be easily ported to many other languages.</li> </ul> <div class='highlight'><pre>superchild.JSONSieve = require('./json-sieve'); superchild.unlogger = require('./unlogger');</pre></div> <div class="fleur">h</div> </div> </div> </body> </html>