node-redis-queue

<h1> <a id="user-content-node-redis-queue" class="anchor" href="#node-redis-queue" aria-hidden="true"></a>node-redis-queue</h1> <h2> <a id="user-content-overview" class="anchor" href="#overview" aria-hidden="true"></a>Overview</h2> This is a very simple queing wrapper for Redis that is intended for communication between separate processes. It comes with two APIs: <ol> <li> Channel -- the push/pop interface The process creates an instance of Channel. The sending process uses the Channel instance to push data onto the queue via the <code>push</code> function. The receiving process uses the Channel instance to remove data from the same queue via the <code>pop</code> function, which delivers the data to a callback function which accepts a single data parameter. Some variants of the <code>pop</code> function may have a timeout parameter so they don't block indefinitely. </li> <li> WorkQueueMgr -- the send/consume interface The process creates an instance of WorkQueueMgr. Then, it uses that instance to create one or more instances of WorkQueue, each representing a different queue having a unique name. The sending process uses a WorkQueue instance to send data to the corresponding queue via the <code>send</code> function. The receiving process uses a WorkQueue instance to remove data from the corresponding queue via the <code>consume</code> function, which delivers the data to a callback function which accepts a data parameter and an ack parameter. The latter is a function that is called to indicate that the callback function is complete and is ready to accept some additional data, if any, in the queue. See the usage examples below and also the worker03 and worker04 files in the demo src or lib directories for examples of how to do this. To achieve greater throughput with <code>consume</code>, one may specify the number of async callback functions to operate in parallel. The later feature is referred to as 'arity'. One also may specify a timeout parameter so that <code>consume</code> doesn't block indefinitely. <code>consume</code> is different from <code>pop</code> in that a single call to <code>consume</code> can fetch multiple data items from the given queue, while <code>pop</code> must be called repeatedly to fetch items from the queue. </li> </ol> <h2> <a id="user-content-details-of-the-channel-interface" class="anchor" href="#details-of-the-channel-interface" aria-hidden="true"></a>Details of the Channel Interface</h2> <h3> <a id="user-content-the-public-methods-of-the-channel-class-are" class="anchor" href="#the-public-methods-of-the-channel-class-are" aria-hidden="true"></a>The public methods of the <code>Channel</code> class are:</h3> <code>constructor configFilePath</code> -- Takes an optional config file path, which may be overridden by the QUEUE_CONFIG_PATH environment variable. <code>connect onReadyCB</code> -- Obtains a Redis client connection using the config file information. Calls the given callback when the connection is ready. <code>attach client</code> -- This is an alternative to calling <code>connect</code>. It attaches to a given Redis client connection. <code>push queueName, data</code> -- Pushes the given data into the given queue. <code>pop key, onDataCB</code> -- Accepts a single key (or queue name), blocking indefinitely until data becomes available. Calls the given callback when the data is available. The callback is called with a single data parameter. <code>popTimeout key, timeout, onDataCB</code> -- Accepts a single key (or queue name) and waits for input on that key. It blocks but times out and emits a <code>timeout</code> event when the specified timeout interval is exceeded without any data becoming available. Calls the given callback when data is available. The callback is called with a single data parameter. <code>popAny keys..., onDataCB</code> -- Accepts one or more keys (or queue names) and waits for input on any of those, blocking indefinitely until data becomes available. Calls the given callback when data is available. The callback is called with a keyName parameter and a data parameter. <code>popAnyTimeout keys..., timeout, onDataCB</code> -- Accepts one or more keys (or queue names) and waits for input on any of those. It blocks but times out and emits a <code>timeout</code> event when the specified timeout interval is exceeded without any data becoming available. Calls the given callback when data is available. The callback receives a keyName parameter and a dat a parameter. The callback is called with a keyName parameter and a data parameter. <code>clear keys..., onClearCB</code> -- Removes the data from the queues specified by the given keys (or queue names). Calls the given callback when the operation is complete. <code>disconnect</code> -- Quits accepting data and closes the connection. <code>end</code> -- Closes the connection <code>commandQueueLength</code> -- The size of the Redis client's command queue (i.e., commands queued to be sent to Redis). <h2> <a id="user-content-details-of-the-workqueuemgr-interface" class="anchor" href="#details-of-the-workqueuemgr-interface" aria-hidden="true"></a>Details of the WorkQueueMgr Interface</h2> <h3> <a id="user-content-the-public-methods-of-the-workqueue-class-are" class="anchor" href="#the-public-methods-of-the-workqueue-class-are" aria-hidden="true"></a>The public methods of the <code>WorkQueue</code> class are:</h3> <code>send data</code> -- Pushes data into the associated work queue. <code>consume onDataCB, arity, timeout</code> -- Consumes data that becomes available in the associated work queue. The arity and timeout parameters are optional and default to 1 and 0 respectively. <code>clear onClearCB</code> -- Removes the data from the associated queue. Calls the given callback when the operation is complete. <code>destroy</code> -- Destroys the meta-data about this WorkQueue instance in the associated WorkQueueMgr class. <h3> <a id="user-content-the-public-methods-of-the-workqueuemgr-class-are" class="anchor" href="#the-public-methods-of-the-workqueuemgr-class-are" aria-hidden="true"></a>The public methods of the <code>WorkQueueMgr</code> class are:</h3> <code>constructor configFilePath</code> -- Takes an optional config file path, which may be overridden by the QUEUE_CONFIG_PATH environment variable. Creates an internal Channel instance. <code>connect onReadyCB</code> -- Obtains a Redis client connection using the config file information. Calls the given callback when the connection is ready. <code>attach client</code> -- This is an alternative to calling <code>connect</code>. It attaches to a given Redis client connection. <code>createQueue queueName</code> -- Creates a WorkQueue instance for the given queue name. <code>send queueName, data</code> -- Pushes the given data into the queue corresponding to the given queue name. This is an alternative to WorkQueue.send and requires no WorkQueue instance. <code>consume queueName, onDataCB, arity, timeout</code> -- Consumes data that becomes available in the given queue, calling the onDataCB function each time data becomes available. Arity is the number of times the given async callback function is to be executed in parallel. Defaults to 1. Timeout is the number of seconds to wait for data to become available, after which a <code>'timeout'</code> event is emitted. Defaults to zero, in which case it blocks indefinitely. <code>clear keys..., onClearCB</code> -- Removes the data from the queues specified by the given keys (or queue names). Calls the given callback when the operation is complete. <code>disconnect</code> -- Quits accepting data and closes the connection. <code>end</code> -- Closes the connection <code>commandQueueLength</code> -- Returns the size of the Redis client's command queue (i.e., commands queued to be sent to Redis). <h2> <a id="user-content-events-emitted-by-both-interfaces" class="anchor" href="#events-emitted-by-both-interfaces" aria-hidden="true"></a>Events Emitted by Both Interfaces</h2> <code>'error'</code> -- emitted when an error is reported by Redis <code>'end'</code> -- emitted when a connection to the Redis server has been lost <code>'timeout'</code> -- emitted when a timeout occurs on a popTimeout operation, popAnyTimeout operation, or on a consume operation with a timeout specified. The provided callback provided to <code>on 'timeout'</code> receives two parameters: <ol> <li><code>keys</code> -- one or more key values or queue names on which the operation was waiting when the timeout occurred.</li> <li><code>cancel</code> -- a function that may be called to prevent another outstanding blocking operation from being performed. This is useful, for example, to get a clean exit from a test case.</li> </ol> <code>'drain'</code> -- emitted when the TCP connection to the Redis server has been buffering, but is now writable. This event can be used to stream commands in to Redis and adapt to backpressure: Call commandQueueLength to detect when the length is too much, then use the <code>'drain'</code> event to resume sending data to the queue or queues. <h2> <a id="user-content-installation" class="anchor" href="#installation" aria-hidden="true"></a>Installation</h2> <pre><code>npm install node-redis-queue --save </code></pre> <h2> <a id="user-content-configuration" class="anchor" href="#configuration" aria-hidden="true"></a>Configuration</h2> Sample configuration files may be found in the sample-configs directory. In each config file, the redis_provider type setting specifies the strategy to use. The verbose setting, if true, specifies to display the config file settings on startup. The environment variable QUEUE_CONFIG_FILE specifies which config file is to be used. If not set, it defaults to node-redis-queue/redis-queue-config.json, which specifies to use the local Redis server with no password. If you do nothing, that is what you get. Currently implemented strategies are: <ul> <li>connStrategyDefaultLocal -- local Redis server, no password</li> <li>connStrategyCustom -- configurable host, port, and password; defaults to local Redis server, no password</li> <li>connStrategyHerokuRedisCloud -- host, port, and password specified by REDISCLOUD_URL environment variable; if not set, then defaults to local Redis server, no password</li> <li>connStrategyBlueMixRedisCloud -- host, port, and password specified by VCAP_SERVICES environment variable; if not set, then defaults to local Redis server, no password</li> </ul> redisQueueConfig determines which strategy is used to configure the client. It is easy to add your own strategy. <h2> <a id="user-content-usage" class="anchor" href="#usage" aria-hidden="true"></a>Usage</h2> <h3> <a id="user-content-coffescript-usage-examples" class="anchor" href="#coffescript-usage-examples" aria-hidden="true"></a>Coffescript Usage Examples</h3> See the Coffeescript usage examples <a href="COFFEESCRIPT_USAGE_EXAMPLES.md">here</a>. <h3> <a id="user-content-javascript-usage-examples" class="anchor" href="#javascript-usage-examples" aria-hidden="true"></a>Javascript Usage Examples</h3> See the Javascript usage examples <a href="JAVASCRIPT_USAGE_EXAMPLES.md">here</a>. <h2> <a id="user-content-running-the-demos" class="anchor" href="#running-the-demos" aria-hidden="true"></a>Running the demos</h2> Instructions for running the demo code may be found <a href="demo/HOW_TO_RUN_DEMOS.md">here</a>. <h2> <a id="user-content-developer-info" class="anchor" href="#developer-info" aria-hidden="true"></a>Developer Info</h2> For developers who wish to make changes to the code, information on running the test suite and how to use <code>grunt</code> may be found <a href="DEVELOPER_INFO.md">here</a>; <h2> <a id="user-content-change-log" class="anchor" href="#change-log" aria-hidden="true"></a>Change Log</h2> View the change log <a href="CHANGE_LOG.md">here</a>. <h2> <a id="user-content-architecture-notes" class="anchor" href="#architecture-notes" aria-hidden="true"></a>Architecture Notes</h2> The Channel class is a very thin wrapper around existing redis module functions. It delegates all its operations to that module. The Channel class uses different strategies to connect to redis. A config file specifies which strategy to use and also supplies options to redis. The WorkQueueMgr class serves as a factory for WorkQueue class instances. The WorkQueueMgr class delegates queue send and consume to the Channel class. It maintains a hash of queues created (@queue), which defines which queues currently are valid. For each queue that is consuming data, it maintains an entry in a hash of callbacks (@consumingCB) and and an entry in an ordered list of names of queues currently being consumed (@consumingNames). The ordered list of names represents the priority for consumption of queue data and is used by the consume function as the list of keys to be monitored by qmgr.popAny. The WorkQueue class is a very simple envelope wrapping four applicative functions, effectively delegating all operations to WorkQueueMgr class. <h2> <a id="user-content-historical-note" class="anchor" href="#historical-note" aria-hidden="true"></a>Historical Note</h2> Part of this work is derived from node-simple-redis-queue v0.9.3 by James Smith and retains the same license. However, the current version bears almost no resemblance to James' project.