expresser
Version:
A ready-to-use platform for Node.js web apps, built on top of Express.
582 lines (295 loc) • 24.7 kB
HTML
<html>
<head>
<title>downloader.coffee</title>
<meta http-equiv="content-type" content="text/html; charset=UTF-8">
<link rel="stylesheet" media="all" href="public/stylesheets/normalize.css" />
<link rel="stylesheet" media="all" href="docco.css" />
</head>
<body>
<div class="container">
<div class="page">
<div class="header">
<h1>downloader.coffee</h1>
<div class="toc">
<h3>Table of Contents</h3>
<ol>
<li>
<a class="source" href="index.html">
index.coffee
</a>
</li>
<li>
<a class="source" href="app.html">
app.coffee
</a>
</li>
<li>
<a class="source" href="cron.html">
cron.coffee
</a>
</li>
<li>
<a class="source" href="database.html">
database.coffee
</a>
</li>
<li>
<a class="source" href="downloader.html">
downloader.coffee
</a>
</li>
<li>
<a class="source" href="events.html">
events.coffee
</a>
</li>
<li>
<a class="source" href="firewall.html">
firewall.coffee
</a>
</li>
<li>
<a class="source" href="imaging.html">
imaging.coffee
</a>
</li>
<li>
<a class="source" href="logger.html">
logger.coffee
</a>
</li>
<li>
<a class="source" href="mailer.html">
mailer.coffee
</a>
</li>
<li>
<a class="source" href="settings.html">
settings.coffee
</a>
</li>
<li>
<a class="source" href="sockets.html">
sockets.coffee
</a>
</li>
<li>
<a class="source" href="utils.html">
utils.coffee
</a>
</li>
</ol>
</div>
</div>
<h2 id="expresser-downloader">EXPRESSER DOWNLOADER</h2>
<p>Handles external downloads.
<!--
@see Settings.downloader
--></p>
<div class='highlight'><pre><span class="hljs-class"><span class="hljs-keyword">class</span> <span class="hljs-title">Downloader</span></span>
events = <span class="hljs-built_in">require</span> <span class="hljs-string">"./events.coffee"</span>
fs = <span class="hljs-built_in">require</span> <span class="hljs-string">"fs"</span>
http = <span class="hljs-built_in">require</span> <span class="hljs-string">"http"</span>
https = <span class="hljs-built_in">require</span> <span class="hljs-string">"https"</span>
lodash = <span class="hljs-built_in">require</span> <span class="hljs-string">"lodash"</span>
logger = <span class="hljs-built_in">require</span> <span class="hljs-string">"./logger.coffee"</span>
moment = <span class="hljs-built_in">require</span> <span class="hljs-string">"moment"</span>
path = <span class="hljs-built_in">require</span> <span class="hljs-string">"path"</span>
settings = <span class="hljs-built_in">require</span> <span class="hljs-string">"./settings.coffee"</span>
url = <span class="hljs-built_in">require</span> <span class="hljs-string">"url"</span></pre></div>
<p>The download queue and simultaneous count.</p>
<div class='highlight'><pre> queue = []
downloading = []</pre></div>
<h2 id="constructor-and-init">CONSTRUCTOR AND INIT</h2>
<p>Downloader constructor.</p>
<div class='highlight'><pre> <span class="hljs-attribute">constructor</span>:<span class="hljs-function"> -></span>
<span class="hljs-property">@setEvents</span>() <span class="hljs-keyword">if</span> settings.events.enabled</pre></div>
<p>Bind event listeners.</p>
<div class='highlight'><pre> <span class="hljs-attribute">setEvents</span>:<span class="hljs-function"> =></span>
events.<span class="hljs-literal">on</span> <span class="hljs-string">"downloader.download"</span>, <span class="hljs-property">@download</span></pre></div>
<h2 id="methods">METHODS</h2>
<p>Download an external file and save it to the specified location. The <code>callback</code>
has the signature (error, data). Returns the downloader object which is added
to the <code>queue</code>, which has the download properties and a <code>stop</code> helper to force
stopping it. Returns false on error or duplicate.
Tip: if you want to get the downloaded data without having to read the target file
you can get the downloaded contents via the <code>options.downloadedData</code>.
@param [String] remoteUrl The URL of the remote file to be downloaded.
@param [String] saveTo The full local path and destination filename.
@param [Object] options Optional, object with request options, for example auth.
@param [Method] callback Optional, a function (err, result) to be called when download has finished.
@return [Object] Returns the download job having timestamp, remoteUrl, saveTo, options, callback and stop helper.</p>
<div class='highlight'><pre> <span class="hljs-attribute">download</span>: <span class="hljs-function"><span class="hljs-params">(remoteUrl, saveTo, options, callback)</span> =></span>
<span class="hljs-keyword">if</span> <span class="hljs-keyword">not</span> remoteUrl?
logger.warn <span class="hljs-string">"Downloader.download"</span>, <span class="hljs-string">"Aborted, remoteUrl is not defined."</span>
<span class="hljs-keyword">return</span></pre></div>
<p>Check options and callback.</p>
<div class='highlight'><pre> <span class="hljs-keyword">if</span> <span class="hljs-keyword">not</span> callback? <span class="hljs-keyword">and</span> lodash.isFunction options
callback = options
options = <span class="hljs-literal">null</span>
now = <span class="hljs-keyword">new</span> Date().getTime()</pre></div>
<p>Create the download object.</p>
<div class='highlight'><pre> downloadObj = {<span class="hljs-attribute">timestamp</span>: now, <span class="hljs-attribute">remoteUrl</span>: remoteUrl, <span class="hljs-attribute">saveTo</span>: saveTo, <span class="hljs-attribute">options</span>: options, <span class="hljs-attribute">callback</span>: callback}</pre></div>
<p>Prevent duplicates?</p>
<div class='highlight'><pre> <span class="hljs-keyword">if</span> settings.downloader.preventDuplicates
existing = lodash.filter downloading, {<span class="hljs-attribute">remoteUrl</span>: remoteUrl, <span class="hljs-attribute">saveTo</span>: saveTo}</pre></div>
<p>If downloading the same file and to the same location, abort download.</p>
<div class='highlight'><pre> <span class="hljs-keyword">if</span> existing.length > <span class="hljs-number">0</span>
existing = existing[<span class="hljs-number">0</span>]
<span class="hljs-keyword">if</span> existing.saveTo <span class="hljs-keyword">is</span> saveTo
logger.warn <span class="hljs-string">"Downloader.download"</span>, <span class="hljs-string">"Aborted, already downloading."</span>, remoteUrl, saveTo
err = {<span class="hljs-attribute">message</span>: <span class="hljs-string">"Download aborted: same file is already downloading."</span>, <span class="hljs-attribute">duplicate</span>: <span class="hljs-literal">true</span>}
callback(err, downloadObj) <span class="hljs-keyword">if</span> callback?
<span class="hljs-keyword">return</span> <span class="hljs-literal">false</span></pre></div>
<p>Create a <code>stop</code> method to force stop the download by setting the <code>stopFlag</code>.
Accepts a <code>keep</code> boolean, if true the already downloaded data will be kept on forced stop.</p>
<div class='highlight'><pre> <span class="hljs-function"><span class="hljs-title">stopHelper</span> = <span class="hljs-params">(keep)</span> -></span> <span class="hljs-property">@stopFlag</span> = (<span class="hljs-keyword">if</span> keep <span class="hljs-keyword">then</span> <span class="hljs-number">1</span> <span class="hljs-keyword">else</span> <span class="hljs-number">2</span>)</pre></div>
<p>Update download object with stop helper and add to queue.</p>
<div class='highlight'><pre> downloadObj.stop = stopHelper
queue.push downloadObj</pre></div>
<p>Start download immediatelly if not exceeding the <code>maxSimultaneous</code> setting.</p>
<div class='highlight'><pre> next() <span class="hljs-keyword">if</span> downloading.length < settings.downloader.maxSimultaneous
<span class="hljs-keyword">return</span> downloadObj</pre></div>
<h2 id="internal-implementation">INTERNAL IMPLEMENTATION</h2>
<p>Helper to remove a download from the <code>downloading</code> list.</p>
<div class='highlight'><pre> <span class="hljs-function"><span class="hljs-title">removeDownloading</span> = <span class="hljs-params">(obj)</span> -></span>
filter = {<span class="hljs-attribute">timestamp</span>: obj.timestamp, <span class="hljs-attribute">remoteUrl</span>: obj.remoteUrl, <span class="hljs-attribute">saveTo</span>: obj.saveTo}
downloading = lodash.reject downloading, filter</pre></div>
<p>Helper function to proccess download errors.</p>
<div class='highlight'><pre> <span class="hljs-function"><span class="hljs-title">downloadError</span> = <span class="hljs-params">(err, obj)</span> -></span>
logger.debug <span class="hljs-string">"Downloader.downloadError"</span>, err, obj
removeDownloading obj
next()
obj.callback(err, obj) <span class="hljs-keyword">if</span> obj.callback?</pre></div>
<p>Helper function to parse the URL and get its options.</p>
<div class='highlight'><pre> <span class="hljs-function"><span class="hljs-title">parseUrlOptions</span> = <span class="hljs-params">(obj, options)</span> -></span>
<span class="hljs-keyword">if</span> obj.redirectUrl? <span class="hljs-keyword">and</span> obj.redirectUrl <span class="hljs-keyword">isnt</span> <span class="hljs-string">""</span>
urlInfo = url.parse obj.redirectUrl
<span class="hljs-keyword">else</span>
urlInfo = url.parse obj.remoteUrl</pre></div>
<p>Set URL options.</p>
<div class='highlight'><pre> options =
<span class="hljs-attribute">host</span>: urlInfo.hostname
<span class="hljs-attribute">hostname</span>: urlInfo.hostname
<span class="hljs-attribute">port</span>: urlInfo.port
<span class="hljs-attribute">path</span>: urlInfo.path</pre></div>
<p>Check for credentials on the URL.</p>
<div class='highlight'><pre> <span class="hljs-keyword">if</span> urlInfo.auth? <span class="hljs-keyword">and</span> urlInfo.auth <span class="hljs-keyword">isnt</span> <span class="hljs-string">""</span>
options.auth = urlInfo.auth
<span class="hljs-keyword">return</span> options</pre></div>
<p>Helper function to start a download request.</p>
<div class='highlight'><pre> <span class="hljs-function"><span class="hljs-title">reqStart</span> = <span class="hljs-params">(obj, options)</span> -></span>
<span class="hljs-keyword">if</span> obj.remoteUrl.indexOf(<span class="hljs-string">"https"</span>) <span class="hljs-keyword">is</span> <span class="hljs-number">0</span>
options.port = <span class="hljs-number">443</span> <span class="hljs-keyword">if</span> <span class="hljs-keyword">not</span> options.port?
httpHandler = https
<span class="hljs-keyword">else</span>
httpHandler = http</pre></div>
<p>Start the request.</p>
<div class='highlight'><pre> req = httpHandler.get options, <span class="hljs-function"><span class="hljs-params">(response)</span> =></span></pre></div>
<p>Downloaded contents will be appended also to the <code>downloadedData</code>
property of the options object.</p>
<div class='highlight'><pre> obj.downloadedData = <span class="hljs-string">""</span></pre></div>
<p>Set the estination temp file.</p>
<div class='highlight'><pre> saveToTemp = obj.saveTo + settings.downloader.tempExtension</pre></div>
<p>If status is 301 or 302, redirect to the specified location and stop the current request.</p>
<div class='highlight'><pre> <span class="hljs-keyword">if</span> response.statusCode <span class="hljs-keyword">is</span> <span class="hljs-number">301</span> <span class="hljs-keyword">or</span> response.statusCode <span class="hljs-keyword">is</span> <span class="hljs-number">302</span>
obj.redirectUrl = response.headers.location
options = lodash.assign options, parseUrlOptions obj
req.end()
reqStart obj, options</pre></div>
<p>If status is not 200 or 304, it means something went wrong so do not proceed
with the download. Otherwise proceed and listen to the <code>data</code> and <code>end</code> events.</p>
<div class='highlight'><pre> <span class="hljs-keyword">else</span> <span class="hljs-keyword">if</span> response.statusCode <span class="hljs-keyword">isnt</span> <span class="hljs-number">200</span> <span class="hljs-keyword">and</span> response.statusCode <span class="hljs-keyword">isnt</span> <span class="hljs-number">304</span>
err = {<span class="hljs-attribute">code</span>: response.statusCode, <span class="hljs-attribute">message</span>: <span class="hljs-string">"Server returned an unexpected status code: <span class="hljs-subst">#{response.statusCode}</span>"</span>}
downloadError err, obj
<span class="hljs-keyword">else</span></pre></div>
<p>Create the file stream with a .download extension. This will be renamed after the
download has finished and the file is totally written.</p>
<div class='highlight'><pre> fileWriter = fs.createWriteStream saveToTemp, {<span class="hljs-string">"flags"</span>: <span class="hljs-string">"w+"</span>}</pre></div>
<p>Helper called response gets new data. The data will also be
appended to <code>options.data</code> property.</p>
<div class='highlight'><pre> <span class="hljs-function"><span class="hljs-title">onData</span> = <span class="hljs-params">(data)</span> -></span>
<span class="hljs-keyword">if</span> obj.stopFlag
req.end()
onEnd()
<span class="hljs-keyword">else</span>
fileWriter.write data
obj.downloadedData += data</pre></div>
<p>Helper called when response ends.</p>
<div class='highlight'><pre> <span class="hljs-function"><span class="hljs-title">onEnd</span> = -></span>
response.removeListener <span class="hljs-string">"data"</span>, onData
fileWriter.addListener <span class="hljs-string">"close"</span>,<span class="hljs-function"> -></span></pre></div>
<p>Check if temp file exists.</p>
<div class='highlight'><pre> <span class="hljs-keyword">if</span> fs.existsSync?
tempExists = fs.existsSync saveToTemp
<span class="hljs-keyword">else</span>
tempExists = path.existsSync saveToTemp</pre></div>
<p>If temp download file can’t be found, set error message.
If <code>stopFlag</code> is 2 means download was stopped and should not keep partial data.</p>
<div class='highlight'><pre> <span class="hljs-keyword">if</span> <span class="hljs-keyword">not</span> tempExists
err = {<span class="hljs-attribute">message</span>:<span class="hljs-string">"Can't find downloaded file: <span class="hljs-subst">#{saveToTemp}</span>"</span>}
<span class="hljs-keyword">else</span>
fs.unlinkSync saveToTemp <span class="hljs-keyword">if</span> obj.stopFlag <span class="hljs-keyword">is</span> <span class="hljs-number">2</span></pre></div>
<p>Check if destination file already exists.</p>
<div class='highlight'><pre> <span class="hljs-keyword">if</span> fs.existsSync?
fileExists = fs.existsSync obj.saveTo
<span class="hljs-keyword">else</span>
fileExists = path.existsSync obj.saveTo</pre></div>
<p>Only proceed with renaming if <code>stopFlag</code> wasn’t set and destionation is valid.</p>
<div class='highlight'><pre> <span class="hljs-keyword">if</span> <span class="hljs-keyword">not</span> obj.stopFlag? <span class="hljs-keyword">or</span> obj.stopFlag < <span class="hljs-number">1</span>
fs.unlinkSync obj.saveTo <span class="hljs-keyword">if</span> fileExists
fs.renameSync saveToTemp, obj.saveTo <span class="hljs-keyword">if</span> tempExists</pre></div>
<p>Remove from <code>downloading</code> list and proceed with the callback.</p>
<div class='highlight'><pre> removeDownloading obj
obj.callback(err, obj) <span class="hljs-keyword">if</span> obj.callback?
logger.debug <span class="hljs-string">"Downloader.next"</span>, <span class="hljs-string">"End"</span>, obj.remoteUrl, obj.saveTo
fileWriter.end()
fileWriter.destroySoon()
next()</pre></div>
<p>Attachd response listeners.</p>
<div class='highlight'><pre> response.addListener <span class="hljs-string">"data"</span>, onData
response.addListener <span class="hljs-string">"end"</span>, onEnd</pre></div>
<p>Unhandled error, call the downloadError helper.</p>
<div class='highlight'><pre> req.<span class="hljs-literal">on</span> <span class="hljs-string">"error"</span>, <span class="hljs-function"><span class="hljs-params">(err)</span> =></span>
downloadError err, obj</pre></div>
<p>Process next download.</p>
<div class='highlight'><pre> <span class="hljs-function"><span class="hljs-title">next</span> = -></span>
<span class="hljs-keyword">return</span> <span class="hljs-keyword">if</span> queue.length < <span class="hljs-number">0</span></pre></div>
<p>Get first download from queue.</p>
<div class='highlight'><pre> obj = queue.shift()</pre></div>
<p>Check if download is valid.</p>
<div class='highlight'><pre> <span class="hljs-keyword">if</span> <span class="hljs-keyword">not</span> obj?
logger.debug <span class="hljs-string">"Downloader.next"</span>, <span class="hljs-string">"Skip"</span>, <span class="hljs-string">"Downloader object is invalid."</span>
<span class="hljs-keyword">return</span>
<span class="hljs-keyword">else</span>
logger.debug <span class="hljs-string">"Downloader.next"</span>, obj</pre></div>
<p>Add to downloading array.</p>
<div class='highlight'><pre> downloading.push obj
<span class="hljs-keyword">if</span> settings.downloader.headers? <span class="hljs-keyword">and</span> settings.downloader.headers <span class="hljs-keyword">isnt</span> <span class="hljs-string">""</span>
headers = settings.web.downloaderHeaders
<span class="hljs-keyword">else</span>
headers = <span class="hljs-literal">null</span></pre></div>
<p>Set default options.</p>
<div class='highlight'><pre> options =
<span class="hljs-attribute">headers</span>: headers
<span class="hljs-attribute">rejectUnauthorized</span>: settings.downloader.rejectUnauthorized</pre></div>
<p>Extend options.</p>
<div class='highlight'><pre> options = lodash.assign options, obj.options, parseUrlOptions(obj)</pre></div>
<p>Start download</p>
<div class='highlight'><pre> <span class="hljs-keyword">if</span> obj.stopFlag? <span class="hljs-keyword">and</span> obj.stopFlag > <span class="hljs-number">0</span>
logger.debug <span class="hljs-string">"Downloader.next"</span>, <span class="hljs-string">"Skip, 'stopFlag' is <span class="hljs-subst">#{obj.stopFlag}</span>."</span>, obj
removeDownloading obj
next()
<span class="hljs-keyword">else</span>
reqStart obj, options</pre></div>
<h2 id="singleton-implementation">Singleton implementation</h2>
<div class='highlight'><pre>Downloader.<span class="hljs-function"><span class="hljs-title">getInstance</span> = -></span>
<span class="hljs-property">@instance</span> = <span class="hljs-keyword">new</span> Downloader() <span class="hljs-keyword">if</span> <span class="hljs-keyword">not</span> <span class="hljs-property">@instance</span>?
<span class="hljs-keyword">return</span> <span class="hljs-property">@instance</span>
<span class="hljs-built_in">module</span>.<span class="hljs-built_in">exports</span> = <span class="hljs-built_in">exports</span> = Downloader.getInstance()</pre></div>
<div class="fleur">h</div>
</div>
</div>
</body>
</html>