browserchannel

<!DOCTYPE html> <html> <head> <title>server.coffee</title> <meta http-equiv="content-type" content="text/html; charset=UTF-8"> <link rel="stylesheet" media="all" href="docco.css" /> </head> <body> <div id="container"> <div id="background"></div> <table cellpadding="0" cellspacing="0"> <thead> <tr> <th class="docs"> <h1> server.coffee </h1> </th> <th class="code"> </th> </tr> </thead> <tbody> <tr id="section-1"> <td class="docs"> <div class="pilwrap"> <a class="pilcrow" href="#section-1">¶</a> </div> <h1>A BrowserChannel server.</h1> <ul> <li>Its still pretty young, so there's probably bugs lurking around and the API will still change quickly.</li> <li>Its missing integration tests</li> </ul> It works in all the browsers I've tried. I've written this using the literate programming style to try it out. So, thats why there's a million comments everywhere. The server is implemented as connect middleware. Its intended to be used like this: <code> server = connect( browserChannel (client) -> client.send 'hi' ) </code> </td> <td class="code"> <div class="highlight"><pre></pre></div> </td> </tr> <tr id="section-2"> <td class="docs"> <div class="pilwrap"> <a class="pilcrow" href="#section-2">¶</a> </div> <h2>Dependancies, helper methods and constant data</h2> </td> <td class="code"> <div class="highlight"><pre></pre></div> </td> </tr> <tr id="section-3"> <td class="docs"> <div class="pilwrap"> <a class="pilcrow" href="#section-3">¶</a> </div> <code>parse</code> helps us decode URLs in requests </td> <td class="code"> <div class="highlight"><pre>{parse} = require 'url'</pre></div> </td> </tr> <tr id="section-4"> <td class="docs"> <div class="pilwrap"> <a class="pilcrow" href="#section-4">¶</a> </div> <code>querystring</code> will help decode the URL-encoded forward channel data </td> <td class="code"> <div class="highlight"><pre>querystring = require 'querystring'</pre></div> </td> </tr> <tr id="section-5"> <td class="docs"> <div class="pilwrap"> <a class="pilcrow" href="#section-5">¶</a> </div> <code>fs</code> is used to read & serve the client library </td> <td class="code"> <div class="highlight"><pre>fs = require 'fs'</pre></div> </td> </tr> <tr id="section-6"> <td class="docs"> <div class="pilwrap"> <a class="pilcrow" href="#section-6">¶</a> </div> Client sessions are `EventEmitters </td> <td class="code"> <div class="highlight"><pre>{EventEmitter} = require 'events'</pre></div> </td> </tr> <tr id="section-7"> <td class="docs"> <div class="pilwrap"> <a class="pilcrow" href="#section-7">¶</a> </div> Client session Ids are generated using <code>node-hat</code> </td> <td class="code"> <div class="highlight"><pre>hat = require('hat').rack(40, 36)</pre></div> </td> </tr> <tr id="section-8"> <td class="docs"> <div class="pilwrap"> <a class="pilcrow" href="#section-8">¶</a> </div> <code>randomInt(n)</code> generates and returns a random int smaller than n (0 <= k < n) </td> <td class="code"> <div class="highlight"><pre>randomInt = (n) -> Math.floor(Math.random() * n)</pre></div> </td> </tr> <tr id="section-9"> <td class="docs"> <div class="pilwrap"> <a class="pilcrow" href="#section-9">¶</a> </div> <code>randomArrayElement(array)</code> Selects and returns a random element from array </td> <td class="code"> <div class="highlight"><pre>randomArrayElement = (array) -> array[randomInt(array.length)]</pre></div> </td> </tr> <tr id="section-10"> <td class="docs"> <div class="pilwrap"> <a class="pilcrow" href="#section-10">¶</a> </div> For testing we'll override <code>setInterval</code>, etc with special testing stub versions (so we don't have to actually wait for actual time. To do that, we need local variable versions (I don't want to edit the global versions). ... and they'll just point to the normal versions anyway. </td> <td class="code"> <div class="highlight"><pre>{setInterval, clearInterval, setTimeout, clearTimeout, Date} = global</pre></div> </td> </tr> <tr id="section-11"> <td class="docs"> <div class="pilwrap"> <a class="pilcrow" href="#section-11">¶</a> </div> The module is configurable </td> <td class="code"> <div class="highlight"><pre>defaultOptions =</pre></div> </td> </tr> <tr id="section-12"> <td class="docs"> <div class="pilwrap"> <a class="pilcrow" href="#section-12">¶</a> </div> An optional array of host prefixes. Each browserchannel client will randomly pick from the list of host prefixes when it connects. This reduces the impact of per-host connection limits. All host prefixes should point to the same server. Ie, if your server's hostname is example.com and your hostPrefixes contains ['a', 'b', 'c'], a.example.com, b.example.com and c.example.com should all point to the same host as example.com. </td> <td class="code"> <div class="highlight"><pre> hostPrefixes: null</pre></div> </td> </tr> <tr id="section-13"> <td class="docs"> <div class="pilwrap"> <a class="pilcrow" href="#section-13">¶</a> </div> You can specify the base URL which browserchannel connects to. Change this if you want to scope browserchannel in part of your app, or if you want /channel to mean something else, or whatever. </td> <td class="code"> <div class="highlight"><pre> base: '/channel'</pre></div> </td> </tr> <tr id="section-14"> <td class="docs"> <div class="pilwrap"> <a class="pilcrow" href="#section-14">¶</a> </div> We'll send keepalives every so often to make sure the http connection isn't closed by eagar clients. The standard timeout is 30 seconds, so we'll default to sending them every 20 seconds or so. </td> <td class="code"> <div class="highlight"><pre> keepAliveInterval: 20 * 1000</pre></div> </td> </tr> <tr id="section-15"> <td class="docs"> <div class="pilwrap"> <a class="pilcrow" href="#section-15">¶</a> </div> After awhile (30 seconds or so) of not having a backchannel connected, we'll evict the session completely. This will happen whenever a user closes their browser. </td> <td class="code"> <div class="highlight"><pre> sessionTimeoutInterval: 30 * 1000</pre></div> </td> </tr> <tr id="section-16"> <td class="docs"> <div class="pilwrap"> <a class="pilcrow" href="#section-16">¶</a> </div> All server responses set some standard HTTP headers. To be honest, I don't know how many of these are necessary. I just copied them from google. The nocache headers in particular seem unnecessary since each client request includes a randomized <code>zx=junk</code> query parameter. </td> <td class="code"> <div class="highlight"><pre>standardHeaders = 'Content-Type': 'text/plain' 'Cache-Control': 'no-cache, no-store, max-age=0, must-revalidate' 'Pragma': 'no-cache' 'Expires': 'Fri, 01 Jan 1990 00:00:00 GMT' 'X-Content-Type-Options': 'nosniff'</pre></div> </td> </tr> <tr id="section-17"> <td class="docs"> <div class="pilwrap"> <a class="pilcrow" href="#section-17">¶</a> </div> Gmail also sends this, though I'm not really sure what it does... 'X-Xss-Protection': '1; mode=block' </td> <td class="code"> <div class="highlight"><pre></pre></div> </td> </tr> <tr id="section-18"> <td class="docs"> <div class="pilwrap"> <a class="pilcrow" href="#section-18">¶</a> </div> The one exception to that is requests destined for iframes. They need to have content-type: text/html set for IE to process the juicy JS inside. </td> <td class="code"> <div class="highlight"><pre>ieHeaders = Object.create standardHeaders ieHeaders['Content-Type'] = 'text/html'</pre></div> </td> </tr> <tr id="section-19"> <td class="docs"> <div class="pilwrap"> <a class="pilcrow" href="#section-19">¶</a> </div> Google's browserchannel server adds some junk after the first message data is sent. I assume this stops some whole-page buffering in IE. I assume the data used is noise so it doesn't compress. I don't really know why google does this. I'm assuming there's a good reason to it though. </td> <td class="code"> <div class="highlight"><pre>ieJunk = "7cca69475363026330a0d99468e88d23ce95e222591126443015f5f462d9a177186c8701fb45a6ffe e0daf1a178fc0f58cd309308fba7e6f011ac38c9cdd4580760f1d4560a84d5ca0355ecbbed2ab715a3350fe0c47 9050640bd0e77acec90c58c4d3dd0f5cf8d4510e68c8b12e087bd88cad349aafd2ab16b07b0b1b8276091217a44 a9fe92fedacffff48092ee693af\n"</pre></div> </td> </tr> <tr id="section-20"> <td class="docs"> <div class="pilwrap"> <a class="pilcrow" href="#section-20">¶</a> </div> If the user is using IE, instead of using XHR backchannel loaded using a forever iframe. When data is sent, it is wrapped in <script></script> tags which call functions in the browserchannel library. This method wraps the normal <code>.writeHead()</code>, <code>.write()</code> and <code>.end()</code> methods by special versions which produce output based on the request's type. This is not used for: <ul> <li>The first channel test</li> <li>The first bind connection a client makes. The server sends arrays there, but the connection is a POST and it returns immediately. So that request happens using XHR/Trident like regular forward channel requests.</li> </ul> </td> <td class="code"> <div class="highlight"><pre>messagingMethods = (query, res) -> type = query.TYPE if type == 'html' junkSent = false methods = writeHead: -> res.writeHead 200, 'OK', ieHeaders res.write '<html><body>' domain = query.DOMAIN</pre></div> </td> </tr> <tr id="section-21"> <td class="docs"> <div class="pilwrap"> <a class="pilcrow" href="#section-21">¶</a> </div> If the iframe is making the request using a secondary domain, I think we need to set the <code>domain</code> to the original domain so that we can call the response methods. </td> <td class="code"> <div class="highlight"><pre> if domain and domain != ''</pre></div> </td> </tr> <tr id="section-22"> <td class="docs"> <div class="pilwrap"> <a class="pilcrow" href="#section-22">¶</a> </div> Make sure the domain doesn't contain anything by naughty by <code>JSON.stringify()</code>-ing it before passing it to the client. There are XSS vulnerabilities otherwise. </td> <td class="code"> <div class="highlight"><pre> res.write "<script>try{document.domain=#{JSON.stringify domain};}catch(e){}</script>\n" write: (data) -></pre></div> </td> </tr> <tr id="section-23"> <td class="docs"> <div class="pilwrap"> <a class="pilcrow" href="#section-23">¶</a> </div> The data is passed to <code>m()</code>, which is bound to onTridentRpcMessage_ in the client. </td> <td class="code"> <div class="highlight"><pre> res.write "<script>try {parent.m(#{JSON.stringify data})} catch(e) {}</script>\n" unless junkSent res.write ieJunk junkSent = true end: -></pre></div> </td> </tr> <tr id="section-24"> <td class="docs"> <div class="pilwrap"> <a class="pilcrow" href="#section-24">¶</a> </div> Once the data has been received, the client needs to call <code>d()</code>, which is bound to onTridentDone_ with success=true. The weird spacing of this is copied from browserchannel. Its really not necessary. </td> <td class="code"> <div class="highlight"><pre> res.end "<script>try {parent.d(); }catch (e){}</script>\n"</pre></div> </td> </tr> <tr id="section-25"> <td class="docs"> <div class="pilwrap"> <a class="pilcrow" href="#section-25">¶</a> </div> This is a helper method for signalling an error in the request back to the client. </td> <td class="code"> <div class="highlight"><pre> writeError: (statusCode, message) -></pre></div> </td> </tr> <tr id="section-26"> <td class="docs"> <div class="pilwrap"> <a class="pilcrow" href="#section-26">¶</a> </div> The HTML (iframe) handler has no way to discover that the embedded script tag didn't complete successfully. To signal errors, we return 200 OK and call an exposed rpcClose() method on the page. </td> <td class="code"> <div class="highlight"><pre> methods.writeHead() res.end "<script>try {parent.rpcClose(#{JSON.stringify message})} catch(e){}</script>\n"</pre></div> </td> </tr> <tr id="section-27"> <td class="docs"> <div class="pilwrap"> <a class="pilcrow" href="#section-27">¶</a> </div> For some reason, sending data during the second test (111112) works slightly differently for XHR, but its identical for html encoding. We'll use a writeRaw() method in that case, which is copied in the case of html. </td> <td class="code"> <div class="highlight"><pre> methods.writeRaw = methods.write methods else</pre></div> </td> </tr> <tr id="section-28"> <td class="docs"> <div class="pilwrap"> <a class="pilcrow" href="#section-28">¶</a> </div> For normal XHR requests, we send data normally. </td> <td class="code"> <div class="highlight"><pre> writeHead: -> res.writeHead 200, 'OK', standardHeaders write: (data) -> res.write "#{data.length}\n#{data}" writeRaw: (data) -> res.write data end: -> res.end() writeError: (statusCode, message) -> res.writeHead statusCode, standardHeaders res.end message</pre></div> </td> </tr> <tr id="section-29"> <td class="docs"> <div class="pilwrap"> <a class="pilcrow" href="#section-29">¶</a> </div> For telling the client its done bad. It turns out google's server isn't particularly fussy about signalling errors using the proper html RPC stuff, so this is useful for html connections too. </td> <td class="code"> <div class="highlight"><pre>sendError = (res, statusCode, message) -> res.writeHead statusCode, message res.end "<html><body><h1>#{message}</h1></body></html>" return</pre></div> </td> </tr> <tr id="section-30"> <td class="docs"> <div class="pilwrap"> <a class="pilcrow" href="#section-30">¶</a> </div> <h2>Parsing client maps from the forward channel</h2> The client sends data in a series of url-encoded maps. The data is encoded like this: <code> count=2&ofs=0&req0_x=3&req0_y=10&req1_abc=def </code> First, we need to buffer up the request response and query string decode it. </td> <td class="code"> <div class="highlight"><pre>bufferPostData = (req, callback) -> data = [] req.on 'data', (chunk) -> data.push chunk.toString 'utf8' req.on 'end', -> data = data.join '' callback data</pre></div> </td> </tr> <tr id="section-31"> <td class="docs"> <div class="pilwrap"> <a class="pilcrow" href="#section-31">¶</a> </div> Next, we'll need to decode the incoming client data into an array of objects. The data could be in two different forms: <ul> <li>Classical browserchannel format, which is a bunch of string->string url-encoded maps</li> <li>A JSON object</li> </ul> We can tell what format the data is in by inspecting the content-type header <h2>URL Encoded data</h2> Essentially, url encoded the data looks like this: <code> { count: '2', ofs: '0', req0_x: '3', req0_y: '10', req1_abc: 'def' } </code> ... and we will return an object in the form of <code>[{x:'3', y:'10'}, {abc: 'def'}, ...]</code> <h2>JSON Encoded data</h2> JSON encoded the data looks like: <code> { ofs: 0 , data: [null, {...}, 1000.4, 'hi', ...] } </code> or <code>null</code> if there's no data. This function returns null if there's no data or {ofs, json:[...]} or {ofs, maps:[...]} </td> <td class="code"> <div class="highlight"><pre>decodeData = (req, data) -> if req.headers['content-type'] == 'application/json' data = JSON.parse data return null if data is null # There's no data. This is a valid response.</pre></div> </td> </tr> <tr id="section-32"> <td class="docs"> <div class="pilwrap"> <a class="pilcrow" href="#section-32">¶</a> </div> We'll restructure it slightly to mark the data as JSON rather than maps. </td> <td class="code"> <div class="highlight"><pre> {ofs, data} = data {ofs, json:data} else</pre></div> </td> </tr> <tr id="section-33"> <td class="docs"> <div class="pilwrap"> <a class="pilcrow" href="#section-33">¶</a> </div> Maps. Ugh. </td> <td class="code"> <div class="highlight"><pre> data = querystring.parse data count = parseInt data.count return null if count is 0</pre></div> </td> </tr> <tr id="section-34"> <td class="docs"> <div class="pilwrap"> <a class="pilcrow" href="#section-34">¶</a> </div> ofs will be missing if count is zero </td> <td class="code"> <div class="highlight"><pre> ofs = parseInt data.ofs throw new Error 'invalid map data' if isNaN count or isNaN ofs throw new Error 'Invalid maps' unless count == 0 or (count > 0 and data.ofs?) maps = new Array count</pre></div> </td> </tr> <tr id="section-35"> <td class="docs"> <div class="pilwrap"> <a class="pilcrow" href="#section-35">¶</a> </div> Scan through all the keys in the data. Every key of the form: <code>req123_xxx</code> will be used to populate its map. </td> <td class="code"> <div class="highlight"><pre> regex = /^req(\d+)_(.+)$/ for key, val of data match = regex.exec key if match id = match[1] mapKey = match[2] map = (maps[id] ||= {})</pre></div> </td> </tr> <tr id="section-36"> <td class="docs"> <div class="pilwrap"> <a class="pilcrow" href="#section-36">¶</a> </div> The client uses <code>mapX_type=_badmap</code> to signify an error encoding a map. </td> <td class="code"> <div class="highlight"><pre> continue if id == 'type' and mapKey == '_badmap' map[mapKey] = val {ofs, maps}</pre></div> </td> </tr> <tr id="section-37"> <td class="docs"> <div class="pilwrap"> <a class="pilcrow" href="#section-37">¶</a> </div> This is a helper method to order the handling of messages / requests / whatever. Use it like this: inOrder = order 0 inOrder 1, -> console.log 'second' inOrder 0, -> console.log 'first' Start is the ID of the first element we expect to receive. If we get data for earlier elements, we'll play them anyway if playOld is truthy. </td> <td class="code"> <div class="highlight"><pre>order = (start, playOld) -></pre></div> </td> </tr> <tr id="section-38"> <td class="docs"> <div class="pilwrap"> <a class="pilcrow" href="#section-38">¶</a> </div> Base is the ID of the (missing) element at the start of the queue </td> <td class="code"> <div class="highlight"><pre> base = start</pre></div> </td> </tr> <tr id="section-39"> <td class="docs"> <div class="pilwrap"> <a class="pilcrow" href="#section-39">¶</a> </div> The queue will start with about 10 elements. Elements of the queue are undefined if we don't have data for that queue element. </td> <td class="code"> <div class="highlight"><pre> queue = new Array 10 (seq, callback) -></pre></div> </td> </tr> <tr id="section-40"> <td class="docs"> <div class="pilwrap"> <a class="pilcrow" href="#section-40">¶</a> </div> Its important that all the cells of the array are truthy if we have data. We'll use an empty function instead of null. </td> <td class="code"> <div class="highlight"><pre> callback or= -></pre></div> </td> </tr> <tr id="section-41"> <td class="docs"> <div class="pilwrap"> <a class="pilcrow" href="#section-41">¶</a> </div> Ignore old messages, or play them back immediately if playOld=true </td> <td class="code"> <div class="highlight"><pre> if seq < base callback() if playOld else queue[seq - base] = callback while queue[0] callback = queue.shift() base++ callback()</pre></div> </td> </tr> <tr id="section-42"> <td class="docs"> <div class="pilwrap"> <a class="pilcrow" href="#section-42">¶</a> </div> We need access to the client's sourcecode. I'm going to get it using a synchronous file call (it'll be fast anyway, and only happen once). I'm also going to set an etag on the client data so the browser client will be cached. I'm kind of uncomfortable about adding complexity here because its not like this code hasn't been written before, but.. I think a lot of people will use this API. I should probably look into hosting the client code as a javascript module using that client-side npm thing. </td> <td class="code"> <div class="highlight"><pre>clientFile = "#{__dirname}/../dist/bcsocket.js" clientStats = fs.statSync clientFile try clientCode = fs.readFileSync clientFile, 'utf8' catch e console.error 'Could not load the client javascript. Run `cake client` to generate it.' throw e</pre></div> </td> </tr> <tr id="section-43"> <td class="docs"> <div class="pilwrap"> <a class="pilcrow" href="#section-43">¶</a> </div> This is mostly to help development, but if the client is recompiled, I'll pull in a new version. This isn't tested by the unit tests - but its not a big deal. </td> <td class="code"> <div class="highlight"><pre>fs.watchFile clientFile, persistent: false, (curr, prev) -> if curr.mtime.getTime() != prev.mtime.getTime()</pre></div> </td> </tr> <tr id="section-44"> <td class="docs"> <div class="pilwrap"> <a class="pilcrow" href="#section-44">¶</a> </div> Putting a synchronous file call here will stop the whole server while the client is reloaded. Again, this will only happen during development so its not a big deal. </td> <td class="code"> <div class="highlight"><pre> console.log "Reloading client JS" clientCode = fs.readFileSync clientFile, 'utf8' clientStats = curr</pre></div> </td> </tr> <tr id="section-45"> <td class="docs"> <div class="pilwrap"> <a class="pilcrow" href="#section-45">¶</a> </div> <hr /> <h1>The server middleware</h1> The server module returns a function, which you can call with your configuration options. It returns your configured connect middleware, which is actually another function. </td> <td class="code"> <div class="highlight"><pre>module.exports = browserChannel = (options, onConnect) -> if typeof onConnect == 'undefined' onConnect = options options = {} options ||= {} options[option] ?= value for option, value of defaultOptions</pre></div> </td> </tr> <tr id="section-46"> <td class="docs"> <div class="pilwrap"> <a class="pilcrow" href="#section-46">¶</a> </div> Strip off a trailing slash in base. </td> <td class="code"> <div class="highlight"><pre> base = options.base base = base[... base.length - 1] if base.match /\/$/ </pre></div> </td> </tr> <tr id="section-47"> <td class="docs"> <div class="pilwrap"> <a class="pilcrow" href="#section-47">¶</a> </div> Add a leading slash back on base </td> <td class="code"> <div class="highlight"><pre> base = "/#{base}" unless base.match /^\// sessions = {}</pre></div> </td> </tr> <tr id="section-48"> <td class="docs"> <div class="pilwrap"> <a class="pilcrow" href="#section-48">¶</a> </div> Host prefixes provide a way to skirt around connection limits. They're only really important for old browsers. </td> <td class="code"> <div class="highlight"><pre> getHostPrefix = -> if options.hostPrefixes randomArrayElement options.hostPrefixes else null</pre></div> </td> </tr> <tr id="section-49"> <td class="docs"> <div class="pilwrap"> <a class="pilcrow" href="#section-49">¶</a> </div> <h1>Create a new client session.</h1> This method will start a new client session. Session ids are generated by <a href="https://github.com/substack/node-hat">node-hat</a>. They are guaranteed to be unique. This method is synchronous, because a database will never be involved in browserchannel session management. Browserchannel sessions only last as long as the user's browser is open. If there's any connection turbulence, the client will reconnect and get a new session id. Sometimes a client will specify an old session ID and old array ID. In this case, the client is reconnecting and we should evict the named session (if it exists). </td> <td class="code"> <div class="highlight"><pre> createSession = (address, query, headers) -> {RID:initialRid, CVER:appVersion, OSID:oldSessionId, OAID:oldArrayId} = query if oldSessionId? and (oldSession = sessions[oldSessionId]) oldSession._acknowledgeArrays oldArrayId oldSession.close 'Reconnected'</pre></div> </td> </tr> <tr id="section-50"> <td class="docs"> <div class="pilwrap"> <a class="pilcrow" href="#section-50">¶</a> </div> Create a new session. Sessions extend node's <a href="http://nodejs.org/docs/v0.4.12/api/events.html">EventEmitter</a> so they have access to goodies like <code>session.on(event, handler)</code>, <code>session.emit('paarty')</code>, etc. </td> <td class="code"> <div class="highlight"><pre> session = new EventEmitter</pre></div> </td> </tr> <tr id="section-51"> <td class="docs"> <div class="pilwrap"> <a class="pilcrow" href="#section-51">¶</a> </div> The session's unique ID for this connection </td> <td class="code"> <div class="highlight"><pre> session.id = hat()</pre></div> </td> </tr> <tr id="section-52"> <td class="docs"> <div class="pilwrap"> <a class="pilcrow" href="#section-52">¶</a> </div> The client stores its IP address and headers from when it first opened the session. The handler can use this information for authentication or something. </td> <td class="code"> <div class="highlight"><pre> session.address = address session.headers = headers</pre></div> </td> </tr> <tr id="section-53"> <td class="docs"> <div class="pilwrap"> <a class="pilcrow" href="#section-53">¶</a> </div> The session is a little state machine. It has the following states: <ul> <li>