epubjs

<?xml version="1.0" encoding="UTF-8" standalone="no"?> <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.1//EN" "http://www.w3.org/TR/xhtml11/DTD/xhtml11.dtd"> <html xmlns="http://www.w3.org/1999/xhtml"><head><title>Chapter 2. Doing Interesting Things</title><link rel="stylesheet" href="core.css" type="text/css"/><meta name="generator" content="DocBook XSL Stylesheets V1.74.0"/></head><body><div class="chapter" title="Chapter 2. Doing Interesting Things"><div class="titlepage"><div><div><h1 class="title"><a id="chap2a"/>Chapter 2. Doing Interesting Things</h1></div></div></div><p>The programming trends of the last few years have made it progressively easier to write more complex applications with ease. It’s important that we don’t lose that, but Node is specifically focused on solving the problem of building <span class="emphasis"><em>network</em></span> applications—that is, applications that do a lot of input/output (I/O). Let’s build a few I/O-type apps and see how easy it is to do this with Node in a way that really scales.</p><div class="sect1" title="Building a Chat Server"><div class="titlepage"><div><div><h1 class="title"><a id="I_sect12_d1e1211"/>Building a Chat Server</h1></div></div></div><p>In a world that’s <a id="I_indexterm2_d1e1216" class="indexterm"/><a id="we2.1" class="indexterm"/><a id="I_indexterm2_d1e1226" class="indexterm"/><a id="no2.1" class="indexterm"/>increasingly real-time, what is more real-time than chat? So where should we begin? Let’s start with a TCP-based chat server we can connect to with Telnet. Not only is it a simple place to start, but it’s also something we can write 100% in Node.</p><p>The first thing we need to do is include the <a id="I_indexterm2_d1e1239" class="indexterm"/>TCP libraries from Node and create a new <a id="I_indexterm2_d1e1245" class="indexterm"/>TCP server (see <a class="xref" href="ch02.html#example2-1" title="Example 2-1. Creating a new TCP server">Example 2-1</a>).</p><div class="example"><a id="example2-1"/><p class="title">Example 2-1. Creating a new TCP server</p><div class="example-contents"><pre class="programlisting">var net = require('net') var chatServer = net.createServer() chatServer.on('connection', function(client) { client.write('Hi!\n'); client.write('Bye!\n'); client.end() }) chatServer.listen(9000)</pre></div></div><p>First, we include the <code class="literal">net</code> module. <a id="I_indexterm2_d1e1264" class="indexterm"/>This contains all the TCP stuff for Node. From that, we can create a TCP server by calling the <code class="literal">net.createServer()</code> <a id="I_indexterm2_d1e1273" class="indexterm"/>method. Now that we have a server, we want it to do stuff. So we add an event listener by using the <code class="literal">on()</code> method. Whenever the <code class="literal">connection</code> event <a id="I_indexterm2_d1e1286" class="indexterm"/>happens, the event listener will call the function we gave it. A <code class="literal">connection</code> event <a id="I_indexterm2_d1e1293" class="indexterm"/>happens when a new client connects to the server.</p><p>The connection event passes us a reference to the TCP socket for our new client when it calls our callback function. We named this reference <code class="literal">client</code>. By calling <code class="literal">client.write()</code>, <a id="I_indexterm2_d1e1307" class="indexterm"/><a id="I_indexterm2_d1e1312" class="indexterm"/><a id="I_indexterm2_d1e1317" class="indexterm"/>we can send messages to the newly connected client. To start with, we just say “Hi!” and then “Bye!”, and we call the <code class="literal">client.end()</code> method, which closes the connection. It’s simple, but it’s a starting point for our chat server. Finally, we need to call <code class="literal">listen()</code> so <a id="I_indexterm2_d1e1330" class="indexterm"/><a id="I_indexterm2_d1e1335" class="indexterm"/>Node knows which port to listen on. Let’s test it.</p><p>We can test our new server by connecting to it with the Telnet program, which is installed on most operating systems.<sup>[<a id="id752685" href="#ftn.id752685" class="footnote">2</a>]</sup> First, we need to start our <a id="I_indexterm2_d1e1347" class="indexterm"/>server by calling <code class="literal">node</code> with the filename. Then we can connect by opening a Telnet connection to <code class="literal">localhost</code> on port 9000, as we <a id="I_indexterm2_d1e1359" class="indexterm"/><a id="I_indexterm2_d1e1364" class="indexterm"/>specified in our Node program. See <a class="xref" href="ch02.html#example2-2" title="Example 2-2. Connecting to a Node TCP server with Telnet">Example 2-2</a>.</p><div class="example"><a id="example2-2"/><p class="title">Example 2-2. Connecting to a Node TCP server with Telnet</p><div class="example-contents"><pre class="programlisting">Console Window 1 ---------------- Enki:~ $ <strong class="userinput"><code>node chat.js</code></strong> Chat server started Console Window 2 ---------------- Last login: Tue Jun 7 20:35:14 on ttys000 Enki:~ $ <strong class="userinput"><code>telnet 127.0.0.1 9000</code></strong> Trying 127.0.0.1... Connected to localhost. Escape character is '^]'. Hi! Bye! Connection closed by foreign host. Enki:~ $</pre></div></div><p>So far we’ve made a server that clients can connect to, and we’ve sent them a message before kicking them out, but it’s hardly a chat server. Let’s add a few more things. First, we <a id="I_indexterm2_d1e1386" class="indexterm"/><a id="I_indexterm2_d1e1391" class="indexterm"/>should make sure we can get messages from the clients, as shown in <a class="xref" href="ch02.html#example2-3" title="Example 2-3. Listening to each connection">Example 2-3</a>.</p><div class="example"><a id="example2-3"/><p class="title">Example 2-3. Listening to each connection</p><div class="example-contents"><pre class="programlisting">var net = require('net') var chatServer = net.createServer() chatServer.on('connection', function(client) { client.write('Hi!\n'); client.on('data', function(data) { console.log(data) }) }) chatServer.listen(9000)</pre></div></div><p>Here we’ve added another event listener, and this time it’s <code class="literal">client.on()</code>. Notice how we’ve added the event listener in the scope of the <code class="literal">connection</code> callback function. Doing this means we have access to the <code class="literal">client</code> that is passed to that event. The listener we’ve added is for an event called <code class="literal">data</code>. This is the event that is called each time <code class="literal">client</code> sends some data to the server. We’ve had to lose the <code class="literal">client.end()</code>, though. If we closed the connection to the client, how could we listen for new data? Now whenever we send data to the server, it will be outputted to the console. Let’s try <a id="I_indexterm2_d1e1428" class="indexterm"/><a id="I_indexterm2_d1e1433" class="indexterm"/><a id="I_indexterm2_d1e1438" class="indexterm"/>that in <a class="xref" href="ch02.html#example2-4" title="Example 2-4. Sending data to the server from Telnet">Example 2-4</a>.</p><div class="example"><a id="example2-4"/><p class="title">Example 2-4. Sending data to the server from Telnet</p><div class="example-contents"><pre class="programlisting">Console 1 ------------- Enki:~ $ <strong class="userinput"><code>node chat.js</code></strong> Chat server started <Buffer 48 65 6c 6c 6f 2c 20 79 6f 75 72 73 65 6c 66 0d 0a> Console 2 ------------ Enki:~ $ <strong class="userinput"><code>telnet 127.0.0.1 9000</code></strong> Trying 127.0.0.1... Connected to localhost. Escape character is '^]'. Hi! Hello, yourself</pre></div></div><p>What has happened here? We ran the server and connected to it with Telnet. The server said “Hi!” and we responded with “Hello, yourself”. At this point, Node spat out a bunch of seeming gibberish in a data type you’ve never seen before. Because JavaScript doesn’t have a good way to deal with binary data, Node added one. It’s called<a id="I_indexterm2_d1e1459" class="indexterm"/> <code class="literal">Buffer</code>, and it lets the server represent binary data. Node doesn’t know what kind of data Telnet sent, so Node simply stores the data as binary until we ask for it in some other kind of encoding. The sequence of letters and numbers is actually bytes in hex (see <a class="xref" href="ch04.html#chap6_id35817399" title="Buffers">Buffers</a> in <a class="xref" href="ch04.html" title="Chapter 4. Core APIs">Chapter 4</a> for more on this). Each byte represents one of the letters or characters in the string “Hello, yourself”. We can use the <code class="literal">toString()</code> <a id="I_indexterm2_d1e1476" class="indexterm"/>method to translate <code class="literal">Buffer</code> data into a regular string again if we want, or we can just pass it around as it is because TCP and Telnet understand the binary, too.</p><p>Now that we can get messages from each client, we should let them send each other messages. To do this, we need a way of letting them communicate with each other. It’s great that we can call <code class="literal">client.write()</code>, but that works on only one client at a time. What we need is a way to reference other clients. We can do this by creating a list of clients that we want to write data to. Whenever we get a new client, we’ll add it to our list and use the list to communicate <a id="I_indexterm2_d1e1491" class="indexterm"/><a id="I_indexterm2_d1e1496" class="indexterm"/>between the clients (see <a class="xref" href="ch02.html#example2-5" title="Example 2-5. Communicating between clients">Example 2-5</a>).</p><div class="example"><a id="example2-5"/><p class="title">Example 2-5. Communicating between clients</p><div class="example-contents"><pre class="programlisting">var net = require('net') var chatServer = net.createServer(), clientList = [] chatServer.on('connection', function(client) { client.write('Hi!\n'); clientList.push(client) client.on('data', function(data) { for(var i=0;i<clientList.length;i+=1) { //write this data to all clients clientList[i].write(data) } }) }) chatServer.listen(9000)</pre></div></div><p>Now when we run it in <a class="xref" href="ch02.html#example2-6" title="Example 2-6. Sending messages between clients">Example 2-6</a>, we can connect multiple <a id="I_indexterm2_d1e1513" class="indexterm"/><a id="I_indexterm2_d1e1518" class="indexterm"/>clients to the server to see them sending messages to each other.</p><div class="example"><a id="example2-6"/><p class="title">Example 2-6. Sending messages between clients</p><div class="example-contents"><pre class="programlisting">Console 1 ------------ Enki:~ $ <strong class="userinput"><code>node chat.js</code></strong> Console 2 ------------ Enki:~ $ <strong class="userinput"><code>telnet 127.0.0.1 9000</code></strong> Trying 127.0.0.1... Connected to localhost. Escape character is '^]'. Hi! Hello, yourself Hello, yourself Console 3 ------------ Enki:~ $ <strong class="userinput"><code>telnet 127.0.0.1 9000</code></strong> Trying 127.0.0.1... Connected to localhost. Escape character is '^]'. Hi! Hello, yourself</pre></div></div><p>This time, the server isn’t logging any messages it receives, and instead we loop through the array and send them back to the clients. Notice that when the Telnet client in terminal 2 sends a message, it gets sent to the Telnet client in terminal 3, but it also gets sent back to Telnet in terminal 2 as well. This is because when we send the message, we aren’t checking who the sender was; we just send to our entire array of clients. It’s also not clear just by looking at Telnet which messages were things we sent and which were things we received. We can improve on this. In <a class="xref" href="ch02.html#example2-7" title="Example 2-7. Improving the sending of messages">Example 2-7</a>, let’s create a function to <a id="I_indexterm2_d1e1544" class="indexterm"/><a id="I_indexterm2_d1e1549" class="indexterm"/>send messages to all the clients, and use it to tidy up some of these issues.</p><div class="example"><a id="example2-7"/><p class="title">Example 2-7. Improving the sending of messages</p><div class="example-contents"><pre class="programlisting">var net = require('net') var chatServer = net.createServer(), clientList = [] chatServer.on('connection', function(client) { client.name = client.remoteAddress + ':' + client.remotePort client.write('Hi ' + client.name + '!\n'); clientList.push(client) client.on('data', function(data) { broadcast(data, client) }) }) function broadcast(message, client) { for(var i=0;i<clientList.length;i+=1) { if(client !== clientList[i]) { clientList[i].write(client.name + " says " + message) } } } chatServer.listen(9000)</pre></div></div><p>The first thing we’ve added to the <code class="literal">connection</code> event listener is a command to add a name property to each <code class="literal">client</code>. Note how we are able to decorate the <code class="literal">client</code> object with additional properties. This is because the closure binds each <code class="literal">client</code> object to a specific request. The existing properties of the <code class="literal">client</code> are used to create the name, and the <code class="literal">client.remoteAddress</code> <a id="I_indexterm2_d1e1582" class="indexterm"/><a id="I_indexterm2_d1e1587" class="indexterm"/>is the IP address the <code class="literal">client</code> is connecting from. <a id="I_indexterm2_d1e1596" class="indexterm"/>The <code class="literal">client</code><code class="literal">.remotePort</code> is the TCP port that the client asked the server to send data back to. When multiple clients connect from the same IP, they will each have a unique <code class="literal">remotePort</code>. When we issue a greeting to the <code class="literal">client</code>, we can now do it using a unique name for that client.</p><p>We also extracted the client write loop from the <code class="literal">data</code> event listener. We now have a function called <code class="literal">broadcast</code> and, using it, we can send a message to all the connected clients. However, this time we pass the <code class="literal">client</code> that is sending the message (<code class="literal">data</code>) so we can exclude it from getting the message. We also include the sending client name (now that it has one) when sending the message to the other clients. This is a much better version of the server, as <a id="I_indexterm2_d1e1628" class="indexterm"/><a id="I_indexterm2_d1e1633" class="indexterm"/>shown in <a class="xref" href="ch02.html#example2-8" title="Example 2-8. Running the improved chat server">Example 2-8</a>.</p><div class="example"><a id="example2-8"/><p class="title">Example 2-8. Running the improved chat server</p><div class="example-contents"><pre class="programlisting">Console 1 --------- Enki:~ $ <strong class="userinput"><code>node chat.js</code></strong> Console 2 --------- Enki:~ $ <strong class="userinput"><code>telnet 127.0.0.1 9000</code></strong> Trying 127.0.0.1... Connected to localhost. Escape character is '^]'. Hi 127.0.0.1:56022! <strong class="userinput"><code>Hello</code></strong> 127.0.0.1:56018 says Back atcha Console 3 --------- Enki:~ $ <strong class="userinput"><code>telnet 127.0.0.1 9000</code></strong> Trying 127.0.0.1... Connected to localhost. Escape character is '^]'. Hi 127.0.0.1:56018! 127.0.0.1:56022 says Hello <strong class="userinput"><code>Back atcha</code></strong></pre></div></div><p>This is a much friendlier and more useful service. It’s still not perfect, but we are making progress. Note that the exact port numbers used in the names will almost certainly vary for you when you run this example. Different operating systems allow different port ranges, and the assignment will also depend on which ones you are already using, as well as a random factor. You may have already encountered this, but our server has a fatal flaw! If one of the clients disconnects, the <a id="I_indexterm2_d1e1663" class="indexterm"/><a id="I_indexterm2_d1e1668" class="indexterm"/><a id="I_indexterm2_d1e1673" class="indexterm"/>server will fail horribly, as demonstrated in <a class="xref" href="ch02.html#example2-9" title="Example 2-9. Causing the server to fail by disconnecting a client">Example 2-9</a>.</p><div class="example"><a id="example2-9"/><p class="title">Example 2-9. Causing the server to fail by disconnecting a client</p><div class="example-contents"><pre class="programlisting">Console 1 ---------- Enki:~ $ <strong class="userinput"><code>node book-chat.js</code></strong> <a id="ex1-9-1"/><img src="callouts/1.png" alt="1"/> net.js:392 <a id="ex1-9-4a"/><img src="callouts/2.png" alt="2"/> throw new Error('Socket is not writable'); ^ Error: Socket is not writable at Socket._writeOut (net.js:392:11) at Socket.write (net.js:378:17) at broadcast (/Users/sh1mmer/book-chat.js:21:21) at Socket.<anonymous> (/Users/sh1mmer/book-chat.js:13:5) at Socket.emit (events.js:64:17) at Socket._onReadable (net.js:679:14) at IOWatcher.onReadable [as callback] (net.js:177:10) Enki:~ $ Console 2 --------- Enki:~ $ <strong class="userinput"><code>telnet 127.0.0.1 9000</code></strong> <a id="ex1-9-2a"/><img src="callouts/3.png" alt="3"/> Trying 127.0.0.1... Connected to localhost. Escape character is '^]'. Hi 127.0.0.1:56910! <strong class="userinput"><code>^]</code></strong> telnet> quit <a id="ex1-9-3"/><img src="callouts/4.png" alt="4"/> Connection closed. Enki:~ $ Console 3 --------- Enki:~ $ <strong class="userinput"><code>telnet 127.0.0.1 9000</code></strong> <a id="ex1-9-2b"/><img src="callouts/5.png" alt="5"/> Trying 127.0.0.1... Connected to localhost. Escape character is '^]'. Hi 127.0.0.1:56911! You still there? <a id="ex1-9-4b"/><img src="callouts/6.png" alt="6"/> Connection closed by foreign host. <a id="ex1-9-5"/><img src="callouts/7.png" alt="7"/> Enki:~ $</pre></div></div><p>We start the server as normal <a class="xref" href="ch02.html#ex1-9-1"><img src="callouts/1.png" alt="1"/></a> and connect some clients <a class="xref" href="ch02.html#ex1-9-2a"><img src="callouts/3.png" alt="3"/></a><a class="xref" href="ch02.html#ex1-9-2b"><img src="callouts/5.png" alt="5"/></a>, but when the client in Console 2 disconnects <a class="xref" href="ch02.html#ex1-9-3"><img src="callouts/4.png" alt="4"/></a>, we have a bit of a problem. The next time we use <code class="literal">broadcast()</code>, in this case when Console 3 sends a message <a class="xref" href="ch02.html#ex1-9-4b"><img src="callouts/6.png" alt="6"/></a>, the server tries to write to a disconnected client <a class="xref" href="ch02.html#ex1-9-4a"><img src="callouts/2.png" alt="2"/></a>. When the client from Console 2 disconnected <a class="xref" href="ch02.html#ex1-9-3"><img src="callouts/4.png" alt="4"/></a>, its socket stopped being writable or readable. When we try to call <code class="literal">write()</code> on a socket that is closed, we get an exception in the Node process. This also causes the disconnection of all the remaining clients <a class="xref" href="ch02.html#ex1-9-5"><img src="callouts/7.png" alt="7"/></a>. Obviously, this is extremely brittle and not acceptable for a server.</p><p>We should fix this in two ways. First, we should make sure that when a client disconnects, we remove it from the <code class="literal">clients</code> array so it stops getting <code class="literal">write()</code> method calls. This will also allow V8 to garbage-collect the socket object and reclaim that memory. Second, we should be a bit more defensive when we write to a socket. We want to make sure that between the last time the socket was written and the current pass, nothing has stopped us from being able to call <code class="literal">write()</code>. Happily, <a id="I_indexterm2_d1e1748" class="indexterm"/><a id="I_indexterm2_d1e1753" class="indexterm"/>Node has easy ways to achieve both of these things. The first is shown in <a class="xref" href="ch02.html#example2-10" title="Example 2-10. Making the chat server more robust">Example 2-10</a>.</p><div class="example"><a id="example2-10"/><p class="title">Example 2-10. Making the chat server more robust</p><div class="example-contents"><pre class="programlisting">chatServer.on('connection', function(client) { client.name = client.remoteAddress + ':' + client.remotePort client.write('Hi ' + client.name + '!\n'); clientList.push(client) client.on('data', function(data) { broadcast(data, client) }) client.on('end', function() { clientList.splice(clientList.indexOf(client), 1) }) })</pre></div></div><p>First, let’s deal with those disconnecting clients. When a client disconnects, we want to be able to remove it from the list of clients. This is easy to achieve with the <code class="literal">end</code> event. When a socket disconnects, it fires the <code class="literal">end</code> event to indicate that it’s about to close. We can call <code class="literal">Array.splice()</code> when this happens to remove the client from the <code class="literal">clientList</code> array. Using <code class="literal">Array.indexOf()</code>, we are able to find the position of this <code class="literal">client</code>. <code class="literal">splice()</code> then removes from the array one item, which is the <code class="literal">client</code>. Now when the next <code class="literal">client</code> uses the broadcast call, the disconnected client will no longer be in the list.</p><p>We can still be a bit more defensive, though, as <a id="I_indexterm2_d1e1798" class="indexterm"/><a id="I_indexterm2_d1e1803" class="indexterm"/>demonstrated in <a class="xref" href="ch02.html#example2-11" title="Example 2-11. Checking the write status of sockets">Example 2-11</a>.</p><div class="example"><a id="example2-11"/><p class="title">Example 2-11. Checking the write status of sockets</p><div class="example-contents"><pre class="programlisting">function broadcast(message, client) { var cleanup = [] for(var i=0;i<clientList.length;i+=1) { if(client !== clientList[i]) { if(clientList[i].writable) { clientList[i].write(client.name + " says " + message) } else { cleanup.push(clientList[i]) clientList[i].destroy() } } } //Remove dead Nodes out of write loop to avoid trashing loop index for(i=0;i<cleanup.length;i+=1) { clientList.splice(clientList.indexOf(cleanup[i]), 1) } }</pre></div></div><p>By adding a check for the write status of the socket during the broadcast call, we can make sure that any sockets that are not available to be written don’t cause an exception. Moreover, we can make sure that any sockets that can’t be written to are closed (using <code class="literal">Socket.destroy()</code>) and <a id="I_indexterm2_d1e1823" class="indexterm"/><a id="I_indexterm2_d1e1828" class="indexterm"/>then removed from the <code class="literal">clientList</code>. Note that we don’t remove the sockets from the <code class="literal">clientList</code> while we are looping through it, because we don’t want to cause side effects on the current loop we are in. Our server is now much more robust. There is one more thing we should do before we are really <a id="I_indexterm2_d1e1840" class="indexterm"/><a id="I_indexterm2_d1e1846" class="indexterm"/>ready to deploy it: log the errors (<a class="xref" href="ch02.html#example2-12" title="Example 2-12. Logging errors">Example 2-12</a>).</p><div class="example"><a id="example2-12"/><p class="title">Example 2-12. Logging errors</p><div class="example-contents"><pre class="programlisting">chatServer.on('connection', function(client) { client.name = client.remoteAddress + ':' + client.remotePort client.write('Hi ' + client.name + '!\n'); console.log(client.name + ' joined') clientList.push(client) client.on('data', function(data) { broadcast(data, client) }) client.on('end', function() { console.log(client.name + ' quit') clientList.splice(clientList.indexOf(client), 1) }) client.on('error', function(e) { console.log(e) }) })</pre></div></div><p>By adding a <code class="literal">console.log()</code> call to the <code class="literal">error</code> event <a id="I_indexterm2_d1e1868" class="indexterm"/><a id="I_indexterm2_d1e1871" class="indexterm"/>for the <code class="literal">client</code> objects, we can ensure that any errors that occur to clients are logged, even as our previous code makes sure that clients throwing errors are not able to cause the server to abort with <a id="I_indexterm2_d1e1880" class="indexterm"/><a id="I_indexterm2_d1e1883" class="indexterm"/>an exception.</p></div><div class="sect1" title="Let’s Build Twitter"><div class="titlepage"><div><div><h1 class="title"><a id="I_sect12_d1e1886"/>Let’s Build Twitter</h1></div></div></div><p>The previous <a id="I_indexterm2_d1e1891" class="indexterm"/><a id="we2.2" class="indexterm"/><a id="no2.2" class="indexterm"/>example shows how easy it is to write something extremely real-time with Node, but often you just want to write a web application. Let’s try to create something similar to Twitter with Node so we can see what it’s like to make a web application. The first thing we should do is install the<a id="I_indexterm2_d1e1907" class="indexterm"/> Express module (<a class="xref" href="ch02.html#example2-13" title="Example 2-13. Installing the Express module">Example 2-13</a>). This web framework for Node makes it much easier to create web applications by adding support for common tasks, such as MVC, to the <a id="I_indexterm2_d1e1915" class="indexterm"/><a id="I_indexterm2_d1e1920" class="indexterm"/><a id="I_indexterm2_d1e1926" class="indexterm"/>existing <code class="literal">http</code> server.</p><div class="example"><a id="example2-13"/><p class="title">Example 2-13. Installing the Express module</p><div class="example-contents"><pre class="programlisting">Enki:~ $ <strong class="userinput"><code>npm install express</code></strong> express@2.3.12 ./node_modules/express ├── mime@1.2.2 ├── connect@1.5.1 └── qs@0.1.0 Enki:~ $</pre></div></div><p>Installing Express is easy using the Node Package Manager (<code class="literal">npm</code>)<a id="I_indexterm2_d1e1948" class="indexterm"/><a id="I_indexterm2_d1e1953" class="indexterm"/><a id="I_indexterm2_d1e1958" class="indexterm"/><a id="I_indexterm2_d1e1963" class="indexterm"/>. Once we have the framework installed, we can make a <a id="I_indexterm2_d1e1969" class="indexterm"/>basic web application (<a class="xref" href="ch02.html#example2-14" title="Example 2-14. A basic web server with Express">Example 2-14</a>). This looks a lot like the application we <a id="I_indexterm2_d1e1978" class="indexterm"/>built in <a class="xref" href="ch01.html" title="Chapter 1. A Very Brief Introduction to Node.js">Chapter 1</a>.</p><div class="note" title="Note"><h3 class="title">Note</h3><p>You can read more about <code class="literal">npm</code> in Chapters <a class="xref" href="ch06.html" title="Chapter 6. Data Access">6</a> and <a class="xref" href="ch07.html" title="Chapter 7. Important External Modules">7</a>.</p></div><div class="example"><a id="example2-14"/><p class="title">Example 2-14. A basic web server with Express</p><div class="example-contents"><pre class="programlisting">var express = require('express') var app = express.createServer() app.get('/', function(req, res) { res.send('Welcome to Node Twitter') }) app.listen(8000)</pre></div></div><p>This code looks pretty similar to the basic web server code from <a class="xref" href="ch01.html" title="Chapter 1. A Very Brief Introduction to Node.js">Chapter 1</a>. Instead of including the <code class="literal">http</code> module, however, we include <code class="literal">express</code>. Express is actually getting <code class="literal">http</code> behind the scenes, but we don’t have to get that ourselves, because Node will automatically resolve the dependencies. Like with <code class="literal">http</code> and <code class="literal">net</code>, we call<a id="I_indexterm2_d1e2021" class="indexterm"/> <code class="literal">createServer()</code> to make a server and call <code class="literal">listen()</code> <a id="I_indexterm2_d1e2033" class="indexterm"/>to make it listen to a specific port. Instead of attaching an event listener to the <code class="literal">request</code> event with Express, we can call methods matching the HTTP verbs. In this case, when we call <code class="literal">get()</code>, we can create a callback function that will match GET requests only to a URL that matches the first argument of the call. This has immediately added two things that the <code class="literal">http</code> server didn’t have: the ability to filter based on HTTP verbs, and the ability to filter based on specific URLs.</p><p>When we get the callback, it looks a lot like the one from the <code class="literal">http</code> server—because it is. However, Express has added some extra methods. With the <code class="literal">http</code> server, we needed to create the HTTP headers and send them to the client before sending the body of the request. Express provides a convenience method on the <code class="literal">res</code> (<code class="literal">http.response</code>) object call named <code class="literal">send()</code>, and this method issues both the HTTP headers as well as a <code class="literal">response.end()</code> call. So far, we haven’t done much more than the original Hello World server from <a class="xref" href="ch01.html" title="Chapter 1. A Very Brief Introduction to Node.js">Chapter 1</a>. However, this server will respond only to a GET request to <code class="literal">/</code> without throwing an error. This is in contrast to the previous example, which would respond to any request with any path.</p><p>Let’s start adding some features to this server in order to provide some of the Twitter functionality (<a class="xref" href="ch02.html#example2-15" title="Example 2-15. Adding a basic API">Example 2-15</a>). At least to start with, we aren’t going to worry about making it super-robust or scalable. We <a id="I_indexterm2_d1e2079" class="indexterm"/><a id="I_indexterm2_d1e2084" class="indexterm"/>are going to make a few assumptions so you can see how to create applications.</p><div class="example"><a id="example2-15"/><p class="title">Example 2-15. Adding a basic API</p><div class="example-contents"><pre class="programlisting">var express = require('express') var app = express.createServer() app.listen(8000) var tweets = [] app.get('/', function(req, res) { res.send('Welcome to Node Twitter') }) app.post('/send', express.bodyParser(), function(req, res) { if (req.body && req.body.tweet) { tweets.push(req.body.tweet) res.send({status:"ok", message:"Tweet received"}) } else { //no tweet? res.send({status:"nok", message:"No tweet received"}) } }) app.get('/tweets', function(req,res) { res.send(tweets) })</pre></div></div><p>Building on the basic Express app, we’ve added a couple of functions to provide an extremely basic API. But first let’s talk about another change we made. We moved the <code class="literal">app.listen()</code> call to the top of the file. It’s important to understand why this doesn’t cause a race condition for the functions that respond to requests. You might imagine that when we call <code class="literal">app.listen()</code>, any requests that happen between the <code class="literal">app.listen()</code> call and the time it takes to run those functions will be ignored. This is incorrect for two reasons. The first is that in JavaScript everything happens in an event loop. That means new events don’t get called until we’ve finished evaluating the code of the existing loop pass. In this case, no <code class="literal">request</code> events will be called (and thus our <code class="literal">request</code>-based functions) until we’ve evaluated all the initialization code in the file. The other reason is that the <code class="literal">app.listen()</code> call is actually asynchronous because binding to a TCP port takes time. The addition of event listeners (via <code class="literal">app.get()</code> and <code class="literal">app.post()</code>), on the other hand, is synchronous.</p><p>To get some very basic tweeting action going, we’ve added a POST “route” for <code class="literal">/send</code> using the <code class="literal">app.post()</code> call. This call is a little bit different from the previous example. Obviously, it’s an <code class="literal">app.post()</code> rather than an <code class="literal">app.get()</code> request. This simply means it accepts HTTP POST requests instead of HTTP GET requests. The significant difference is that we’ve passed an extra argument to the function. You don’t need to do this on all <code class="literal">app.post()</code> calls, or any, in fact. The extra argument after the <code class="literal">url</code> is a <span class="emphasis"><em>middleware</em></span>.</p><p>A middleware is a <a id="I_indexterm2_d1e2151" class="indexterm"/>small piece of code that sits in between the original <code class="literal">request</code> event and the route we defined with <code class="literal">app.post()</code>. We use middleware to reuse code for common tasks such as authentication or logging. In this case the middleware’s job is to stream the POST data from the client and then turn it into a JavaScript object that we can use. This middleware is one that is included in Express itself, called <code class="literal">bodyParser</code>. We simply include it by specifying it in the arguments we give to the <code class="literal">app.post()</code> route. Notice that we call <code class="literal">express.bodyParser()</code>. <a id="I_indexterm2_d1e2173" class="indexterm"/><a id="I_indexterm2_d1e2178" class="indexterm"/>This function call actually returns another function. We use this standard style for middleware to allow you to pass configuration to the middleware if you want to.</p><p>If we didn’t include the middleware, we would have to manually write code to accept the <code class="literal">data</code> event provided by the <code class="literal">request</code> (<code class="literal">req</code>) object. Only after we had streamed in all the POST data could we call the code in the <code class="literal">app.post()</code> route. Using the middleware not only helps with code reuse but also with clarity.</p><p>The <code class="literal">express.bodyParser</code> adds a property to <code class="literal">req</code> called <code class="literal">req.body</code>. This property (if it exists) contains an object representing the POST data. The <code class="literal">express.bodyParser</code> middleware will work only for POST requests with the <code class="literal">content-type</code> HTTP header of <code class="literal">application/x-www-form-urlencoded</code> or <code class="literal">application/json</code>. Both of these are easy to parse into key/value pairs as properties of the <code class="literal">req.body</code> object.</p><p>This means that in the <code class="literal">app.post()</code> route we made, the first thing we do is check whether <code class="literal">express.bodyParser</code> found any data. We can simply check to see whether <code class="literal">req.body</code> was created. If it was, we look for a property called <code class="literal">req.body.tweet</code> to represent the tweet. If we find a tweet, we stash it in a global array called <code class="literal">tweets</code> and send a JSON string back to the client noting success. If we couldn’t find <code class="literal">req.body</code> or <code class="literal">req.body.tweet</code>, we send JSON back to the client, noting the failure. Notice how we didn’t serialize the data in the <code class="literal">res.send()</code> calls. If we give <code class="literal">res.send()</code> an object, it automatically serializes it as JSON and sends the correct HTTP headers.</p><p>Finally, to make our basic API complete, we create an <code class="literal">app.get()</code> route that listens to <code class="literal">/tweets</code>. This route simply sends back JSON for the <code class="literal">tweets</code> array.</p><p>We can write a few tests for our simple API to make sure it’s working (<a class="xref" href="ch02.html#example2-16" title="Example 2-16. A test for the POST API">Example 2-16</a>). This<a id="I_indexterm2_d1e2268" class="indexterm"/><a id="I_indexterm2_d1e2273" class="indexterm"/><a id="I_indexterm2_d1e2278" class="indexterm"/> is a good habit to get into, even if you don’t do full test-driven development (TDD).</p><div class="example"><a id="example2-16"/><p class="title">Example 2-16. A test for the POST API</p><div class="example-contents"><pre class="programlisting">var http = require('http'), assert = require('assert') var opts = { host: 'localhost', port: 8000, path: '/send', method: 'POST', headers: {'content-type':'application/x-www-form-urlencoded'} } var req = http.request(opts, function(res) { res.setEncoding('utf8') var data = "" res.on('data', function(d) { data += d }) res.on('end', function() { assert.strictEqual(data, '{"status":"ok","message":"Tweet received"}') }) }) req.write('tweet=test') req.end()</pre></div></div><p>We need the <code class="literal">http</code> and <code class="literal">assert</code><sup>[<a id="id598177" href="#ftn.id598177" class="footnote">3</a>]</sup> modules in order to <a id="I_indexterm2_d1e2305" class="indexterm"/>send HTTP requests and then test the values returned. <code class="literal">assert</code> is a core module in Node that lets us test return values in various ways. When a value doesn’t match the expected criteria, an exception is thrown. By making test scripts that check an expected behavior of our program, we can ensure that it is doing what it should be.</p><p>The <code class="literal">http</code> library <a id="I_indexterm2_d1e2319" class="indexterm"/>doesn’t just contain objects to serve HTTP; it also provides a client. In this test program, we use the <code class="literal">http.request()</code> factory method to create a new <code class="literal">http.Request</code> object. To create an <code class="literal">http.Request</code>, we need an <span class="emphasis"><em>options object</em></span>. This is a configuration object we pass that has a list of properties defining the functionality we want the <code class="literal">http.Request</code> to exhibit. You’ll see config objects used for constructing other Node objects. In this case, we include the <code class="literal">hostname</code> (which will be resolved by <code class="literal">dns</code>), the port, URL path, HTTP method, and some HTTP headers. Here the settings of the config object reflect what we used when creating our Express server.</p><p>The <code class="literal">http.request()</code> <a id="I_indexterm2_d1e2352" class="indexterm"/>constructor takes two arguments: the first is the config object, and the second is a callback. The callback is attached to the <code class="literal">response</code> event for the <code class="literal">http.Request</code>. It’s similar to an <code class="literal">http.Server</code>, except we have only one object in the response.</p><p>The first thing we do with the response is call<a id="I_indexterm2_d1e2370" class="indexterm"/><a id="I_indexterm2_d1e2375" class="indexterm"/><a id="I_indexterm2_d1e2378" class="indexterm"/><a id="I_indexterm2_d1e2383" class="indexterm"/> <code class="literal">setEncoding()</code>. This allows us to define the encoding of all the received data. By setting this to <code class="literal">utf8</code>, we ensure that any data we receive will be treated as the right kind of string. Next, we define a variable, <code class="literal">data</code>, which we are going to use to stream all the responses from the server. In Express, we can use <code class="literal">express.bodyDecoder</code> <a id="I_indexterm2_d1e2405" class="indexterm"/>to catch all the data in a request and stream it, but we don’t have the same luxury in the client, so we’ll do it by hand. It’s really easy. We simply attach a function to the <code class="literal">data</code> event on <code class="literal">response</code>. Whenever <code class="literal">data</code> happens, we append it to our <code class="literal">data</code> variable. We can listen for the <code class="literal">end</code> event of the <code class="literal">response</code> and then take further action on all of the data. The API is set up this way because there are many applications in which it is possible to stream data. In these cases, we can do all of the work in the <code class="literal">data</code> event listener rather than aggregating first.</p><p>When we get the <code class="literal">end</code> event on <code class="literal">response</code>, it’s because we have all the data from the server. Now we can run our test on whatever the server sent. Our test in this case is to check whether the <code class="literal">data</code> variable has received what we expected from the server. If the server is acting correctly, it should send us back a piece of JSON. By using<a id="I_indexterm2_d1e2446" class="indexterm"/> <code class="literal">assert.strict</code><code class="literal">Equal</code>, we are checking that data matches the expected data using <code class="literal">===</code>. If it doesn’t, an <code class="literal">assert</code> exception is thrown. We are using the <code class="literal">x-www-form-urlencoded</code> format because that’s what a web page form would send.</p><p>Now that we’ve set up the <code class="literal">request</code> and the event handlers for it, we need to write some data to the server. Calling <code class="literal">write()</code> on <code class="literal">request</code> lets us send data (since this is a POST request). We send some test data to ensure that the server will respond correctly. Finally, we call <code class="literal">end()</code> to indicate that we are finished sending data with the <code class="literal">request</code> object.</p><p>When we call this script, it will access the server we set up (if it is running) and send a POST request. If it gets back the correct data, it will finish without output. If it can’t connect to the server or if the server responds with the wrong output, it will throw an exception. The goal is to have a set of scripts we can run to check that the server is behaving correctly as we build it.</p><p>Now that we have an API, we can start adding a web interface so that people can use our app. Right now, it’s basic, but the API allows people to send messages that everyone can receive. Let’s make an interface to that.</p><p>Express supports an MVC (model, view, controller) approach oriented around the routing of requests. The routes act like controllers, providing a way to join the data model with a view. We’ve already used a route (<code class="literal">app.get('/', function)</code>). In the folder structure shown in <a class="xref" href="ch02.html#example2-17" title="Example 2-17. The basic folder structure of an Express app">Example 2-17</a>, we can see where we host the different parts of the views. By convention, the <em class="filename">views</em> folder holds the view templates, and within it a <em class="filename">partials</em> folder contains the “partial views” (we’ll discuss these more later). For applications that don’t use a content delivery network (CDN), the <em class="filename">public</em> folder is used to store static files, such as <a id="I_indexterm2_d1e2505" class="indexterm"/><a id="I_indexterm2_d1e2510" class="indexterm"/>CSS and JavaScript.</p><div class="example"><a id="example2-17"/><p class="title">Example 2-17. The basic folder structure of an Express app</p><div class="example-contents"><pre class="programlisting">. ├── app.js ├── public └── views └── partials</pre></div></div><p>To start connecting our very simple model (<code class="literal">var tweets = []</code>) with a view, we need to create some views first. We are going to create some basic view files and put them in the <em class="filename">views</em> folder. Express offers a few different templating languages and is extensible to allow the addition of more. We are going to start with EJS.<sup>[<a id="id824494" href="#ftn.id824494" class="footnote">4</a>]</sup> EJS simply embeds JavaScript into the templates with a few simple tags to define how the JavaScript is interpreted. Let’s take a look at an example of EJS, starting with the <a id="I_indexterm2_d1e2535" class="indexterm"/><a id="I_indexterm2_d1e2538" class="indexterm"/>layout file in <a class="xref" href="ch02.html#example2-18" title="Example 2-18. EJS layout file">Example 2-18</a>.</p><div class="example"><a id="example2-18"/><p class="title">Example 2-18. EJS layout file</p><div class="example-contents"><pre class="programlisting"><!DOCTYPE html> <html lang="en"> <head> <meta charset="utf-8"> <%- partial('partials/stylesheet', stylesheets) %> <title><%= title %></title> </head> <body> <h1><%= header %></h1> <%- body %> </body> </html></pre></div></div><p>The layout file in Express <a id="I_indexterm2_d1e2553" class="indexterm"/>defines a skeleton to use for your site. It’s some basic view boilerplate you will use almost everywhere. In this case, we’ve used a very simple HTML5 page. It has a head with some stylesheet definitions and a body. The body consists of an <code class="literal">h1</code> header element and some content. Notice the <code class="literal"><%</code> tags. These are the places in which we are going to insert JavaScript variables. The JavaScript to be evaluated is between the <code class="literal"><%</code> and <code class="literal">%></code> tags. The tags can also start with <code class="literal">=</code> or <code class="literal">-</code>, which we will discuss in more detail shortly. Mostly you’ll just reference a piece of data. You can simply list the variable or reference you wish to include in the page. For example, <code class="literal"><h1></code><code class="literal"><%= header %></h1></code> includes the variable <code class="literal">header</code> into the <code class="literal">h1</code> element.</p><p>There are two special things used in this template. The first is the call to <code class="literal">partial()</code>. Partials are mini-templates for code that is expected to repeat again and again with different data. For example, you can imagine the comments on a blog post are the same snippet of HTML repeating many times, but with different pieces of information for each commenter and the comment she made. The actual HTML template doesn’t change. Partials are a way to represent and store those small pieces of code that repeat often, independently of the pages that include them, to make it easy to update the code on all the pages at once. The other special thing in this layout template is the <code class="literal">body</code> variable. Because we use the layout template on all the pages on the site (unless we turn it off), we need some way to say where the specific template being rendered goes. Express provides the <code class="literal">body</code> variable for this task. This variable will contain the rendered contents of the specific template we load.</p><p>Let’s make a render call from a route to see what that looks <a id="I_indexterm2_d1e2606" class="indexterm"/>like before we explore the other templates we’ll need (<a class="xref" href="ch02.html#example2-19" title="Example 2-19. Rendering the index template from the '/' route">Example 2-19</a>).</p><div class="example"><a id="example2-19"/><p class="title">Example 2-19. Rendering the index template from the '/' route</p><div class="example-contents"><pre class="programlisting">app.get('/', function(req, res) { var title = 'Chirpie', header = 'Welcome to Chirpie' res.render('index', { locals: { 'title': title, 'header': header, 'tweets': tweets, stylesheets: ['/public/style.css'] } }) })</pre></div></div><p>The route code looks like the other route code we’ve used. However, instead of calling <code class="literal">res