coffeescript

<!DOCTYPE html> <html> <head> <title>rewriter.coffee</title> <meta http-equiv="content-type" content="text/html; charset=UTF-8"> <link rel="stylesheet" media="all" href="docco.css" /> </head> <body> <div id="container"> <div id="background"></div> <div id="jump_to"> Jump To … <div id="jump_wrapper"> <div id="jump_page"> <a class="source" href="browser.html"> browser.coffee </a> <a class="source" href="cake.html"> cake.coffee </a> <a class="source" href="coffee-script.html"> coffee-script.coffee </a> <a class="source" href="command.html"> command.coffee </a> <a class="source" href="grammar.html"> grammar.coffee </a> <a class="source" href="helpers.html"> helpers.coffee </a> <a class="source" href="index.html"> index.coffee </a> <a class="source" href="lexer.html"> lexer.coffee </a> <a class="source" href="nodes.html"> nodes.coffee </a> <a class="source" href="optparse.html"> optparse.coffee </a> <a class="source" href="repl.html"> repl.coffee </a> <a class="source" href="rewriter.html"> rewriter.coffee </a> <a class="source" href="scope.html"> scope.coffee </a> </div> </div> </div> <table cellpadding="0" cellspacing="0"> <thead> <tr> <th class="docs"> <h1> rewriter.coffee </h1> </th> <th class="code"> </th> </tr> </thead> <tbody> <tr id="section-1"> <td class="docs"> <div class="pilwrap"> <a class="pilcrow" href="#section-1">¶</a> </div> The CoffeeScript language has a good deal of optional syntax, implicit syntax, and shorthand syntax. This can greatly complicate a grammar and bloat the resulting parse table. Instead of making the parser handle it all, we take a series of passes over the token stream, using this Rewriter to convert shorthand into the unambiguous long form, add implicit indentation and parentheses, balance incorrect nestings, and generally clean things up. </td> <td class="code"> <div class="highlight"><pre></pre></div> </td> </tr> <tr id="section-2"> <td class="docs"> <div class="pilwrap"> <a class="pilcrow" href="#section-2">¶</a> </div> The Rewriter class is used by the <a href="lexer.html">Lexer</a>, directly against its internal array of tokens. </td> <td class="code"> <div class="highlight"><pre>class exports.Rewriter</pre></div> </td> </tr> <tr id="section-3"> <td class="docs"> <div class="pilwrap"> <a class="pilcrow" href="#section-3">¶</a> </div> Helpful snippet for debugging: console.log (t[0] + '/' + t[1] for t in @tokens).join ' ' </td> <td class="code"> <div class="highlight"><pre></pre></div> </td> </tr> <tr id="section-4"> <td class="docs"> <div class="pilwrap"> <a class="pilcrow" href="#section-4">¶</a> </div> Rewrite the token stream in multiple passes, one logical filter at a time. This could certainly be changed into a single pass through the stream, with a big ol' efficient switch, but it's much nicer to work with like this. The order of these passes matters -- indentation must be corrected before implicit parentheses can be wrapped around blocks of code. </td> <td class="code"> <div class="highlight"><pre> rewrite: (@tokens) -> @removeLeadingNewlines() @removeMidExpressionNewlines() @closeOpenCalls() @closeOpenIndexes() @addImplicitIndentation() @tagPostfixConditionals() @addImplicitBraces() @addImplicitParentheses() @ensureBalance BALANCED_PAIRS @rewriteClosingParens() @tokens</pre></div> </td> </tr> <tr id="section-5"> <td class="docs"> <div class="pilwrap"> <a class="pilcrow" href="#section-5">¶</a> </div> Rewrite the token stream, looking one token ahead and behind. Allow the return value of the block to tell us how many tokens to move forwards (or backwards) in the stream, to make sure we don't miss anything as tokens are inserted and removed, and the stream changes length under our feet. </td> <td class="code"> <div class="highlight"><pre> scanTokens: (block) -> {tokens} = this i = 0 i += block.call this, token, i, tokens while token = tokens[i] true detectEnd: (i, condition, action) -> {tokens} = this levels = 0 while token = tokens[i] return action.call this, token, i if levels is 0 and condition.call this, token, i return action.call this, token, i - 1 if not token or levels < 0 if token[0] in EXPRESSION_START levels += 1 else if token[0] in EXPRESSION_END levels -= 1 i += 1 i - 1</pre></div> </td> </tr> <tr id="section-6"> <td class="docs"> <div class="pilwrap"> <a class="pilcrow" href="#section-6">¶</a> </div> Leading newlines would introduce an ambiguity in the grammar, so we dispatch them here. </td> <td class="code"> <div class="highlight"><pre> removeLeadingNewlines: -> break for [tag], i in @tokens when tag isnt 'TERMINATOR' @tokens.splice 0, i if i</pre></div> </td> </tr> <tr id="section-7"> <td class="docs"> <div class="pilwrap"> <a class="pilcrow" href="#section-7">¶</a> </div> Some blocks occur in the middle of expressions -- when we're expecting this, remove their trailing newlines. </td> <td class="code"> <div class="highlight"><pre> removeMidExpressionNewlines: -> @scanTokens (token, i, tokens) -> return 1 unless token[0] is 'TERMINATOR' and @tag(i + 1) in EXPRESSION_CLOSE tokens.splice i, 1 0</pre></div> </td> </tr> <tr id="section-8"> <td class="docs"> <div class="pilwrap"> <a class="pilcrow" href="#section-8">¶</a> </div> The lexer has tagged the opening parenthesis of a method call. Match it with its paired close. We have the mis-nested outdent case included here for calls that close on the same line, just before their outdent. </td> <td class="code"> <div class="highlight"><pre> closeOpenCalls: -> condition = (token, i) -> token[0] in [')', 'CALL_END'] or token[0] is 'OUTDENT' and @tag(i - 1) is ')' action = (token, i) -> @tokens[if token[0] is 'OUTDENT' then i - 1 else i][0] = 'CALL_END' @scanTokens (token, i) -> @detectEnd i + 1, condition, action if token[0] is 'CALL_START' 1</pre></div> </td> </tr> <tr id="section-9"> <td class="docs"> <div class="pilwrap"> <a class="pilcrow" href="#section-9">¶</a> </div> The lexer has tagged the opening parenthesis of an indexing operation call. Match it with its paired close. </td> <td class="code"> <div class="highlight"><pre> closeOpenIndexes: -> condition = (token, i) -> token[0] in [']', 'INDEX_END'] action = (token, i) -> token[0] = 'INDEX_END' @scanTokens (token, i) -> @detectEnd i + 1, condition, action if token[0] is 'INDEX_START' 1</pre></div> </td> </tr> <tr id="section-10"> <td class="docs"> <div class="pilwrap"> <a class="pilcrow" href="#section-10">¶</a> </div> Object literals may be written with implicit braces, for simple cases. Insert the missing braces here, so that the parser doesn't have to. </td> <td class="code"> <div class="highlight"><pre> addImplicitBraces: -> stack = [] start = null startIndent = 0 condition = (token, i) -> {(i+1): one, (i+2): two, (i+3): three} = @tokens return false if 'HERECOMMENT' is one?[0] [tag] = token (tag in ['TERMINATOR', 'OUTDENT'] and not (two?[0] is ':' or one?[0] is '@' and three?[0] is ':')) or (tag is ',' and one and one[0] not in ['IDENTIFIER', 'NUMBER', 'STRING', '@', 'TERMINATOR', 'OUTDENT', '(']) action = (token, i) -> @tokens.splice i, 0, ['}', '}', token[2]] @scanTokens (token, i, tokens) -> if (tag = token[0]) in EXPRESSION_START stack.push [(if tag is 'INDENT' and @tag(i - 1) is '{' then '{' else tag), i] return 1 if tag in EXPRESSION_END start = stack.pop() return 1 return 1 unless tag is ':' and ((ago2 = @tag i - 2) is ':' or (ago1 = @tag i - 1) is ')' and @tag(start[1] - 1) is ':' or stack[stack.length - 1]?[0] isnt '{') stack.push ['{'] idx = if ago1 is ')' start[1] else if ago2 is '@' i - 2 else i - 1 idx -= 2 if @tag(idx - 2) is 'HERECOMMENT' value = new String('{') value.generated = yes tok = ['{', value, token[2]] tok.generated = yes tokens.splice idx, 0, tok @detectEnd i + 2, condition, action 2</pre></div> </td> </tr> <tr id="section-11"> <td class="docs"> <div class="pilwrap"> <a class="pilcrow" href="#section-11">¶</a> </div> Methods may be optionally called without parentheses, for simple cases. Insert the implicit parentheses here, so that the parser doesn't have to deal with them. </td> <td class="code"> <div class="highlight"><pre> addImplicitParentheses: -> noCall = no action = (token, i) -> idx = if token[0] is 'OUTDENT' then i + 1 else i @tokens.splice idx, 0, ['CALL_END', ')', token[2]] @scanTokens (token, i, tokens) -> tag = token[0] noCall = yes if tag in ['CLASS', 'IF', 'UNLESS'] {(i-1): prev, (i+1): next} = tokens callObject = not noCall and tag is 'INDENT' and next and next.generated and next[0] is '{' and prev and prev[0] in IMPLICIT_FUNC seenSingle = no noCall = no if tag in LINEBREAKS token.call = yes if prev and not prev.spaced and tag is '?' return 1 unless callObject or prev?.spaced and (prev.call or prev[0] in IMPLICIT_FUNC) and (tag in IMPLICIT_CALL or not (token.spaced or token.newLine) and tag in IMPLICIT_UNSPACED_CALL) tokens.splice i, 0, ['CALL_START', '(', token[2]] @detectEnd i + (if callObject then 2 else 1), (token, i) -> return yes if not seenSingle and token.fromThen [tag] = token seenSingle = yes if tag in ['IF', 'ELSE', 'UNLESS', '->', '=>'] return yes if tag in ['.', '?.', '::'] and @tag(i - 1) is 'OUTDENT' not token.generated and @tag(i - 1) isnt ',' and tag in IMPLICIT_END and (tag isnt 'INDENT' or (@tag(i - 2) isnt 'CLASS' and @tag(i - 1) not in IMPLICIT_BLOCK and not ((post = @tokens[i + 1]) and post.generated and post[0] is '{'))) , action prev[0] = 'FUNC_EXIST' if prev[0] is '?' 2</pre></div> </td> </tr> <tr id="section-12"> <td class="docs"> <div class="pilwrap"> <a class="pilcrow" href="#section-12">¶</a> </div> Because our grammar is LALR(1), it can't handle some single-line expressions that lack ending delimiters. The Rewriter adds the implicit blocks, so it doesn't need to. ')' can close a single-line block, but we need to make sure it's balanced. </td> <td class="code"> <div class="highlight"><pre> addImplicitIndentation: -> @scanTokens (token, i, tokens) -> [tag] = token if tag is 'TERMINATOR' and @tag(i + 1) is 'THEN' tokens.splice i, 1 return 0 if tag is 'ELSE' and @tag(i - 1) isnt 'OUTDENT' tokens.splice i, 0, @indentation(token)... return 2 if tag is 'CATCH' and @tag(i + 2) in ['OUTDENT', 'TERMINATOR', 'FINALLY'] tokens.splice i + 2, 0, @indentation(token)... return 4 if tag in SINGLE_LINERS and @tag(i + 1) isnt 'INDENT' and not (tag is 'ELSE' and @tag(i + 1) is 'IF') starter = tag [indent, outdent] = @indentation token indent.fromThen = true if starter is 'THEN' indent.generated = outdent.generated = true tokens.splice i + 1, 0, indent condition = (token, i) -> token[1] isnt ';' and token[0] in SINGLE_CLOSERS and not (token[0] is 'ELSE' and starter not in ['IF', 'THEN']) action = (token, i) -> @tokens.splice (if @tag(i - 1) is ',' then i - 1 else i), 0, outdent @detectEnd i + 2, condition, action tokens.splice i, 1 if tag is 'THEN' return 1 return 1</pre></div> </td> </tr> <tr id="section-13"> <td class="docs"> <div class="pilwrap"> <a class="pilcrow" href="#section-13">¶</a> </div> Tag postfix conditionals as such, so that we can parse them with a different precedence. </td> <td class="code"> <div class="highlight"><pre> tagPostfixConditionals: -> condition = (token, i) -> token[0] in ['TERMINATOR', 'INDENT'] @scanTokens (token, i) -> return 1 unless token[0] in ['IF', 'UNLESS'] original = token @detectEnd i + 1, condition, (token, i) -> original[0] = 'POST_' + original[0] if token[0] isnt 'INDENT' 1</pre></div> </td> </tr> <tr id="section-14"> <td class="docs"> <div class="pilwrap"> <a class="pilcrow" href="#section-14">¶</a> </div> Ensure that all listed pairs of tokens are correctly balanced throughout the course of the token stream. </td> <td class="code"> <div class="highlight"><pre> ensureBalance: (pairs) -> levels = {} openLine = {} for token in @tokens [tag] = token for [open, close] in pairs levels[open] |= 0 if tag is open openLine[open] = token[2] if <span class="nx