coffeescript

<!DOCTYPE html> <html> <head> <title>lexer.coffee</title> <meta http-equiv="content-type" content="text/html; charset=UTF-8"> <link rel="stylesheet" media="all" href="docco.css" /> </head> <body> <div id="container"> <div id="background"></div> <div id="jump_to"> Jump To … <div id="jump_wrapper"> <div id="jump_page"> <a class="source" href="browser.html"> browser.coffee </a> <a class="source" href="cake.html"> cake.coffee </a> <a class="source" href="coffee-script.html"> coffee-script.coffee </a> <a class="source" href="command.html"> command.coffee </a> <a class="source" href="grammar.html"> grammar.coffee </a> <a class="source" href="helpers.html"> helpers.coffee </a> <a class="source" href="index.html"> index.coffee </a> <a class="source" href="lexer.html"> lexer.coffee </a> <a class="source" href="nodes.html"> nodes.coffee </a> <a class="source" href="optparse.html"> optparse.coffee </a> <a class="source" href="repl.html"> repl.coffee </a> <a class="source" href="rewriter.html"> rewriter.coffee </a> <a class="source" href="scope.html"> scope.coffee </a> </div> </div> </div> <table cellpadding="0" cellspacing="0"> <thead> <tr> <th class="docs"> <h1> lexer.coffee </h1> </th> <th class="code"> </th> </tr> </thead> <tbody> <tr id="section-1"> <td class="docs"> <div class="pilwrap"> <a class="pilcrow" href="#section-1">¶</a> </div> The CoffeeScript Lexer. Uses a series of token-matching regexes to attempt matches against the beginning of the source code. When a match is found, a token is produced, we consume the match, and start again. Tokens are in the form: <pre><code>[tag, value, lineNumber] </code></pre> Which is a format that can be fed directly into <a href="http://github.com/zaach/jison">Jison</a>. </td> <td class="code"> <div class="highlight"><pre>{Rewriter} = require './rewriter'</pre></div> </td> </tr> <tr id="section-2"> <td class="docs"> <div class="pilwrap"> <a class="pilcrow" href="#section-2">¶</a> </div> Import the helpers we need. </td> <td class="code"> <div class="highlight"><pre>{count, starts, compact, last} = require './helpers'</pre></div> </td> </tr> <tr id="section-3"> <td class="docs"> <div class="pilwrap"> <a class="pilcrow" href="#section-3">¶</a> </div> <h2>The Lexer Class</h2> </td> <td class="code"> <div class="highlight"><pre></pre></div> </td> </tr> <tr id="section-4"> <td class="docs"> <div class="pilwrap"> <a class="pilcrow" href="#section-4">¶</a> </div> The Lexer class reads a stream of CoffeeScript and divvies it up into tagged tokens. Some potential ambiguity in the grammar has been avoided by pushing some extra smarts into the Lexer. </td> <td class="code"> <div class="highlight"><pre>exports.Lexer = class Lexer</pre></div> </td> </tr> <tr id="section-5"> <td class="docs"> <div class="pilwrap"> <a class="pilcrow" href="#section-5">¶</a> </div> tokenize is the Lexer's main method. Scan by attempting to match tokens one at a time, using a regular expression anchored at the start of the remaining code, or a custom recursive token-matching method (for interpolations). When the next token has been recorded, we move forward within the code past the token, and begin again. Each tokenizing method is responsible for returning the number of characters it has consumed. Before returning the token stream, run it through the <a href="rewriter.html">Rewriter</a> unless explicitly asked not to. </td> <td class="code"> <div class="highlight"><pre> tokenize: (code, opts = {}) -> code = code.replace(/\r/g, '').replace TRAILING_SPACES, '' @code = code # The remainder of the source code. @line = opts.line or 0 # The current line. @indent = 0 # The current indentation level. @indebt = 0 # The over-indentation at the current level. @outdebt = 0 # The under-outdentation at the current level. @indents = [] # The stack of all current indentation levels. @tokens = [] # Stream of parsed tokens in the form `['TYPE', value, line]`.</pre></div> </td> </tr> <tr id="section-6"> <td class="docs"> <div class="pilwrap"> <a class="pilcrow" href="#section-6">¶</a> </div> At every position, run through this list of attempted matches, short-circuiting if any of them succeed. Their order determines precedence: <code>@literalToken</code> is the fallback catch-all. </td> <td class="code"> <div class="highlight"><pre> i = 0 while @chunk = code.slice i i += @identifierToken() or @commentToken() or @whitespaceToken() or @lineToken() or @heredocToken() or @stringToken() or @numberToken() or @regexToken() or @jsToken() or @literalToken() @closeIndentation() return @tokens if opts.rewrite is off (new Rewriter).rewrite @tokens</pre></div> </td> </tr> <tr id="section-7"> <td class="docs"> <div class="pilwrap"> <a class="pilcrow" href="#section-7">¶</a> </div> <h2>Tokenizers</h2> </td> <td class="code"> <div class="highlight"><pre></pre></div> </td> </tr> <tr id="section-8"> <td class="docs"> <div class="pilwrap"> <a class="pilcrow" href="#section-8">¶</a> </div> Matches identifying literals: variables, keywords, method names, etc. Check to ensure that JavaScript reserved words aren't being used as identifiers. Because CoffeeScript reserves a handful of keywords that are allowed in JavaScript, we're careful not to tag them as keywords when referenced as property names here, so you can still do <code>jQuery.is()</code> even though <code>is</code> means <code>===</code> otherwise. </td> <td class="code"> <div class="highlight"><pre> identifierToken: -> return 0 unless match = IDENTIFIER.exec @chunk [input, id, colon] = match if id is 'all' and @tag() is 'FOR' @token 'ALL', id return id.length forcedIdentifier = colon or (prev = last @tokens) and not prev.spaced and prev[0] in ['.', '?.', '@', '::'] tag = 'IDENTIFIER' if id in JS_KEYWORDS or not forcedIdentifier and id in COFFEE_KEYWORDS tag = id.toUpperCase() if tag is 'WHEN' and @tag() in LINE_BREAK tag = 'LEADING_WHEN' else if tag is 'FOR' @seenFor = yes else if tag in UNARY tag = 'UNARY' else if tag in RELATION if tag isnt 'INSTANCEOF' and @seenFor @seenFor = no tag = 'FOR' + tag else tag = 'RELATION' if @value() is '!' @tokens.pop() id = '!' + id if id in JS_FORBIDDEN if forcedIdentifier tag = 'IDENTIFIER' id = new String id id.reserved = yes else if id in RESERVED @identifierError id unless forcedIdentifier id = COFFEE_ALIASES[id] if COFFEE_ALIASES.hasOwnProperty id tag = switch id when '!' then 'UNARY' when '==', '!=' then 'COMPARE' when '&&', '||' then 'LOGIC' when 'true', 'false', 'null', 'undefined' then 'BOOL' when 'break', 'continue', 'debugger' then 'STATEMENT' else tag @token tag, id @token ':', ':' if colon input.length</pre></div> </td> </tr> <tr id="section-9"> <td class="docs"> <div class="pilwrap"> <a class="pilcrow" href="#section-9">¶</a> </div> Matches numbers, including decimals, hex, and exponential notation. Be careful not to interfere with ranges-in-progress. </td> <td class="code"> <div class="highlight"><pre> numberToken: -> return 0 unless match = NUMBER.exec @chunk number = match[0] @token 'NUMBER', number number.length</pre></div> </td> </tr> <tr id="section-10"> <td class="docs"> <div class="pilwrap"> <a class="pilcrow" href="#section-10">¶</a> </div> Matches strings, including multi-line strings. Ensures that quotation marks are balanced within the string's contents, and within nested interpolations. </td> <td class="code"> <div class="highlight"><pre> stringToken: -> switch @chunk.charAt 0 when "'" return 0 unless match = SIMPLESTR.exec @chunk @token 'STRING', (string = match[0]).replace MULTILINER, '\\\n' when '"' return 0 unless string = @balancedString @chunk, [['"', '"'], ['#{', '}']] if 0 < string.indexOf '#{', 1 @interpolateString string.slice 1, -1 else @token 'STRING', @escapeLines string else return 0 @line += count string, '\n' string.length</pre></div> </td> </tr> <tr id="section-11"> <td class="docs"> <div class="pilwrap"> <a class="pilcrow" href="#section-11">¶</a> </div> Matches heredocs, adjusting indentation to the correct level, as heredocs preserve whitespace, but ignore indentation to the left. </td> <td class="code"> <div class="highlight"><pre> heredocToken: -> return 0 unless match = HEREDOC.exec @chunk heredoc = match[0] quote = heredoc.charAt 0 doc = @sanitizeHeredoc match[2], quote: quote, indent: null if quote is '"' and 0 <= doc.indexOf '#{' @interpolateString doc, heredoc: yes else @token 'STRING', @makeString doc, quote, yes @line += count heredoc, '\n' heredoc.length</pre></div> </td> </tr> <tr id="section-12"> <td class="docs"> <div class="pilwrap"> <a class="pilcrow" href="#section-12">¶</a> </div> Matches and consumes comments. </td> <td class="code"> <div class="highlight"><pre> commentToken: -> return 0 unless match = @chunk.match COMMENT [comment, here] = match @line += count comment, '\n' if here @token 'HERECOMMENT', @sanitizeHeredoc here, herecomment: true, indent: Array(@indent + 1).join(' ') @token 'TERMINATOR', '\n' comment.length</pre></div> </td> </tr> <tr id="section-13"> <td class="docs"> <div class="pilwrap"> <a class="pilcrow" href="#section-13">¶</a> </div> Matches JavaScript interpolated directly into the source via backticks. </td> <td class="code"> <div class="highlight"><pre> jsToken: -> return 0 unless @chunk.charAt(0) is '`' and match = JSTOKEN.exec @chunk @token 'JS', (script = match[0]).slice 1, -1 script.length</pre></div> </td> </tr> <tr id="section-14"> <td class="docs"> <div class="pilwrap"> <a class="pilcrow" href="#section-14">¶</a> </div> Matches regular expression literals. Lexing regular expressions is difficult to distinguish from division, so we borrow some basic heuristics from JavaScript and Ruby. </td> <td class="code"> <div class="highlight"><pre> regexToken: -> return 0 if @chunk.charAt(0) isnt '/' return @heregexToken match if match = HEREGEX.exec @chunk prev = last @tokens return 0 if prev and (prev[0] in (if prev.spaced then NOT_REGEX else NOT_SPACED_REGEX)) return 0 unless match = REGEX.exec @chunk [regex] = match @token 'REGEX', if regex is '//' then '/(?:)/' else regex regex.length</pre></div> </td> </tr> <tr id="section-15"> <td class="docs"> <div class="pilwrap"> <a class="pilcrow" href="#section-15">¶</a> </div> Matches multiline extended regular expressions. </td> <td class="code"> <div class="highlight"><pre> heregexToken: (match) -> [heregex, body, flags] = match if 0 > body.indexOf '#{' re = body.replace(HEREGEX_OMIT, '').replace(/\//g, '\\/') @token 'REGEX', "/#{ re or '(?:)' }/#{flags}" return heregex.length @token 'IDENTIFIER', 'RegExp' @tokens.push ['CALL_START', '('] tokens = [] for [tag, value] in @interpolateString(body, regex: yes) if tag is 'TOKENS' tokens.push value... else continue unless value = value.replace HEREGEX_OMIT, '' value = value.replace /\\/g, '\\\\' tokens.push ['STRING', @makeString(value, '"', yes)] tokens.push ['+', '+'] tokens.pop() @tokens.push ['STRING', '""'], ['+', '+'] unless tokens[0]?[0] is 'STRING' @tokens.push tokens... @tokens.push [',', ','], ['STRING', '"' + flags + '"'] if flags @token ')', ')' heregex.length</pre></div> </td> </tr> <tr id="section-16"> <td class="docs"> <div class="pilwrap"> <a class="pilcrow" href="#section-16">¶</a> </div> Matches newlines, indents, and outdents, and determines which is which. If we can detect that the current line is continued onto the the next line, then the newline is suppressed: <pre><code>elements .each( ... ) .map( ... ) </code></pre> Keeps track of the level of indentation, because a single outdent token can close multiple indents, so we need to know how far in we happen to be. </td> <td class="code"> <div class="highlight"><pre> lineToken: -> return 0 unless match = MULTI_DENT.exec @chunk indent = match[0] @line += count indent, '\n' prev = last @tokens, 1 size = indent.length - 1 - indent.lastIndexOf '\n' noNewlines = @unfinished() if size - @indebt is @indent if noNewlines then @suppressNewlines() else @newlineToken() return indent.length if size > @indent if noNewlines @indebt = size - @indent @suppressNewlines() return indent.length diff = size - @indent + @outdebt @token 'INDENT', diff @indents.push diff @outdebt = @indebt = 0 else @indebt = 0 @outdentToken @indent - size, noNewlines @indent = size indent.length</pre></div> </td> </tr> <tr id="section-17"> <td class="docs"> <div class="pilwrap"> <a class="pilcrow" href="#section-17">¶</a> </div> Record an outdent token or multiple tokens, if we happen to be moving back inwards past several recorded indents. </td> <td class="code"> <div class="highlight"><pre> outdentToken: (moveOut, noNewlines, close) -> while moveOut > 0 len = @indents.length - 1 if @indents[len] is undefined moveOut = 0 else if @indents[len] is @outdebt moveOut -= @outdebt @outdebt = 0 else if @indents[len] < @outdebt @outdebt -= @indents[len] moveOut -= @indents[len] else dent = @indents.pop() - @outdebt moveOut -= dent @outdebt = 0 @token 'OUTDENT', dent @outdebt -= moveOut if dent @token 'TERMINATOR', '\n' unless @tag() is 'TERMINATOR' or noNewlines this</pre></div> </td> </tr> <tr id="section-18"> <td class="docs"> <div class="pilwrap"> <a class="pilcrow" href="#section-18">¶</a> </div> Matches and consumes non-meaningful whitespace. Tag the previous token as being "spaced", because there are some cases where it makes a difference. </td> <td class="code"> <div class="highlight"><pre> whitespaceToken: -> return 0 unless (match = WHITESPACE.exec @chunk) or (nline = @chunk.charAt(0) is '\n') prev = last @tokens prev[if match then 'spaced' else 'newLine'] = true if prev if match then match[0].length else 0</pre></div> </td> </tr> <tr id="section-19"> <td class="docs"> <div class="pilwrap"> <a class="pilcrow" href="#section-19">¶</a> </div> Generate a newline token. Consecutive newlines get merged together. </td> <td class="code"> <div class="highlight"><pre> newlineToken: -> @token 'TERMINATOR', '\n' unless @tag() is 'TERMINATOR' this</pre></div> </td> </tr> <tr id="section-20"> <td class="docs"> <div class="pilwrap"> <a class="pilcrow" href="#section-20">¶</a> </div> Use a <code>\</code> at a line-ending to suppress the newline. The slash is removed here once its job is done. </td> <td class="code"> <div class="highlight"><pre> suppressNewlines: -> @tokens.pop() if @value() is '\\' this</pre></div> </td> </tr> <tr id="section-21"> <td class="docs"> <div class="pilwrap"> <a class="pilcrow" href="#section-21">¶</a> </div> We treat all other single characters as a token. E.g.: <code>( ) , . !</code> Multi-character operators are also literal tokens, so that Jison can assign the proper order of operations. There are some symbols that we tag specially here. <code>;</code> and newlines are both treated as a <code>TERMINATOR</code>, we distinguish parentheses that indicate a method call from regular parentheses, and so on. </td> <td class="code"> <div class="highlight"><pre> literalToken: -> if match = OPERATOR.exec @chunk [value] = match @tagParameters() if CODE.test value else value = @chunk.charAt 0 tag = value prev = last<