jade

<!DOCTYPE html> <html> <head> <title>lexer.coffee</title> <meta http-equiv="content-type" content="text/html; charset=UTF-8"> <link rel="stylesheet" media="all" href="docco.css" /> </head> <body> <div id="container"> <div id="background"></div> <div id="jump_to"> Jump To … <div id="jump_wrapper"> <div id="jump_page"> <a class="source" href="browser.html"> browser.coffee </a> <a class="source" href="cake.html"> cake.coffee </a> <a class="source" href="coffee-script.html"> coffee-script.coffee </a> <a class="source" href="command.html"> command.coffee </a> <a class="source" href="grammar.html"> grammar.coffee </a> <a class="source" href="helpers.html"> helpers.coffee </a> <a class="source" href="index.html"> index.coffee </a> <a class="source" href="lexer.html"> lexer.coffee </a> <a class="source" href="nodes.html"> nodes.coffee </a> <a class="source" href="optparse.html"> optparse.coffee </a> <a class="source" href="repl.html"> repl.coffee </a> <a class="source" href="rewriter.html"> rewriter.coffee </a> <a class="source" href="scope.html"> scope.coffee </a> </div> </div> </div> <table cellpadding="0" cellspacing="0"> <thead> <tr> <th class="docs"> <h1> lexer.coffee </h1> </th> <th class="code"> </th> </tr> </thead> <tbody> <tr id="section-1"> <td class="docs"> <div class="octowrap"> <a class="octothorpe" href="#section-1">#</a> </div> The CoffeeScript Lexer. Uses a series of token-matching regexes to attempt matches against the beginning of the source code. When a match is found, a token is produced, we consume the match, and start again. Tokens are in the form: <pre><code>[tag, value, lineNumber] </code></pre> Which is a format that can be fed directly into <a href="http://github.com/zaach/jison">Jison</a>. </td> <td class="code"> <div class="highlight"><pre>{Rewriter} = require './rewriter'</pre></div> </td> </tr> <tr id="section-2"> <td class="docs"> <div class="octowrap"> <a class="octothorpe" href="#section-2">#</a> </div> Import the helpers we need. </td> <td class="code"> <div class="highlight"><pre>{include, count, starts, compact} = require('./helpers').helpers</pre></div> </td> </tr> <tr id="section-3"> <td class="docs"> <div class="octowrap"> <a class="octothorpe" href="#section-3">#</a> </div> <h2>The Lexer Class</h2> </td> <td class="code"> <div class="highlight"><pre></pre></div> </td> </tr> <tr id="section-4"> <td class="docs"> <div class="octowrap"> <a class="octothorpe" href="#section-4">#</a> </div> The Lexer class reads a stream of CoffeeScript and divvys it up into tagged tokens. Some potential ambiguity in the grammar has been avoided by pushing some extra smarts into the Lexer. </td> <td class="code"> <div class="highlight"><pre>exports.Lexer = class Lexer</pre></div> </td> </tr> <tr id="section-5"> <td class="docs"> <div class="octowrap"> <a class="octothorpe" href="#section-5">#</a> </div> tokenize is the Lexer's main method. Scan by attempting to match tokens one at a time, using a regular expression anchored at the start of the remaining code, or a custom recursive token-matching method (for interpolations). When the next token has been recorded, we move forward within the code past the token, and begin again. Each tokenizing method is responsible for incrementing <code>@i</code> by the number of characters it has consumed. <code>@i</code> can be thought of as our finger on the page of source. Before returning the token stream, run it through the <a href="rewriter.html">Rewriter</a> unless explicitly asked not to. </td> <td class="code"> <div class="highlight"><pre> tokenize: (code, options) -> code = code.replace /(\r|\s+$)/g, '' o = options or {} @code = code # The remainder of the source code. @i = 0 # Current character position we're parsing. @line = o.line or 0 # The current line. @indent = 0 # The current indentation level. @indebt = 0 # The over-indentation at the current level. @outdebt = 0 # The under-outdentation at the current level. @indents = [] # The stack of all current indentation levels. @tokens = [] # Stream of parsed tokens in the form ['TYPE', value, line] while @i < @code.length @chunk = @code[@i..] @extractNextToken() @closeIndentation() return @tokens if o.rewrite is off (new Rewriter).rewrite @tokens</pre></div> </td> </tr> <tr id="section-6"> <td class="docs"> <div class="octowrap"> <a class="octothorpe" href="#section-6">#</a> </div> At every position, run through this list of attempted matches, short-circuiting if any of them succeed. Their order determines precedence: <code>@literalToken</code> is the fallback catch-all. </td> <td class="code"> <div class="highlight"><pre> extractNextToken: -> return if @identifierToken() return if @commentToken() return if @whitespaceToken() return if @lineToken() return if @heredocToken() return if @stringToken() return if @numberToken() return if @regexToken() return if @jsToken() return @literalToken()</pre></div> </td> </tr> <tr id="section-7"> <td class="docs"> <div class="octowrap"> <a class="octothorpe" href="#section-7">#</a> </div> <h2>Tokenizers</h2> </td> <td class="code"> <div class="highlight"><pre></pre></div> </td> </tr> <tr id="section-8"> <td class="docs"> <div class="octowrap"> <a class="octothorpe" href="#section-8">#</a> </div> Matches identifying literals: variables, keywords, method names, etc. Check to ensure that JavaScript reserved words aren't being used as identifiers. Because CoffeeScript reserves a handful of keywords that are allowed in JavaScript, we're careful not to tag them as keywords when referenced as property names here, so you can still do <code>jQuery.is()</code> even though <code>is</code> means <code>===</code> otherwise. </td> <td class="code"> <div class="highlight"><pre> identifierToken: -> return false unless id = @match IDENTIFIER, 1 @i += id.length forcedIdentifier = @tagAccessor() or @match ASSIGNED, 1 tag = 'IDENTIFIER' tag = id.toUpperCase() if include(JS_KEYWORDS, id) or (not forcedIdentifier and include(COFFEE_KEYWORDS, id)) tag = 'LEADING_WHEN' if tag is 'WHEN' and include LINE_BREAK, @tag() tag = 'ALL' if id is 'all' and @tag() is 'FOR' tag = 'UNARY' if include UNARY, tag if include(JS_FORBIDDEN, id) if forcedIdentifier tag = 'STRING' id = "\"#{id}\"" if forcedIdentifier is 'accessor' close_index = true @tokens.pop() if @tag() isnt '@' @token 'INDEX_START', '[' else if include(RESERVED, id) @identifierError id unless forcedIdentifier tag = id = CONVERSIONS[id] if include COFFEE_ALIASES, id tag = 'LOGIC' if include LOGIC, id tag = 'UNARY' if id is '!' @token tag, id @token ']', ']' if close_index true</pre></div> </td> </tr> <tr id="section-9"> <td class="docs"> <div class="octowrap"> <a class="octothorpe" href="#section-9">#</a> </div> Matches numbers, including decimals, hex, and exponential notation. Be careful not to interfere with ranges-in-progress. </td> <td class="code"> <div class="highlight"><pre> numberToken: -> return false unless number = @match NUMBER, 1 return false if @tag() is '.' and starts number, '.' @i += number.length @token 'NUMBER', number true</pre></div> </td> </tr> <tr id="section-10"> <td class="docs"> <div class="octowrap"> <a class="octothorpe" href="#section-10">#</a> </div> Matches strings, including multi-line strings. Ensures that quotation marks are balanced within the string's contents, and within nested interpolations. </td> <td class="code"> <div class="highlight"><pre> stringToken: -> return false unless starts(@chunk, '"') or starts(@chunk, "'") return false unless string = @balancedToken(['"', '"'], ['#{', '}']) or @balancedToken ["'", "'"] @interpolateString string.replace /\n/g, '\\\n' @line += count string, "\n" @i += string.length true</pre></div> </td> </tr> <tr id="section-11"> <td class="docs"> <div class="octowrap"> <a class="octothorpe" href="#section-11">#</a> </div> Matches heredocs, adjusting indentation to the correct level, as heredocs preserve whitespace, but ignore indentation to the left. </td> <td class="code"> <div class="highlight"><pre> heredocToken: -> return false unless match = @chunk.match HEREDOC quote = match[1].substr 0, 1 doc = @sanitizeHeredoc match[2] or match[4] or '', {quote} @interpolateString quote + doc + quote, heredoc: yes @line += count match[1], "\n" @i += match[1].length true</pre></div> </td> </tr> <tr id="section-12"> <td class="docs"> <div class="octowrap"> <a class="octothorpe" href="#section-12">#</a> </div> Matches and consumes comments. </td> <td class="code"> <div class="highlight"><pre> commentToken: -> return false unless match = @chunk.match(COMMENT) @line += count match[1], "\n" @i += match[1].length if match[2] @token 'HERECOMMENT', @sanitizeHeredoc match[2], herecomment: true, indent: Array(@indent + 1).join(' ') @token 'TERMINATOR', '\n' true</pre></div> </td> </tr> <tr id="section-13"> <td class="docs"> <div class="octowrap"> <a class="octothorpe" href="#section-13">#</a> </div> Matches JavaScript interpolated directly into the source via backticks. </td> <td class="code"> <div class="highlight"><pre> jsToken: -> return false unless starts @chunk, '`' return false unless script = @balancedToken ['`', '`'] @token 'JS', script.replace JS_CLEANER, '' @i += script.length true</pre></div> </td> </tr> <tr id="section-14"> <td class="docs"> <div class="octowrap"> <a class="octothorpe" href="#section-14">#</a> </div> Matches regular expression literals. Lexing regular expressions is difficult to distinguish from division, so we borrow some basic heuristics from JavaScript and Ruby, borrow slash balancing from <code>@balancedToken</code>, and borrow interpolation from <code>@interpolateString</code>. </td> <td class="code"> <div class="highlight"><pre> regexToken: -> return false unless first = @chunk.match REGEX_START return false if first[1] is ' ' and @tag() not in ['CALL_START', '='] return false if include NOT_REGEX, @tag() return false unless regex = @balancedToken ['/', '/'] return false unless end = @chunk.substr(regex.length).match REGEX_END regex += flags = end[2] if end[2] if regex.match REGEX_INTERPOLATION str = regex.substring(1).split('/')[0] str = str.replace REGEX_ESCAPE, (escaped) -> '\\' + escaped @tokens = @tokens.concat [['(', '('], ['NEW', 'new'], ['IDENTIFIER', 'RegExp'], ['CALL_START', '(']] @interpolateString "\"#{str}\"", escapeQuotes: yes @tokens.splice @tokens.length, 0, [',', ','], ['STRING', "\"#{flags}\""] if flags @tokens.splice @tokens.length, 0, [')', ')'], [')', ')'] else @token 'REGEX', regex @i += regex.length true</pre></div> </td> </tr> <tr id="section-15"> <td class="docs"> <div class="octowrap"> <a class="octothorpe" href="#section-15">#</a> </div> Matches a token in which which the passed delimiter pairs must be correctly balanced (ie. strings, JS literals). </td> <td class="code"> <div class="highlight"><pre> balancedToken: (delimited...) -> @balancedString @chunk, delimited</pre></div> </td> </tr> <tr id="section-16"> <td class="docs"> <div class="octowrap"> <a class="octothorpe" href="#section-16">#</a> </div> Matches newlines, indents, and outdents, and determines which is which. If we can detect that the current line is continued onto the the next line, then the newline is suppressed: <pre><code>elements .each( ... ) .map( ... ) </code></pre> Keeps track of the level of indentation, because a single outdent token can close multiple indents, so we need to know how far in we happen to be. </td> <td class="code"> <div class="highlight"><pre> lineToken: -> return false unless indent = @match MULTI_DENT, 1 @line += count indent, "\n" @i += indent.length prev = @prev(2) size = indent.match(LAST_DENTS).reverse()[0].match(LAST_DENT)[1].length nextCharacter = @match NEXT_CHARACTER, 1 noNewlines = nextCharacter is '.' or nextCharacter is ',' or @unfinished() if size - @indebt is @indent return @suppressNewlines() if noNewlines return @newlineToken indent else if size > @indent if noNewlines @indebt = size - @indent return @suppressNewlines() diff = size - @indent + @outdebt @token 'INDENT', diff @indents.push diff @outdebt = @indebt = 0 else @indebt = 0 @outdentToken @indent - size, noNewlines @indent = size true</pre></div> </td> </tr> <tr id="section-17"> <td class="docs"> <div class="octowrap"> <a class="octothorpe" href="#section-17">#</a> </div> Record an outdent token or multiple tokens, if we happen to be moving back inwards past several recorded indents. </td> <td class="code"> <div class="highlight"><pre> outdentToken: (moveOut, noNewlines, close) -> while moveOut > 0 len = @indents.length - 1 if @indents[len] is undefined moveOut = 0 else if @indents[len] is @outdebt moveOut -= @outdebt @outdebt = 0 else if @indents[len] < @outdebt @outdebt -= @indents[len] moveOut -= @indents[len] else dent = @indents.pop() dent -= @outdebt moveOut -= dent @outdebt = 0 @token 'OUTDENT', dent @outdebt -= moveOut if dent @token 'TERMINATOR', "\n" unless @tag() is 'TERMINATOR' or noNewlines true</pre></div> </td> </tr> <tr id="section-18"> <td class="docs"> <div class="octowrap"> <a class="octothorpe" href="#section-18">#</a> </div> Matches and consumes non-meaningful whitespace. Tag the previous token as being "spaced", because there are some cases where it makes a difference. </td> <td class="code"> <div class="highlight"><pre> whitespaceToken: -> return false unless space = @match WHITESPACE, 1 prev = @prev() prev.spaced = true if prev @i += space.length true</pre></div> </td> </tr> <tr id="section-19"> <td class="docs"> <div class="octowrap"> <a class="octothorpe" href="#section-19">#</a> </div> Generate a newline token. Consecutive newlines get merged together. </td> <td class="code"> <div class="highlight"><pre> newlineToken: (newlines) -> @token 'TERMINATOR', "\n" unless @tag() is 'TERMINATOR' true</pre></div> </td> </tr> <tr id="section-20"> <td class="docs"> <div class="octowrap"> <a class="octothorpe" href="#section-20">#</a> </div> Use a <code>\</code> at a line-ending to suppress the newline. The slash is removed here once its job is done. </td> <td class="code"> <div class="highlight"><pre> suppressNewlines: -> @tokens.pop() if @value() is "\\" true</pre></div> </td> </tr> <tr id="section-21"> <td class="docs"> <div class="octowrap"> <a class="octothorpe" href="#section-21">#</a> </div> We treat all other single characters as a token. Eg.: <code>( ) , . !</code> Multi-character operators are also literal tokens, so that Jison can assign the proper order of operations. There are some symbols that we tag specially here. <code>;</code> and newlines are both treated as a <code>TERMINATOR</code>, we distinguish parentheses that indicate a method call from regular parentheses, and so on. </td> <td class="code"> <div class="highlight"><pre> literalToken: -> match = @chunk.match OPERATOR value = match and match[1] space = match and match[2] @tagParameters() if value and value.match CODE value or= @chunk.substr 0, 1 @i += value.length spaced = (prev = @prev()) and prev.spaced tag = value if value is '=' @assignmentError() if include JS_FORBIDDEN, @value() if @value() in ['or', 'and'] @tokens.splice(@tokens.length - 1, 1, ['COMPOUND_ASSIGN', CONVERSIONS[@value()] + '=', prev[2]]) return true if value is ';' then tag = 'TERMINATOR' else if include(LOGIC, value) then tag = 'LOGIC' else if include(MATH, value) then tag = 'MATH' else if include(COMPARE, value) then tag = 'COMPARE' else if include(COMPOUND_ASSIGN, value) then tag = 'COMPOUND_ASSIGN' else if include(UNARY, value) then tag = 'UNARY' else if i