@masala/parser

<!doctype html> <html class="default no-js"> <head> <meta charset="utf-8"> <meta http-equiv="X-UA-Compatible" content="IE=edge"> <title>@masala/parser</title> <meta name="description" content=""> <meta name="viewport" content="width=device-width, initial-scale=1"> <link rel="stylesheet" href="assets/css/main.css"> </head> <body> <header> <div class="tsd-page-toolbar"> <div class="container"> <div class="table-wrap"> <div class="table-cell" id="tsd-search" data-index="assets/js/search.js" data-base="."> <div class="field"> <label for="tsd-search-field" class="tsd-widget search no-caption">Search</label> <input id="tsd-search-field" type="text" /> </div> <ul class="results"> <li class="state loading">Preparing search index...</li> <li class="state failure">The search index is not available</li> </ul> <a href="index.html" class="title">@masala/parser</a> </div> <div class="table-cell" id="tsd-widgets"> <div id="tsd-filter"> <a href="#" class="tsd-widget options no-caption" data-toggle="options">Options</a> <div class="tsd-filter-group"> <div class="tsd-select" id="tsd-filter-visibility"> All <ul class="tsd-select-list"> <li data-value="public">Public</li> <li data-value="protected">Public/Protected</li> <li data-value="private" class="selected">All</li> </ul> </div> <input type="checkbox" id="tsd-filter-inherited" checked /> <label class="tsd-widget" for="tsd-filter-inherited">Inherited</label> <input type="checkbox" id="tsd-filter-only-exported" /> <label class="tsd-widget" for="tsd-filter-only-exported">Only exported</label> </div> </div> <a href="#" class="tsd-widget menu no-caption" data-toggle="menu">Menu</a> </div> </div> </div> </div> <div class="tsd-page-title"> <div class="container"> <ul class="tsd-breadcrumb"> <li> <a href="globals.html">Globals</a> </li> </ul> <h1> @masala/parser</h1> </div> </div> </header> <div class="container container-main"> <div class="row"> <div class="col-8 col-content"> <div class="tsd-panel tsd-typography"> <h1 id="masala-parser-javascript-parser-combinators">Masala Parser: Javascript Parser Combinators</h1> <a href="https://badge.fury.io/js/%40masala%2Fparser"><img src="https://badge.fury.io/js/%40masala%2Fparser.svg" alt="npm version"></a> <a href="https://travis-ci.org/d-plaindoux/masala-parser"><img src="https://travis-ci.org/d-plaindoux/masala-parser.svg" alt="Build Status"></a> <a href="https://coveralls.io/r/d-plaindoux/masala-parser?branch=master"><img src="https://coveralls.io/repos/d-plaindoux/masala-parser/badge.png?branch=master" alt="Coverage Status"></a> <a href="http://github.com/badges/stability-badges"><img src="http://badges.github.io/stability-badges/dist/stable.svg" alt="stable"></a> Masala Parser is inspired by the paper titled: <a href="https://www.microsoft.com/en-us/research/wp-content/uploads/2016/02/parsec-paper-letter.pdf">Direct Style Monadic Parser Combinators For The Real World</a>. Masala Parser is a Javascript implementation of the Haskell Parsec. It is plain Javascript that works in the browser, is tested with more than 450 unit tests, covering 100% of code lines. <h3 id="use-cases">Use cases</h3> <ul> <li>It can create a full parser from scratch</li> <li>It can extract data from a big text and replace complex regexp</li> <li>It works in any browser</li> <li>There is a good typescript type declaration</li> <li>It can validate complete structure with variations</li> <li>It's a great starting point for parser education. It's way simpler than Lex & Yacc.</li> <li>It's designed to be written in other languages (Python, Java, Rust) with the same interface</li> </ul> Masala Parser keywords are simplicity, variations and maintainability. You won't need theoretical bases on languages for extraction or validation use cases. Masala Parser has relatively good performances, however Javascript is obviously not the fastest machine. <h1 id="usage">Usage</h1> With Node Js or modern build <pre><code> npm install -S @masala/parser</code></pre>Or in the browser <ul> <li><a href="https://github.com/d-plaindoux/masala-parser/releases">download Release</a></li> <li><code><script src="masala-parser.min.js"/></code></li> </ul> Check the <a href="./changelog.md">Change Log</a> if you can from a previous version. <h1 id="reference">Reference</h1> You will find an <a href="documentation/typedoc/modules/_masala_parser_d_.html">Masala Parser online reference</a>, generated from typescript interface. <h1 id="quick-examples">Quick Examples</h1> <h2 id="hello-world">Hello World</h2> <pre><code class="language-js">const helloParser = C.string('hello'); const white = C.char(' '); const worldParser = C.char('world'); const combinator = helloParser.then(white.rep()).then(worldParser);</code></pre> <h2 id="floor-notation">Floor notation</h2> <pre><code class="language-js">// N: Number Bundle, C: Chars Bundle const {Streams, N, C}= require('@masala/parser'); const stream = Stream.ofString('|4.6|'); const floorCombinator = C.char('|').drop() .then(N.number()) // we have ['|', 4.6], we drop '|' .then(C.char('|').drop()) // we have [4.6, '|'], we keep [4.6] .single() // we had [4.6], now just 4.6 .map(x =>Math.floor(x)); // The parser parses a stream of characters const parsing = floorCombinator.parse(stream); assertEquals( 4, parsing.value, 'Floor parsing');</code></pre> <h2 id="explanations">Explanations</h2> According to Wikipedia "in functional programming, a parser combinator is a higher-order function that accepts several parsers as input and returns a new parser as its output." <h2 id="the-parser">The Parser</h2> Let's say we have a document : <blockquote> <blockquote> <blockquote> The James Bond series, by writer Ian Fleming, focuses on a fictional British Secret Service agent created in 1953, who featured him in twelve novels and two short-story collections. Since Fleming's death in 1964, eight other authors have written authorised Bond novels or novelizations: Kingsley Amis, Christopher Wood, John Gardner, Raymond Benson, Sebastian Faulks, Jeffery Deaver, William Boyd and Anthony Horowitz. </blockquote> </blockquote> </blockquote> The parser could fetch every names, ie two consecutive words starting with uppercase. The parser will read through the document and aggregate a Response, which contains a value and the current offset in the text. This value will evolve when the parser will meet new characters, but also with some function calls, such as the <code>map()</code> function. <img src="./documentation/parsec-monoid.png" alt=""> <h2 id="the-response">The Response</h2> By definition, a Parser takes text as an input, and the Response is a structure that represents your problem. After parsing, there are two subtypes of <code>Response</code>: <ul> <li><code>Accept</code> when it found something. </li> <li><code>Reject</code> if it could not.</li> </ul> <pre><code class="language-js"> let response = C.char('a').rep().parse(Streams.ofString('aaaa')); assertEquals(response.value.join(''), 'aaaa' ); assertEquals(response.offset, 4 ); assertTrue(response.isAccepted()); assertTrue(response.isConsumed()); // Partially accepted response = C.char('a').rep().parse(Streams.ofString('aabb')); assertEquals(response.value.join(''), 'aa' ); assertEquals(response.offset, 2 ); assertTrue(response.isAccepted()); assertFalse(response.isConsumed()); </code></pre> <h2 id="building-the-parser-and-execution">Building the Parser, and execution</h2> Like a language, the parser is built then executed. With Masala, we build using other parsers. <pre><code class="language-js">const helloParser = C.string('hello'); const white = C.char(' '); const worldParser = C.char('world'); const combinator = helloParser.then(white.rep()).then(worldParser);</code></pre> There is a compiling time when you combine your parser, and an execution time when the parser runs its <code>parse(stream)</code> function. You will have the <code>Response</code> after parsing. So after building, the parser is executed against a stream of token. For simplicity, we will use a stream of characters, which is a text :) <h2 id="hello-gandhi">Hello Gandhi</h2> The goal is check that we have Hello 'someone', then to grab that name <pre><code class="language-js">// Plain old javascript const {Streams, C}= require('@masala/parser'); var helloParser = C.string("Hello") .then(C.char(' ').rep()) .then(C.letters()) // succession of A-Za-z letters .last(); // keeping previous letters var value = helloParser.val("Hello Gandhi"); // val(x) is a shortcut for parse(Stream.ofString(x)).value; assertEquals('Gandhi', value);</code></pre> <h1 id="parser-combinations">Parser Combinations</h1> Let's use a real example. We combine many functions that returns a new Parser. And each new Parser is a combination of Parsers given by the standard bundles or previous functions. <pre><code class="language-js">import {Streams, N,C, F} from '@masala/parser'; const blanks = ()=>C.char(' ').optrep(); function operator(symbol) { return blanks().drop() .then(C.char(symbol)) // '+' or '*' .then(blanks().drop()) .single(); } function sum() { return N.integer() .then(operator('+').drop()) .then(N.integer()) // then(x) creates a tuple - here, one value was dropped .map(tuple => tuple.at(0) + tuple.at(1)); } function multiplication() { return N.integer() .then(operator('*').drop()) .then(N.integer()) .array() // we can have access to the value of the tuple .map( ([left,right])=> left * right); // more modern js } function scalar() { return N.integer(); } function combinator() { return F.try(sum()) .or(F.try(multiplication())) // or() will often work with try() .or(scalar()); } function parseOperation(line) { return combinator().parse(Streams.ofString(line)); } assertEquals(4, parseOperation('2 +2').value, 'sum: '); assertEquals(6, parseOperation('2 * 3').value, 'multiplication: '); assertEquals(8, parseOperation('8').value, 'scalar: ');</code></pre> A curry paste is an higher order ingredient made from a good combination of spices. <img src="./documentation/images/curry-paste.jpg" alt=""> <h2 id="precedence">Precedence</h2> Precedence is a technical term for priority. Using: <pre><code class="language-js">function combinator() { return F.try(sum()) .or(F.try(multiplication())) // or() will often work with try() .or(scalar()); } console.info('sum: ',parseOperation('2+2').value);</code></pre> We will give priority to sum, then multiplication, then scalar. If we had put <code>scalar()</code> first, we would have first accepted <code>2</code>, then what could we do with <code>+2</code> alone ? It's not a valid sum ! Moreover <code>+2</code> and <code>-2</code> are acceptable scalars. <h2 id="try-x-or-y-">try(x).or(y)</h2> <code>or()</code> will often be used with <code>try()</code>, that makes <a href="https://en.wikipedia.org/wiki/Backtracking">backtracking</a> : it saves the current offset, then tries an option. And as soon that it's not satisfied, it goes back to the original offset and use the parser inside the <code>.or(P)</code> expression.`. Like Haskell's Parsec, Masala Parser can parse infinite look-ahead grammars but performs best on predictive (LL[1]) grammars. Let see how with <code>try()</code>, we can look a bit ahead of next characters, then go back: <pre><code> F.try(sum()).or(F.try(multiplication())).or(scalar()) // try(sum()) parser in action 2 *2 ..ok..ok ↑oups: go back and try multiplication. Should be OK.</code></pre>Suppose we do not <code>try()</code> but use <code>or()</code> directly: <pre><code> sum().or(multiplication()).or(scalar()) // testing sum() 2 *2 ..ok..ok ↑oups: cursor is NOT going back. So now we must test '*2' ; Is it (multiplication())? No ; or(scalar()) ? neither</code></pre><h1 id="recursion">Recursion</h1> Masala-Parser (like Parsec) is a top-down parser and doesn't like <a href="https://cs.stackexchange.com/a/9971">Left Recursion</a>. However, it is a resolved problem for this kind of parsers, with a lot of documentation. You can read more on <a href="./documentation/recursion.md">recursion with Masala</a>, and checkout examples on our Github repository ( <a href="https://github.com/d-plaindoux/masala-parser/blob/master/integration-npm/examples/recursion/aaab-lazy-recursion.js">simple recursion</a>, or <a href="https://github.com/d-plaindoux/masala-parser/blob/master/integration-npm/examples/operations/plus-minus.js">calculous expressions</a> ). <h1 id="simple-documentation-of-core-bundles">Simple documentation of Core bundles</h1> <h2 id="core-parser-functions">Core Parser Functions</h2> Here is a link for <a href="./documentation/parser-core-functions.md">Core functions documentation</a>. It will explain <code>then()</code>, <code>drop()</code>, <code>map()</code>, <code>rep()</code>, <code>opt()</code> and other core functions of the Parser with code examples. ### <h2 id="the-chars-bundle">The Chars Bundle</h2> Example: <pre><code class="language-js">C.char('-') .then(C.letters()) .then(C.char('-')) // accepts '-hello-' ; value is ['-','hello','-'] // reject '-hel lo-' because space is not a letter </code></pre> <a href="./documentation/chars-bundle.md">General use</a> <ul> <li><code>letter()</code>: accept a european letter (and moves the cursor)</li> <li><code>letters()</code>: accepts many letters and returns a string</li> <li><code>letterAs(symbol)</code>: accepts a european(default), ascii, or utf8 Letter. <a href="./documentation/chars-bundle.md">More here</a></li> <li><code>lettersAs(symbol)</code>: accepts many letters and returns a string</li> <li><code>emoji()</code>: accept any emoji sequence. <a href="https://github.com/d-plaindoux/masala-parser/issues/86">Opened Issue</a>.</li> <li><code>notChar(x)</code>: accept if next input is not <code>x</code></li> <li><code>char(x)</code>: accept if next input is <code>x</code></li> <li><code>charIn('xyz')</code>: accept if next input is <code>x</code>, <code>y</code> or <code>z</code></li> <li><code>charNotIn('xyz')</code>: accept if next input is not <code>x</code>, <code>y</code> or <code>z</code></li> <li><code>subString(length)</code>: accept any next length characters and returns the equivalent string</li> <li><code>string(word)</code>: accept if next input is the given <code>word</code> </li> <li><code>stringIn(words)</code>: accept if next input is the given <code>words</code> <a href="./documentation/chars-bundle.md">More here</a></li> <li><code>notString(word)</code>: accept if next input is not the given <code>word</code></li> <li><code>charLiteral()</code>: single quoted char element in C/Java : <code>'a'</code> is accepted</li> <li><code>stringLiteral()</code>: double quoted string element in java/json: <code>"hello world"</code> is accepted</li> <li><code>lowerCase()</code>: accept any next lower case inputs</li> <li><code>upperCase()</code>: accept any next uppercase inputs</li> </ul> Other example: <pre><code class="language-js">C.string('Hello') .then(C.char(' ')) .then(C.lowerCase().rep().join('')) // accepts Hello johnny ; value is ['Hello', ' ', 'johnny'] // rejects Hello Johnny : J is not lowercase ; no value</code></pre> <h2 id="the-numbers-bundle">The Numbers Bundle</h2> <ul> <li><code>number()</code>: accept any float number, such as -2.3E+24, and returns a float </li> <li><code>digit()</code>: accept any single digit, and returns a number</li> <li><code>digits()</code>: accept many digits, and returns a number. Warning: it does not accept +- signs symbols.</li> <li><code>integer()</code>: accept any positive or negative integer</li> </ul> <h2 id="the-flow-bundle">The Flow Bundle</h2> The flow bundle will mix ingredients together. For example if you have a Parser <code>p</code>, <code>F.not(p)</code> will accept anything that does not satisfy <code>p</code> All of these functions will return a brand new Parser that you can combine with others. Most important: <ul> <li><code>F.try(parser).or(otherParser)</code>: Try a parser and come back to <code>otherParser</code> if failed</li> <li><code>F.any()</code>: Accept any character (and so moves the cursor)</li> <li><code>F.not(parser)</code>: Accept anything that is not a parser. Often used to accept until a given stop </li> <li><code>F.eos()</code>: Accepted if the Parser has reached the E*nd *Of Stream</li> <li><code>F.moveUntil(string|stopParser)</code>: Alternative for regex. Will traverse the document until the stop parser<ul> <li>returns <code>undefined</code> if stop is not found</li> <li>returns all characters if stop is found, and set the cursor at the spot of the stop</li> </ul> </li> <li><code>F.dropTo(string|stopParser)</code>: Will traverse the document including the stop parser</li> </ul> Others: <ul> <li><code>F.lazy(parser, ?params)</code>: Makes a lazy evaluation. May be used for Left recursion (difficult)</li> <li><code>F.parse(parserFunction)</code>: Create a new Parser from a function. Usually, you won't start here.</li> <li><code>F.subStream(length)</code>: accept any next characters </li> <li><code>F.returns(value)</code>: forces a returned value</li> <li><code>F.error()</code>: returns an error. Parser will never be accepted</li> <li><code>F.satisfy(predicate)</code>: check if condition is satisfied</li> <li><code>F.startsWith(value)</code>: create a no-op parser with initial value </li> </ul> <h2 id="license">License</h2> Copyright (C)2016-2019 D. Plaindoux. This program is free software; you can redistribute it and/or modify it under the terms of the GNU Lesser General Public License as published by the Free Software Foundation; either version 2, or (at your option) any later version. This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU Lesser General Public License for more details. You should have received a copy of the GNU Lesser General Public License along with this program; see the file COPYING. If not, write to the Free Software Foundation, 675 Mass Ave, Cambridge, MA 02139, USA. </div> </div> <div class="col-4 col-menu menu-sticky-wrap menu-highlight"> <nav class="tsd-navigation primary"> <ul> <li class="globals "> <a href="globals.html">Globals</a> </li> <li class=" tsd-kind-external-module"> <a href="modules/_masala_parser_d_.html">"masala-parser.d"</a> </li> </ul> </nav> <nav class="tsd-navigation secondary menu-sticky"> <ul class="before-current"> </ul> </nav> </div> </div> </div> <footer class="with-border-bottom"> <div class="container"> <h2>Legend</h2> <div class="tsd-legend-group"> <ul class="tsd-legend"> <li class="tsd-kind-module">Module</li> <li class="tsd-kind-object-literal">Object literal</li> <li class="tsd-kind-variable">Variable</li> <li class="tsd-kind-function">Function</li> <li class="tsd-kind-function tsd-has-type-parameter">Function with type parameter</li> <li class="tsd-kind-index-signature">Index signature</li> <li class="tsd-kind-type-alias">Type alias</li> </ul> <ul class="tsd-legend"> <li class="tsd-kind-enum">Enumeration</li> <li class="tsd-kind-enum-member">Enumeration member</li> <li class="tsd-kind-property tsd-parent-kind-enum">Property</li> <li class="tsd-kind-method tsd-parent-kind-enum">Method</li> </ul> <ul class="tsd-legend"> <li class="tsd-kind-interface">Interface</li> <li class="tsd-kind-interface tsd-has-type-parameter">Interface with type parameter</li> <li class="tsd-kind-constructor tsd-parent-kind-interface">Constructor</li> <li class="tsd-kind-property tsd-parent-kind-interface">Property</li> <li class="tsd-kind-method tsd-parent-kind-interface">Method</li> <li class="tsd-kind-index-signature tsd-parent-kind-interface">Index signature</li> </ul> <ul class="tsd-legend"> <li class="tsd-kind-class">Class</li> <li class="tsd-kind-class tsd-has-type-parameter">Class with type parameter</li> <li class="tsd-kind-constructor tsd-parent-kind-class">Constructor</li> <li class="tsd-kind-property tsd-parent-kind-class">Property</li> <li class="tsd-kind-method tsd-parent-kind-class">Method</li> <li class="tsd-kind-accessor tsd-parent-kind-class">Accessor</li> <li class="tsd-kind-index-signature tsd-parent-kind-class">Index signature</li> </ul> <ul class="tsd-legend"> <li class="tsd-kind-constructor tsd-parent-kind-class tsd-is-inherited">Inherited constructor</li> <li class="tsd-kind-property tsd-parent-kind-class tsd-is-inherited">Inherited property</li> <li class="tsd-kind-method tsd-parent-kind-class tsd-is-inherited">Inherited method</li> <li class="tsd-kind-accessor tsd-parent-kind-class tsd-is-inherited">Inherited accessor</li> </ul> <ul class="tsd-legend"> <li class="tsd-kind-property tsd-parent-kind-class tsd-is-protected">Protected property</li> <li class="tsd-kind-method tsd-parent-kind-class tsd-is-protected">Protected method</li> <li class="tsd-kind-accessor tsd-parent-kind-class tsd-is-protected">Protected accessor</li> </ul> <ul class="tsd-legend"> <li class="tsd-kind-property tsd-parent-kind-class tsd-is-private">Private property</li> <li class="tsd-kind-method tsd-parent-kind-class tsd-is-private">Private method</li> <li class="tsd-kind-accessor tsd-parent-kind-class tsd-is-private">Private accessor</li> </ul> <ul class="tsd-legend"> <li class="tsd-kind-property tsd-parent-kind-class tsd-is-static">Static property</li> <li class="tsd-kind-call-signature tsd-parent-kind-class tsd-is-static">Static method</li> </ul> </div> </div> </footer> <div class="container tsd-generator"> Generated using <a href="http://typedoc.org/" target="_blank">TypeDoc</a> </div> <div class="overlay"></div> <script src="assets/js/main.js"></script> <script>if (location.protocol == 'file:') document.write('<script src="assets/js/search.js"><' + '/script>');</script> </body> </html>