@ciao-lang/ts-ciao-interface

Version:

Simple Ciao interface for node.

42 lines • 8.65 kB

HTML

<!DOCTYPE HTML><html><head><meta http-equiv="Content-Type" content="text/html; charset=utf-8"/><meta name="viewport" content="width=device-width, initial-scale=1"/><meta name="theme-color" content="#273f79"/><link rel="stylesheet" href="lpdoc.css" type="text/css"/><script type="text/javascript" src="lpdoc.js"></script><title>Tokenizer — The Ciao System v1.22</title></head><body><div class="lpdoc-page fixleftbar"><a href="#" id="sidebar-toggle-button" class="lpdoc-navbutton"><span id="sidebar-button-arrow">☰</span></a><div id="sidebar" class="lpdoc-sidebar"><div style="height: 40px; margin-left: auto; margin-right: auto"><img src="ciao-logo_autofig.png" width=auto height=100%></div><div class="lpdoc-nav"><span class="lpdoc-on-right"><a class="lpdoc-navbutton" href="StdLibs.html">↑</a><a class="lpdoc-navbutton" href="stream_utils.html">←</a><a class="lpdoc-navbutton" href="read.html">→</a><a class="lpdoc-navbutton" href="ciaosearch.html">🔍</a></span><span><a href="ciaofulltoc.html">TOC</a></span></div><hr></hr><ul class="lpdoc-itemize-sectpath"><li><a href="ciao.html">The Ciao System</a> »<br/> </li><li><a href="StdLibs.html">PART VII - Standard libraries</a> »<br/> </li><li><a href=""><strong>Tokenizer</strong></a></li></ul><hr></hr><em>ON THIS PAGE</em><ul><li><a href="#Unicode source code">Unicode source code</a></li><li><a href="#Usage and interface">Usage and interface</a></li><li><a href="#Documentation on exports">Documentation on exports</a></li><li><a href="#Documentation on multifiles">Documentation on multifiles</a></li><li><a href="#Documentation on imports">Documentation on imports</a></li></ul></div><div class="lpdoc-main"><div id=""><h1>Tokenizer</h1><a class="lpdoc-idx-anchor" href="ciaosearch.html#tokenize"></a> <strong>Author(s):</strong> <a class="lpdoc-idx-anchor" id="The Ciao Development Team" href="ciaosearch.html#The Ciao Development Team">The Ciao Development Team</a>, <a class="lpdoc-idx-anchor" id="Jose F. Morales" href="ciaosearch.html#Jose F. Morales">Jose F. Morales (curly blocks, Unicode source support)</a>.<p> This module defines the tokenizer for Ciao. In addition to optional flags, the main differences w.r.t. the ISO-Prolog standard are:<p><ul> <li><tt>`</tt> is a graphic char, there are no <tt>back_quoted_strings</tt>. <li><tt>\</tt> followed by any layout char (not only <tt>new_line</tt>) in a string is a <tt>continuation_escape_sequence</tt>. <li><tt>\^</tt> starts a <tt>control_escape_char</tt> in a string. <li><tt>\c</tt> skips layout in a string. <li><tt>\e</tt> = <tt>ESC</tt>, <tt>\d</tt> = <tt>DEL</tt>, <tt>\s</tt> = <tt>SPACE</tt>. <li><tt>13'23</tt> is 23 in base 13 (same for other bases). <li><tt>0'</tt><tt>'</tt> is accepted as <tt>0'</tt><tt>'</tt><tt>'</tt> (if not followed by <tt>'</tt>). <li>Quoted atoms and strings support Unicode escape <tt>\uDDDD</tt> (four hexadecimal digits) and <tt>\UDDDDDDDD</tt> (eight hexadecimal digits) <li>Support for Unicode identifiers (see <a href="#Unicode source code">Unicode source code</a>). </ul> <p><div id="Unicode source code"><h2>Unicode source code</h2> <p><a class="lpdoc-idx-anchor" id="0" href="ciaosearch.html#unicode"></a> <p>The tokenizer for Ciao extends ISO-Prolog with support for Unicode source identifiers, based on the the <a href="https://unicode.org/reports/tr31/#Case_and_Stability">Unicode Standard Annex 31</a>, as follows:<p><ul> <li>Identifiers can begin with <tt>XID_Start</tt> characters and must be followed with zero or more <tt>XID_Continue</tt> (see the Unicode Derived Core Properties), extended with categorty <tt>No</tt>. Variables are those identifiers that start with characters in the <tt>Lu</tt> category. <li>Use <tt>Z*</tt> as layout characters, as well as other control characters (<tt>Cc</tt> categoty) with bidirectional category <tt>WS</tt> (whitespace), <tt>S</tt> (segment separator), or <tt>B</tt> (paragraph separator). <li>Use <tt>S*</tt> (<tt>Sm</tt>, <tt>Sc</tt>, <tt>Sk</tt>, <tt>So</tt>) and <tt>P*</tt> (<tt>Pc</tt>, <tt>Pd</tt>, <tt>Ps</tt>, <tt>Pe</tt>, <tt>Pi</tt>, <tt>Pf</tt>, <tt>Po</tt>) as symbols. <li>Identifiers that begin with <tt>XID_Continue</tt> are treated as <em>solo</em> tokens. </ul> </div><br/><div id="Usage and interface"><h2>Usage and interface</h2><div class="lpdoc-cartouche"><ul><li><strong>Library usage:</strong><br/><tt>:- use_module(library(tokenize)).</tt><li><strong>Exports:</strong><br/><ul class="lpdoc-itemize-minus"><li><em>Predicates:</em><br/><a class="lpdoc-idx-anchor" id="1" href="#read_tokens/2"><tt>read_tokens/2</tt></a>. <li><em>Regular Types:</em><br/><a class="lpdoc-idx-anchor" id="2" href="#token/1"><tt>token/1</tt></a>. <li><em>Multifiles:</em><br/><a class="lpdoc-idx-anchor" id="3" href="#define_flag/3"><tt>define_flag/3</tt></a>. </ul></ul></div></div><div id="Documentation on exports"><h2>Documentation on exports</h2><div><div class="lpdoc-defname"><span class="lpdoc-predtag">REGTYPE</span><a class="lpdoc-idx-anchor" id="token/1" href="ciaosearch.html#token/1">token/1</a></div><div class="lpdoc-deftext">A regular type, defined as follows:<pre class="lpdoc-codeblock">token(atom(A)) :- atm(A). token(badatom(S)) :- string(S). token(number(N)) :- num(N). token(string(S)) :- string(S). token(var(T,S)) :- term(T), string(S). token('/* ...'). token(','). token('('). token(' ('). token(')'). token('['). token(']'). token('|'). token('{'). token('}'). token('.'). </pre> </div></div><p> <div><div class="lpdoc-defname"><span class="lpdoc-predtag">PREDICATE</span><a class="lpdoc-idx-anchor" id="read_tokens/2" href="ciaosearch.html#read_tokens/2">read_tokens/2</a></div><div class="lpdoc-deftext"><p><span class="lpdoc-usage-header">Usage:</span><span class="lpdoc-usage-decl"><tt>read_tokens(TokenList,Dictionary)</tt> </span><p></p><ul class="lpdoc-itemize-minus"><li><em>The following properties should hold upon exit:</em><br/><span class="lpdoc-on-right"> (<a class="lpdoc-idx-anchor" id="4" href="basic_props.html#list/2"><tt>basic_props:list/2</tt></a>)</span><span><span class="lpdoc-var">TokenList</span> is a list of <span class="lpdoc-var">token</span>s. </span><br/><span class="lpdoc-on-right"> (<a class="lpdoc-idx-anchor" id="5" href="assertions_props.html#dictionary/1"><tt>dict:dictionary/1</tt></a>)</span><span><span class="lpdoc-var">Dictionary</span> is a dictionary. </span> </ul></div></div><p> </div><div id="Documentation on multifiles"><h2>Documentation on multifiles</h2><div><div class="lpdoc-defname"><span class="lpdoc-predtag">PREDICATE</span><a class="lpdoc-idx-anchor" id="define_flag/3" href="ciaosearch.html#define_flag/3">define_flag/3</a></div><div class="lpdoc-deftext"><p><span class="lpdoc-usage-header">Usage:</span><span class="lpdoc-usage-decl"><tt>define_flag(Flag,FlagValues,Default)</tt> </span><p></p><ul class="lpdoc-itemize-minus"><li><em>The following properties hold upon exit:</em><br/><span class="lpdoc-on-right"> (<a class="lpdoc-idx-anchor" id="6" href="basic_props.html#atm/1"><tt>basic_props:atm/1</tt></a>)</span><span><span class="lpdoc-var">Flag</span> is an atom. </span><br/><span class="lpdoc-on-right"> (<a class="lpdoc-idx-anchor" id="7" href="basic_props.html#flag_values/1"><tt>basic_props:flag_values/1</tt></a>)</span><span>Define the valid flag values </span> </ul> The predicate is <em>multifile</em>.<br/></div></div><p> </div><div id="Documentation on imports"><h2>Documentation on imports</h2>This module has the following direct dependencies:<ul class="lpdoc-itemize-minus"><li><em>System library modules:</em><br/><a class="lpdoc-idx-anchor" id="8" href="read.html"><tt>read</tt></a>, <a class="lpdoc-idx-anchor" id="9" href="dict.html"><tt>dict</tt></a>. <li><em>Packages:</em><br/><a class="lpdoc-idx-anchor" id="10" href="ciaosearch.html#prelude"><tt>prelude</tt></a>, <a class="lpdoc-idx-anchor" id="11" href="ciaosearch.html#initial"><tt>initial</tt></a>, <a class="lpdoc-idx-anchor" id="12" href="condcomp_doc.html"><tt>condcomp</tt></a>, <a class="lpdoc-idx-anchor" id="13" href="assertions_doc.html"><tt>assertions</tt></a>, <a class="lpdoc-idx-anchor" id="14" href="ciaosearch.html#assertions/assertions_basic"><tt>assertions/assertions_basic</tt></a>, <a class="lpdoc-idx-anchor" id="15" href="ciaosearch.html#define_flag"><tt>define_flag</tt></a>. </ul></div></div><div class="lpdoc-footer">Generated with LPdoc using Ciao</div></div><div class="lpdoc-clearer"></div></div></body></html>