node-zoom
Version:
292 lines (264 loc) • 7.81 kB
text/xml
<refentry id="yaz-icu">
<refentryinfo>
<productname>YAZ</productname>
<productnumber>&version;</productnumber>
<info><orgname>Index Data</orgname></info>
</refentryinfo>
<refmeta>
<refentrytitle>yaz-icu</refentrytitle>
<manvolnum>1</manvolnum>
<refmiscinfo class="manual">Commands</refmiscinfo>
</refmeta>
<refnamediv>
<refname>yaz-icu</refname>
<refpurpose>YAZ ICU utility</refpurpose>
</refnamediv>
<refsynopsisdiv>
<cmdsynopsis>
<command>yaz-icu</command>
<arg>-c <replaceable>config</replaceable></arg>
<arg>-p <replaceable>opt</replaceable></arg>
<arg>-s</arg>
<arg>-x</arg>
<arg choice="opt">infile</arg>
</cmdsynopsis>
</refsynopsisdiv>
<refsect1><title>DESCRIPTION</title>
<para>
<command>yaz-icu</command> is utility which demonstrates
the ICU chain module of yaz. (<filename>yaz/icu.h</filename>).
</para>
<para>
The utility can be used in two ways. It may read some text
using an XML configuration for configuring ICU and show text analysis.
This mode is triggered by option <literal>-c</literal> which specififies
the configuration to be used. The input file is read from standard
input or from a file if <literal>infile</literal> is specified.
</para>
<para>
The utility may also show ICU information. This is triggered by
option <literal>-p</literal>.
</para>
</refsect1>
<refsect1><title>OPTIONS</title>
<variablelist>
<varlistentry>
<term>-c <replaceable>config</replaceable></term>
<listitem><para>
Specifies the file containing ICU chain configuration
which is XML based.
</para></listitem>
</varlistentry>
<varlistentry>
<term>-p <replaceable>type</replaceable></term>
<listitem><para>
Specifies extra information to be printed about the ICU system.
If <replaceable>type</replaceable> is <literal>c</literal>
then ICU converters are printed.
If <replaceable>type</replaceable> is <literal>l</literal>
available locales are printed.
If <replaceable>type</replaceable> is <literal>t</literal>
available transliterators are printed.
</para></listitem>
</varlistentry>
<varlistentry>
<term>-s</term>
<listitem><para>
Specifies that output should include sort key as well. Note that
sort key differs between ICU versions.
</para></listitem>
</varlistentry>
<varlistentry>
<term>-x</term>
<listitem><para>
Specifies that output should be XML based rather than
"text" based.
</para></listitem>
</varlistentry>
</variablelist>
</refsect1>
<refsect1><title>ICU chain configuration</title>
<para>
The ICU chain configuration speicifies one or more rules to convert
text data into tokens. The configuration format is XML based.
</para>
<para>
The toplevel element must be named <literal>icu_chain</literal>.
The <literal>icu_chain</literal> element has one required attribute
<literal>locale</literal> which specifies the ICU locale to be used
in the conversion steps.
</para>
<para>
The <literal>icu_chain</literal> element must include elements where
each element specifies a conversion step. The conversion is performed
in the order in which the conversion steps are specified.
Each conversion element takes one attribute: <literal>rule</literal>
which serves as argument to the conversion step.
</para>
<para>
The following conversion elements are available:
<variablelist>
<varlistentry>
<term>casemap</term>
<listitem><para>
Converts case and rule specifies how:
<variablelist>
<varlistentry>
<term>l</term>
<listitem>
<para>Lowercase using ICU function u_strToLower. </para>
</listitem>
</varlistentry>
<varlistentry>
<term>u</term>
<listitem>
<para>Upper case using ICU function u_strToUpper.</para>
</listitem>
</varlistentry>
<varlistentry>
<term>t</term>
<listitem>
<para>To title using UCU function u_strToTitle.</para>
</listitem>
</varlistentry>
<varlistentry>
<term>f</term>
<listitem>
<para>Fold case using ICU function u_strFoldCase.</para>
</listitem>
</varlistentry>
</variablelist>
</para></listitem>
</varlistentry>
<varlistentry>
<term>display</term>
<listitem><para>
This is a meta step which specifies that a term/token is to
be displayed. This term is retrieved in an application
using function icu_chain_token_display (<filename>yaz/icu.h</filename>).
</para></listitem>
</varlistentry>
<varlistentry>
<term>transform</term>
<listitem><para>
Specifies an ICU transform rule using a transliterator
Identifier.
The rule attribute is the transliterator Identifier.
See <ulink url="&url.icu.transform;">ICU Transforms</ulink> for
more information.
</para></listitem>
</varlistentry>
<varlistentry>
<term>transliterate</term>
<listitem><para>
Specifies a rule-based transliterator.
The rule attribute is the custom transformation rule to be used.
See <ulink url="&url.icu.transform;">ICU Transforms</ulink> for
more information.
</para></listitem>
</varlistentry>
<varlistentry>
<term>tokenize</term>
<listitem><para>
Breaks / tokenizes a string into components using
ICU functions ubrk_open, ubrk_setText, .. . The rule is
one of:
<variablelist>
<varlistentry>
<term>l</term>
<listitem>
<para>Line. ICU: UBRK_LINE.</para>
</listitem>
</varlistentry>
<varlistentry>
<term>s</term>
<listitem>
<para>Sentence. ICU: UBRK_SENTENCE.</para>
</listitem>
</varlistentry>
<varlistentry>
<term>w</term>
<listitem>
<para>Word. ICU: UBRK_WORD.</para>
</listitem>
</varlistentry>
<varlistentry>
<term>c</term>
<listitem>
<para>Character. ICU: UBRK_CHARACTER.</para>
</listitem>
</varlistentry>
<varlistentry>
<term>t</term>
<listitem>
<para>Title. ICU: UBRK_TITLE.</para>
</listitem>
</varlistentry>
</variablelist>
</para></listitem>
</varlistentry>
<varlistentry>
<term>join</term>
<listitem>
<para>
Joins tokens into one string. The rule attribute is the joining
string, which may be empty. The join conversion element was added
in YAZ 4.2.49.
</para>
</listitem>
</varlistentry>
</variablelist>
</para>
</refsect1>
<refsect1><title>EXAMPLES</title>
<para>
The following command analyzes text in file <filename>text</filename>
using ICU chain configuration <filename>chain.xml</filename>:
<screen>
cat text | yaz-icu -c chain.xml
</screen>
The chain.xml might look as follows:
<screen><![CDATA[
<icu_chain locale="en">
<transform rule="[:Control:] Any-Remove"/>
<tokenize rule="w"/>
<transform rule="[[:WhiteSpace:][:Punctuation:]] Remove"/>
<transliterate rule="xy > z;"/>
<display/>
<casemap rule="l"/>
</icu_chain>
]]>
</screen>
</para>
</refsect1>
<refsect1><title>SEE ALSO</title>
<para>
<citerefentry>
<refentrytitle>yaz</refentrytitle>
<manvolnum>7</manvolnum>
</citerefentry>
</para>
<para>
<ulink url="&url.icu;">ICU Home</ulink>
</para>
<para>
<ulink url="&url.icu.transform;">ICU Transforms</ulink>
</para>
</refsect1>
</refentry>
<!-- Keep this comment at the end of the file
Local variables:
mode: nxml
nxml-child-indent: 1
End:
-->