@yozora/tokenizer-math
Version:
Tokenizer for processing fenced math block (formulas)
394 lines (351 loc) • 17.6 kB
Markdown
<!-- :begin use tokenizer/banner -->
<header>
<h1 align="center">
<a href="https://github.com/yozorajs/yozora/tree/v2.3.13/tokenizers/math#readme">@yozora/tokenizer-math</a>
</h1>
<div align="center">
<a href="https://www.npmjs.com/package/@yozora/tokenizer-math">
<img
alt="Npm Version"
src="https://img.shields.io/npm/v/@yozora/tokenizer-math.svg"
/>
</a>
<a href="https://www.npmjs.com/package/@yozora/tokenizer-math">
<img
alt="Npm Download"
src="https://img.shields.io/npm/dm/@yozora/tokenizer-math.svg"
/>
</a>
<a href="https://www.npmjs.com/package/@yozora/tokenizer-math">
<img
alt="Npm License"
src="https://img.shields.io/npm/l/@yozora/tokenizer-math.svg"
/>
</a>
<a href="#install">
<img
alt="Module formats: cjs, esm"
src="https://img.shields.io/badge/module_formats-cjs%2C%20esm-green.svg"
/>
</a>
<a href="https://github.com/nodejs/node">
<img
alt="Node.js Version"
src="https://img.shields.io/node/v/@yozora/tokenizer-math"
/>
</a>
<a href="https://github.com/facebook/jest">
<img
alt="Tested with Jest"
src="https://img.shields.io/badge/tested_with-jest-9c465e.svg"
/>
</a>
<a href="https://github.com/prettier/prettier">
<img
alt="Code Style: prettier"
src="https://img.shields.io/badge/code_style-prettier-ff69b4.svg?style=flat-square"
/>
</a>
</div>
</header>
<br/>
<!-- :end -->
[@yozora/tokenizer-math] produce [Math][node-type] type nodes. See [documentation][docpage] for
details.
<!-- :begin use tokenizer/usage -->
## Install
- npm
```bash
npm install --save @yozora/tokenizer-math
```
## Usage
[@yozora/tokenizer-math][] has been integrated into [@yozora/parser][], so you can use
`YozoraParser` directly.
### Basic Usage
[@yozora/tokenizer-math][] cannot be used alone, it needs to be registered in _YastParser_ as a
plugin-in before it can be used.
```typescript {4,9}
import { DefaultParser } from '@yozora/core-parser'
import ParagraphTokenizer from '@yozora/tokenizer-paragraph'
import TextTokenizer from '@yozora/tokenizer-text'
import MathTokenizer from '@yozora/tokenizer-math'
const parser = new DefaultParser()
.useFallbackTokenizer(new ParagraphTokenizer())
.useFallbackTokenizer(new TextTokenizer())
.useTokenizer(new MathTokenizer())
// parse source markdown content
parser.parse(`
$$x^2 + y^2=z^2$$
$$
f(x)=\left\lbrace\begin{aligned}
&x^2, &x < 0\\
&0, &x = 0\\
&x^3, &x > 0
\end{aligned}\right.
$$
`)
```
### Use within [@yozora/parser][]
```typescript
import YozoraParser from '@yozora/parser'
const parser = new YozoraParser()
// parse source markdown content
parser.parse(`
$$x^2 + y^2=z^2$$
$$
f(x)=\left\lbrace\begin{aligned}
&x^2, &x < 0\\
&0, &x = 0\\
&x^3, &x > 0
\end{aligned}\right.
$$
`)
```
### Use with [@yozora/parser-gfm][]
```typescript {2,5}
import GfmParser from '@yozora/parser-gfm'
import MathTokenizer from '@yozora/tokenizer-math'
const parser = new GfmParser()
parser.useTokenizer(new MathTokenizer())
// parse source markdown content
parser.parse(`
$$x^2 + y^2=z^2$$
$$
f(x)=\left\lbrace\begin{aligned}
&x^2, &x < 0\\
&0, &x = 0\\
&x^3, &x > 0
\end{aligned}\right.
$$
`)
```
### Use within [@yozora/parser-gfm-ex][]
```typescript {2,5}
import GfmExParser from '@yozora/parser-gfm-ex'
import MathTokenizer from '@yozora/tokenizer-math'
const parser = new GfmExParser()
parser.useTokenizer(new MathTokenizer())
// parse source markdown content
parser.parse(`
$$x^2 + y^2=z^2$$
$$
f(x)=\left\lbrace\begin{aligned}
&x^2, &x < 0\\
&0, &x = 0\\
&x^3, &x > 0
\end{aligned}\right.
$$
`)
```
### Options
| Name | Type | Required | Default |
| :--------: | :------: | :------: | :------------------------------: |
| `name` | `string` | `false` | `"@yozora/tokenizer-math"` |
| `priority` | `number` | `false` | `TokenizerPriority.FENCED_BLOCK` |
- `name`: The unique name of the tokenizer, used to bind the token it generates, to determine the
tokenizer that should be called in each life cycle of the token in the entire _matching / parsing_
phase.
- `priority`: Priority of the tokenizer, determine the order of processing, high priority priority
execution. interruptable. In addition, in the `match-block` stage, a high-priority tokenizer can
interrupt the matching process of a low-priority tokenizer.
<!-- :end -->
## Related
- [@yozora/ast][]
- [@yozora/parser][]
- [@yozora/parser-gfm][]
- [@yozora/parser-gfm-ex][]
- [@yozora/tokenizer-inline-math][]
- [@yozora/react-math][]
- [@yozora/react-markdown][]
- [Live Examples][live-examples]
- [Math | Yozora AST][node-type]
- [Documentation][docpage]
[node-type]: http://yozora.guanghechen.com/docs/package/ast#math
<!-- :begin use tokenizer/definitions -->
[live-examples]: https://yozora.guanghechen.com/docs/package/tokenizer-math#live-examples
[docpage]: https://yozora.guanghechen.com/docs/package/tokenizer-math
[homepage]: https://github.com/yozorajs/yozora/tree/v2.3.13/tokenizers/math#readme
[gfm-spec]: https://github.github.com/gfm
[mdast-homepage]: https://github.com/syntax-tree/mdast
[@yozora/ast]: https://github.com/yozorajs/yozora/tree/v2.3.13/packages/ast#readme
[@yozora/ast-util]: https://github.com/yozorajs/yozora/tree/v2.3.13/packages/ast-util#readme
[@yozora/character]: https://github.com/yozorajs/yozora/tree/v2.3.13/packages/character#readme
[@yozora/eslint-config]:
https://github.com/yozorajs/yozora/tree/release-2.x.x/packages/eslint-config#readme
[@yozora/core-parser]: https://github.com/yozorajs/yozora/tree/v2.3.13/packages/core-parser#readme
[@yozora/core-tokenizer]:
https://github.com/yozorajs/yozora/tree/v2.3.13/packages/core-tokenizer#readme
[@yozora/invariant]: https://github.com/yozorajs/yozora/tree/v2.3.13/packages/invariant#readme
[@yozora/jest-for-tokenizer]:
https://github.com/yozorajs/yozora/tree/release-2.x.x/packages/jest-for-tokenizer#readme
[@yozora/parser]: https://github.com/yozorajs/yozora/tree/v2.3.13/packages/parser#readme
[@yozora/parser-gfm]: https://github.com/yozorajs/yozora/tree/v2.3.13/packages/parser-gfm#readme
[@yozora/parser-gfm-ex]:
https://github.com/yozorajs/yozora/tree/v2.3.13/packages/parser-gfm-ex#readme
[@yozora/template-tokenizer]:
https://github.com/yozorajs/yozora/tree/release-2.x.x/packages/template-tokenizer#readme
[@yozora/tokenizer-admonition]:
https://github.com/yozorajs/yozora/tree/v2.3.13/tokenizers/admonition#readme
[@yozora/tokenizer-autolink]:
https://github.com/yozorajs/yozora/tree/v2.3.13/tokenizers/autolink#readme
[@yozora/tokenizer-autolink-extension]:
https://github.com/yozorajs/yozora/tree/v2.3.13/tokenizers/autolink-extension#readme
[@yozora/tokenizer-blockquote]:
https://github.com/yozorajs/yozora/tree/v2.3.13/tokenizers/blockquote#readme
[@yozora/tokenizer-break]: https://github.com/yozorajs/yozora/tree/v2.3.13/tokenizers/break#readme
[@yozora/tokenizer-definition]:
https://github.com/yozorajs/yozora/tree/v2.3.13/tokenizers/definition#readme
[@yozora/tokenizer-delete]: https://github.com/yozorajs/yozora/tree/v2.3.13/tokenizers/delete#readme
[@yozora/tokenizer-ecma-import]:
https://github.com/yozorajs/yozora/tree/v2.3.13/tokenizers/ecma-import#readme
[@yozora/tokenizer-emphasis]:
https://github.com/yozorajs/yozora/tree/v2.3.13/tokenizers/emphasis#readme
[@yozora/tokenizer-fenced-block]:
https://github.com/yozorajs/yozora/tree/v2.3.13/tokenizers/fenced-block#readme
[@yozora/tokenizer-fenced-code]:
https://github.com/yozorajs/yozora/tree/v2.3.13/tokenizers/fenced-code#readme
[@yozora/tokenizer-footnote]:
https://github.com/yozorajs/yozora/tree/v2.3.13/tokenizers/footnote#readme
[@yozora/tokenizer-footnote-definition]:
https://github.com/yozorajs/yozora/tree/v2.3.13/tokenizers/footnote-definition#readme
[@yozora/tokenizer-footnote-reference]:
https://github.com/yozorajs/yozora/tree/v2.3.13/tokenizers/footnote-reference#readme
[@yozora/tokenizer-heading]:
https://github.com/yozorajs/yozora/tree/v2.3.13/tokenizers/heading#readme
[@yozora/tokenizer-html-block]:
https://github.com/yozorajs/yozora/tree/v2.3.13/tokenizers/html-block#readme
[@yozora/tokenizer-html-inline]:
https://github.com/yozorajs/yozora/tree/v2.3.13/tokenizers/html-inline#readme
[@yozora/tokenizer-image]: https://github.com/yozorajs/yozora/tree/v2.3.13/tokenizers/image#readme
[@yozora/tokenizer-image-reference]:
https://github.com/yozorajs/yozora/tree/v2.3.13/tokenizers/image-reference#readme
[@yozora/tokenizer-indented-code]:
https://github.com/yozorajs/yozora/tree/v2.3.13/tokenizers/indented-code#readme
[@yozora/tokenizer-inline-code]:
https://github.com/yozorajs/yozora/tree/v2.3.13/tokenizers/inline-code#readme
[@yozora/tokenizer-inline-math]:
https://github.com/yozorajs/yozora/tree/v2.3.13/tokenizers/inline-math#readme
[@yozora/tokenizer-link]: https://github.com/yozorajs/yozora/tree/v2.3.13/tokenizers/link#readme
[@yozora/tokenizer-link-reference]:
https://github.com/yozorajs/yozora/tree/v2.3.13/tokenizers/link-reference#readme
[@yozora/tokenizer-list]: https://github.com/yozorajs/yozora/tree/v2.3.13/tokenizers/list#readme
[@yozora/tokenizer-math]: https://github.com/yozorajs/yozora/tree/v2.3.13/tokenizers/math#readme
[@yozora/tokenizer-paragraph]:
https://github.com/yozorajs/yozora/tree/v2.3.13/tokenizers/paragraph#readme
[@yozora/tokenizer-setext-heading]:
https://github.com/yozorajs/yozora/tree/v2.3.13/tokenizers/setext-heading#readme
[@yozora/tokenizer-table]: https://github.com/yozorajs/yozora/tree/v2.3.13/tokenizers/table#readme
[@yozora/tokenizer-text]: https://github.com/yozorajs/yozora/tree/v2.3.13/tokenizers/text#readme
[@yozora/tokenizer-thematic-break]:
https://github.com/yozorajs/yozora/tree/v2.3.13/tokenizers/thematic-break#readme
[@yozora/react-admonition]:
https://github.com/yozorajs/yozora-react/tree/main/packages/admonition#readme
[@yozora/react-blockquote]:
https://github.com/yozorajs/yozora-react/tree/main/packages/blockquote#readme
[@yozora/react-break]: https://github.com/yozorajs/yozora-react/tree/main/packages/break#readme
[@yozora/react-delete]: https://github.com/yozorajs/yozora-react/tree/main/packages/delete#readme
[@yozora/react-emphasis]:
https://github.com/yozorajs/yozora-react/tree/main/packages/emphasis#readme
[@yozora/react-code]: https://github.com/yozorajs/yozora-react/tree/main/packages/code#readme
[@yozora/react-code-live]:
https://github.com/yozorajs/yozora-react/tree/main/packages/code-live#readme
[@yozora/react-footnote-definitions]:
https://github.com/yozorajs/yozora-react/tree/main/packages/footnote-definitions#readme
[@yozora/react-footnote-reference]:
https://github.com/yozorajs/yozora-react/tree/main/packages/footnote-reference#readme
[@yozora/react-heading]: https://github.com/yozorajs/yozora-react/tree/main/packages/heading#readme
[@yozora/react-image]: https://github.com/yozorajs/yozora-react/tree/main/packages/image#readme
[@yozora/react-inline-code]:
https://github.com/yozorajs/yozora-react/tree/main/packages/inline-code#readme
[@yozora/react-inline-math]:
https://github.com/yozorajs/yozora-react/tree/main/packages/inline-math#readme
[@yozora/react-link]: https://github.com/yozorajs/yozora-react/tree/main/packages/link#readme
[@yozora/react-list]: https://github.com/yozorajs/yozora-react/tree/main/packages/list#readme
[@yozora/react-list-item]:
https://github.com/yozorajs/yozora-react/tree/main/packages/list-item#readme
[@yozora/react-markdown]:
https://github.com/yozorajs/yozora-react/tree/main/packages/markdown#readme
[@yozora/react-math]: https://github.com/yozorajs/yozora-react/tree/main/packages/math#readme
[@yozora/react-paragraph]:
https://github.com/yozorajs/yozora-react/tree/main/packages/paragraph#readme
[@yozora/react-strong]: https://github.com/yozorajs/yozora-react/tree/main/packages/strong#readme
[@yozora/react-table]: https://github.com/yozorajs/yozora-react/tree/main/packages/table#readme
[@yozora/react-text]: https://github.com/yozorajs/yozora-react/tree/main/packages/text#readme
[@yozora/react-thematic-break]:
https://github.com/yozorajs/yozora-react/tree/main/packages/thematic-break#readme
[doc-live-examples/gfm]: https://yozora.guanghechen.com/docs/example/gfm
[doc-@yozora/ast]: https://yozora.guanghechen.com/docs/package/ast
[doc-@yozora/ast-util]: https://yozora.guanghechen.com/docs/package/ast-util
[doc-@yozora/core-parser]: https://yozora.guanghechen.com/docs/package/core-parser
[doc-@yozora/core-tokenizer]: https://yozora.guanghechen.com/docs/package/core-tokenizer
[doc-@yozora/parser]: https://yozora.guanghechen.com/docs/package/parser
[doc-@yozora/parser-gfm]: https://yozora.guanghechen.com/docs/package/parser-gfm
[doc-@yozora/parser-gfm-ex]: https://yozora.guanghechen.com/docs/package/parser-gfm-ex
[doc-@yozora/tokenizer-admonition]: https://yozora.guanghechen.com/docs/package/tokenizer-admonition
[doc-@yozora/tokenizer-autolink]: https://yozora.guanghechen.com/docs/package/tokenizer-autolink
[doc-@yozora/tokenizer-autolink-extension]:
https://yozora.guanghechen.com/docs/package/tokenizer-autolink-extension
[doc-@yozora/tokenizer-blockquote]: https://yozora.guanghechen.com/docs/package/tokenizer-blockquote
[doc-@yozora/tokenizer-break]: https://yozora.guanghechen.com/docs/package/tokenizer-break
[doc-@yozora/tokenizer-delete]: https://yozora.guanghechen.com/docs/package/tokenizer-delete
[doc-@yozora/tokenizer-emphasis]: https://yozora.guanghechen.com/docs/package/tokenizer-emphasis
[doc-@yozora/tokenizer-fenced-code]:
https://yozora.guanghechen.com/docs/package/tokenizer-fenced-code
[doc-@yozora/tokenizer-heading]: https://yozora.guanghechen.com/docs/package/tokenizer-heading
[doc-@yozora/tokenizer-html-block]: https://yozora.guanghechen.com/docs/package/tokenizer-html-block
[doc-@yozora/tokenizer-html-inline]:
https://yozora.guanghechen.com/docs/package/tokenizer-html-inline
[doc-@yozora/tokenizer-image]: https://yozora.guanghechen.com/docs/package/tokenizer-image
[doc-@yozora/tokenizer-image-reference]:
https://yozora.guanghechen.com/docs/package/tokenizer-image-reference
[doc-@yozora/tokenizer-indented-code]:
https://yozora.guanghechen.com/docs/package/tokenizer-indented-code
[doc-@yozora/tokenizer-inline-code]:
https://yozora.guanghechen.com/docs/package/tokenizer-inline-code
[doc-@yozora/tokenizer-inline-math]:
https://yozora.guanghechen.com/docs/package/tokenizer-inline-math
[doc-@yozora/tokenizer-link]: https://yozora.guanghechen.com/docs/package/tokenizer-link
[doc-@yozora/tokenizer-definition]: https://yozora.guanghechen.com/docs/package/tokenizer-definition
[doc-@yozora/tokenizer-link-reference]:
https://yozora.guanghechen.com/docs/package/tokenizer-link-reference
[doc-@yozora/tokenizer-list]: https://yozora.guanghechen.com/docs/package/tokenizer-list
[doc-@yozora/tokenizer-math]: https://yozora.guanghechen.com/docs/package/tokenizer-math
[doc-@yozora/tokenizer-paragraph]: https://yozora.guanghechen.com/docs/package/tokenizer-paragraph
[doc-@yozora/tokenizer-setext-heading]:
https://yozora.guanghechen.com/docs/package/tokenizer-setext-heading
[doc-@yozora/tokenizer-table]: https://yozora.guanghechen.com/docs/package/tokenizer-table
[doc-@yozora/tokenizer-text]: https://yozora.guanghechen.com/docs/package/tokenizer-text
[doc-@yozora/tokenizer-thematic-break]:
https://yozora.guanghechen.com/docs/package/tokenizer-thematic-break
[doc-@yozora/jest-for-tokenizer]: https://yozora.guanghechen.com/docs/package/jest-for-tokenizer
[doc-@yozora/parser-gfm]: https://yozora.guanghechen.com/docs/package/parser-gfm
[gfm-atx-heading]: https://github.github.com/gfm/#atx-heading
[gfm-autolink]: https://github.github.com/gfm/#autolinks
[gfm-autolink-extension]: https://github.github.com/gfm/#autolinks-extension-
[gfm-blockquote]: https://github.github.com/gfm/#block-quotes
[gfm-bullet-list]: https://github.github.com/gfm/#bullet-list
[gfm-delete]: https://github.github.com/gfm/#strikethrough-extension-
[gfm-emphasis]: https://github.github.com/gfm/#can-open-emphasis
[gfm-fenced-code]: https://github.github.com/gfm/#fenced-code-block
[gfm-hard-line-break]: https://github.github.com/gfm/#hard-line-break
[gfm-html-block]: https://github.github.com/gfm/#html-block
[gfm-html-inline]: https://github.github.com/gfm/#raw-html
[gfm-image]: https://github.github.com/gfm/#images
[gfm-image-reference]: https://github.github.com/gfm/#example-590
[gfm-indented-code]: https://github.github.com/gfm/#indented-code-block
[gfm-inline-code]: https://github.github.com/gfm/#code-span
[gfm-link]: https://github.github.com/gfm/#inline-link
[gfm-definition]: https://github.github.com/gfm/#link-reference-definition
[gfm-link-reference]: https://github.github.com/gfm/#reference-link
[gfm-list]: https://github.github.com/gfm/#lists
[gfm-list-item]: https://github.github.com/gfm/#list-items
[gfm-list-task-item]: https://github.github.com/gfm/#task-list-items-extension-
[gfm-paragraph]: https://github.github.com/gfm/#paragraph
[gfm-setext-heading]: https://github.github.com/gfm/#setext-heading
[gfm-soft-line-break]: https://github.github.com/gfm/#soft-line-breaks
[gfm-strong]: https://github.github.com/gfm/#can-open-strong-emphasis
[gfm-tab]: https://github.github.com/gfm/#tabs
[gfm-table]: https://github.github.com/gfm/#table
[gfm-text]: https://github.github.com/gfm/#soft-line-breaks
[gfm-thematic-break]: https://github.github.com/gfm/#thematic-break
<!-- :end -->