smartdown/docs/lib/gallery/Stdlib.md

A library for translating, rendering and interacting with Smartdown documents. Smartdown is an extension of Markdown that provides richer media support and reactive programming capability.

## `stdlib` Experiments

[stdlib](https://stdlib.io) is a Javascript library supporting both NodeJS and web browsers. Quoting the authors:

> Stdlib is a standard library for JavaScript and Node.js, with an emphasis on numeric computing. The library provides a collection of robust, high performance libraries for mathematics, statistics, streams, utilities, and more.

The examples below are adapted from various `stdlib` documentation examples, although many uses of `console.log()` have been replaced with writing to Smartdown variables and having these variables rendered via Smartdown cells. The other minor change needed to use `stdlib` within Smartdown is to replace usage of `require('@stdlib/foo/bar')` with a reference to Smartdown's bundled version `Stdlib.foo.bar`. This is currently necessary because `stdlib` is being bundled (via Webpack) with Smartdown. Eventually, this will be replaced with a more dynamic mechanism.

The source for this example can be viewed and transiently edited at [Smartdown Stdlib Example](https://smartdown.site/?url=lib/gallery/Stdlib.md)


### CDF Experiments

Trying to plot the CDF for various $\sigma$ and $\mu$ versions of the normal distribution.

Based upon the Stdlib examples:

- https://stdlib.io/develop/docs/api/@stdlib/math/base/dists/normal/cdf
- https://stdlib.io/develop/docs/api/@stdlib/plot/ctor/


```javascript/playable
const thisDiv = this.div;

var toHTML = Stdlib.vdomToHtml;
var randn = Stdlib.random.base.boxMuller;
var plot = Stdlib.plot.ctor;
var cdf = Stdlib.math.base.dists.normal.cdf;


var x = new Float64Array( 100 );
var y1 = new Float64Array( x.length );
var y2 = new Float64Array( x.length );
var y3 = new Float64Array( x.length );
var y4 = new Float64Array( x.length );
for (var i = 0; i < x.length; i++) {
  x[ i ] = (((i + 1) * 1.0) - x.length / 2) / 25.0;
  y1[ i ] = cdf(x[i], 0, 1);
  y2[ i ] = cdf(x[i], 0, 0.5);
  y3[ i ] = cdf(x[i], 0, 0.2);
  y4[ i ] = cdf(x[i], 0, 0.1);
}

var h = plot( [x, x, x, x], [y1, y2, y3, y4], {
    yMin: -0.1,
    yMax: 1.1,
    'description': 'Plotting the CDF of the Normal Distribution',
    'title': 'CDF of the Normal Distribution for various σ',
    'labels': [
      'σ = 1',
      'σ = 0.5',
      'σ = 0.2',
      'σ = 0.1'
    ],
    lineWidth: 5
});

thisDiv.innerHTML = Stdlib.vdomToHtml( h.render() );

```

---

### Machine Learning

From this example: https://stdlib.io/develop/docs/api/@stdlib/ml/online-binary-classification

I honestly don't understand this example (yet), I've just transliterated it to Smartdown.

```javascript/playable
var binomial = Stdlib.random.base.binomial;
var normal = Stdlib.random.base.normal;
var exp = Stdlib.math.base.special.exp;
var onlineBinaryClassification = Stdlib.ml.onlineBinaryClassification;

var phat;
var lp;
var x1;
var x2;
var y;
var i;

// Create model:
var model = onlineBinaryClassification({
  'lambda': 1e-3,
  'loss': 'log',
  'intercept': true
});

// Update model as data comes in...
for ( i = 0; i < 10000; i++ ) {
  x1 = normal( 0.0, 1.0 );
  x2 = normal( 0.0, 1.0 );
  lp = (3.0 * x1) - (2.0 * x2) + 1.0;
  phat = 1.0 / ( 1.0 + exp( -lp ) );
  y = binomial( 1, phat ) ? 1.0 : -1.0;
  model.update( [ x1, x2 ], y );
}

// Extract model coefficients:
var markdownCoefficients =
`
### Model Coefficients

$$
${model.coefs}
$$
`;

// Predict new observations:
// console.log( 'Pr(Y=1)_hat = %d; x1 = %d; x2 = %d', model.predict( [0.9, 0.1], 'probability' ), 0.9, 0.1 );
// console.log( 'y_hat = %d; x1 = %d; x2 = %d', model.predict( [0.1, 0.9], 'link' ), 0.1, 0.9 );
// console.log( 'y_hat = %d; x1 = %d; x2 = %d', model.predict( [0.9, 0.9], 'link' ), 0.9, 0.9 );

const p1 = 0.9;
const p2 = 0.1;
const predictionProbability = model.predict( [p1, p2], 'probability' );
const predictionLink12 = model.predict( [p1, p2], 'link' );
const predictionLink22 = model.predict( [p1, p2], 'link' );

var markdownOutput =
`
### Output

|Expression|Value|$x_1$|$x_2$|
|:---:|---:|---:|---:|
|$\\hat{P_r(Y=1)}$|${predictionProbability}|${x1}|${x2}|
|$\\hat{y}$|${predictionLink12}|${x1}|${x2}|
|$\\hat{y}$|${predictionLink22}|${x2}|${x2}|
`;

smartdown.setVariable('MarkdownCoefficients', markdownCoefficients);
smartdown.setVariable('MarkdownOutput', markdownOutput);
```

[](:!MarkdownCoefficients|markdown)
[](:!MarkdownOutput|markdown)

---

### Unicode Sparklines

From https://stdlib.io/develop/docs/api/@stdlib/plot/sparklines/unicode/column

For this example, we modify the use of `console.log()` to instead place the Unicode sparkline into a Smartdown variable, where it will be rendered automatically as it is updated.

#### The Generated Plot
[](:!SparklinePlot)


#### The `Stdlib.plot.sparklines.unicode.column` script

```javascript/playable
var randu = Stdlib.random.base.randu;
var columnChart = Stdlib.plot.sparklines.unicode.column;

var chart;
var data;
var id;
var i;

// Generate some random data...
data = new Float64Array( 30 );
for ( i = 0; i < data.length; i++ ) {
  data[ i ] = randu() * 100.0;
}

// Create a new column chart:
chart = columnChart();

// Set the chart data:
chart.data = data;

// Configure the chart to support streaming data:
chart.window = data.length;
chart.yMin = 0.0;
chart.yMax = 100.0;

// Update the terminal chart with new data every second:
id = setInterval( update, 1000 );

// After some time, stop updating and close:
setTimeout( stop, 20000 );

function update() {
  // Update the chart with new data:
  chart.push( randu() * 100.0 );

  var rendered = chart.render();
  smartdown.setVariable('SparklinePlot', rendered);
}

function stop() {
  clearInterval( id );
}
```

---

### Plots

Adapted from https://stdlib.io/develop/docs/api/@stdlib/plot/ctor

This example generates a plot as *virtual DOM*, which is then converted to HTML prior to rendering within the Smartdown-created playable's div (`this.div`).


```javascript/playable
const thisDiv = this.div;

var toHTML = Stdlib.vdomToHtml;
var randn = Stdlib.random.base.boxMuller;
var plot = Stdlib.plot.ctor;

var now;
var x;
var y;
var i;

// Create some data...
now = ( new Date() ).getTime();
x = new Float64Array( 100 );
y = new Float64Array( x.length );
for ( i = 0; i < x.length; i++ ) {
  x[ i ] = now + (i * 360000);
  y[ i ] = 50.0 + (10.0 * randn());
}

// Create a new plot:
var h = plot( [x], [y], {
    'width': 500,
    'height': 500,
    'xScale': 'time',
    'xTickFormat': '%H:%M'
});

// Render as a virtual DOM tree:
var vtree = h.render();
// console.log( JSON.stringify( vtree ) );
smartdown.setVariable('treeJSON', vtree);

// Transform the virtual DOM tree to HTML:
var html = toHTML( vtree );
// console.log( html );
thisDiv.innerHTML = html;

```

##### The generated virtual DOM Tree

[](:!treeJSON|json)


---

### NLP (Natural Language Processing)

Derived from this example https://stdlib.io/develop/docs/api/@stdlib/nlp/lda

This version of Smartdown includes a subset (1930-2010) of the full 1790-2016 [SOTU State of the Union Dataset](https://stdlib.io/develop/docs/api/@stdlib/datasets/sotu), because the data is currently being bundled with the rest of Smartdown, and is quite large. In the future, the `stdlib` datasets will be optionally and dynamically loadable, which will reduce the default Smartdown bundle size and enable a much richer set of data to be used.

This example also uses the [English Stop Words Dataset](https://stdlib.io/develop/docs/api/@stdlib/datasets/stopwords-en).


#### Smartdown outputs

The following smartdown cells will be populated with values generated by the `stdlib` script below. They are initially empty or undefined.

##### Stopwords
[](:!stopwords)

##### State of the Union Texts (first 100 characters)
[](:!speechTexts)

##### Average $\theta$ per topic
[](:!yearsMD|markdown)

##### Top words associated with each topic
[](:!topicMD|markdown)


##### The playable `stdlib` example.

```javascript/playable
var that = this;

// Hack to keep the progress bar up while everything happens in
// this long-running script. Better would be a smartdown.setProgress()
// method that would not leak the implementation details.
//
var saveProgress = this.progress; // Oh, the hack
this.progress = null;

Stdlib.loadSOTU(function() {
  var roundn = Stdlib.math.base.special.roundn;
  var stopwords = Stdlib.datasets['stopwords-en'];
  var lda = Stdlib.nlp.lda;

  var STOPWORDS = stopwords();
  var terms;
  var model;
  var str;
  var i;

  smartdown.setVariable('stopwords', STOPWORDS);

  function getText(e) {
    function remove(word) {
      var RE = new RegExp('\\b' + word + '\\b', 'gi');
      str = str.replace(RE, '');
    }

    str = e.text.toLowerCase();
    STOPWORDS.forEach(remove);
    return str;
  }

  var startYear = 1930;
  var endYear = 2010;

  var speechTexts = null;
  var speeches = Stdlib.datasets['sotu-data'];

  speechTexts = speeches.reduce(
                      function (
                        accumulator,
                        speech,
                        currentIndex,
                        array) {
                        if (speech.year >= startYear && speech.year <= endYear) {
                          accumulator.push(getText(speech));
                        }
                        return accumulator;
                      },
                      []);

  var trimmedTexts = speeches.map(function(speech, index, array) {
    return speech.text.slice(0, 100);
  });
  smartdown.setVariable('speechTexts', trimmedTexts);

  model = lda(speechTexts, 3);

  // model.fit(1000, 100, 10);
  model.fit(100, 50, 10);

  // Safari (at least on my machine) will timeout and Chrome will give a warning
  // if we try to execute too many iterations, so I'm using smaller parameters than would
  // be used in a non-browser context. Perhaps Service Workers would alleviate this?
  //

  var yearsMD = '|Year|Topic 1 Average $\\theta$|Topic 2 Average $\\theta$|Topic 3 Average $\\theta$|\n|:---|---:|---:|---:|\n';
  for (i = 0; i <= 80; i++) {
    var year = (startYear + i);
    var theta0 = roundn(model.avgTheta.get(i, 0), -3);
    var theta1 = roundn(model.avgTheta.get(i, 1), -3);
    var theta2 = roundn(model.avgTheta.get(i, 2), -3);

    str = 'Year: ' + year + '\t';
    str += 'Topic 1: ' + theta0 + '\t';
    str += 'Topic 2: ' + theta1 + '\t';
    str += 'Topic 3: ' + theta2;
    yearsMD += `|${year}|${theta0}|${theta1}|${theta2}|\n`;
  }
  yearsMD += '\n';

  smartdown.setVariable('yearsMD', yearsMD);

  var topicMD = '|Topic|Words Most Associated with Topic|\n|:---|:---|\n';
  var trim = Stdlib.string.trim;
  var removePunctuation = Stdlib.string.removePunctuation;

  for (var whichTopic = 0; whichTopic < 3; ++whichTopic) {
    terms = model.getTerms(whichTopic, 20);
    var topicString = `|Topic ${whichTopic}||\n`;

    for (i = 0; i < terms.length; i++) {
      terms[i] = terms[i].word;
      const stripped = trim(removePunctuation(terms[i]));
      if (stripped !== '' && stripped !== '-') {
        topicString += `||${terms[i]}|\n`;
      }
    }

    var termsString = terms.join(', ');

    topicMD += topicString;
  }

  smartdown.setVariable('topicMD', topicMD);

  saveProgress.style.display = 'none';
});
```

---

### Use Graphviz to display `stdlib` functions

This is a *very quick hack* to demonstrate the synergy between Smartdown's playables, variables, and cells. It uses Graphviz to display the heirarchical structure of the `stdlib` namespace. For this example, we are only displaying the heirarchy under `stdlib.math`.

[More Graphviz examples](:@Graphviz)

[](:!plot|code)
[](:!plot|graphviz)

```javascript/playable

var index = 1;

function generateTree(rootIndex, rootfIndex, rootName, root) {
  var source =
`
"node${rootIndex}" [
    shape = "record"
    label = "`;
  var keys = Object.keys(root);
  var fIndex = 0;
  var subroot = [];

  keys.forEach(function(k) {
    ++fIndex;
    ++index;
    var v = root[k];
    var line = `  "${rootName}" -> "${k}";\n`;
    line = `<f${fIndex}> ${k}|`;
    source += line;

    if (typeof v === 'object') {
      subroot.push({
        v: v,
        k: k,
        index: index,
        fIndex: fIndex
      });
    }
  });
  source = source.slice(0, source.length - 1);
  source += '"\n];';

  subroot.forEach(function(subroot) {
    source += generateTree(subroot.index, subroot.fIndex, subroot.k, subroot.v);
    source += `\n"node${rootIndex}":f${subroot.fIndex} -> "node${subroot.index}":f0;\n`;
  });

  return source;
}

var tree = generateTree(1, 0, 'stdlib', Stdlib.math);
var plot =
`
digraph G {
  rankdir = "LR"
  ranksep = 1.5
  ratio="compact"
  node [
    fontsize = "10"
    margin = "0"
    shape = "rectangle"
  ];
  edge [
  ];
  "node0" [
    label = "<f0> math"
    shape = "record"
  ];
  ${tree}
  "node0":f0 -> "node1":f1
}
`;
smartdown.setVariable('plot', plot);
```

---


[Back to Home](:@Home)