scuttlesort/README.md

ScuttleSort - incremental convergent topological sort for Secure Scuttlebutt

# ScuttleSort -- Incremental Convergent Topological Sort for Secure Scuttlebutt (SSB)

## Synopsis

ScuttleSort transforms SSB's tangled append-only logs into a single
linear sequence. Because this sequence changes as new log entries are
incrementally added, an append-only stream of instructions ('insert'
and 'move' commands) is derived as a side effect that permits to
replicate and update copies of the sequence. ScuttleSort is convergent
which means that the resulting sequence is the same for all receivers
having seen the same set of log entries, regardless of the order in
which they process a new log entry as long as it is added in log order.


## ScuttleSort Demo

SSB's append-only logs are implemented by hash chains where each
entry in a log has a unique ID. Basically, a log entry is four-tuple
and its ID is some hash value computed over the entry's binary
representation:

```
log_entry = (author, prev, payload, signature);
id_of_log_entry = hash(log_entry);
```

where
- ```author``` is some public key, used to validate the signature
- ```prev``` is the ID of the predecessor log entry
- ```payload``` is the content of the log entry
- ```signature``` bytes to verify the authenticty and integrity of (the encoding of) all above fields

The first entry in the chain has an empty ```prev``` field. Because of
the randomness of the author key, this entry is unique, and so is its
ID, and so are the subsequent entries in the chain and their IDs. Each
author only appends to their own chain, and receives unchanged copies
of the others' logs.

```
 1st entry      2nd entry    3rd entry                 In Secure Scuttlebutt,
 .------.       .----.       .----.                    each append-only log
 |AUTH  |       |AUTH|       |AUTH|                    is implemented as an
 |PREV=0|    .--|PREV|    .--|PREV    ...              independend hash chain.
 |PAYL  |    |  |PAYL|    |  |                         Only the author can
 |SIGN  |    |  |SIGN|    |  |                         write to their chain.
 `------'    |  '----'    |  `---
   hash|     |  hash|     |
       v     |      v     |
    ID_0 ----'   ID_1 ----'    ID_2


    e0    <----   e1    <----   e2   <----   e3   ... chained log entries
```

In the demo we assume that four append-only logs (see figure below)
are involved in some joint activity where their owners need clarity
about which log entry was appended before another one. The logs only
define a partial order, namely within the log itself and in form of
cross-referenced IDs of other entries (that are carried in the
payloads). The desired "total order" cannot be determined in general,
as some event can be logically independent i.e.,
concurrent. ScuttleSort nevertheless provides such a total order
thanks to a convention how to derive the global order, and based on the
causality relationship of events.

Other total orders will exist, but ScuttleSort's aim is to construct
one that has strong eventual consistency, meaning that all observers
seeing all logs will all compute the same total order that is not
dependend on the sequence in which log entries are ingested (as long
as they are ingested as chained).

As an example, consider the following scenario with four append-only
logs (in form of hash chains) and eight highlighted entries

```
  chain 1   ...............     .-- F <-- E  ....
                               /         /
  chain 2   ...   X <-- A <-- B <-- D <-'   .....
                  ^     ^          /
  chain 3   ...    \     `--- C <-'   ...........
                    \ 
  chain 4   .....    `- Y  ......................

                                               ---> time
```

Causality is shown as arrows from effect to cause i.e., dependency. In
the example above, A depends on X as A was appended later to the chain
than X (and needed to know entry X's ID). We also say that "A points
to X" via its ```prev``` field, for example, and "D points to B" via
its ```prev``` field and contains another hash value (in its payload)
that points to C.

ScuttleSort has bascially one method:
```
add(name, after_list)
```

which adds edges to the dependency graph between a node with ID
```name``` and the names in the ```after_list``` which are hash values
of past events in other chains.

The total order is called a ```Timeline```. On creation of a Timeline
object, one can parametrize it with a callback. In these callbacks,
```insert``` and ```move``` commands are conveyed which permit to
(re-) construct the sorted array with all events added so far.

If the entries are added in the following sequence

```
let g = {
    'X': [],
    'A': ['X'],
    'F': ['B'],
    'E': ['D', 'F'],
    'B': ['A'],
    'Y': ['X'],
    'D': ['B', 'C'],
    'C': ['A']
};
```

then the stream of instructions as generated by ScuttleSort looks like this:

```
// when adding X
  [ 'ins', 'X', 0 ]
// when adding A
  [ 'ins', 'A', 1 ]
// when adding F
  [ 'ins', 'F', 0 ]
// when adding E
  [ 'ins', 'E', 3 ]
// when adding B
  [ 'ins', 'B', 4 ]
  [ 'mov', 0, 4 ]
  [ 'mov', 2, 4 ]
// when adding Y
  [ 'ins', 'Y', 2 ]
// when adding D
  [ 'ins', 'D', 4 ]
// when adding C
  [ 'ins', 'C', 4 ]
```

which corresponds to the array ```[X, A, Y, B, C, D, F, E]```.

## JavaScript Implementation

The full demo code is given below. It also includes a generator
for all possible ingestion schedules. The demo then exhaustively
invokes ScuttleSort for all schedules and each time produces
the same result, as required for strong eventual consistency:

```
  ingest
   order   resulting (total) ScuttleSort order
--------   -----------------------------------
FEXABDCY   ["X","A","Y","B","C","D","F","E"]
FEXABDYC   ["X","A","Y","B","C","D","F","E"]
FEXABCDY   ["X","A","Y","B","C","D","F","E"]
FEXABCYD   ["X","A","Y","B","C","D","F","E"]
FEXABYDC   ["X","A","Y","B","C","D","F","E"]
...
```

## Appendix: Demo Code

```
#!/usr/bin/env node

// scuttlesort/demo.js
// 2022-05-14 <christian.tschudin@unibas.ch


"use strict"

const Timeline = require("scuttlesort")

let g = {
    'X': [],
    'A': ['X'],
    'F': ['B'],
    'E': ['D', 'F'],
    'B': ['A'],
    'Y': ['X'],
    'D': ['B', 'C'],
    'C': ['A']
};

let timeline = new Timeline( (x) => { console.log(" ", x); } );

for (let n in g) {
    console.log("// when adding", n);
    timeline.add(n, g[n]);
}

console.log("\nResulting timeline: (pos, name, rank, successors)")
for (let e of timeline.linear)
    console.log(" ", e.indx, e.name, e.rank,
                e.succ.map( x => {return x.name;} ) );


let chains = [
    [ ['F',['B']], ['E',['D','F']] ],
    [ ['X',[]], ['A',['X']], ['B',['A']], ['D',['B','C']] ],
    [ ['C',['A']] ],
    [ ['Y',['X']] ]
];

function interleave(pfx, config, lst) {
    var empty = true;
    for (let i=0; i < config.length; i++) {
        if (config[i].length > 0) {
            let config2 = JSON.parse(JSON.stringify(config));
            let e = config2[i].shift();
            interleave(pfx + e[0], config2, lst);
            empty = false;
        }
    }
    if (empty) {
        lst.push(pfx);
        return;
    }
}

var lst = [];
interleave('', chains, lst);

console.log("\nRunning ScuttleSort for all", lst.length,
            "possible ingestion schedules:\n");
console.log("  ingest");
console.log("   order   resulting (total) ScuttleSort order");
console.log("--------   -----------------------------------")
for (let pfx of lst) {
    let tl2 = new Timeline();
    for (let nm of pfx) {
        tl2.add(nm, g[nm]);
    }
    console.log(pfx, " ", JSON.stringify(tl2.linear.map( x => {return x.name;} )))
}

// eof
```