terriajs/architecture/0005-root-group-ids-and-shareKeys.md

Geospatial data visualization platform.

# 5. Root group IDs, `shareKeys` and share link compatibility

Date: 2020-12-03  
Version: 3

## Status

Accepted

## Context

### Glossary

- **v7 JSON catalog**: An old v7 catalog that lives in a JSON file, typically specified in a TerriaMap's config.json.
- **v8 JSON catalog**: Like an old v7 catalog, but v8.
- **v8 JSON catalog loaded from Magda (Magda v8 JSON)**: A v8 json catalog sitting in the map-config record for that map in Magda.
- **Magda catalog**: Each catalog item and group is a magda record.
- **Root group**: The group that contains all catalog members - it is essentially the Data Catalog.

### Autogenerated IDs (autoIDs)

Most maps (v7 and v8) use autogenerated `id` for catalog members - these are used if catalog members don't have an `id` explicitly defined.

The root group `id` is very important for autoIDs. By default v8 uses `"/"` and v7 uses `"Root Group"`

- v7 ID has format `Root Group/$someContainerId/$someLowerContainerId/$catalogName`
- v8 ID has format `//$someContainerId/$someLowerContainerId/$catalogName`

Using autoIDs solidifies catalog - you can't move items around without changing IDs, which will break share links.

### Catalog converter

Currently, catalog converter will transfer `id` (only if explicitly defined - i.e. **not autogenerated**) from v7 to v8, and it doesn't write anything to `shareKeys`.

There is a PR - https://github.com/TerriaJS/catalog-converter/pull/28 - to add optional random ID generation.

#### Share conversion (using catalog-converter)

When we convert v7 share JSON to v8 - if shared models are using v7 autoIDs (they don't have an `id` defined) we convert v7 autoIDs to v8 autoIDs ([see `catalong-converter` share assumptions ADR](https://github.com/TerriaJS/catalog-converter/blob/master/architecture/0001-share-conversion-assumptions.md))

## v8 Map deployment types - ID handling

- v8 JSON catalog maps (TerriaMap)
- Magda map-config (v8 JSON)
- Magda map-config (Magda catalog)

### v8 JSON catalog maps (TerriaMap)

If `ids` aren't defined for catalog members, `terriajs` will use autoIDs outlined above.

### Magda map-config

Previously, if a map used a Magda config, the root group ID was `"$magda-config-record-id"`.

As a quick fix, we forced this Magda root group ID to be `"/"` instead of `$magda-config-record-id`. This fixed sharing for Magda JSON/initURL based maps, but broke sharing for Magda catalog maps (not JSON based) - eg digital twins.

#### Magda v8 JSON

Magda map-config maps can still use static v8 JSON catalogs (similar to TerriaMap).

autoIDs are used in the same way. But, if we are using `"$magda-config-record-id"` root group ID, we would see **all** autoIDs change:

- `//$someContainerId/$someLowerContainerId/$catalogName` will become
- `$magda-config-record-id/$someContainerId/$someLowerContainerId/$catalogName`

Thus, breaking share links.

#### Magda catalog maps (not v8 JSON)

Magda records for each catalog member will always have an explicit ID - for example `41f21ec3-95f8-429a-a62b-44a6183034aa`. Therefore, the changes in autoID outlined above don't apply.

### ShareKeys

Any change in catalog member IDs across Terria maps will break share links. To overcome this we have `shareKeys`, which acts as a string Map from previous IDs to the current ID.

For example (using Magda v8 JSON):

- If `//$someContainerId/$someLowerContainerId/$catalogName` ID is now
- `$magda-config-record-id/$someContainerId/$someLowerContainerId/$catalogName`

We must set set `shareKeys` to:

- `shareKeys=["//$someContainerId/$someLowerContainerId/$catalogName"]`

**AND** we must also set `shareKeys` for all parent items...

- `$magda-config-record-id/$someContainerId/$someLowerContainerId` -  
  `shareKeys=["//$someContainerId/$someLowerContainerId"]`

- `$magda-config-record-id/$someContainerId` - `shareKeys=["//$someContainerId"]`

- `$magda-config-record-id` - `shareKeys=["/"]`

### Up/Downgrading maps

It is common to move maps between deployment types:

- v7 JSON catalog
- v8 JSON catalog
- Magda v7-v8 converted JSON-based catalog`*`
- Magda v8 JSON-based catalog`*`
- Magda v8 catalog

`*` indicates on-the-fly conversion - i.e. catalog JSON file is untouched.

These are possible transformations:

- v7 JSON catalog -> v8 JSON catalog
- v7 JSON catalog -> Magda v7-v8 converted JSON-based catalog
- v8 JSON catalog -> Magda v8 JSON-based catalog
- v8 JSON catalog -> Magda v8 catalog
- Magda v8 catalog -> v8 JSON catalog

#### v7 JSON catalog -> v8 JSON catalog

Approaches:

1. Catalog converter with autoIDs (no IDs are written to catalog JSON)
   - Share links converted on the fly using v7 autoIDs -> v8 autoIDs
   - `shareKeys` not required unless catalog structure changes (which would change v8 autoIDs)
2. Catalog converter with new random IDs for each item
   - `shareKeys` must contain v7 autoID to preserve share links
   - Can't re-run catalog converter, as you will get new random IDs
   - Can change catalog structure without breaking share links

#### v7 JSON catalog -> Magda v7-v8 converted JSON-based catalog

This deployment uses a v7 JSON catalog, but converts it to v8 on-the-fly (in `terria-js`).

The approach is similar to "v7 JSON catalog -> v8 JSON catalog".

#### v8 JSON catalog -> Magda v8 JSON-based catalog

Depends on Magda Reference root group ID.

- If we keep it as `/`, then the catalog will behave the same as a v8 JSON catalog

- If Magda Reference root group ID is reverted to `$magda-config-record-id`, autoIDs will change for every catalog member, therefore `shareKeys` will have to be added to every catalog member to preserve shares (for v8 and/or v7 autoIDs).

#### v8 JSON catalog -> Magda v8 catalog

When converting to a "Magda catalog", each catalog member is given a new magda record with a new random ID. Therefore, all members will need `shareKeys` with v8 autoIDs (or previous catalog IDs if they were defined) **AND** v7 autoIDs (if applicable).

Changes in Magda Reference root group ID have no effect.

#### Magda v8 catalog -> v8 JSON catalog

If magda record IDs are copied into v8 catalog, then share links should work.

#### Notes on combining transformations

A common scenario will be v7 JSON catalog -> Magda v7-v8 converted JSON-based catalog -> v8 JSON catalog -> Magda v8 JSON-based catalog -> Magda v8 catalog

If we use `$magda-config-record-id`, then we may have 3 `shareKeys` for each catalog item to deal with, as will have to add `shareKeys` for the following transformations:

- v8 JSON catalog -> Magda v8 JSON-based catalog
  - v8 autoIDs
  - v7 autoIDs
- Magda v8 JSON-based catalog -> Magda v8 catalog
  - v8 autoIDs with `$magda-config-record-id` root group ID

If we use `/`, then we will only need to deal with adding `shareKeys` for:

- Magda v8 JSON-based catalog -> Magda v8 catalog
  - v8 autoIDs
  - v7 autoIDs

## Decision 1: Magda Reference root group ID

We have two options:

- Keeping magda root group as `/`
- Reverting to `$magda-config-record-id`

### Reverting Magda forced group ID

#### Sharekeys for catalog members

`shareKeys` will need to be added for the following transformations:

- v8 JSON catalog -> Magda v8 JSON-based catalog
- v8 JSON catalog -> Magda v8 catalog

This is because iff we revert this, then **ALL** autoIDs for catalog items will change:

- `//$someContainerId/$someLowerContainerId/$catalogName` will become
- `$magda-config-record-id/$someContainerId/$someLowerContainerId/$catalogName`

This also means that if we chain transformations - for example:

- v7 JSON catalog
- -> Magda v7-v8 converted JSON-based catalog
- -> v8 JSON catalog
- -> Magda v8 JSON-based catalog
- -> Magda v8 catalog

We will have 3 `shareKeys` to maintain:

- v7 autoIDs: `Root Group/$someContainerId/$someLowerContainerId/$catalogName`
- Old magda JSON based autoIDs: `$magda-config-record-id/$someContainerId/$someLowerContainerId/$catalogName`

Also, if we ever change the `$magda-config-record-id` - this will result in a whole new set of IDs that need to be added to `shareKeys` (for Magda JSON-based maps).

In PR - **Unmerged** - https://github.com/TerriaJS/terriajs/pull/5042

#### Catalog-converter

We must then:

- Stop converting autoIDs to v8 in sharelinks - we will keep them as v7 autoIDs.
- Add `shareKeys` with v7 autoIDs.

#### Add `shareKeys` for Magda `map-config` root group

We will have to handle share links which have been generated since the magda root group was changed to `/`. To do so, we must add `"shareKeys": ["/"]` to the root group:

This can be added in the `terria` aspect of the Magda map config record:

```json
{
  "aspects": {
    "terria": {
      "id": "map-config-de-australia",
      "shareKeys": [
        "/"
      ],
      "type": "group"
    },
    ...
  },
  "id": "map-config-de-australia",
  ...
}
```

In PR - **Unmerged** - https://github.com/TerriaJS/terriajs/pull/5042

### Leaving magda root group ID as `/`

Why do we need `$magda-config-record-id` as the root group ID?

#### Catalog-converter

Leaving `"/"` is very convenient - it means we don't need `shareKeys` with our current approach of converting share links (converting v7 autoIDs to v8).

#### Sharekeys for catalog members

No `shareKeys` are needed except for v8 JSON catalog -> Magda v8 catalog transformation

#### Add `shareKeys` for Magda `map-config` root group

To fix sharelinks which used `$magda-config-record-id` - we need to add `shareKeys=["$magda-config-record-id"]` in the `terria` aspect of the magda map config record (similar to above).

## Decision 2: Catalog converter ID handling

### Sharekeys

Regardless of previous decision, should we still write v7 autoID into `shareKeys`?

This would ensure old v7 share-links continue to work if catalog structure changes.

### Random ID generation default

Should we default to writing random IDs to encourage ID usage in catalogs?

If we do, we must also add `shareKeys` for v7 autoIDs.

The whole automatic ID generation thing is painful.

## Decisions Summary

### Decision 1: Magda Reference root group ID

| option                           | pros                                                                                                             | cons                                                                                                                                        |
| -------------------------------- | ---------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------- |
| Revert to `$magda-map-config-id` | Can identify map-config from sharelinks.<br/> Support multiple root groups?                                      | Need to add `shareKeys` for Magda autoIDs (with `$magda-map-config-id` root group ID).<br/> Have to deal with magda map config IDs changing |
| Leave as `/`                     | All JSON based catalogs are treated the same.<br/>Don't need to add `shareKeys` until we move into Magda catalog | Reliance on converting v7 autoIDs to v8 is a little opaque (this can be overcome by adding random `id` when converting catalog)             |

Option 2 is a go!

### Decision 2: Catalog converter ID handling

| option                                               | pros                                                                                                                                    | cons                                                                 |
| ---------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------- |
| Write v7 autoID `shareKeys`                          | Can change catalog structure and preserve old v7 share links.<br/>Can remove opaque autoID conversion in share link conversion          | Makes catalog JSON less readable, may be confusing to some users.    |
| Create random IDs by default for all catalog members | Encourages usage of proper IDs. <br/>Can remove autoID conversion.<br/>Prevents changing IDs with `$magda-map-config-id` root group ID. | Makes catalog JSON less readable.<br/>Requires v7 autoID `shareKeys` |

Option 1: yes + yes for default

Option 2: yes + give warning to users to not change IDs

## Consequences

- We will have to add a `shareKey` for the root group of Magda catalog maps - to translate `$magda-map-config-id` to `/`.
- No longer able to identify maps from `$magda-map-config-id` root group ID.
- Users may misunderstand importance of keeping IDs stable when converting catalog
- `shareKeys` (and `ids`) may clutter JSON catalog
- v7-v8 share link conversion will only work if v8 catalog has `shareKeys`
  - The v7-v8 sharelink autoID conversion is going to be removed
- There will be more unseen consequences to come...