@selfage/datastore_client/README.md

Provides a type-safe client library to interact with Google Cloud Datastore.

# @selfage/datastore_client

## Install

`npm install @selfage/datastore_client`

## Overview

Written in TypeScript and compiled to ES6 with inline source map & source. See [@selfage/tsconfig](https://www.npmjs.com/package/@selfage/tsconfig) for full compiler options. Provides type-safe Google Cloud Datastore APIs as a thin layer on top of `@google-cloud/datastore`, though requiring `DatastoreModelDescriptor`s and `QueryBuilder`s generated from `@selfage/generator_cli`.

You are encouraged to understand how Datastore works essentially before using this lib.

## Example generated code

See [@selfage/generator_cli#datastore-client](https://github.com/selfage/generator_cli#datastore-client) for how to generate `DatastoreModelDescriptor`s and `QueryBuilder`s. Suppose the following has been generated and committed as `task_model.ts`. We will continue using the example below.

```TypeScript
import { DatastoreQuery, DatastoreFilter, DatastoreModelDescriptor } from '@selfage/datastore_client/model_descriptor';
import { Task, TASK } from './task'; // Also generated from @selfage/generator_cli.

export let TASK_MODEL: DatastoreModelDescriptor<Task> = {
  name: "Task",
  key: "id",
  excludedIndexes: ["id", "payload"],
  valueDescriptor: TASK,
}

export class TaskDoneQueryBuilder {
  private datastoreQuery: DatastoreQuery<Task> = {
    modelDescriptor: TASK_MODEL,
    filters: new Array<DatastoreFilter>(),
    orderings: [
      {
        fieldName: "created",
        descending: true
      },
      {
        fieldName: "priority",
        descending: false
      },
    ]
  };

  public start(cursor: string): this {
    this.datastoreQuery.startCursor = cursor;
    return this;
  }
  public limit(num: number): this {
    this.datastoreQuery.limit = num;
    return this;
  }
  public equalToDone(value: boolean): this {
    this.datastoreQuery.filters.push({
      fieldName: "done",
      fieldValue: value,
      operator: "=",
    });
    return this;
  }
  public build(): DatastoreQuery<Task> {
    return this.datastoreQuery;
  }
}

export class TaskDoneSinceQueryBuilder {
  private datastoreQuery: DatastoreQuery<Task> = {
    modelDescriptor: TASK_MODEL,
    filters: new Array<DatastoreFilter>(),
    orderings: [
      {
        fieldName: "created",
        descending: true
      },
      {
        fieldName: "priority",
        descending: false
      },
    ]
  };

  public start(cursor: string): this {
    this.datastoreQuery.startCursor = cursor;
    return this;
  }
  public limit(num: number): this {
    this.datastoreQuery.limit = num;
    return this;
  }
  public equalToDone(value: boolean): this {
    this.datastoreQuery.filters.push({
      fieldName: "done",
      fieldValue: value,
      operator: "=",
    });
    return this;
  }
  public greaterThanCreated(value: number): this {
    this.datastoreQuery.filters.push({
      fieldName: "created",
      fieldValue: value,
      operator: ">",
    });
    return this;
  }
  public build(): DatastoreQuery<Task> {
    return this.datastoreQuery;
  }
}
```

## Create DatastoreClient

You can simply create a `DatastoreClient` with default Datastore configuration, which assumes you are running under Google Cloud environment, e.g., on a Compute Engine. Or pass in your own configured `Datastore` instance. See `@google-cloud/datastore` for their documents.

```TypeScript
import { DatastoreClient } from '@selfage/datastore_client';
import { Datastore } from '@google-cloud/datastore';

let client = DatastoreClient.create();
let client2 = new DatastoreClient(new Datastore());
```

## Save values

To save values, you can use `DatastoreClient`'s `save()`, which takes a generated `DatastoreModelDescriptor`, e.g. `TASK_MODEL`. The name of the model `Task` and the `id` field, because of `"key": "id"`, will be used together as the Datastore key, which also means you have to populate `id` field ahead of time. Note that only fields that are used by queries are indexed, and the rest are excluded.

```TypeScript
import { DatastoreClient } from '@selfage/datastore_client';
import { TASK_MODEL } from './task_model'; // Generated by @selfage/generator_cli.
import { Task, Priority } from './task'; // Generated by @selfage/generator_cli.

async function main(): void {
  let client = DatastoreClient.create();
  // Nothing is returned by save().
  await client.save([{
    id: '12345',
    payload: 'some params',
    done: false,
    priority: Priority.HIGH,
    created: 162311234
  }],
  TASK_MODEL,
  // Can also be 'update' or 'upsert'. See Datastore's doc for what they do.
  'insert');
}
```

Note that the `id` field is stripped and converted to Datastore key when saving. If you inspect your Datastore dashboard/console, or query directly from Datastore, you should expect the `id` field to not be set. The `id` field will be populated if you use get/query method described below.

## Allocate keys/ids

Because we have to populate `id` field (or whatever field you specified for `"key": ...`) before saving, you can either use a your own random number generator or use `DatastoreClient`'s `allocateKeys()`.

```TypeScript
import { DatastoreClient } from '@selfage/datastore_client';
import { TASK_MODEL } from './task_model'; // Generated by @selfage/generator_cli.
import { Task, Priority } from './task'; // Generated by @selfage/generator_cli.

async function main(): void {
  let client = DatastoreClient.create();
  // The `id` field will be populated in the returned `values`.
  let values = await client.allocateKeys([{
    payload: 'some params',
    done: false,
    priority: Priority.HIGH,
    created: 162311234
  }], TASK_MODEL);
}
```

Note the field for key has to be of `string` type and thus we will always store Datastore key as `[kind, name]`. This decision is opinionated that we don't have to struggle with number vs string when coding, reading or debugging.

Datastore actually allocate ids as int64 numbers, but JavaScript's number cannot be larger than 2^53. Therefore the response from Datastore is actually a 10-based string. We here further convert it to a base64 string to save a bit storage.

## Get values

Getting values is straightforward with a list of `id`.

```TypeScript
import { DatastoreClient } from '@selfage/datastore_client';
import { TASK_MODEL } from './task_model'; // Generated by @selfage/generator_cli.
import { Task, Priority } from './task'; // Generated by @selfage/generator_cli.

async function main(): void {
  let client = DatastoreClient.create();
  let values = await client.get(['12345', '23456'], TASK_MODEL);
}
```

## Query with QueryBuilder

`QueryBuilder`s are generated from `"queries": ...` field. Each of them is named as `${query's name}QueryBuilder`, and with `${operator name}${captalized field name}()` function(s) which takes a `value` with proper type as its only argument.

```TypeScript
import { DatastoreClient } from '@selfage/datastore_client';
import { TASK_MODEL, TaskDoneQueryBuilder } from './task_model'; // Generated by @selfage/generator_cli.

async function main(): void {
  let client = DatastoreClient.create();
  let taskDoneQuery = new TaskDoneSinceQueryBuilder()
    .equalToDone(true)
    .greaterThanCreated(1000100100)
    // .start(cursor) if you have one to use.
    .limit(10)
    .build();
  let {values, cursor} = await client.query(taskDoneQuery);
}
```

Note that you need to update the generated `index.yaml` to Datastore to build those indexes first.

Because query order has already been specified in `queries` field, you only need to set the values to filter by. And you MUST set all filters, otherwise Datastore might complain the lack of a corresponding composite index.

## Delete values

Simply providing a list of `id`.

```TypeScript
import { DatastoreClient } from '@selfage/datastore_client';
import { TASK_MODEL } from './task_model'; // Generated by @selfage/generator_cli.
import { Task, Priority } from './task'; // Generated by @selfage/generator_cli.

async function main(): void {
  let client = DatastoreClient.create();
  await client.delete(['12345', '23456'], TASK_MODEL);
}
```

## Transaction

`DatastoreClient` also acts as a factory to create transactions, which then can do all operations above but in a transaction. Finally you'd need to commit it.

```TypeScript
import { DatastoreClient } from '@selfage/datastore_client';

async function main(): void {
  let client = DatastoreClient.create();
  let transaction = await client.startTransaction();
  // await transaction.save([{}], TASK_MODEL, 'insert');
  // let values = await transaction.allocateKeys([{}], TASK_MODEL);
  // let values = await transaction.get(['12345', '23456'], TASK_MODEL);
  // await client.delete(['12345', '23456'], TASK_MODEL);
  // let {values, cursor} = await transaction.query(taskDoneQuery);
  await transaction.commit();
}
```

## Design considerations

We choose to define `datastore` field inside `message` because any change of `message` must be also reflected in the generated `DatastoreModelDescriptor` and `QueryBuilder` in one PR/git commit, to make sure fields are properly indexed. Otherwise, they might not be excluded from indexing or composite indexes might need to be back-filled.