@moicky/dynamodb
Version:
Contains a collection of convenience functions for working with AWS DynamoDB
542 lines (430 loc) • 15.4 kB
Markdown
# @moicky/dynamodb
## Description
Contains convenience functions for all major dynamodb operations. Requires very little code to interact with items from aws dynamodb. Uses **aws sdk v3** and fixes several issues:
- 🎁 Will **automatically marshall and unmarshall** items
- 📦 Will **group items into batches** to avoid aws limits and improve performance
- ⏱ Will **automatically** add `createdAt` and `updatedAt` attributes on all items to track their most recent create/update operation timestamp. Example value: `Date.now() -> 1685138436000`
- 🔄 Will **retry** `getItems`, `deleteItems` **up to 3 times** on unprocessed items and `queryAllItems` until finished
- 🔒 When specifying an item using its keySchema, all additional attributes (apart from keySchema attributes from `initSchema` or `PK` & `SK` as default) will be removed to avoid errors
- 👻 Will **use placeholders** to avoid colliding with [reserved words](https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/ReservedWords.html) if applicable
- 🌎 Supports globally defined default arguments for each operation ([example](#configuring-global-defaults))
- 🔨 Supports fixes for several issues with dynamodb ([example](#applying-fixes))
- 📖 Offers a convenient way to use pagination with queries ([example](#paginated-items))
- 🗂️ Supports **transactGetItems** & **transactWriteItems**
## Installation
```bash
npm i @moicky/dynamodb
```
## Setup
Automatically grabs `DYNAMODB_TABLE` as an **environment variable** and assumes `PK` and `SK` as it's schema. Can be customized using `initSchema` with one or more tables:
```ts
import { initSchema } from "@moicky/dynamodb";
// Should be called once at the start of the runtime before any operation is executed
initSchema({
// first one will be used by default if no TableName is specified when calling functions
[process.env.DEFAULT_TABLE]: {
hash: "PK",
range: "SK",
},
[process.env.SECOND_TABLE]: {
hash: "somePK",
},
});
```
## Working with multiple tables
Every function accepts `args` which can include a `TableName` property that specifies the table and uses the keySchema from `initSchema()`
```ts
import { getItem, putItem, deleteItem } from "@moicky/dynamodb";
await putItem(
{
PK: "User/1",
someSortKey: "Book/1",
title: "The Great Gatsby",
author: "F. Scott Fitzgerald",
released: 1925,
},
{ TableName: process.env.SECOND_TABLE }
);
const item = await getItem(
{ PK: "User/1", someSortKey: "Book/1" },
{ TableName: process.env.SECOND_TABLE }
);
await deleteItem(item, { TableName: process.env.SECOND_TABLE });
```
## Usage Examples
### Put Items
Every put operation also adds the `createdAt` attribute with the current timestamp on each item.
```ts
import { putItem, putItems } from "@moicky/dynamodb";
// Put single item into dynamodb
await putItem({
PK: "User/1",
SK: "Book/1",
title: "The Great Gatsby",
author: "F. Scott Fitzgerald",
released: 1925,
});
// Put multiple items into dynamodb
await putItems([
{
PK: "User/1",
SK: "Book/1",
title: "The Great Gatsby",
author: "F. Scott Fitzgerald",
released: 1925,
},
// ... infinite more items (will be grouped into batches of 25 due to aws limit)
]);
```
### Get Items
```ts
import { getItem, getItems, getAllItems } from "@moicky/dynamodb";
// Passing more than just the key is possible, but will be removed to avoid errors
// Get single item
await getItem({
PK: "User/1",
SK: "Book/1",
title: "The Great Gatsby", // additional fields will be removed before sending
author: "F. Scott Fitzgerald",
released: 1925,
});
// Get multiple items
// Items will be grouped into batches of 100 and will be retried up to 3 times if there are unprocessed items
// Will also only request each keySchema once, even if it is present multiple times in the array to improve performance
await getItems([
{
PK: "User/1",
SK: "Book/1",
title: "The Great Gatsby", // additional fields will be removed before sending
author: "F. Scott Fitzgerald",
released: 1925,
},
// ... infinite more items (will be grouped into batches of 100 due to aws limit) and retried up to 3 times
]);
// Retrieve all items using ScanCommand
await getAllItems();
```
### Delete Items
```ts
import { deleteItem, deleteItems } from "@moicky/dynamodb";
// Delete a single item
await deleteItem({
PK: "User/1",
SK: "Book/1",
title: "The Great Gatsby", // additional fields will be removed before sending to avoid errors
author: "F. Scott Fitzgerald",
released: 1925,
});
// Delete multiple items
// Will only delete each keySchema once, even if it is present multiple times in the array to improve performance and avoid aws errors
await deleteItems([
{ PK: "User/1", SK: "Book/1" },
// ... infinite more items (will be grouped into batches of 25 due to aws limit) and retried up to 3 times
]);
```
### Update Items
Every update operation also upserts the `updatedAt` attribute with the current timestamp on each item.
```ts
import { updateItem, removeAttributes } from "@moicky/dynamodb";
// Update the item and overwrite all supplied fields
await updateItem(
{ PK: "User/1", SK: "Book/1" }, // reference to item
{ description: "A book about a rich guy", author: "F. Scott Fitzgerald" } // fields to update
);
await updateItem(
{ PK: "User/1", SK: "Book/1" },
{ released: 2000, maxReleased: 1950 }, // maxReleased will not be updated on the item, since it is referenced inside the ConditionExpression
{ ConditionExpression: "#released < :maxReleased" }
);
const newItem = await updateItem(
{ PK: "User/1", SK: "Book/1" },
{ released: 2000 },
{ ReturnValues: "ALL_NEW" }
);
console.log(newItem); // { "PK": "User/1", "SK": "Book/1", "released": 2000 }
await removeAttributes({ PK: "User/1", SK: "Book/1" }, ["description"]);
```
### Query Items
```ts
import { query, queryItems, queryAllItems } from "@moicky/dynamodb";
// You HAVE TO use placeholders for the keyCondition & filterExpression:
// Prefix the attributeNames with a hash (#) and the attributeValues with a colon (:)
// Query only using keyCondition and retrieve complete response
const booksResponse = await query("#PK = :PK and begins_with(#SK, :SK)", {
PK: "User/1",
SK: "Book/",
});
// Query and retrieve unmarshalled items array
const books = await queryItems("#PK = :PK and begins_with(#SK, :SK)", {
PK: "User/1",
SK: "Book/",
});
// Query and retry until all items are retrieved (due to aws limit of 1MB per query)
const allBooks = await queryAllItems("#PK = :PK and begins_with(#SK, :SK)", {
PK: "User/1",
SK: "Book/",
});
// Query with filterExpression (also specifiy attributes inside the key object)
const booksWithFilter = await queryAllItems(
"#PK = :PK and begins_with(#SK, :SK)", // keyCondition
{
// definition for all attributes
PK: "User/1",
SK: "Book/",
from: 1950,
to: 2000,
},
// additional args with filterExpression for example
{ FilterExpression: "#released BETWEEN :from AND :to" }
);
```
#### Paginated Items
```ts
// Pagination
const { items, hasNextPage, hasPreviousPage, currentPage } =
await queryPaginatedItems(
"#PK = :PK and begins_with(#SK, :SK)",
{ PK: "User/1", SK: "Book/" },
{ pageSize: 100 }
);
// items: The items on the current page.
// currentPage: { number: 1, firstKey: { ... }, lastKey: { ... } }
const { items: nextItems, currentPage: nextPage } = await queryPaginatedItems(
"#PK = :PK and begins_with(#SK, :SK)",
{ PK: "User/1", SK: "Book/" },
{ pageSize: 100, currentPage } // args.direction: 'next' or 'previous'
);
// items: The items on the second page.
// currentPage: { number: 2, firstKey: { ... }, lastKey: { ... } }
```
### Miscellaneous
```ts
import { itemExists, getAscendingId } from "@moicky/dynamodb";
// Check if an item exists using keySchema
const exists = await itemExists({ PK: "User/1", SK: "Book/1" });
console.log(exists); // true or false
// Generate ascending ID
// Specify Partition-Key and optionally the Sort-Key.
// Example Structure 1: PK: "User/1", SK: "{{ ASCENDING_ID }}"
// Last item: { PK: "User/1", SK: "00000009" }
const id1 = await getAscendingId({ PK: "User/1" });
console.log(id1); // "00000010"
// Example Structure 2: PK: "User/1", SK: "Book/{{ ASCENDING_ID }}"
// Last item: { PK: "User/1", SK: "Book/00000009" }
const id2 = await getAscendingId({ PK: "User/1", SK: "Book/" });
console.log(id2); // "00000010"
// Specify length of ID
const id3 = await getAscendingId({ PK: "User/1", SK: "Book/", length: 4 });
console.log(id3); // "0010"
// Example Structure 3: somePartitionKey: "User/1", SK: "Book/{{ ASCENDING_ID }}"
// Last item: { somePartitionKey: "User/1", SK: "Book/00000009" }
const id4 = await getAscendingId({
somePartitionKey: "User/1",
SK: "Book/",
});
console.log(id4); // "00000010"
```
### TransactWriteItems
```ts
import { transactWriteItems } from "@moicky/dynamodb";
// Perform a TransactWriteItems operation
const response = await transactWriteItems([
{
Put: {
item: {
PK: "User/1",
SK: "Book/1",
title: "The Great Gatsby",
author: "F. Scott Fitzgerald",
released: 1925,
},
},
},
{
Update: {
key: { PK: "User/1", SK: "Book/1" },
updateData: { title: "The Great Gatsby - Updated" },
},
},
{
Delete: {
key: { PK: "User/1", SK: "Book/1" },
},
},
{
ConditionCheck: {
key: { PK: "User/1", SK: "Book/1" },
ConditionExpression: "#title = :title",
conditionData: { title: "The Great Gatsby" },
},
},
]);
console.log(response);
```
### TransactGetItems
```ts
import { transactGetItems } from "@moicky/dynamodb";
// Perform a TransactGetItems operation
const items = await transactGetItems([
{ key: { PK: "User/1", SK: "Book/1" } },
{ key: { PK: "User/1", SK: "Book/2" } },
{ key: { PK: "User/1", SK: "Book/3" } },
]);
console.log(items);
```
## Configuring global defaults
Global defaults can be configured using the `initDefaults` function. This allows to provide but still override every property of the `args` parameter.
Should be called before any DynamoDB operations are performed.
```ts
import { initDefaultArguments, getItem } from "@moicky/dynamodb";
// This example enables consistent reads for all DynamoDB operations which support it.
initDefaultArguments({
getItem: { ConsistentRead: true },
getAllItems: { ConsistentRead: true },
itemExists: { ConsistentRead: true },
query: { ConsistentRead: true },
queryItems: { ConsistentRead: true },
queryAllItems: { ConsistentRead: true },
});
// It is still possible to override any arguments when calling a function
const itemWithoutConsistentRead = await getItem(
{ PK: "User/1", SK: "Book/001" },
{ ConsistentRead: false }
);
```
## Applying fixes
Arguments which are passed to [marshall](https://docs.aws.amazon.com/AWSJavaScriptSDK/v3/latest/interfaces/_aws_sdk_util_dynamodb.marshallOptions.html) and [unmarshall](https://docs.aws.amazon.com/AWSJavaScriptSDK/v3/latest/interfaces/_aws_sdk_util_dynamodb.unmarshallOptions.html) from `@aws-sdk/util-dynamodb` can be configured using
```ts
import { initFixes } from "@moicky/dynamodb";
initFixes({
marshallOptions: {
removeUndefinedValues: true,
},
unmarshallOptions: {
wrapNumbers: true,
},
});
```
When using `GlobalSecondaryIndexes`, DynamoDb does not support using `ConsistantRead`. This is fixed by default (`ConsistantRead` is turned off) and can be configured using:
```ts
import { initFixes } from "@moicky/dynamodb";
initFixes({
disableConsistantReadWhenUsingIndexes: {
enabled: true, // default,
// Won't disable ConsistantRead if IndexName is specified here.
// This works because DynamoDB supports ConsistantRead on LocalSecondaryIndexes
stillUseOnLocalIndexes: ["localIndexName1", "localIndexName2"],
},
});
```
## What are the benefits and why should I use it?
Generally it makes it easier to interact with the dynamodb from AWS. Here are some before and after examples using the new aws-sdk v3:
### Put
```js
const demoItem = {
PK: "User/1",
SK: "Book/1",
title: "The Great Gatsby",
author: "F. Scott Fitzgerald",
released: 1925,
};
// Without helpers:
import { DynamoDBClient, PutItemCommand } from "@aws-sdk/client-dynamodb";
import { marshall, unmarshall } from "@aws-sdk/util-dynamodb";
const client = new DynamoDBClient({
region: process.env.AWS_REGION,
});
const newItem = await client
.send(
new PutItemCommand({
TableName: process.env.DYNAMODB_TABLE,
Item: marshall(demoItem),
ReturnValues: "ALL_NEW",
})
)
.then((result) => unmarshall(result.Attributes));
// With helpers:
import { putItem } from "@moicky/dynamodb";
const newItem = await putItem(demoItem, { ReturnValues: "ALL_NEW" });
```
### Query
```js
// Without helpers:
import { DynamoDBClient, QueryCommand } from "@aws-sdk/client-dynamodb";
import { marshall, unmarshall } from "@aws-sdk/util-dynamodb";
const client = new DynamoDBClient({
region: process.env.AWS_REGION,
});
const results = await client
.send(
new QueryCommand({
TableName: process.env.DYNAMODB_TABLE,
KeyConditionExpression: "#PK = :PK and begins_with(#SK, :SK)",
ExpressionAttributeNames: {
"#PK": "PK",
"#SK": "SK",
},
ExpressionAttributeValues: {
":PK": marshall("User/1"),
":SK": marshall("Book/"),
},
})
)
.then((result) => result.Items.map((item) => unmarshall(item)));
// With helpers
import { queryItems } from "@moicky/dynamodb";
const results = await queryItems("#PK = :PK and begins_with(#SK, :SK)", {
PK: "User/1",
SK: "Book/",
});
```
### Update
```js
// Without helpers
import { DynamoDBClient, UpdateItemCommand } from "@aws-sdk/client-dynamodb";
import { marshall, unmarshall } from "@aws-sdk/util-dynamodb";
const client = new DynamoDBClient({
region: process.env.AWS_REGION,
});
const result = await client
.send(
new UpdateItemCommand({
TableName: process.env.DYNAMODB_TABLE,
Key: marshall({ PK: "User/1", SK: "Book/1" }),
UpdateExpression: "SET #released = :released, #title = :title",
ExpressionAttributeNames: {
"#released": "released",
"#title": "title",
},
ExpressionAttributeValues: marshall({
":released": 2000,
":title": "New Title",
}),
ReturnValues: "ALL_NEW",
})
)
.then((result) => unmarshall(result.Attributes));
// With helpers
import { updateItem } from "@moicky/dynamodb";
const result = await updateItem(
{ PK: "User/1", SK: "Book/1" },
{ released: 2000, title: "New Title" },
{ ReturnValues: "ALL_NEW" }
);
```
## Tests
### Setup
Requires environment variables to be present for the tests to successfully connect to dynamodb tables. You can find a list of required environment variables here:
[.env.template](.env.template)
They can be obtained using the **template.yml** which can be deployed on aws using:
```bash
sam deploy
```
Will then return the table-names as the output of the template
### Execution
Finally executing all tests:
```bash
npm run test
```