atomic-saga
Version:
A comprehensive npm package for ensuring atomic API operations in distributed Node.js applications using Saga patterns, compensating transactions, and idempotent operations
413 lines (322 loc) • 17.2 kB
Markdown
# atomic-saga
### ✨ Implement rock-solid, distributed transactions in Node.js with confidence.
[](https://www.google.com/search?q=https://www.npmjs.com/package/atomic-saga)
[](https://www.google.com/search?q=https://travis-ci.com/ankitsharma97/atomic-saga)
[](https://www.google.com/search?q=https://github.com/ankitsharma97/atomic-saga/blob/main/LICENSE)
[](https://www.google.com/search?q=https://www.npmjs.com/package/atomic-saga)
Building reliable applications with microservices is hard. How do you handle a business transaction that spans multiple services, like processing a payment, updating inventory, and sending a confirmation email? If one step fails, how do you prevent leaving your system in an inconsistent state?
**`atomic-saga`** provides a complete, production-ready toolkit to solve these challenges using proven architectural patterns. It empowers you to build resilient and reliable distributed systems on Node.js.
## 🤔 Why `atomic-saga`?
- ✅ **Production-Ready Patterns**: Implements the Saga, Transactional Outbox, and Idempotency patterns right out of the box, saving you from building complex infrastructure from scratch.
- 👨💻 **Developer-Friendly API**: A clean, fluent, and strongly-typed API makes defining, executing, and monitoring complex workflows straightforward.
- 🧩 **Flexible & Extensible**: Bring your own storage and messaging systems. The package provides interfaces for databases (PostgreSQL, MongoDB) and message brokers (Kafka, RabbitMQ), allowing you to integrate with your existing stack.
- 🛡️ **Built for Reliability**: With features like automatic rollbacks (compensations), configurable retries, and atomic event publishing, you can handle failures gracefully and ensure data consistency across services.
## 📖 Table of Contents
- [Features](https://www.google.com/search?q=%23-features)
- [Installation](https://www.google.com/search?q=%23-installation)
- [Quick Start](https://www.google.com/search?q=%23-quick-start)
- [Core Concepts](https://www.google.com/search?q=%23-core-concepts)
- [Saga Pattern (Orchestration)](https://www.google.com/search?q=%23saga-pattern-orchestration)
- [Compensating Transactions](https://www.google.com/search?q=%23compensating-transactions)
- [Transactional Outbox](https://www.google.com/search?q=%23transactional-outbox-pattern)
- [API Usage](https://www.google.com/search?q=%23-api-usage)
- [Idempotent APIs](https://www.google.com/search?q=%23-idempotent-apis)
- [Transactional Outbox](https://www.google.com/search?q=%23-transactional-outbox)
- [Configuration](https://www.google.com/search?q=%23-configuration)
- [Monitoring & Observability](https://www.google.com/search?q=%23-monitoring--observability)
- [Store Implementations](https://www.google.com/search?q=%23%EF%B8%8F-store-implementations)
- [Testing](https://www.google.com/search?q=%23-testing)
- [Best Practices](https://www.google.com/search?q=%23-best-practices)
- [Contributing](https://www.google.com/search?q=%23-contributing)
- [License](https://www.google.com/search?q=%23-license)
## ✨ Features
- **Saga Orchestration**: Define, execute, and monitor complex business workflows as a sequence of steps.
- **Automatic Compensation**: If any step fails, the Saga automatically runs compensating actions to roll back previous steps.
- **Configurable Retries**: Robust retry policies with exponential backoff for transient failures.
- **Idempotent API Middleware**: A simple Express middleware to make your API endpoints idempotent, preventing duplicate operations.
- **Transactional Outbox**: Guarantees that events are published if and only if the corresponding database transaction succeeds.
- **Reliable Event Delivery**: A background processor ensures outbox messages are reliably delivered to your message broker.
- **Full TypeScript Support**: Strongly typed from end to end for superior developer experience and fewer runtime errors.
## 📦 Installation
```bash
npm install atomic-saga
```
## 🚀 Quick Start
### 1\. Initialization
First, create an instance of `AtomicApiOperations` with your chosen storage and logging implementations. For development, you can use the provided in-memory stores.
```typescript
import { AtomicApiOperations, InMemoryIdempotencyStore, InMemorySagaStore } from 'atomic-saga';
// Use your preferred logger (e.g., Winston)
const logger = console;
// Configure the package with storage implementations
const atomicApi = new AtomicApiOperations({
idempotencyStore: new InMemoryIdempotencyStore(),
sagaStore: new InMemorySagaStore(),
logger: logger
});
```
### 2\. Define a Saga
Define the steps of your business transaction. Each step has an `action` to perform the work and an optional `compensation` to undo it.
```typescript
import { SagaDefinition } from 'atomic-saga';
const paymentSaga: SagaDefinition = {
id: 'payment-processing',
name: 'Payment Processing Saga',
steps: [
{
id: 'deduct-payment',
name: 'Deduct Payment from Account',
action: async (context) => {
// 1. Call your payment service
const result = await paymentService.deduct(context.amount, context.userId);
return { transactionId: result.id }; // Pass output to the next step or compensation
},
compensation: async (context, actionOutput) => {
// 1a. If a later step fails, refund the payment
await paymentService.refund(actionOutput.transactionId);
},
retryPolicy: { maxAttempts: 3, backoffMs: 1000, backoffMultiplier: 2 }
},
{
id: 'update-inventory',
name: 'Update Inventory',
action: async (context) => {
// 2. Call your inventory service
await inventoryService.reserve(context.productId, context.quantity);
return { inventoryReserved: true };
},
compensation: async (context) => {
// 2a. Release the inventory reservation
await inventoryService.release(context.productId, context.quantity);
}
},
{
id: 'send-confirmation',
name: 'Send Confirmation Email',
action: async (context) => {
// 3. Send a confirmation email
await emailService.sendConfirmation(context.userId, context.orderId);
return { emailSent: true };
}
// No compensation needed, as we usually don't "un-send" an email.
}
],
onSuccess: async (context) => {
logger.log(`Saga [${paymentSaga.id}] completed successfully for order ${context.orderId}.`);
},
onFailure: async (context, error) => {
logger.error(`Saga [${paymentSaga.id}] failed for order ${context.orderId}:`, error);
}
};
```
### 3\. Execute the Saga
Run the Saga with the required initial data. The orchestrator will manage the entire flow.
```typescript
async function processOrder() {
const executionContext = {
amount: 100.00,
userId: 'user-123',
productId: 'prod-456',
quantity: 2,
orderId: 'order-789'
};
const result = await atomicApi.executeSaga(paymentSaga, executionContext);
console.log('Saga execution finished with status:', result.status); // e.g., 'COMPLETED' or 'COMPENSATED'
}
```
## 🏗️ Core Concepts
### Saga Pattern (Orchestration)
This package implements the **Saga Orchestration** pattern, where a central coordinator manages a distributed transaction.
1. **Start**: A client requests to start a Saga.
2. **Execute Step**: The orchestrator executes the first step's `action`.
3. **Continue**: If the step succeeds, it moves to the next one.
4. **Failure & Compensation**: If any step fails, the orchestrator executes the `compensation` function for all previously completed steps, in reverse order.
5. **State Management**: The state of the Saga is persisted, allowing it to be resumed after a crash.
### Compensating Transactions
A compensating transaction is an operation that semantically reverses the effect of a previous step. It's the key to achieving "all or nothing" behavior.
- **Action**: `paymentService.deduct(amount)`
- **Compensation**: `paymentService.refund(transactionId)`
Your compensation logic should be idempotent and designed to succeed even in failure scenarios.
### Transactional Outbox Pattern
This pattern ensures atomic updates between a database and a message broker. It prevents a common distributed system failure: the database commit succeeds, but the message to notify other services fails to send.
1. **Atomic Operation**: The business data (e.g., `User` record) and the event (`UserUpdated` message) are saved to the database in the *same transaction*. The event is stored in a dedicated `outbox` table.
2. **Background Publisher**: A separate, reliable process polls the `outbox` table for unpublished events.
3. **Publish & Mark**: It publishes the events to a message broker (like Kafka or RabbitMQ) and, upon success, marks them as `published` in the database to prevent duplicates.
## 🔄 API Usage
### Idempotent APIs
Protect your `POST` or `PUT` endpoints from duplicate requests caused by client retries or network issues.
#### Express Middleware
```typescript
import express from 'express';
import { AtomicApiOperations } from 'atomic-saga';
const app = express();
const atomicApi = new AtomicApiOperations(/* ... config ... */);
// Add the middleware to your app.
// It automatically checks for the 'X-Idempotency-Key' header.
app.use(atomicApi.getIdempotencyMiddleware());
app.post('/api/payments', async (req, res) => {
// This operation will now be idempotent. If a request with the same
// idempotency key arrives, the middleware will return the cached response.
const paymentResult = await processPayment(req.body);
res.json(paymentResult);
});
```
#### Client Usage
The client must generate a unique key and send it in the header.
```typescript
// Generate a unique key for the operation
const idempotencyKey = atomicApi.generateIdempotencyKey(); // Or use a UUID library
// Send the key in the request header
await fetch('/api/payments', {
method: 'POST',
headers: {
'Content-Type': 'application/json',
'X-Idempotency-Key': idempotencyKey,
},
body: JSON.stringify(paymentData),
});
```
### 📨 Transactional Outbox
Use the outbox to atomically save data and publish an event.
```typescript
// 1. Configure AtomicApiOperations with an OutboxStore and MessageBroker
const atomicApi = new AtomicApiOperations({
// ... other stores
outboxStore: new PostgresOutboxStore(dbConnection),
messageBroker: new KafkaMessageBroker(kafkaClient),
});
// 2. Start the background processor (typically on app startup)
atomicApi.startOutbox();
// 3. Use it in your business logic
async function updateUser(userId: string, userData: any) {
// Get a transaction handle from the outbox
const outboxTransaction = atomicApi.createOutboxTransaction();
// Execute your database logic and define the event to be published
await outboxTransaction.execute(
// A function that contains your database updates
async (transactionalClient) => {
await database.updateUser(userId, userData, { client: transactionalClient });
await database.updateRelatedData(userId, { client: transactionalClient });
},
// The event to publish upon success
{
eventType: 'UserUpdated',
payload: { userId, ...userData }
}
);
}
```
## 🔧 Configuration
### `AtomicApiConfig`
```typescript
interface AtomicApiConfig {
idempotencyStore: IdempotencyStore; // Required: Manages idempotency keys.
sagaStore: SagaStore; // Required: Persists Saga state.
outboxStore?: OutboxStore; // Optional: Required for Transactional Outbox.
messageBroker?: MessageBroker; // Optional: Required for Transactional Outbox.
logger?: Logger; // Optional: For logging. Defaults to console.
defaultRetryPolicy?: RetryPolicy; // Optional: Default retry policy for Saga steps.
}
```
### `RetryPolicy`
```typescript
interface RetryPolicy {
maxAttempts: number; // Maximum number of attempts for an action.
backoffMs: number; // Initial delay in milliseconds.
backoffMultiplier: number; // Factor to multiply delay by for each retry (e.g., 2 for exponential).
}
```
## 📊 Monitoring & Observability
### Saga Execution Tracking
```typescript
// Get details of a specific Saga execution
const execution = await atomicApi.getSagaExecution('execution-id-123');
console.log('Execution Status:', execution.status);
console.log('Step Results:', execution.stepResults);
// List all completed executions for a specific Saga
const executions = await atomicApi.listSagaExecutions({
sagaId: 'payment-processing',
status: 'COMPLETED'
});
```
### Outbox Statistics
```typescript
const stats = await atomicApi.getOutboxStats();
console.log('Pending Events:', stats.pending);
console.log('Published Events:', stats.published);
console.log('Failed Events:', stats.failed);
```
## 🛠️ Store Implementations
The package is unopinionated about your data storage. You provide the implementation by conforming to these interfaces. Example implementations for popular databases are planned for the future.
- **`IdempotencyStore`**:
- `set(key, response)`
- `get(key)`
- *Recommended Backend: Redis (for its speed and TTL support)*
- **`SagaStore`**:
- `createExecution(data)`
- `getExecution(id)`
- `updateExecution(id, data)`
- *Recommended Backend: PostgreSQL, MongoDB*
- **`OutboxStore`**:
- `add(event, transaction)`
- `getUnpublished()`
- `markAsPublished(eventId)`
- *Recommended Backend: PostgreSQL, MongoDB*
## 🧪 Testing
The package is designed to be easily testable. Use the `InMemory` stores to test your Saga logic without external dependencies.
```typescript
import { AtomicApiOperations, InMemorySagaStore, InMemoryIdempotencyStore } from 'atomic-saga';
describe('Payment Saga', () => {
let atomicApi: AtomicApiOperations;
let mockPaymentService;
beforeEach(() => {
atomicApi = new AtomicApiOperations({
idempotencyStore: new InMemoryIdempotencyStore(),
sagaStore: new InMemorySagaStore(),
});
// Mock external services
mockPaymentService = {
deduct: jest.fn().mockResolvedValue({ id: 'txn-123' }),
refund: jest.fn().mockResolvedValue(true),
};
});
it('should complete successfully when all steps succeed', async () => {
// ... setup mocks to succeed
const result = await atomicApi.executeSaga(paymentSaga, paymentContext);
expect(result.status).toBe('COMPLETED');
expect(mockPaymentService.refund).not.toHaveBeenCalled();
});
it('should compensate successfully when a step fails', async () => {
// Mock the inventory service to throw an error
mockInventoryService.reserve.mockRejectedValue(new Error('Inventory not available'));
const result = await atomicApi.executeSaga(paymentSaga, paymentContext);
expect(result.status).toBe('COMPENSATED');
expect(mockPaymentService.refund).toHaveBeenCalledWith('txn-123');
});
});
```
## 🔒 Best Practices
- **Design Idempotent Compensations**: Your compensation logic might be retried. Ensure it can be run multiple times without causing issues (e.g., don't refund twice).
- **Keep Steps Small and Focused**: Each step should ideally interact with a single service or transactional boundary.
- **Avoid Business Logic in the Orchestrator**: The Saga definition should only coordinate the steps. The actual business logic belongs in your services.
- **Monitor for Failed Compensations**: A failed compensation is a critical error that requires manual intervention. Set up alerts to detect these scenarios.
## 🤝 Contributing
Contributions are welcome\! Please feel free to fork the repository, create a feature branch, and submit a pull request.
1. Fork the repository.
2. Create your feature branch (`git checkout -b feature/amazing-feature`).
3. Commit your changes (`git commit -m 'Add some amazing feature'`).
4. Push to the branch (`git push origin feature/amazing-feature`).
5. Open a pull request.
## 🙏 Acknowledgments
This package is heavily inspired by the patterns and principles from giants in the field:
- "Saga" by Hector Garcia-Molina and Kenneth Salem
- "Patterns of Enterprise Application Architecture" by Martin Fowler
- "Building Microservices" by Sam Newman
## 📄 License
This project is licensed under the MIT License - see the [LICENSE](https://www.google.com/search?q=LICENSE) file for details.
-----
For questions, issues, or contributions:
"description": "A comprehensive npm package for ensuring atomic API operations in distributed Node.js applications using Saga patterns, compensating transactions, and idempotent operations",
- GitHub Issues: [Create an issue](https://github.com/ankitsharma97/atomic-saga/issues)
- Documentation: [Read the docs](https://github.com/ankitsharma97/atomic-saga/wiki)
- Examples: [View examples](https://github.com/ankitsharma97/atomic-sagag/examples)