@cavai/adonis-queue
Version:
Basic AdonisJS queue provider
310 lines (225 loc) • 8.15 kB
Markdown
# Adonis Queue
> Queue for AdonisJS 6
NPM: https://www.npmjs.com/package/@cavai/adonis-queue
Github: https://github.com/cavai-research/Adonis-Queue
## Why queue?
When application grows big there might be CPU heavy tasks that should be thrown
off from main event loop, long-running tasks outside of HTTP context or certain
jobs should be executed at certain time, then queueing jobs is perfect solution
Adonis queue provider makes using queues a lot easier and cleaner
## About queue
It provides database based queue to run jobs and works out-of-box with Lucid
Supports running jobs in queue, retries, scheduling, re-scheduling, basic
failure tracking and delaying job execution
## Prerequisites
- [Adonis Lucid](https://docs.adonisjs.com/guides/database/introduction) to be installed and configured
## Installation
- Install package `npm install /adonis-queue`
- Generate base files `node ace configure /adonis-queue`
- app/jobs/example_job.ts
- config/queue.ts
- database/migrations/TIMESTAMP_adonis_queue.ts
- Run database migrations `node ace migration:run`
## Configuration
Queue configuration file is in `config/queue.ts` Most parameters are documented
in there
### Jobs themselves
Jobs have multiple configurable parameters
- `retries` - Nr of times job is re-tried before it is marked as failed,
`defaults to 0`
- `retryAfter` - Delay for retries in seconds, so other jobs get chance to run,
`defaults to 5` sec
- `classPath` - Filesystem path to job class, defaults to
`app.relativePath(fileURLToPath(new URL('./', import.meta.url)))`
### Driver - database
Database driver is currently only driver, others (memory, Redis etc.) will be
added soon.
It offers great persistence, is good to use both in dev and in production
environments and powerful enough for most applications.
There's no need for extra deployments to manage queue supportive tooling like
Redis, Kafka, MemCache etc
Deployments are easy as just restarting queue. Since NodeJS caches files to memory it's needed to restart to load in latest job classes from last release
## Usage
Usage is relatively similar to events
### Creating jobs
New job can be made with `node ace make:job JobName`, where `JobName` is name of
the job, for example `node ace make:job EncodeVideo` which will be made to
`app/Jobs`
Job can be made or moved manually wherever. Might have to change `classPath`
accordingly if default dynamic one fails to work
Example job:
```ts
import { fileURLToPath } from 'node:url'
import app from '/core/services/app'
import { BaseJob } from '/adonis-queue'
export default class ExampleJob extends BaseJob {
/**
* Nr of times job is re-tried before it is marked as failed
*/
// static retries = 0
/**
* Delay for retries in seconds, so other jobs get chance to run
*/
// static retryAfter = 5
/**
* Filesystem path to job class
*/
static classPath = app.relativePath(fileURLToPath(new URL('./', import.meta.url)))
/**
* Jobs accept additional payload that can be typed for easier usage
*/
constructor(public payload: { name: string; id: number; signup_date: Date }) {
super()
}
/**
* Job handler function, write your own code in here
*/
async handle() {
// Code...
}
}
```
### Dispatching job
Once job class is made, and it's ready to be used, you can **dispatch** job to
queue up for execution
```js
// First have to import job we want to queue up
import MailJob from '#jobs/mail_job'
// And then dispatch it with optional payload
const job = await MailJob.dispatch({
name: '123',
id: 123,
signup_date: new Date(),
})
console.log(job) // { id: 7902 }
```
Awaiting dispatch does **not** wait for execution, it waits for job to be stored
to queue
`.dispatch()` accepts payload that will be accessible in job class `handle()`
method. They will be added to class instance, so in current example `name` will
be accessible with `this.payload.name`
Job dispatch returns object with job ID, which can be later used for job progress tracking
### Delaying job
Job execution can be delayed with `.delay(NOT_BEFORE_TIME)`
```ts
// First have to import job we want to queue up
import MailJob from '#jobs/mail_job'
// Import DateTime from Luxon for easier date management
import { DateTime } from 'luxon'
// Dispatch job and delay it's execution for one day
await MailJob.dispatch().delay(DateTime.now().plus({ days: 1 }))
// You can also specify date as string
// This job won't execute before given date
await MailJob.dispatch().delay('2025-02-24 15:30:00')
```
### Start the queue
> It's not needed for sync memory queue, since it will use share NodeJS event
> loop with main application
It's good to start queue as separate process, especially in production
environment, to not block main event loop
To start (default) queue run `node ace queue:start` and keep it running in the background
Or in case of multiple queues can specify which queue to start by providing Ace
command a name `node ace queue:start mailQueue`, will start mailQueue
### Scaling queue
To scale queue to multiple runners, just start new instances with
`node ace queue:start`. Jobs are locked by default, so there's no worry about
multiple runners picking up same job
### Extending with custom drivers
Let's say we want to have driver for testing that never runs jobs,
just deletes them as soon as they are dispatched
First off, need to create new queue driver, for that let's create `providers/never_queue/driver.ts`
This driver needs to extend abstract `QueueDriver` to ensure everything works correctly
```ts
// never_queue/driver.ts
import { QueueDriver } from '/adonis-queue/types'
export class NeverQueueDriver extends QueueDriver {
/**
* Do nothing, NeverQueue will never store any jobs
*/
async store() {
console.log('Stored nothing')
}
/**
* Just return null, since there is never going to be job to return
*/
async getNext() {
return null
}
/**
* Always return null, since there are no jobs
*/
async getJob() {
return null
}
/**
* Keeping on with never having jobs theme
*/
async reSchedule() {}
/**
* Do nothing
*/
async markFailed() {}
/**
* Do nothing
*/
async remove() {}
}
```
Now also need to make provider, that loads in this new driver
`node ace make:provider never_queue/queue_provider`
Inside provider `register()` method we are going to register our own driver
```ts
// never_queue/queue_provider.ts
import { DriversCollection } from '/adonis-queue'
import { ApplicationService } from '/core/types'
import { NeverQueueDriver } from './driver.js'
export default class NeverQueueProvider {
constructor(protected app: ApplicationService) {}
register() {
// Register your own bindings
DriversCollection.extend('never', () => {
// TS error, solution below
return new NeverQueueDriver()
})
}
}
```
Even tho our new queue is working now and we can configure it to be used inside `config/queue.ts`, we are going to get some TypeScript errors, about `never` queue not acceptable queue
> Argument of type '"never"' is not assignable to parameter of type '"database"'.ts(2345)
To fix that, need to create TS typings file: `contracts/queue.ts`
```ts
// contracts/queue.ts
// Importing in new queue
import { NeverQueueDriver } from '#providers/never_queue/driver'
declare module '/adonis-queue' {
export interface QueueDriverList {
// Appending it to drivers list and naming it
never: () => NeverQueueDriver
}
}
```
Now last thing to do is to update config inside `config/queue.ts` adding new queue to there with it's custom driver
```ts
/**
* Queue configuration file, all queue config variables are in here
*/
import { defineConfig } from '/adonis-queue'
export default defineConfig({
/**
* Which driver to use by default
*/
default: 'neverQueue',
queues: {
database: {
...
},
// Added new custom driver to here
neverQueue: {
driver: 'never',
},
},
})
```
## TODO:
- Add memory queue
- Add more tests