pg-transactional-outbox/CHANGELOG.md

A PostgreSQL based transactional outbox and inbox pattern implementation to support exactly once message processing (with at least once message delivery).

# Change Log

All notable changes to the pg-transactional-outbox library will be documented in
this file.

## [0.6.0] - 2025-01-14

### Added

- For databases under heavy load the logical replication can notify about a new
  message that is not yet returned via a SELECT on the database. A new strategy
  `MessageNotFoundRetryStrategy` was added to decide to retry in case of a not
  found message. The config settings `maxMessageNotFoundAttempts` can be used to
  define how many retries should be done. The default is zero. The setting
  `maxMessageNotFoundDelayInMs` can be used to add a delay before another retry
  is done.

## [0.5.9] - 2024-12-03

### Changed

- Package upgrade for the cross-spawn dependency with critical vulnerability.
  This package was used in build related scripts but was not part of the
  `pg-transactional-outbox` library.

## [0.5.8] - 2024-09-18

### Changed

- Library dependencies upgraded to their latest version.

## [0.5.7] - 2024-05-10

### Changed

- Errors are now consistently logged as `Error` objects. Where possible enriched
  with additional data.

## [0.5.6] - 2024-04-23

### Added

- Exposed the `applyDefaultPollingListenerConfigValues` function.

## [0.5.5] - 2024-03-19

### Added

- When logging is enabled and the log level is set to "trace" then the client
  will track the executed queries and add them to the PostgreSQL error object.

## [0.5.4] - 2024-03-19

### Changed

- When getting a new database client from the Pool the error handler is attached
  to the retrieved PoolClient.

## [0.5.3] - 2024-03-04

### Changed

- The "best-effort" attempt to increase the message finished attempts is retried
  multiple times in case of database locking errors.

## [0.5.2] - 2024-02-29

### Changed

- When a message handling timeout happens, the message handler database
  transaction is rolled back. To ensure, that the message handler logic does not
  use the database client anymore after the rollback the client is released.

## [0.5.1] - 2024-02-28

### Changed

- The polling functions (e.g. `next_outbox_messages`) did not correctly lock the
  inbox/outbox messages. Please update the SQL functions in your projects to the
  latest version created e.g. based on the
  `examples/setup/out/example-trx-polling.sql` adjustments.

## [0.5.0] - 2024-02-01

### Added

- Support for a polling subscriber was added as an alternative to the logical
  replication listener.
- The following fields have to be added to the inbox table:

  ```sql
  ALTER TABLE public.inbox ADD COLUMN segment TEXT;
  ALTER TABLE public.inbox ADD COLUMN locked_until TIMESTAMPTZ NOT NULL DEFAULT to_timestamp(0);
  ALTER TABLE public.inbox ADD COLUMN concurrency TEXT NOT NULL DEFAULT 'sequential';
  ALTER TABLE public.inbox ADD COLUMN abandoned_at TIMESTAMPTZ;
  ALTER TABLE public.inbox ADD CONSTRAINT inbox_concurrency_check
    CHECK (concurrency IN ('sequential', 'parallel'));

  GRANT UPDATE (started_attempts, finished_attempts, processed_at, abandoned_at, locked_until) ON public.inbox TO db_inbox_handler;
  ```

- These fields must also be added to the outbox table:

  ```sql
  ALTER TABLE public.outbox ADD COLUMN segment TEXT;
  ALTER TABLE public.outbox ADD COLUMN locked_until TIMESTAMPTZ NOT NULL DEFAULT to_timestamp(0);
  ALTER TABLE public.outbox ADD COLUMN concurrency TEXT NOT NULL DEFAULT 'sequential';
  ALTER TABLE public.outbox ADD COLUMN abandoned_at TIMESTAMPTZ;
  ALTER TABLE public.outbox ADD CONSTRAINT outbox_concurrency_check
    CHECK (concurrency IN ('sequential', 'parallel'));

  GRANT UPDATE (started_attempts, finished_attempts, processed_at, abandoned_at, locked_until) ON public.outbox TO db_outbox_handler;
  ```

- Added automatic message cleanup. Please check the settings in the
  configuration to activate it and configure the times in seconds after which
  processed, abandoned, and in general messages should be deleted.

### Changed

- Aligned the outbox message handling logic to the inbox table. This allows to
  include maximum retries and poisonous message handling but slightly increases
  the time to send outbox messages.
- The following fields are now required (also) on the outbox table:
  ```sql
  ALTER TABLE public.outbox ADD COLUMN started_attempts smallint NOT NULL DEFAULT 0;
  ALTER TABLE public.outbox ADD COLUMN finished_attempts smallint NOT NULL DEFAULT 0;
  ALTER TABLE public.outbox ADD COLUMN processed_at TIMESTAMPTZ;
  ```
- Function names were renamed to not include "inbox" or "outbox" specifically
  anymore as both use the same underlying concept now. Generally, the methods
  and type names reflect now more their roles:
  - "replication" is used for the logical replication based listener approach
  - "polling" on the other hand is used now for the polling listener approach
  - "handler" is used for code that is handling the outbox or inbox message
- To get (close) to the prior outbox handling logic you can set the settings
  fields `enableMaxAttemptsProtection` and `enablePoisonousMessageProtection` to
  `false`.
- The created_at date should use the `clock_timestamp` function to accurately
  define the created date when the insert happened and not when the transaction
  was committed. This is especially important for bulk inserts to maintain the
  message sort order.  
  For the inbox table:
  ```sql
  ALTER TABLE public.inbox ALTER COLUMN created_at SET DEFAULT clock_timestamp();
  ```
  For the outbox table:
  ```sql
  ALTER TABLE public.outbox ALTER COLUMN created_at SET DEFAULT clock_timestamp();
  ```
- The discriminating controller and the multi-concurrency controller were
  changed to use the "segment" value from the messages as discriminators. This
  was done to align that usage between the polling and replication listener.

## [0.4.0] - 2024-02-01

### Changed

- BREAKING CHANGE: the outbox and inbox listeners accept now a logger instance
  to not depend on a global logger so custom names and settings can be provided
  to the outbox and inbox listener.
- BREAKING CHANGE: changed the term "service" to "listener" in multiple places
  to more accurately reflect what the code does.
- BREAKING CHANGE: renamed "attempts" to "finished_attempts" and added the
  column "started_attempts" in the transactional inbox. This was done to
  implement the poisonous message handling. Please run the following command to
  update your database (adjust the namespace to yours):

  ```sql
  ALTER TABLE public.inbox RENAME COLUMN attempts TO finished_attempts;
  ALTER TABLE public.inbox ADD COLUMN started_attempts smallint NOT NULL DEFAULT 0;
  GRANT UPDATE (started_attempts, finished_attempts, processed_at) ON public.inbox TO db_inbox_handler;
  ```

### Added

- Added an optional `strategies` object to the inbox and outbox listeners. It
  allows you to optionally define fine granular strategies on how to define
  specific logic around handling inbox and outbox messages.
- To manage the concurrency of message processing on a granular level, a
  concurrency manager can now be provided as part of the strategies for the
  outbox and inbox listener. The default will use a mutex to guarantee
  sequential message processing. There are the following pre-build ones but you
  can also write your own:
  - `createMutexConcurrencyController` - this controller guarantees sequential
    message processing.
  - `createFullConcurrencyController` - this controller allows the parallel
    processing of messages without guarantees on the processing order.
  - `createSemaphoreConcurrencyController` - this controller allows the
    processing of messages in parallel up to a configurable number.
  - `createDiscriminatingMutexConcurrencyController` - this controller enables
    sequential message processing based on a specified discriminator. This could
    be the message type or some other (calculated) value.
  - `createMultiConcurrencyController` - this is a combined concurrency
    controller. You can define for every message which from the above
    controllers the message should use.
- Messages are processed via message handlers as part of a database transaction.
  Some handlers may require a different database handler user. In this case, you
  can use the `messageProcessingDbClientStrategy` to return a database client
  from the desired database pool.
- The `messageProcessingTimeoutStrategy` allows you to define a message-based
  timeout on how long the message is allowed to be processed in milliseconds.
  This allows you to allow some more expensive messages to take longer while
  still keeping others on a short timeout. By default, it uses the configured
  messageProcessingTimeout or falls back to a 15-second timeout.
- The inbox listener lets you define the
  `messageProcessingTransactionLevelStrategy` per message. Some message
  processing logic may have higher isolation level requirements than for
  processing other messages. If no custom strategy is provided it uses the
  default database transaction level via `BEGIN`.
- Messages can fail when they are processed. The `messageRetryStrategy` allows
  you to define how often a message can be retried. And in case a message is
  (likely) a poisonous message that crashes the service you can use the
  `poisonousMessageRetryStrategy` to customize if and how often such a message
  can be retried.
- When the inbox or outbox listener fails due to an error it is restarted. The
  `listenerRestartStrategy` is used to define how long it should wait before it
  attempts to start again. It allows you to decide (based on the error) to log
  or track the caught error or try to resolve the underlying issue.

## [0.3.0] - 2023-10-23

### Changed

- BREAKING CHANGE: added support for additional "metadata" in outbox and inbox
  messages. A new database column `metadata JSONB` must be added to the inbox
  and outbox database table. This setting can hold any additional metadata e.g.
  routing information, message signature etc. Please run the following two
  commands to update your database (adjust the namespace to yours):
  ```sql
  ALTER TABLE public.inbox ADD COLUMN IF NOT EXISTS metadata JSONB;
  ALTER TABLE app_hidden.outbox ADD COLUMN IF NOT EXISTS metadata JSONB;
  ```
- BREAKING CHANGE: renamed "retries" to "attempts" for the transactional inbox
  to make it clear that "retries" include the initial attempt. Please run the
  following command to update your database (adjust the namespace to yours):
  ```sql
  ALTER TABLE app_public.inbox RENAME COLUMN retries TO attempts;
  ```

## [0.2.0] - 2023-10-26

### Changed

- BREAKING CHANGE: renamed "event type" to "message type" in the library and in
  the database columns. This was done to better transport the meaning that the
  transactional outbox and inbox can be used both for commands and events and
  not just for events. Please rename for your outbox and inbox table the
  `event_type` column to `message_type`. And in your code the message
  `eventType` field with `messageType`.

### Added

- The function `initializeGeneralOutboxMessageStorage` can now be used for a
  general outbox storage function that does not encapsulate the settings to
  store a specific message and aggregate type.

## [0.1.8] - 2023-09-22

### Changed

- Fixed an issue where "this" was not correctly bound when executing message
  handlers when they are provided as object methods.

## [0.1.7] - 2023-09-18

### Changed

- Improved published package contents to exclude unit test files.

## [0.1.6] - 2023-09-15

### Changed

- The logical replication service will now guarantee sequential message
  processing in the order how the messages were received. So far the messages
  were only started in the desired order but could finish in different order
  depending how long the message handler ran.

### Added

- Only one service can connect to the publication of a replication slot. When
  services are scaled, the first one will succeed to connect but the others will
  fail. There is now a new setting `restartDelaySlotInUse` to define the delay
  before trying to connect again if the replication slot is in use.

## [0.1.5] - 2023-09-11

### Added

- Debug log for replication start added. This way the actual start of the
  service and restarts can be tracked.

## [0.1.4] - 2023-05-15

### Changed

- Fixed an issue where messages were sometimes processed even after the maximum
  message retry for the inbox message was exceeded.

## [0.1.1 - 0.1.3] - 2023-01-28

### Changed

- Updated the readme files and referenced images.

## [0.1.0] - 2023-01-28

### Added

- Initial version of the library.