kafka-pub-sub/CHANGELOG.md

Enterprise-grade Kafka publish/subscribe library for Node.js — producer pool, batch sending, DLQ, SSL/SASL, multi-broker, and real-world industry examples.

## [2.0.0] - 2026-06-30

### ⚠ Breaking Changes

- **Consumer API changed**: `ConsumeEvent` no longer returns a one-shot Promise. It now accepts a handler callback and returns a `stop()` function. See migration guide below.
- **Env variable renamed**: `KAFKA_BROKER_URL` → `KAFKA_BROKER_URLS` (plural, comma-separated for multi-broker support).
- **Topic validation relaxed**: min length changed from 5 to 1, max from 20 to 249 (Kafka's actual hard limit). Topics must now match `[a-zA-Z0-9._-]`.
- **Event validation relaxed**: min length changed from 5 to 1, max from 20 to 100.

#### Migration: ConsumeEvent

```js
// v1
const data = await ConsumeEvent('MY_TOPIC');
console.log(data);

// v2
const stop = await ConsumeEvent('MY_TOPIC', async (msg) => {
  console.log(msg.value);
});
// Call stop() for graceful shutdown
await stop();
```

---

### Added

- **Singleton producer pool** — connection is reused across calls instead of connecting/disconnecting per message. Eliminates the most common performance bottleneck.
- **`BatchProduceEvent`** — sends multiple messages across one or more topics in a single broker round-trip via `producer.sendBatch`. Significantly more efficient for high-throughput scenarios.
- **`KafkaAdmin`** — admin client factory for programmatic topic management: `createTopics`, `deleteTopics`, `listTopics`, `getTopicMetadata`, `getConsumerGroupOffsets`, `topicsExist`.
- **`HealthCheck`** — lightweight broker probe for Kubernetes liveness/readiness endpoints. Returns broker count, cluster ID, and error details on failure.
- **Dead-letter queue (DLQ)** — failed consumer messages are automatically routed to `<topic>.dlq` after exhausting retries. Configurable per consumer via `dlq: true/false`.
- **Exponential back-off retry** — consumer handler retries with 100ms → 200ms → 400ms delays before DLQ routing.
- **SSL/TLS support** — `KAFKA_SSL_ENABLED`, `KAFKA_SSL_CA_PATH`, `KAFKA_SSL_KEY_PATH`, `KAFKA_SSL_CERT_PATH`.
- **SASL authentication** — `plain`, `scram-sha-256`, `scram-sha-512`, `oauthbearer` mechanisms. Compatible with Confluent Cloud and AWS MSK.
- **Multi-broker support** — `KAFKA_BROKER_URLS` accepts a comma-separated list for high-availability clusters.
- **Message compression** — `KAFKA_COMPRESSION_TYPE`: `none` / `gzip` / `snappy` / `lz4` / `zstd`. Overridable per message via options.
- **Idempotent producer** — `KAFKA_IDEMPOTENT=true` enables exactly-once semantics at the broker level.
- **Custom partition key** — `ProduceEvent(..., { partitionKey: 'CUST-42' })` for deterministic partition routing.
- **Correlation ID propagation** — `ProduceEvent(..., { correlationId: 'req-xyz' })` auto-adds the header.
- **`produced-at` header** — every message gets a source timestamp header.
- **Structured JSON logging** — daily-rotating log files in `logs/` with separate error log. Console output suppressed in `NODE_ENV=test`.
- **Graceful shutdown** — `SIGTERM`/`SIGINT` handlers on producer and consumer. `ProduceEvent.disconnect()` and `BatchProduceEvent.disconnect()` for manual control.
- **Configurable consumer options** — `sessionTimeout`, `heartbeatInterval`, `maxWaitTimeInMs`, `groupId` per consumer.
- **`index.js` entry point** — all modules exported from a single `require('kafka-pub-sub')`.
- **Package `exports` map** — sub-path imports supported: `require('kafka-pub-sub/ProduceEvent')`.
- **`docker-compose.yml`** — upgraded to Kafka 3.8 in KRaft mode (no ZooKeeper), with Kafka UI on port 8080.
- **Industry examples** — three self-contained runnable examples:
  - `examples/quickstart/` — minimal publish and subscribe scripts
  - `examples/ecommerce/` — order saga (placed → inventory → payment → fulfillment)
  - `examples/financial/` — payment transactions, fraud detection, DLQ review, settlement batch
  - `examples/audit/` — GDPR/SOC 2 compliance logging with 7-year retention
- **`.npmignore`** — test files, Docker, dev tooling excluded from published package.
- **`engines` field** — declares Node.js ≥ 16 requirement.

### Changed

- `winstonKafkaLogger` now uses `winston-daily-rotate-file` instead of a plain `File` transport.
- Error messages fixed: "lenght" typo corrected to "length" throughout.
- All dev dependencies updated to latest compatible versions.
- `package.json` description updated to reflect enterprise scope.
- `package.json` keywords expanded for better npm discoverability.

### Fixed

- **Critical**: Consumer `Promise.resolve()` on first message only — now runs continuously for all messages.
- **Critical**: Producer connect/disconnect on every single call — replaced with persistent singleton pool.
- Topic validation was rejecting all real-world Kafka topic names (e.g. `order.placed`, `payment.transaction`).
- Event validation was rejecting common event names shorter than 5 chars or longer than 20.

---

### [1.1.7] - 2023-10-19

#### Added

- Sample project structure image directory added

#### Changed

- Project structure image url modified in README.md

</br>

### [1.1.6] - 2023-09-03

#### Added

- Image added in README.md for test project / sample structure

</br>

### [1.1.5] - 2023-06-01

#### Added

- husky.sh added to check all committed code

</br>

### [1.1.4] - 2023-06-01

#### Added

- ESLint, Prettier code formatter and linter support
- Husky pre-commit, post-commit and post-checkout rules

#### Fixed

- Test validations fixed from previous release (v1.1.3)

</br>

### [1.1.3] - 2023-05-29

#### Added

- Kafka custom headers, headers validation, test coverage for custom headers

</br>

### [1.0.3] - 2023-05-24

#### Added

- Jest testing support
- Test examples in README.md

</br>

### [1.0.2] - 2023-05-23

#### Added

- MIT License
- npm monthly downloads badge

#### Changed

- Updated README.md example

</br>

### [1.0.0] - 2023-05-22

#### Added

- Initial kafka-pub-sub functionality