rotating-file-stream/README.md

Opens a stream.Writable to a file rotated by interval and/or size. A logrotate alternative.

# rotating-file-stream

[![Build Status][travis-badge]][travis-url]
[![Code Climate][code-badge]][code-url]
[![Test Coverage][cover-badge]][code-url]

[![NPM version][npm-badge]][npm-url]
[![NPM downloads][npm-downloads-badge]][npm-url]
[![Stars][stars-badge]][stars-url]

[![Types][types-badge]][npm-url]
[![Dependents][deps-badge]][npm-url]
[![Donate][donate-badge]][donate-url]

[code-badge]: https://codeclimate.com/github/iccicci/rotating-file-stream/badges/gpa.svg
[code-url]: https://codeclimate.com/github/iccicci/rotating-file-stream
[cover-badge]: https://codeclimate.com/github/iccicci/rotating-file-stream/badges/coverage.svg
[deps-badge]: https://img.shields.io/librariesio/dependents/npm/rotating-file-stream?logo=npm
[deps-url]: https://www.npmjs.com/package/rotating-file-stream?activeTab=dependents
[donate-badge]: https://img.shields.io/static/v1?label=donate&message=bitcoin&color=blue&logo=bitcoin
[donate-url]: https://blockchain.info/address/12p1p5q7sK75tPyuesZmssiMYr4TKzpSCN
[github-url]: https://github.com/iccicci/rotating-file-stream
[npm-downloads-badge]: https://img.shields.io/npm/dw/rotating-file-stream?logo=npm
[npm-badge]: https://img.shields.io/npm/v/rotating-file-stream?color=green&logo=npm
[npm-url]: https://www.npmjs.com/package/rotating-file-stream
[stars-badge]: https://img.shields.io/github/stars/iccicci/rotating-file-stream?logo=github&style=flat&color=green
[stars-url]: https://github.com/iccicci/rotating-file-stream/stargazers
[travis-badge]: https://img.shields.io/travis/com/iccicci/rotating-file-stream?logo=travis
[travis-url]: https://app.travis-ci.com/github/iccicci/rotating-file-stream
[types-badge]: https://img.shields.io/static/v1?label=types&message=included&color=green&logo=typescript

### Description

Creates a [stream.Writable](https://nodejs.org/api/stream.html#stream_class_stream_writable) to a file which is
rotated. Rotation behavior can be deeply customized; optionally, classical UNIX **logrotate** behavior can be used.

### Usage

```javascript
const rfs = require("rotating-file-stream");
const stream = rfs.createStream("file.log", {
  size: "10M", // rotate every 10 MegaBytes written
  interval: "1d", // rotate daily
  compress: "gzip" // compress rotated files
});
```

### Installation

With [npm](https://www.npmjs.com/package/rotating-file-stream):

```sh
$ npm install --save rotating-file-stream
```

### Table of contents

- [Upgrading from v2 to v3](#upgrading-from-v2-to-v3)
- [Upgrading from v1 to v2](#upgrading-from-v1-to-v2)
- [API](#api)
  - [rfs.createStream(filename[, options])](#rfscreatestreamfilename-options)
    - [filename](#filename)
      - [filename(time[, index])](#filenametime-index)
      - [filename(index)](#filenameindex)
  - [Class: RotatingFileStream](#class-rotatingfilestream)
    - [Event: 'external'](#event-external)
    - [Event: 'history'](#event-history)
    - [Event: 'open'](#event-open)
    - [Event: 'removed'](#event-removed)
    - [Event: 'rotation'](#event-rotation)
    - [Event: 'rotated'](#event-rotated)
    - [Event: 'warning'](#event-warning)
  - [options](#options)
    - [compress](#compress)
    - [encoding](#encoding)
    - [history](#history)
    - [immutable](#immutable)
    - [initialRotation](#initialrotation)
    - [interval](#interval)
    - [intervalBoundary](#intervalboundary)
    - [intervalUTC](#intervalutc)
    - [maxFiles](#maxfiles)
    - [maxSize](#maxsize)
    - [mode](#mode)
    - [omitExtension](#omitextension)
    - [path](#path)
    - [rotate](#rotate)
    - [size](#size)
    - [teeToStdout](#teeToStdout)
- [Rotation logic](#rotation-logic)
- [Under the hood](#under-the-hood)
- [Compatibility](#compatibility)
- [TypeScript](#typescript)
- [License](#license)
- [Bugs](#bugs)
- [ChangeLog](#changelog)
- [Donating](#donating)

# Upgrading from v2 to v3

In **v3** the package was completely refactored using **async / await**.

**TypeScript** types for events and the [external](#event-external) event were added.

**Breaking change**: by default the `.gz` extension is added to the rotated compressed files.

**Breaking change**: the way the _external compression command_ is executed was slightly changed; possible breaking
change.

To maintain back compatibility upgrading from **v2** to **v3**, just follow this rules:

- using a _file name generator_ or not using [`options.compress`](#compress): nothing to do
- using a _file name_ and using [`options.compress`](#compress): use [`options.omitExtension`](#omitextension) or check
  how rotated files are treated.

# Upgrading from v1 to v2

There are two main changes in package interface.

In **v1** the _default export_ of the package was directly the **RotatingFileStream** _constructor_ and the caller
have to use it; while in **v2** there is no _default export_ and the caller should use the
[createStream](#rfscreatestreamfilename-options) exported function and should not directly use
[RotatingFileStream](#class-rotatingfilestream) class.
This is quite easy to discover: if this change is not applied, nothing than a runtime error can happen.

The other important change is the removal of option **rotationTime** and the introduction of **intervalBoundary**.
In **v1** the `time` argument passed to the _filename generator_ function, by default, is the time when _rotation job_
started, while if [`options.interval`](#interval) option is used, it is the lower boundary of the time interval within
_rotation job_ started. Later I was asked to add the possibility to restore the default value for this argument so I
introduced `options.rotationTime` option with this purpose. At the end the result was something a bit confusing,
something I never liked.
In **v2** the `time` argument passed to the _filename generator_ function is always the time when _rotation job_
started, unless [`options.intervalBoundary`](#intervalboundary) option is used. In a few words, to maintain back
compatibility upgrading from **v1** to **v2**, just follow this rules:

- using [`options.rotation`](#rotation): nothing to do
- not using [`options.rotation`](#rotation):
  - not using [`options.interval`](#interval): nothing to do
  - using [`options.interval`](#interval):
    - using `options.rotationTime`: to remove it
    - not using `options.rotationTime`: then use [`options.intervalBoundary`](#intervalboundary).

# API

```javascript
const rfs = require("rotating-file-stream");
```

## rfs.createStream(filename[, options])

- `filename` [<string>](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Data_structures#String_type) |
  [<Function>](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Function) The name
  of the file or the function to generate it, called _file name generator_. See below for
  [details](#filename-stringfunction).
- `options` [<Object>](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Object)
  Rotation options, See below for [details](#options).
- Returns: [<RotatingFileStream>](#class-rotatingfilestream) The **rotating file stream**!

This interface is inspired to
[fs.createWriteStream](https://nodejs.org/api/fs.html#fs_fs_createwritestream_path_options) one. The file is rotated
following _options_ rules.

### filename

The most complex problem about file name is: _how to call the rotated file name?_

The answer to this question may vary in many forms depending on application requirements and/or specifications.
If there are no requirements, a `string` can be used and _default rotated file name generator_ will be used;
otherwise a `Function` which returns the _rotated file name_ can be used.

**Note:**
if part of returned destination path does not exists, the rotation job will try to create it.

#### filename(time[, index])

- `time` [<Date>](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Date)

  - By default: the time when rotation job started;
  - if both [`options.interval`](#interval) and [`intervalBoundary`](#intervalboundary) options are enabled: the start
    time of rotation period.

  If `null`, the _not-rotated file name_ must be returned.

- `index` [<number>](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Data_structures#Number_type) The
  progressive index of rotation by size in the same rotation period.

An example of a complex _rotated file name generator_ function could be:

```javascript
const pad = num => (num > 9 ? "" : "0") + num;
const generator = (time, index) => {
  if (!time) return "file.log";

  var month = time.getFullYear() + "" + pad(time.getMonth() + 1);
  var day = pad(time.getDate());
  var hour = pad(time.getHours());
  var minute = pad(time.getMinutes());

  return `${month}/${month}${day}-${hour}${minute}-${index}-file.log`;
};

const rfs = require("rotating-file-stream");
const stream = rfs.createStream(generator, {
  size: "10M",
  interval: "30m"
});
```

**Note:**
if both of [`options.interval`](#interval) [`options.size`](#size) are used, returned _rotated file name_ **must** be
function of both arguments `time` and `index`.

#### filename(index)

- `index` [<number>](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Data_structures#Number_type) The
  progressive index of rotation. If `null`, the _not-rotated file name_ must be returned.

If classical **logrotate** behavior is enabled (by [`options.rotate`](#rotate)), _rotated file name_ is only a
function of `index`.

## Class: RotatingFileStream

Extends [stream.Writable](https://nodejs.org/api/stream.html#stream_class_stream_writable). It should not be directly
used. Exported only to be used with `instanceof` operator and similar.

### Event: 'external'

- `stdout` [<string>](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Data_structures#String_type) The
  standard output of the external compression command.
- `stderr` [<string>](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Data_structures#String_type) The
  standard error of the external compression command.

The `external` event is emitted once an _external compression command_ completes its execution to give access to the
command output streams.

### Event: 'history'

The `history` event is emitted once the _history check job_ is completed.

### Event: 'open'

- `filename` [<string>](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Data_structures#String_type) Is
  constant unless [`options.immutable`](#immutable) is `true`.

The `open` event is emitted once the _not-rotated file_ is opened.

### Event: 'removed'

- `filename` [<string>](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Data_structures#String_type) The
  name of the removed file.
- `number` [<boolean>](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Data_structures#Boolean_type)
  - `true` if the file was removed due to [`options.maxFiles`](#maxFiles)
  - `false` if the file was removed due to [`options.maxSize`](#maxSize)

The `removed` event is emitted once a _rotated file_ is removed due to [`options.maxFiles`](#maxFiles) or
[`options.maxSize`](#maxSize).

### Event: 'rotation'

The `rotation` event is emitted once the _rotation job_ is started.

### Event: 'rotated'

- `filename` [<string>](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Data_structures#String_type) The
  _rotated file name_ produced.

The `rotated` event is emitted once the _rotation job_ is completed.

### Event: 'warning'

- `error` [<Error>](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Error) The
  non blocking error.

The `warning` event is emitted once a non blocking error happens.

## options

- [`compress`](#compress):
  [<boolean>](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Data_structures#Boolean_type) |
  [<string>](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Data_structures#String_type) |
  [<Function>](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Function)
  Specifies compression method of rotated files. **Default:** `null`.
- [`encoding`](#encoding):
  [<string>](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Data_structures#String_type)
  Specifies the default encoding. **Default:** `'utf8'`.
- [`history`](#history):
  [<string>](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Data_structures#String_type)
  Specifies the _history filename_. **Default:** `null`.
- [`immutable`](#immutable):
  [<boolean>](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Data_structures#Boolean_type)
  Never mutate file names. **Default:** `null`.
- [`initialRotation`](#initialRotation):
  [<boolean>](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Data_structures#Boolean_type)
  Initial rotation based on _not-rotated file_ timestamp. **Default:** `null`.
- [`interval`](#interval):
  [<string>](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Data_structures#String_type)
  Specifies the time interval to rotate the file. **Default:** `null`.
- [`intervalBoundary`](#intervalBoundary):
  [<boolean>](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Data_structures#Boolean_type)
  Makes rotated file name with lower boundary of rotation period. **Default:** `null`.
- [`intervalUTC`](#intervalutc):
  [<boolean>](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Data_structures#Boolean_type)
  Boundaries for rotation are computed in UTC. **Default:** `null`.
- [`maxFiles`](#maxFiles):
  [<number>](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Data_structures#Number_type)
  Specifies the maximum number of rotated files to keep. **Default:** `null`.
- [`maxSize`](#maxSize):
  [<string>](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Data_structures#String_type)
  Specifies the maximum size of rotated files to keep. **Default:** `null`.
- [`mode`](#mode):
  [<number>](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Data_structures#Number_type)
  Forwarded to [fs.createWriteStream](https://nodejs.org/api/fs.html#fs_fs_createwritestream_path_options).
  **Default:** `0o666`.
- [`omitExtension`](#omitextension):
  [<boolean>](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Data_structures#Boolean_type)
  Omits the `.gz` extension from compressed rotated files. **Default:** `null`.
- [`path`](#path):
  [<string>](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Data_structures#String_type)
  Specifies the base path for files. **Default:** `null`.
- [`rotate`](#rotate):
  [<number>](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Data_structures#Number_type)
  Enables the classical UNIX **logrotate** behavior. **Default:** `null`.
- [`size`](#size):
  [<string>](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Data_structures#String_type)
  Specifies the file size to rotate the file. **Default:** `null`.
- [`teeToStdout`](#teeToStdout):
  [<boolean>](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Data_structures#Boolean_type)
  Writes file content to `stdout` as well. **Default:** `null`.

### encoding

Specifies the default encoding that is used when no encoding is specified as an argument to
[stream.write()](https://nodejs.org/api/stream.html#stream_writable_write_chunk_encoding_callback).

### mode

Forwarded to [fs.createWriteStream](https://nodejs.org/api/fs.html#fs_fs_createwritestream_path_options).

### path

If present, it is prepended to generated file names as well as for history file.

### teeToStdout

If `true`, it makes the file content to be written to `stdout` as well. Useful for debugging purposes.

### size

Accepts a positive integer followed by one of these possible letters:

- **B**: Bites
- **K**: KiloBites
- **M**: MegaBytes
- **G**: GigaBytes

```javascript
  size: '300B', // rotates the file when size exceeds 300 Bytes
                // useful for tests
```

```javascript
  size: '300K', // rotates the file when size exceeds 300 KiloBytes
```

```javascript
  size: '100M', // rotates the file when size exceeds 100 MegaBytes
```

```javascript
  size: '1G', // rotates the file when size exceeds a GigaByte
```

### interval

Accepts a positive integer followed by one of these possible letters:

- **s**: seconds. Accepts integer divider of 60.
- **m**: minutes. Accepts integer divider of 60.
- **h**: hours. Accepts integer divider of 24.
- **d**: days. Accepts integer.
- **M**: months. Accepts integer. **EXPERIMENTAL**

```javascript
  interval: '5s', // rotates at seconds 0, 5, 10, 15 and so on
                  // useful for tests
```

```javascript
  interval: '5m', // rotates at minutes 0, 5, 10, 15 and so on
```

```javascript
  interval: '2h', // rotates at midnight, 02:00, 04:00 and so on
```

```javascript
  interval: '1d', // rotates at every midnight
```

```javascript
  interval: '1M', // rotates at every midnight between two distinct months
```

### intervalBoundary

If set to `true`, the argument `time` of _filename generator_ is no longer the time when _rotation job_ started, but
the _lower boundary_ of rotation interval.

**Note:**
this option has effect only if [`options.interval`](#interval) is used.

### intervalUTC

If set to `true`, the boundaries of the rotation interval are computed against UTC time rather than against system time
zone.

**Note:**
this option has effect only if [`options.intervalBoundary`](#intervalboundary) is used.

### compress

The best choice here is to use the value `"gzip"` to use **Node.js** internal compression library.

For historical reasons external compression can be used.

To enable external compression, a _function_ can be used or simply the _boolean_ `true` value to use default
external compression.
The function should accept `source` and `dest` file names and must return the shell command to be executed to
compress the file.
The two following code snippets have exactly the same effect:

```javascript
var rfs = require("rotating-file-stream");
var stream = rfs.createStream("file.log", {
  size: "10M",
  compress: true
});
```

```javascript
var rfs = require("rotating-file-stream");
var stream = rfs.createStream("file.log", {
  size: "10M",
  compress: (source, dest) => `cat ${source} | gzip -c9 > ${dest}`
});
```

**Note:**
this option is ignored if [`options.immutable`](#immutable) is used.

**Note:**
the shell command to compress the rotated file should not remove the source file, it will be removed by the package
if rotation job complete with success.

### omitExtension

From **v3** the package adds by default the `.gz` extension to the rotated compressed files. Simultaneously this option
was added: set this option to `true` to not add the extension, i.e. to keep backward compatibility.

### initialRotation

When program stops in a rotation period then restarts in a new rotation period, logs of different rotation period will
go in the next rotated file; in a few words: a rotation job is lost. If this option is set to `true` an initial check
is performed against the _not-rotated file_ timestamp and, if it falls in a previous rotation period, an initial
rotation job is done as well.

**Note:**
this option has effect only if both [`options.interval`](#interval) and [`options.intervalBoundary`](#intervalboundary)
are used.

**Note:**
this option is ignored if [`options.rotate`](#rotate) is used.

### rotate

If specified, classical UNIX **logrotate** behavior is enabled and the value of this option has same effect in
_logrotate.conf_ file.

**Note:**
if this option is used following ones take no effect: [`options.history`](#history), [`options.immutable`](#immutable),
[`options.initialRotation`](#initialrotation), [`options.intervalBoundary`](#intervalboundary),
[`options.maxFiles`](#maxfiles), [`options.maxSize`](#maxsize).

### immutable

If set to `true`, names of generated files never changes. New files are immediately generated with their rotated
name. In other words the _rotated file name generator_ is never called with a `null` _time_ argument unless to
determinate the _history file_ name; this can happen if [`options.history`](#history) is not used while
[`options.maxFiles`](#maxfiles) or [`options.maxSize`](#maxsize) are used.
The `filename` argument passed to [`'open'`](#event-open) _event_ evaluates now as the newly created file name.

Useful to send logs to logstash through [filebeat](https://www.elastic.co/beats/filebeat).

**Note:**
if this option is used, [`options.compress`](#compress) is ignored.

**Note:**
this option is ignored if [`options.interval`](#interval) is not used.

### history

Due to the complexity that _rotated file names_ can have because of the _filename generator function_, if number or
size of rotated files should not exceed a given limit, the package needs a file where to store this information. This
option specifies the name _history file_. This option takes effect only if at least one of
[`options.maxFiles`](#maxfiles) or [`options.maxSize`](#maxsize) is used. If `null`, the _not rotated filename_ with
the `'.txt'` suffix is used.

### maxFiles

If specified, it's value is the maximum number of _rotated files_ to be kept.

### maxSize

If specified, it's value must respect same syntax of [option.size](#size) and is the maximum size of _rotated files_ to
be kept.

# Rotation logic

Regardless of when and why rotation happens, the content of a single
[stream.write](https://nodejs.org/api/stream.html#stream_writable_write_chunk_encoding_callback)
will never be split among two files.

## by size

Once the _not-rotated_ file is opened first time, its size is checked and if it is greater or equal to
size limit, a first rotation happens. After each
[stream.write](https://nodejs.org/api/stream.html#stream_writable_write_chunk_encoding_callback),
the same check is performed.

## by interval

The package sets a [Timeout](https://nodejs.org/api/timers.html#timers_settimeout_callback_delay_args)
to start a rotation job at the right moment.

# Under the hood

Logs should be handled so carefully, so this package tries to never overwrite files.

At stream creation, if the _not-rotated_ log file already exists and its size exceeds the rotation size,
an initial rotation attempt is done.

At each rotation attempt a check is done to verify that destination rotated file does not exists yet;
if this is not the case a new destination _rotated file name_ is generated and the same check is
performed before going on. This is repeated until a not existing destination file name is found or the
package is exhausted. For this reason the _rotated file name generator_ function could be called several
times for each rotation job.

If requested through [`options.maxFiles`](#maxfiles) or [`options.maxSize`](#maxsize), at the end of a rotation job, a
check is performed to ensure that given limits are respected. This means that
**while rotation job is running both the limits could be not respected**. The same can happen till the end of first
_rotation job_ if [`options.maxFiles`](#maxfiles) or [`options.maxSize`](#maxsize) are changed between two runs.
The first check performed is the one against [`options.maxFiles`](#maxfiles), in case some files are removed, then the
check against [`options.maxSize`](#maxsize) is performed, finally other files can be removed. When
[`options.maxFiles`](#maxfiles) or [`options.maxSize`](#maxsize) are enabled for first time, an _history file_ can be
created with one _rotated filename_ (as returned by _filename generator function_) at each line.

Once an **error** _event_ is emitted, nothing more can be done: the stream is closed as well.

# Compatibility

Requires **Node.js v14**.

The package is tested under [all Node.js versions](https://app.travis-ci.com/github/iccicci/rotating-file-stream)
currently supported accordingly to [Node.js Release](https://github.com/nodejs/Release#readme).

To work with the package under Windows, be sure to configure `bash.exe` as your _script-shell_.

```
> npm config set script-shell bash.exe
```

# TypeScript

**TypeScript** types are distributed with the package itself.

# License

[MIT License](https://github.com/iccicci/rotating-file-stream/blob/master/LICENSE)

# Bugs

Do not hesitate to report any bug or inconsistency [@github](https://github.com/iccicci/rotating-file-stream/issues).

# ChangeLog

[ChangeLog](https://github.com/iccicci/rotating-file-stream/blob/master/CHANGELOG.md)

# Donating

If you find useful this package, please consider the opportunity to donate some satoshis to this bitcoin address:
**12p1p5q7sK75tPyuesZmssiMYr4TKzpSCN**