sky-buckets/README.md

NPM package to redirect file uploads from Multer to MinIO with TypeScript support.

# sky-buckets

[](https://www.google.com/search?q=https://www.npmjs.com/package/sky-buckets)

`sky-buckets` is a TypeScript-powered Node.js package designed to streamline the process of uploading files from an Express backend using Multer directly to MinIO object storage. This package efficiently redirects file data streams, eliminating the need for temporary disk storage on your server, thereby reducing disk I/O and enhancing overall performance.

## `sky-buckets` Overview

In modern web application development, handling file uploads often involves temporarily storing files on the server's local disk before moving them to a persistent storage solution like MinIO. This process can lead to significant disk I/O overhead and complexity in managing temporary files.

`sky-buckets` addresses these challenges by providing a custom Multer storage engine that streams incoming file data directly to MinIO. This means files never touch your server's disk, making it a highly efficient and scalable solution for applications dealing with high volumes of file uploads.

### Key Features

  * **Direct Upload Redirection**: Streams files directly from Multer to MinIO, bypassing temporary disk storage. [1, 2, 3]
  * **Full TypeScript Support**: Built with TypeScript for type safety, maintainability, and an improved development experience. [4, 5]
  * **Configurable Multer Middleware**: Seamless integration with Multer, allowing dynamic determination of MinIO bucket and object names based on request data. [6, 7, 8]
  * **Comprehensive Inline File Operations**: Provides a `MinioService` for programmatic file operations outside the HTTP middleware context, including:
      * Upload from Buffer [1, 9]
      * Upload from Path [10, 11, 12]
      * Upload from Base64 [13, 14]
      * Upload from Blob [15]
      * Multi-file Upload [16]
      * Download from Specific Bucket [17, 10, 11]
      * Download File to Local Path [17, 10, 11]
      * Delete single file
      * Delete multiple file

## Usage Documentation

### Installation

To install the `sky-buckets` package and all its necessary dependencies, run the following command:

```bash
npm install sky-buckets
```

If you are using TypeScript in your project and directly importing types from `express` or `multer` outside of the `sky-buckets` package, you might also need to install the following type definitions as `devDependencies` in your project:

```bash
npm install --save-dev @types/node @types/express @types/multer
```

**Note**: The `minio` library versions `7.1.0` and above ship with built-in type definitions, so `@types/minio` is no longer separately required. [17, 10, 11, 18, 19]

### Configuration

First, initialize your MinIO client. It is highly recommended to use environment variables for your MinIO credentials in production environments. [20, 11, 7]

```typescript
import * as Minio from 'minio';

const minioClient = new Minio.Client({
  endPoint: process.env.MINIO_ENDPOINT ?? 'localhost',
  port: parseInt(process.env.MINIO_PORT ?? '9000', 10),
  useSSL: process.env.MINIO_USE_SSL === 'true',
  accessKey: process.env.MINIO_ACCESS_KEY ?? 'minioadmin',
  secretKey: process.env.MINIO_SECRET_KEY ?? 'minioadmin',
});
```

### Usage as Multer Middleware

`sky-buckets` provides a custom Multer storage engine that redirects file uploads directly to MinIO. You can dynamically configure the bucket and object names based on the request. [6, 21, 22]

```typescript
import express, { Request, Response, NextFunction } from 'express';
import multer from 'multer';
import * as Minio from 'minio';
import { createMinioStorage } from 'sky-buckets';

const app = express();
app.use(express.json());
app.use(express.urlencoded({ extended: true }));

// MinIO Client Configuration (as above)
const minioClient = new Minio.Client({
  endPoint: process.env.MINIO_ENDPOINT ?? 'localhost',
  port: parseInt(process.env.MINIO_PORT ?? '9000', 10),
  useSSL: process.env.MINIO_USE_SSL === 'true',
  accessKey: process.env.MINIO_ACCESS_KEY ?? 'minioadmin',
  secretKey: process.env.MINIO_SECRET_KEY ?? 'minioadmin',
});

// Initialize Multer with the custom MinIO storage engine
const uploadToMinio = multer({
  storage: createMinioStorage({
    minioClient: minioClient,
    minioEndPoint: process.env.MINIO_ENDPOINT ?? 'localhost',
    minioPort: parseInt(process.env.MINIO_PORT ?? '9000', 10),
    minioUseSSL: process.env.MINIO_USE_SSL === 'true',
    bucketName: (req) => req.body.bucketName ?? 'default-bucket',
    objectName: (req, file) => {
      const filename = req.body.filename ?? file.originalname;
      const timestamp = Date.now();
      return `${timestamp}-${filename}`;
    },
    metadata: (req, file) => ({
      'Content-Type': file.mimetype,
      'X-Uploaded-By': req.body.userId ?? 'anonymous',
    }),
  }),
});

// Example route for single file upload
// Use `uploadToMinio.single('fileFieldName')`
app.post('/upload/single', uploadToMinio.single('file'), (req: Request, res: Response) => {
  if (!req.file) {
    return res.status(400).json({ message: 'No file uploaded.' });
  }
  res.status(200).json({
    message: 'File uploaded successfully to MinIO!',
    fileInfo: req.file, // req.file will contain MinIO metadata like bucket, key, etag, location [23, 24]
  });
});

// Example route for multiple file uploads
// Use `uploadToMinio.array('fileFieldName', maxFileCount)`
app.post('/upload/multiple', uploadToMinio.array('files', 10), (req: Request, res: Response) => {
  if (!req.files |
| (Array.isArray(req.files) && req.files.length === 0)) {
    return res.status(400).json({ message: 'No files uploaded.' });
  }
  res.status(200).json({
    message: 'Files uploaded successfully to MinIO!',
    fileInfo: req.files, // req.files will contain an array of MinIO metadata [23, 24]
  });
});

// Multer error handling middleware
app.use((err: any, req: Request, res: Response, next: NextFunction) => {
  if (err instanceof multer.MulterError) {
    return res.status(400).json({
      code: err.code,
      message: err.message,
      field: err.field,
    });
  } else if (err) {
    console.error(err);
    return res.status(500).json({
      message: 'An unexpected error occurred.',
      error: err.message,
    });
  }
  next();
});

// Multer populates `req.body` with text fields and `req.file`/`req.files` with file metadata. [6, 7, 26]
// The custom storage engine can access `req.body` for `bucketName` and `filename` parsed by Multer. [6, 7, 8]
```

### Inline File Operations with `MinioService`

`sky-buckets` also exports `MinioService` for direct interaction with MinIO, enabling programmatic upload and download operations outside the Multer middleware context.

```typescript
import * as Minio from 'minio';
import { MinioService } from 'sky-buckets';

const minioClient = new Minio.Client({
  endPoint: process.env.MINIO_ENDPOINT ?? 'localhost',
  port: parseInt(process.env.MINIO_PORT ?? '9000', 10),
  useSSL: process.env.MINIO_USE_SSL === 'true',
  accessKey: process.env.MINIO_ACCESS_KEY ?? 'minioadmin',
  secretKey: process.env.MINIO_SECRET_KEY ?? 'minioadmin',
});

const minioService = new MinioService(minioClient);

// Upload from buffer
const buffer = Buffer.from(base64String, 'base64');
await minioService.uploadFromBuffer(
  'my-bucket',
  'example.png',
  buffer,
  { 'Content-Type': 'image/png' }
);

// Upload from filepath
await minioService.uploadFromPath(
  'my-bucket',
  'example.jpg',
  '/path/to/file.jpg',
  { 'Content-Type': 'image/jpeg' }
);

// Upload from base64 string
await minioService.uploadFromBase64(
  'my-bucket',
  'example.txt',
  base64String,
  { 'Content-Type': 'text/plain' }
);

// Upload from blob
const blob = new Blob([yourBufferOrString], { type: 'application/pdf' });
await minioService.uploadFromBlob(
  'my-bucket',
  'document.pdf',
  blob,
  { 'Content-Type': 'application/pdf' }
);

// Multifile upload
const files = [
  {
    objectName: 'file1.txt',
    data: Buffer.from('Hello 1'),
    metaData: { 'Content-Type': 'text/plain' }
  },
  {
    objectName: 'file2.txt',
    data: Buffer.from('Hello 2'),
    metaData: { 'Content-Type': 'text/plain' }
  }
];

await minioService.uploadMultipleFiles('my-bucket', files);

// Download stream to HTTP response
await minioService.downloadFile('my-bucket', 'example.png', res);

// Download to local path
await minioService.downloadFileToPath(
  'my-bucket',
  'example.png',
  './downloads/example.png'
);

// Delete single file 
await minioService.deleteFile('my-bucket', 'example.png');

// Delete multiple file
const filesToDelete = [
  'document1.pdf',
  'document2.docx',
  'spreadsheet.xlsx'
];

await minioService.deleteMultipleFiles('my-bucket', filesToDelete);

// Check if file exsist
const check = await minioService.fileExists('my-bucket', 'example.ong'); // Promise<boolean>

```

### Error Handling

The package integrates error handling for both Multer and MinIO. Multer errors can be caught using `instanceof multer.MulterError` within an Express error handling middleware. MinIO operations are wrapped in `try-catch` blocks to handle connectivity issues, unfound buckets, or permission problems.

### Contributing

Contributions are welcome\! Please feel free to open issues or submit pull requests on the GitHub repository.

### License

This package is licensed under the MIT License. See the `LICENSE` file for more details.