s3fs
Version:
Implementation of Node.JS FS interface using Amazon Simple Storage Service (S3).
352 lines (287 loc) • 14 kB
Markdown
# S3FS
[](https://www.npmjs.com/package/s3fs)
[](https://www.npmjs.com/package/s3fs)
[](https://travis-ci.org/RiptideElements/s3fs)
[](https://coveralls.io/r/RiptideElements/s3fs)
[](https://www.codacy.com/public/davidtpate/s3fs)
[](https://codeclimate.com/github/RiptideElements/s3fs)
[](https://david-dm.org/RiptideElements/s3fs)
[](https://david-dm.org/RiptideElements/s3fs)
[](https://david-dm.org/RiptideElements/s3fs)
Implementation of Node.JS [FS interface](http://nodejs.org/api/fs.html) using [Amazon Simple Storage Service (S3)](http://aws.amazon.com/s3/) for storage.
**Lead Maintainer**: [David Pate](https://github.com/DavidTPate)
## Purpose
S3FS provides a drop-in replacement for the File System (FS) implementation that is available with Node.JS allowing a distributed file-system to be used
by Node.JS applications through the well-known [FS interface](http://nodejs.org/api/fs.html) used by Node.JS.
## Minimum IAM Policy
Below is a policy for AWS [Identity and Access Management](http://aws.amazon.com/iam/) which provides the minimum privileges needed to use S3FS.
```json
{
"Statement": [
{
"Action": [
"s3:ListBucket"
],
"Effect": "Allow",
"Resource": [
"arn:aws:s3:::your-bucket"
]
},
{
"Action": [
"s3:AbortMultipartUpload",
"s3:CreateBucket",
"s3:DeleteBucket",
"s3:DeleteBucketPolicy",
"s3:DeleteObject",
"s3:GetBucketPolicy",
"s3:GetLifecycleConfiguration",
"s3:GetObject",
"s3:ListBucket",
"s3:ListBucketMultipartUploads",
"s3:ListMultipartUploadParts",
"s3:PutBucketPolicy",
"s3:PutLifecycleConfiguration",
"s3:PutObject"
],
"Effect": "Allow",
"Resource": [
"arn:aws:s3:::your-bucket/*"
]
}
]
}
```
## Currently Supported Methods
The methods below from Node.JS's [FS interface](http://nodejs.org/api/fs.html) are the only currently supported methods
matching the signature and functionality of the `fs` module. All of the methods support either usage through callbacks
or promises. There isn't any support for synchronous actions currently as there isn't a need.
* [fs.createReadStream(path, [options])](http://nodejs.org/api/fs.html#fs_fs_createreadstream_path_options)
* [fs.createWriteStream(path, [options])](http://nodejs.org/api/fs.html#fs_fs_createwritestream_path_options)
* [fs.exists(path, callback)](http://nodejs.org/api/fs.html#fs_fs_exists_path_callback)
* [fs.stat(path, callback)](http://nodejs.org/api/fs.html#fs_fs_stat_path_callback)
* [fs.lstat(path, callback)](http://nodejs.org/api/fs.html#fs_fs_lstat_path_callback)
* [fs.mkdir(path, [mode], callback)](http://nodejs.org/api/fs.html#fs_fs_mkdir_path_mode_callback)
* [fs.readdir(path, callback)](http://nodejs.org/api/fs.html#fs_fs_readdir_path_callback)
* [fs.readFile(filename, [options], callback)](http://nodejs.org/api/fs.html#fs_fs_readfile_filename_options_callback)
* [fs.rmdir(path, callback)](http://nodejs.org/api/fs.html#fs_fs_rmdir_path_callback)
* [fs.unlink(path, callback)](http://nodejs.org/api/fs.html#fs_fs_unlink_path_callback)
* [fs.writeFile(filename, data, [options], callback)](http://nodejs.org/api/fs.html#fs_fs_writefile_filename_data_options_callback)
### Constructor Details
Creating an instance of `S3fs` takes in the `bucketPath` and `options` which are passed on to the [S3 constructor](http://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/S3.html#constructor-property).
```js
var bucketPath = 'mySuperCoolBucket';
var s3Options = {
region: 'us-east-1',
};
var fsImpl = new S3FS(bucketPath, s3Options);
```
### Example Callback Usage
```js
var S3FS = require('s3fs');
var fsImpl = new S3FS('test-bucket', options);
fsImpl.writeFile('message.txt', 'Hello Node', function (err) {
if (err) throw err;
console.log('It\'s saved!');
});
```
### Example Promise Usage
```js
var S3FS = require('s3fs');
var fsImpl = new S3FS('test-bucket', options);
fsImpl.writeFile('message.txt', 'Hello Node').then(function() {
console.log('It\'s saved!');
}, function(reason) {
throw reason;
});
```
## Custom Supported Methods
Besides the methods from Node.JS's [FS interface](http://nodejs.org/api/fs.html) we also support some custom expansions
to the interface providing various methods such as recursive methods and S3 specific methods. They are described below.
### s3fs.getPath(path)
Provides a location by concatenating the bucket with path(s).
* path `String`. _Optional_. The relative path to the working directory or file
```js
// Create an instance of S3FS which has a current working directory of `test-folder` within the S3 bucket `test-bucket`
var fsImpl = new S3FS('test-bucket/test-folder', options);
// Returns location to directory `test-bucket/test-folder/styles
var fsImplStyles = fsImpl.getPath('styles');
// Returns location to file `test-bucket/test-folder/styles/main.css
var fsImplStyles = fsImpl.getPath('styles/main.css');
// Returns location to file `test-bucket/test-folder
var fsImplStyles = fsImpl.getPath();
```
### s3fs.clone(path)
Provides a clone of the instance of S3FS which has relative access to the specified directory.
* path `String`. _Optional_. The relative path to extend the current working directory
```js
// Create an instance of S3FS which has a current working directory of `test-folder` within the S3 bucket `test-bucket`
var fsImpl = new S3FS('test-bucket/test-folder', options);
// Creates a copy (which uses the same instance of S3FS) which has a current working directory of `test-folder/styles`
var fsImplStyles = fsImpl.clone('styles');
```
### s3fs.copyFile(sourcePath, destinationPath[, options, callback])
Allows a file to be copied from one path to another path within the same bucket. Paths are relative to
the bucket originally provided.
* sourceFile `String`. **Required**. Relative path to the source file
* destinationFile `String`. **Required**. Relative path to the destination file
* options `Object`. _Optional_. The options to be used when copying the file. See [AWS SDK](http://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/S3.html#copyObject-property)
* callback `Function`. _Optional_. Callback to be used, if not provided will return a Promise
```js
var fsImpl = new S3FS('test-bucket', options);
fsImpl.copyFile('test-folder/test-file.txt', 'other-folder/test-file.txt').then(function(data) {
// File was successfully copied
// Data contains details such as the `ETag` about the object. See [AWS SDK](http://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/S3.html#copyObject-property) for details.
}, function(reason) {
// Something went wrong
});
```
### s3fs.copyDir(sourcePath, destinationPath[, callback])
Recursively copies a directory from the source path to the destination path.
* sourcePath `String`. **Required**. The source directory to be copied
* destinationPath `String`. **Required**. The destination directory to be copied to
* callback `Function`. _Optional_. Callback to be used, if not provided will return a Promise
```js
var fsImpl = new S3FS('test-bucket', options);
fsImpl.copyDir('test-folder', 'other-folder').then(function() {
// Directory was successfully copied
}, function(reason) {
// Something went wrong
});
```
### s3fs.create(options[, callback])
Creates a new bucket on S3.
* options `Object`. _Optional_. The options to be used when creating the bucket. See [AWS SDK](http://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/S3.html#createBucket-property)
* callback `Function`. _Optional_. Callback to be used, if not provided will return a Promise
```js
var fsImpl = new S3FS('test-bucket', options);
fsImpl.create().then(function() {
// Bucket was successfully created
}, function(reason) {
// Something went wrong
});
```
### s3fs.delete([callback])
Deletes a bucket on S3, can only be deleted when empty. If you need to delete one that isn't empty use
[`destroy([callback])`](#s3fsdestroycallback) instead.
* callback `Function`. _Optional_. Callback to be used, if not provided will return a Promise
```js
var fsImpl = new S3FS('test-bucket', options);
fsImpl.delete().then(function() {
// Bucket was successfully deleted
}, function(reason) {
// Something went wrong
});
```
### s3fs.destroy([callback])
Recursively deletes all files within the bucket and then deletes the bucket.
* callback `Function`. _Optional_. Callback to be used, if not provided will return a Promise
```js
var fsImpl = new S3FS('test-bucket', options);
fsImpl.destroy().then(function() {
// Bucket was successfully destroyed
}, function(reason) {
// Something went wrong
});
```
### s3fs.headObject(path[, callback])
Retrieves the details about an object, but not the contents.
* path `String`. **Required**. Path to the object to retrieve the head for
* callback `Function`. _Optional_. Callback to be used, if not provided will return a Promise
```js
var fsImpl = new S3FS('test-bucket', options);
fsImpl.headObject('test-file.txt').then(function(details) {
// Details contains details such as the `ETag` about the object. See [AWS SDK](http://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/S3.html#headObject-property) for details.
}, function(reason) {
// Something went wrong
});
```
### s3fs.listContents(path, marker[, callback])
Retrieves a list of all objects within the specific path. The result is similar to that of [`headObject(path[, callback])`](#s3fsheadobjectpath-callback)
expect that it contains an array of objects.
* path `String`. **Required**. The path to list all of the objects for
* marker `String`. **Required**. The key to start with when listing objects
* callback `Function`. _Optional_. Callback to be used, if not provided will return a Promise
```js
var fsImpl = new S3FS('test-bucket', options);
fsImpl.listContents('/', '/').then(function(data) {
// Data.Contents contains details such as the `ETag` about the object. See [AWS SDK](http://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/S3.html#headObject-property) for details.
}, function(reason) {
// Something went wrong
});
```
### s3fs.putBucketLifecycle(name, prefix, days[, callback])
Adds/Updates a lifecycle on a bucket.
* name `String`. **Required**. The name of the lifecycle. The value cannot be longer than 255 characters.
* prefix `String`. **Required**. Prefix identifying one or more objects to which the rule applies.
* days Indicates the lifetime, in days, of the objects that are subject to the rule. The value must be a non-zero positive integer.
* callback `Function`. _Optional_. Callback to be used, if not provided will return a Promise
```js
var fsImpl = new S3FS('test-bucket', options);
// Remove the Cached contents in the `/cache` directory each day.
fsImpl.putBucketLifecycle('expire cache', 'cache', 1).then(function() {
// Bucket Lifecycle was successfully added/updated
}, function(reason) {
// Something went wrong
});
```
### s3fs.readdirp(path[, callback])
Recursively reads a directory.
* path `String`. **Required**. The path to the directory to read from
```js
var fsImpl = new S3FS('test-bucket', options);
fsImpl.readdirp('test-folder').then(function(files) {
// Files contains a list of all of the files similar to [`fs.readdir(path, callback)`](http://nodejs.org/api/fs.html#fs_fs_readdir_path_callback) but with recursive contents
}, function(reason) {
// Something went wrong
});
```
### s3fs.mkdirp(path[, callback])
Recursively creates a directory.
* path The path to the directory to create
* callback `Function`. _Optional_. Callback to be used, if not provided will return a Promise
```js
var fsImpl = new S3FS('test-bucket', options);
fsImpl.mkdirp('test-folder').then(function() {
// Directory has been recursively created
}, function(reason) {
// Something went wrong
});
```
### s3fs.rmdirp(path[, callback])
Recursively deletes a directory.
* path The path to the directory to delete
* callback `Function`. _Optional_. Callback to be used, if not provided will return a Promise
```js
var fsImpl = new S3FS('test-bucket', options);
fsImpl.rmdirp('test-folder').then(function() {
// Directory has been recursively deleted
}, function(reason) {
// Something went wrong
});
```
## Testing
This repository uses [Mocha](http://mochajs.org/) as its test runner. Tests can be run by executing the following command:
```bash
npm test
```
This will run all tests and report on their success/failure in the console, additionally it will include our [Code Coverage](#code-coverage).
## Code Coverage
This repository uses [Istanbul](http://gotwarlost.github.io/istanbul/) as its code coverage tool. Code Coverage will be calculated when executing the following command:
```bash
npm test
```
This will report the Code Coverage to the console similar to the following:
```bash
=============================== Coverage summary ===============================
Statements : 78.07% ( 356/456 )
Branches : 50.23% ( 107/213 )
Functions : 74.77% ( 83/111 )
Lines : 78.07% ( 356/456 )
================================================================================
```
Additionally, an interactive HTML report will be generated in `./coverage/lcov-report/index.html` which allows browsing the coverage by file.
## License
[MIT](LICENSE)
## Copyright
> Copyright (c) 2015 Riptide Software Inc.