@pellegrims/nx-remotecache-s3
Version:
Remote caching for @nrwl/nx using S3 storage
94 lines (69 loc) • 7.24 kB
Markdown
[](https://www.npmjs.com/package/@pellegrims/nx-remotecache-s3)
# @pellegrims/nx-remotecache-s3
> **DEPRECATED:** This package is no longer maintained and relies on the unmaintained `nx-remotecache-custom` package. Users are encouraged to migrate to [Nx Powerpack](https://nx.dev/powerpack/remote-cache) or other community-maintained remote cache runners.
A task runner for [@nrwl/nx](https://nx.dev/react) that uses s3 storage as a remote cache. This enables all team members and CI servers to share a single cache. The concept and benefits of [computation caching](https://nx.dev/angular/guides/computation-caching) are explained in the NX documentation.
This package was built with [nx-remotecache-custom](https://www.npmjs.com/package/nx-remotecache-custom) 🙌
## Setup
```
npm install --save-dev @pellegrims/nx-remotecache-s3
```
| Parameter | Description | Environment Variable / .env | `nx.json` |
| ---------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------- | ---------------- |
| Endpoint | Optional. The fully qualified endpoint of the webservice. This is only required when using a custom (non-AWS) endpoint. | `NXCACHE_S3_ENDPOINT` | `endpoint` |
| Bucket | Optional. Specify which bucket should be used for storing the cache. | `NXCACHE_S3_BUCKET` | `bucket` |
| Prefix | Optional. Specify prefix path of target object key. | `NXCACHE_S3_PREFIX` | `prefix` |
| Region | Optional. The AWS region to which this client will send requests. | `NXCACHE_S3_REGION` | `region` |
| Profile | Optional. The AWS profile to use to authenticate. | `NXCACHE_S3_PROFILE` | `profile` |
| Force Path Style | Optional. Whether to force path style URLs for S3 objects (e.g., `https://s3.amazonaws.com/<bucket>/` instead of `https://<bucket>.s3.amazonaws.com/` | `NXCACHE_S3_FORCE_PATH_STYLE` | `forcePathStyle` |
| Http Proxy | Optional. Useful when your environment must connect to a proxy to access the internet. | `http_proxy` \|\| `HTTP_PROXY` | |
| Https Proxy | Optional. Useful when your environment must connect to a proxy to access the internet. | `https_proxy` \|\| `HTTPS_PROXY` | |
| No Proxy | Optional. Useful when your environment must connect to a proxy to access the internet, but is not needed for connecting to aws. | `no_proxy` \|\| `NO_PROXY` | |
```json
{
"tasksRunnerOptions": {
"default": {
"runner": "@pellegrims/nx-remotecache-s3",
"options": {
"cacheableOperations": ["build", "test", "lint", "e2e"],
"endpoint": "https://some-endpoint.com",
"bucket": "name-of-bucket",
"prefix": "prefix/",
"region": "us-west-000",
"profile": "name-of-aws-profile",
"forcePathStyle": true
}
}
}
}
```
Authentication is handled by [@aws-sdk/credential-provider-node](https://docs.aws.amazon.com/AWSJavaScriptSDK/v3/latest/modules/_aws_sdk_credential_provider_node.html), so credentials will be attempted to be found from the following sources (listed in order of precedence):
- Environment variables exposed via process.env (example: AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY)
- SSO credentials from token cache
- Web identity token credentials
- Shared credentials and config ini files
- The EC2/ECS Instance Metadata Service
## Run it 🚀
Running tasks should now show the storage or retrieval from the remote cache:
```
------------------------------------------------------------------------
Built Angular Package
- from: /Users/name/example-workspace/libs/example-lib
- to: /Users/name/example-workspace/dist/libs/example-lib
------------------------------------------------------------------------
------------------------------------------------------------------------
Stored output to remote cache: s3 storage
Hash: d3d2bea71ea0f3004304c5cc88cf91be50b02bb636ebbdfcc927626fd8edf1ae
------------------------------------------------------------------------
```
## Advanced Configuration
See [nx-remotecache-custom](https://github.com/NiklasPor/nx-remotecache-custom#advanced-configuration) for more options like the ability to disable writes which may be useful if you only want to write to the cache from a CI but not localhost.
### Proxy Configuration
To use a proxy to connect to the S3 bucket, set any of the `http_proxy` `https_proxy` `HTTP_PROXY` or `HTTPS_PROXY` environment variables, with the lowercase version taking precedence. If needed, the proxy can be bypassed by adding the S3 endpoint to the `no_proxy` or `NO_PROXY` environment variables.
According to AWS, the `endpoint` and `region` parameters are meant to be mutually exclusive (though that isn't the case 100% of the time). Due to the complexity of creating a specific and valid s3 endpoint, and that there is no API provided by the aws-sdk libraries to generate one, this project handles checking the `no_proxy` for the s3 `endpoint` in the following manner.
If the s3 `endpoint` has been configured, it will be checked against the `no_proxy` and if found, s3 requests will bypass the proxy.
If the s3 `endpoint` has not been configured, it is assumed that the global s3 endpoint `s3.amazonaws.com` is being used, and the `no_proxy` will be checked for that url. Because of this, you could see the following behavior:
- If the s3 client does use the global s3 `endpoint` to make requests and if that url has been added to the `no_proxy`, then those requests will correctly bypass the proxy.
- If the s3 client does not use the global s3 `endpoint` (which could happen if a region other than us-east-1 is specified), and if that non global s3 `endpoint` has not been added to the `no_proxy`, then requests that should bypass the proxy will in fact NOT bypass the proxy. In this case, it is up to the user to determine which `endpoint` is being used and to add it to the `no_proxy`.
TIP: `no_proxy` is a global env variable. To limit the blast radius of `no_proxy`, include the specific service subdomain in the `no_proxy` entry; e.g. `s3.amazonaws.com` vs `amazonaws.com`.
## More Custom Runners
See [nx-remotecache-custom](https://github.com/NiklasPor/nx-remotecache-custom#all-custom-runners).