s3-cli-js/README.md

A TypeScript-based npm package that replaces AWS CLI for S3 operations using presigned URLs

# S3 CLI JS

A TypeScript-based npm package that replaces AWS CLI for S3 operations using presigned URLs for enhanced performance.

## Features

- **Fast file operations**: Uses presigned URLs + direct HTTP requests instead of AWS SDK methods
- **Concurrent operations**: Supports parallel uploads/downloads for improved performance
- **Advanced sync**: Intelligent sync with timestamp comparison, size checking, and conflict resolution
- **Complete S3 functionality**: Supports all major S3 operations (ls, cp, mv, rm, sync, mb, rb)
- **TypeScript**: Fully typed for better development experience
- **CLI interface**: Can be run directly with `npx`
- **Progress indicators**: Shows progress for file uploads/downloads with overall sync progress
- **Flexible configuration**: Supports AWS credentials from environment variables or AWS credentials file

## Installation

### Global installation
```bash
npm install -g s3-cli-js
```

### Run with npx (no installation required)
```bash
npx s3-cli-js --help
```

## Configuration

The tool uses standard AWS credential configuration methods:

### Environment Variables
```bash
export AWS_ACCESS_KEY_ID=your_access_key
export AWS_SECRET_ACCESS_KEY=your_secret_key
export AWS_DEFAULT_REGION=us-east-1

# custom endpoint for custom S3 implementations
export AWS_ENDPOINT_URL=https://xxx.r2.cloudflarestorage.com
```

### AWS Credentials File
Create `~/.aws/credentials`:
```ini
[default]
aws_access_key_id = your_access_key
aws_secret_access_key = your_secret_key

[profile]
region = us-east-1
```

## Usage

### List buckets
```bash
s3-cli ls
```

### List objects in a bucket
```bash
s3-cli ls s3://my-bucket
s3-cli ls s3://my-bucket/prefix --recursive
```

### Upload files
```bash
# Upload single file
s3-cli cp file.txt s3://my-bucket/

# Upload directory recursively
s3-cli cp ./local-dir s3://my-bucket/remote-dir --recursive

# Upload with custom concurrency (default: 5)
s3-cli cp ./local-dir s3://my-bucket/remote-dir --recursive --concurrency 10
```

### Download files
```bash
# Download single file
s3-cli cp s3://my-bucket/file.txt ./

# Download directory recursively
s3-cli cp s3://my-bucket/remote-dir ./local-dir --recursive

# Download with custom concurrency
s3-cli cp s3://my-bucket/remote-dir ./local-dir --recursive --concurrency 8
```

### Move files
```bash
s3-cli mv file.txt s3://my-bucket/
s3-cli mv s3://my-bucket/file.txt ./
```

### Remove objects
```bash
# Remove single object
s3-cli rm s3://my-bucket/file.txt

# Remove directory recursively
s3-cli rm s3://my-bucket/directory/ --recursive
```

### Create bucket
```bash
s3-cli mb s3://my-new-bucket
```

### Remove bucket
```bash
# Remove empty bucket
s3-cli rb s3://my-bucket

# Force remove bucket and all contents
s3-cli rb s3://my-bucket --force
```

### Sync directories (Advanced)
```bash
# Basic sync with intelligent timestamp comparison
s3-cli sync ./local-dir s3://my-bucket/remote-dir
s3-cli sync s3://my-bucket/remote-dir ./local-dir

# Sync with high concurrency for large datasets
s3-cli sync ./local-dir s3://my-bucket/remote-dir --concurrency 15

# Sync with delete (remove files not in source)
s3-cli sync ./local-dir s3://my-bucket/remote-dir --delete

# Size-only comparison (faster, ignores timestamps)
s3-cli sync ./local-dir s3://my-bucket/remote-dir --size-only

# Exact timestamp matching (no tolerance)
s3-cli sync ./local-dir s3://my-bucket/remote-dir --exact-timestamps

# Force overwrite all files
s3-cli sync ./local-dir s3://my-bucket/remote-dir --force

# Dry run to see what would be synced
s3-cli sync ./local-dir s3://my-bucket/remote-dir --dry-run
```

## Command Options

### Global Options
- `--region <region>`: Override AWS region
- `--profile <profile>`: Use specific AWS profile
- `--endpoint-url <url>`: Use custom S3 endpoint

### Common Options
- `--recursive, -r`: Perform operation recursively
- `--dry-run`: Show what would be done without actually doing it
- `--exclude <patterns>`: Exclude files matching patterns
- `--include <patterns>`: Include only files matching patterns
- `--concurrency <number>`: Number of concurrent operations (default: 5, applies to cp, mv, sync)

### Sync Options
- `--delete`: Delete files in destination that do not exist in source
- `--size-only`: Compare files by size only (skip timestamp comparison)
- `--exact-timestamps`: Require exact timestamp matches (default allows 1s tolerance)
- `--force`: Force overwrite all files regardless of timestamps
- `--checksum`: Compare files using checksums (not implemented yet)

### List Options
- `--human-readable`: Display file sizes in human readable format
- `--summarize`: Display summary information

### Bucket Options
- `--force`: Force operation (for rb command)

## Examples

```bash
# List all buckets
npx s3-cli-js ls

# List objects with human readable sizes
npx s3-cli-js ls s3://my-bucket --human-readable --summarize

# Upload with exclusions
npx s3-cli-js cp ./src s3://my-bucket/src --recursive --exclude "*.log" "*.tmp"

# Dry run to see what would be copied
npx s3-cli-js cp ./data s3://my-bucket/data --recursive --dry-run

# Sync with custom endpoint
npx s3-cli-js sync ./website s3://my-bucket --endpoint-url https://s3.example.com

# High-performance sync with increased concurrency
npx s3-cli-js sync ./large-dataset s3://my-bucket/data --concurrency 20

# Advanced sync with delete and size-only comparison
npx s3-cli-js sync ./website s3://my-bucket/site --delete --size-only

# Exact timestamp sync for critical data
npx s3-cli-js sync ./backup s3://my-bucket/backup --exact-timestamps
```

## Development

### Setup
```bash
git clone <repository>
cd s3-cli-js
npm install
```

### Build
```bash
npm run build
```

### Test
```bash
npm test
```

### Lint
```bash
npm run lint
```

## Architecture

This tool is designed for performance and uses:

- **Presigned URLs**: Generated using AWS SDK for secure, time-limited access
- **Direct HTTP requests**: Uses axios for file uploads/downloads instead of AWS SDK streaming
- **TypeScript**: Provides type safety and better development experience
- **Modular design**: Each command is implemented as a separate module

## Sync Algorithm

The advanced sync feature uses intelligent comparison logic:

### File Comparison Strategy
1. **Size Comparison**: Files with different sizes are always synced
2. **Timestamp Comparison**: By default, allows 1-second tolerance for timestamp differences
3. **Exact Timestamps**: With `--exact-timestamps`, requires exact timestamp matches
4. **Size-Only Mode**: With `--size-only`, skips timestamp comparison entirely
5. **Force Mode**: With `--force`, overwrites all files regardless of timestamps

### Sync Process
1. **Discovery**: Lists all files in source and destination
2. **Comparison**: Compares each file using the selected strategy
3. **Planning**: Creates a sync plan showing what will be uploaded/downloaded/deleted
4. **Execution**: Performs operations concurrently with progress tracking
5. **Summary**: Reports results including errors and statistics

### Conflict Resolution
- **Newer Source**: Source file is newer → Upload/Download
- **Same Size & Timestamp**: Files are identical → Skip
- **Force Mode**: Always overwrite destination → Upload/Download
- **Delete Mode**: Files only in destination → Delete (with `--delete` flag)

## Performance Benefits

By using presigned URLs + direct HTTP requests instead of AWS SDK methods:
- Faster upload/download speeds
- Better progress tracking
- Reduced memory usage for large files
- More control over HTTP request configuration
- Concurrent operations for improved throughput on large datasets
- Intelligent sync with advanced file comparison strategies

## License

MIT