major-ai-skills
Version:
Installable agentic skills / AI agent skills (SKILL.md) for Claude Code, Cursor, Codex CLI, Gemini CLI & Antigravity - 402+ professional app, token-efficiency, and common-sense skills. SEO/GEO ready.
123 lines (92 loc) • 6.57 kB
Markdown
title: "Rclone Cloud Storage Engine AI Skill Guide (Gemini)"
description: "Comprehensive operational skill specification for Google Gemini to visually diagnose, automate, script, and troubleshoot Rclone transfer metrics, remote configurations, and multi-cloud sync."
category: "Cloud Storage Sync & Mount Utility"
tags: ["rclone", "cloud-sync", "gemini", "transfer-metrics", "multi-cloud", "vfs-diagnostics"]
# Rclone Cloud Storage Engine AI Skill Guide (Gemini)
## Overview & Engine Architecture
Rclone serves as the premier multi-cloud data synchronization and virtual filesystem tool across enterprise and personal clouds. Gemini acts as an AI Cloud Storage Engineer and Systems Auditor, specializing in **multimodal transfer speed & IOPS diagnostic analysis**, **remote configuration file verification (`rclone.conf`)**, **bandwidth allocation tuning**, and **automated multi-cloud mirroring**.
### Remote Synchronization & Transfer Pipeline
```
┌─────────────────────────────────────────────────────────────┐
│ Rclone Data Transfer Pipeline │
│ │
│ Data Source Ingestion │
│ ├── Local Filesystem / NFS / SMB / ZFS Snapshots │
│ ├── Cloud Object & API Remotes (S3, GCS, B2, Azure, Drive) │
│ └── Stream Filtering (`--include`, `--exclude-from`) │
│ │
│ Transfer Engine & Verification │
│ ├── Concurrent Multi-Threaded Chunk Streamers │
│ ├── Dynamic Hash Verification (MD5, SHA1, QuickXorHash) │
│ └── Web GUI Dashboard (`rclone rcd --rc-web-gui`) │
└─────────────────────────────────────────────────────────────┘
```
## Operational Capabilities & Agent Directives
1. **Multimodal Metric & Dashboard Analysis**: Analyze screenshots of Rclone Web GUI, Grafana bandwidth meters, and CLI `--stats` progress bars to detect network throttling, disk IOPS bottlenecks, and transfer stalls.
2. **Multi-Cloud Mirroring Pipelines**: Formulate resilient cross-cloud mirroring commands (e.g. Google Cloud Storage to AWS S3) with zero local disk buffering.
3. **Filter Rule Architecture**: Construct granular glob filter patterns (`--filter-from rules.txt`) to selectively synchronize enterprise assets while excluding cache and temporary build artifacts.
4. **Configuration Health Auditing**: Audit `rclone.conf` files to verify token expiration handling, service account key bindings, and regional endpoint configurations.
## Production Python Automation: Cross-Cloud Mirroring Orchestrator
Run this Python script to synchronize an AWS S3 bucket directly to a Google Cloud Storage (GCS) bucket with cryptographic checksum verification:
```python
"""
Rclone Automated Cross-Cloud Storage Synchronizer (S3 -> GCS)
Executes direct cloud-to-cloud transfer with error handling and metrics.
"""
import sys
import subprocess
import json
def mirror_cloud_buckets(src_remote: str, dst_remote: str):
print(f"Starting Cross-Cloud Sync: {src_remote} -> {dst_remote}...")
cmd = [
"rclone", "sync", src_remote, dst_remote,
"--fast-list",
"--transfers", "16",
"--checkers", "32",
"--stats", "10s",
"--stats-one-line",
"--checksum",
"--retries", "3"
]
process = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.STDOUT, text=True)
for line in process.stdout:
sys.stdout.write(f"\r[RCLONE STATS] {line.strip()}")
sys.stdout.flush()
process.wait()
sys.stdout.write("\n")
if process.returncode == 0:
print("Cross-cloud mirroring completed successfully with 100% hash verification.")
else:
print(f"Sync failed with return code: {process.returncode}")
if __name__ == "__main__":
mirror_cloud_buckets("s3_primary:prod-assets", "gcs_backup:prod-assets-mirror")
```
## Technical Troubleshooting Matrix
| Issue & Visual Signature | Root Cause Analysis | Diagnostic & Resolution Pathway |
| :--- | :--- | :--- |
| **Transfer Speed Drops to Zero (Stalled Connection)** | Network firewall dropped idle TCP connection, or cloud provider enforced silent API rate limit. | 1. Add keepalive parameters: `--timeout 2m --contimeout 30s`.<br>2. Lower transfer concurrency: `--transfers 4`.<br>3. Enable `--stats-log-level DEBUG` to inspect raw HTTP retry codes. |
| **`corrupted on transfer: MD5 hash differ` Warning** | File was modified on local source disk while Rclone was reading it, causing hash mismatch. | 1. Add `--local-no-check-updated` or ensure files are quiesced before sync.<br>2. Run `rclone sync` again to update modified files.<br>3. Check for flaky network hardware causing bit flips. |
| **Rclone Web GUI Connection Refused** | Remote Control (RC) daemon is not listening on the requested port or IP binding. | 1. Launch with: `rclone rcd --rc-web-gui --rc-addr :5572 --rc-user admin --rc-pass secret`.<br>2. Check local port 5572 availability with `netstat` or `ss`. |
| **OneDrive `401 Unauthorized` / Expired Token Loop** | Microsoft Graph OAuth refresh token revoked or system clock drifted. | 1. Re-authenticate via `rclone config reconnect remote:`.<br>2. Verify local NTP system clock synchronization. |
## Command Line Syntax & Server Control
```bash
# Launch Rclone Embedded Web GUI for Interactive Monitoring
rclone rcd --rc-web-gui --rc-addr 127.0.0.1:5572
# Compute Total Space & File Count on Remote Cloud Path
rclone size remote:mybucket/media/
# Test Sync Actions with Dry Run Output
rclone sync /local/data remote:archive --dry-run
```
### Essential File Locations
- **Windows Configuration**: `%APPDATA%\rclone\rclone.conf`
- **Linux/macOS Configuration**: `~/.config/rclone/rclone.conf`
- **RC Web GUI Assets**: `~/.cache/rclone/webgui`
## Agent Operational Directive
> **MANDATORY**: For cloud-to-cloud data migrations, use `--fast-list` to reduce API billing costs and memory overhead. Always verify file hash integrity using `--checksum`.