appwrite-utils-cli/README.md

Appwrite Utility Functions to help with database management, data conversion, data import, migrations, and much more. Meant to be used as a CLI tool, I do not recommend installing this in frontend environments.

# appwrite-utils-cli

## Overview

`appwrite-utils-cli` is a powerful, YAML-first command-line interface tool designed for Appwrite developers who need to manage database migrations, schema generation, data import, and comprehensive project management. Built on a modular architecture with enhanced performance, this CLI tool facilitates complex tasks like setting up databases, running migrations, generating schemas, and managing backups efficiently.

Highlights:
- Adapter-first database orchestration (no legacy attribute fall-through)
- Safe diff-based updates for attributes and indexes (no duplicate creations)
- Manual push selections (no auto-push of all DBs)
- Multi-database targeting per table/collection via `databaseIds`
- Per-function configuration via `.fnconfig.yaml` (discovered anywhere in repo)

## Features

### Core Capabilities
- **YAML-First Configuration**: Modern YAML-based configuration with JSON Schema support for IntelliSense
- **Interactive Mode**: Professional guided CLI experience with rich visual feedback and progress tracking
- **Modular Import System**: Completely refactored import architecture with 50%+ complexity reduction
- **Enhanced Performance**: Configurable rate limiting and batch processing for optimal performance
- **Safety First**: Smart confirmation dialogs for destructive operations with explicit confirmations

### Dual API Support (Collections & TablesDB)
- **Collections API**: Traditional Appwrite database operations with document-based terminology
- **TablesDB API**: New high-performance table API with column-based terminology
- **Automatic Detection**: Smart API mode detection (fetch-based detection; no extra packages required)
- **Seamless Migration**: Zero-downtime migration between Collections and TablesDB APIs
- **Terminology Flexibility**: Support for both `collections/documents/attributes` and `tables/rows/columns`

### Data Management
- **Advanced Data Import**: YAML-configured imports with sophisticated file handling, URL support, and user deduplication
- **Relationship Resolution**: Intelligent cross-collection relationship mapping and ID resolution
- **User Management**: Advanced user deduplication with email/phone matching and merge capabilities
- **File Operations**: Complete file handling with URL downloads, local file search, and afterImportActions
- **Backup Management**: Comprehensive backup system with progress tracking and detailed reporting

### Development Tools
- **Database Migrations**: Full migration control with progress tracking and operation summaries
- **Schema Generation**: Generate TypeScript (Zod), JSON, and Python (Pydantic) schemas/models from database configurations
- **Constants Generation**: Generate cross-language constants files (TypeScript, Python, PHP, Dart, JSON, Env) for database, collection, bucket, and function IDs
- **Data Transfer**: Transfer data between databases, collections, and instances with real-time progress
- **Configuration Sync**: Bidirectional synchronization between local YAML configs and Appwrite projects
- **Function Management**: Deploy and manage Appwrite Functions with specification updates

## Installation

To use `appwrite-utils-cli`, you can install it globally via npm to make it accessible from anywhere in your command line:

```bash
npm install -g appwrite-utils-cli
```

However, due to the rapid development of this project, it's recommended to use the following command:

```bash
npx --package=appwrite-utils-cli@latest appwrite-migrate [options]
```

**Note: Do not install this locally into your project. It is meant to be used as a command-line tool only.**

## TablesDB Support

`appwrite-utils-cli` provides comprehensive support for both traditional Appwrite Collections API and the new TablesDB API, enabling high-performance database operations with modern terminology.

### API Mode Detection

The CLI automatically selects Collections or TablesDB using fetch-based server detection (health/version and endpoint probes). No additional SDK packages are required.

### Configuration Comparison

#### Collections API (Traditional)
```yaml
# .appwrite/config.yaml
databases:
  - name: "main"
    id: "main"
    collections:  # Collections terminology
      - name: "Users"
        id: "users"
        attributes:
          - key: "email"
            type: "string"
            required: true
```

#### TablesDB API (New)
```yaml
# .appwrite/config.yaml
databases:
  - name: "main"
    id: "main"
    tables:  # TablesDB terminology
      - name: "Users"
        id: "users"
        attributes:  # Internally consistent
          - key: "email"
            type: "string"
            required: true
```

### Performance Benefits

TablesDB provides significant performance improvements:

- **Bulk Operations**: Native support for bulk inserts, updates, and deletes
- **Advanced Queries**: Enhanced query capabilities with better optimization
- **Reduced Latency**: Direct table operations without document overhead
- **Scalability**: Better performance with large datasets

### Migration Commands

The CLI provides seamless migration between APIs:

```bash
# Migrate Collections config to TablesDB format
npx appwrite-utils-cli appwrite-migrate --migrate-to-tablesdb

# Convert existing Collections to Tables (when TablesDB is available)
npx appwrite-utils-cli appwrite-migrate --sync --use-tablesdb

# Generate schemas for TablesDB
npx appwrite-utils-cli appwrite-migrate --generate --api-mode=tablesdb
```

## YAML-First Configuration

Version 1.0.0 introduces a YAML-first approach for better cross-platform compatibility and enhanced developer experience.

### Configuration Structure

The CLI automatically detects configuration files in this order:
1. `.appwrite/config.yaml` (recommended)
2. `appwrite.yaml`
3. `appwriteConfig.ts` (legacy, still supported)

### YAML Configuration Examples

#### Collections API Configuration
```yaml
# .appwrite/config.yaml with JSON Schema support
# yaml-language-server: $schema=./.yaml_schemas/appwrite-config.schema.json

appwriteEndpoint: "https://cloud.appwrite.io/v1"
appwriteProject: "your-project-id"
appwriteKey: "your-api-key"

databases:
  - name: "main"
    id: "main"
    collections:  # Collections terminology
      - name: "Users"
        id: "users"
        attributes:
          - key: "email"
            type: "string"
            required: true
            format: "email"
          - key: "name"
            type: "string"
            required: true

buckets:
  - name: "documents"
    id: "documents"
    permissions:
      - target: "any"
        permission: "read"

logging:
  enabled: false  # Disabled by default
  level: "info"
  console: true
```

#### TablesDB API Configuration
```yaml
# .appwrite/config.yaml for TablesDB
# yaml-language-server: $schema=./.yaml_schemas/appwrite-config.schema.json

appwriteEndpoint: "https://cloud.appwrite.io/v1"
appwriteProject: "your-project-id"
appwriteKey: "your-api-key"

databases:
  - name: "main"
    id: "main"
    tables:  # TablesDB terminology
      - name: "Users"
        id: "users"
        attributes:  # Internal compatibility
          - key: "email"
            type: "string"
            required: true
            format: "email"
          - key: "name"
            type: "string"
            required: true

buckets:
  - name: "documents"
    id: "documents"
    permissions:
      - target: "any"
        permission: "read"

logging:
  enabled: false  # Disabled by default
  level: "info"
  console: true
```

## Usage

After installation, you can access the tool directly from your command line using the provided commands.

### Interactive Mode

Run the CLI in interactive mode with enhanced visual feedback:

```bash
npx --package=appwrite-utils-cli@latest appwrite-migrate --it
```

This provides a professional guided experience with:
- Rich visual feedback and progress tracking
- Smart confirmation dialogs for destructive operations
- Operation summaries with detailed statistics
- Real-time progress bars with ETA calculations

### Generate Schemas (Zod / JSON / Pydantic)

Interactive schema generation lets you pick the format and output directory:

```bash
npx appwrite-utils-cli appwrite-migrate --it
# Choose: Generate schemas
# Select: TypeScript (Zod), JSON, Python (Pydantic), or All
# Enter output directory (absolute path respected)
```

- Pydantic models use modern typing (str | None, list[str]) and alias mapping for Appwrite system fields:
  - Base model (written as `base.py`) defines aliases for `$id`, `$createdAt`, `$updatedAt`, `$permissions`, `$databaseId`, `$collectionId`, `$sequence`.
  - Each collection/table generates a model extending `BaseAppwriteModel`.
  - Serialize with aliases via `model_dump(by_alias=True)` and validate from Appwrite docs via `model_validate(...)`.
  - Output directory is selectable; files are written directly to your chosen path (no extra subfolder).

### Push (manual selection)

Pushing local schema is now an explicit, manual selection flow to avoid unintended changes:

```bash
npx appwrite-utils-cli appwrite-migrate --push
```

- Select databases from the remote project (no default auto-selection)
- Select tables/collections to push per selected database
- Summary confirmation is shown before applying changes

Flags:
- `--dbIds=id1,id2` pre-selects databases by ID (skips DB prompt)
- `--collectionIds=c1,c2` pre-selects tables/collections by ID for the selected DB(s)

Eligibility per DB:
- A table/collection is considered eligible for a database if:
  - `databaseIds` includes the database ID, or
  - `databaseId` equals the database ID, or
  - neither field is set (eligible everywhere)

Attribute/Index behavior:
- Attributes and indexes are compared and only created/updated when changed
- Unchanged definitions are skipped
- Status is monitored until available (with sensible timeouts)

### Non-Interactive Mode

You can also use specific flags to run tasks without the interactive prompt:

```bash
npx --package=appwrite-utils-cli@latest appwrite-migrate [options]
```

## YAML Import Configuration System

Version 1.0.0 introduces a powerful YAML-based import system with sophisticated data handling capabilities.

### Import Configuration Examples

#### Collections API Import
```yaml
# .appwrite/import/users-import.yaml
# yaml-language-server: $schema=../.yaml_schemas/import-config.schema.json

source:
  file: "importData/users.json"
  basePath: "RECORDS"
  type: "json"

target:
  collection: "Users"  # Collections terminology
  type: "create"
  primaryKey: "user_id"
  createUsers: true

mapping:
  attributes:
    - oldKey: "user_id"
      targetKey: "userId"
      converters: ["anyToString"]
      validation:
        - rule: "required"
          params: ["{userId}"]

    - oldKey: "profile_image_url"
      targetKey: "avatar"
      fileData:
        path: "{profile_image_url}"
        name: "{user_id}_avatar"
      afterImport:
        - action: "createFileAndUpdateField"
          params: ["{dbId}", "{collId}", "{docId}", "avatar", "{bucketId}", "{filePath}", "{fileName}"]

  relationships:
    - sourceField: "department_id"
      targetField: "departmentId"
      targetCollection: "Departments"

options:
  batchSize: 50
  continueOnError: true
```

#### TablesDB API Import
```yaml
# .appwrite/import/users-tablesdb-import.yaml
# yaml-language-server: $schema=../.yaml_schemas/import-config.schema.json

source:
  file: "importData/users.json"
  basePath: "RECORDS"
  type: "json"

target:
  table: "Users"  # TablesDB terminology (alternatively: collection: "Users" works too)
  type: "create"
  primaryKey: "user_id"
  createUsers: true

mapping:
  attributes:  # Note: Still uses 'attributes' for compatibility
    - oldKey: "user_id"
      targetKey: "userId"
      converters: ["anyToString"]
      validation:
        - rule: "required"
          params: ["{userId}"]

    - oldKey: "profile_image_url"
      targetKey: "avatar"
      fileData:
        path: "{profile_image_url}"
        name: "{user_id}_avatar"
      afterImport:
        - action: "createFileAndUpdateRowField"  # TablesDB-specific action
          params: ["{dbId}", "{tableId}", "{rowId}", "avatar", "{bucketId}", "{filePath}", "{fileName}"]

  relationships:
    - sourceField: "department_id"
      targetField: "departmentId"
      targetTable: "Departments"  # TablesDB terminology

options:
  batchSize: 100  # TablesDB supports larger batches
  continueOnError: true
  useBulkOperations: true  # Enable TablesDB bulk operations
```

### Import System Features

- **File Handling**: Complete support for URL downloads and local file search
- **User Deduplication**: Sophisticated email/phone matching with merge capabilities
- **Rate Limiting**: Configurable p-limit for optimal performance (5 concurrent by default)
- **Relationship Resolution**: Intelligent cross-collection ID mapping
- **Validation**: Pre-import validation with detailed error reporting
- **Progress Tracking**: Real-time progress bars with batch processing statistics
- **AfterImportActions**: Complete post-import action system for file uploads and field updates

## Command-Line Options

Available options:

### Core Operations
- `--it`: Run in interactive mode with enhanced visual feedback
- `--setup`: Create setup files (supports both YAML and TypeScript)
- `--generate`: Generate TypeScript and JSON schemas from database configurations
- `--import`: Import data using YAML configurations with advanced processing
- `--backup`: Comprehensive backup with progress tracking and detailed reporting

### Data Management
- `--dbIds`: Comma-separated list of database IDs to operate on
- `--collectionIds`: Comma-separated list of collection IDs to operate on (also accepts `--tableIds`)
- `--bucketIds`: Comma-separated list of bucket IDs to operate on
- `--wipe`: Wipe data with smart confirmation (all: everything, docs: only documents, users: only user data)
- `--wipeCollections`: Non-destructively wipe specified collections (also `--wipeTables`)
- `--writeData`: Write converted imported data to file for validation

### TablesDB-Specific Options
- `--api-mode`: Force API mode (`collections` or `tablesdb`)
- `--use-tablesdb`: Force TablesDB API usage when available
- `--migrate-to-tablesdb`: Convert Collections configuration to TablesDB format
- `--bulk-operations`: Enable bulk operations for TablesDB (default: true when available)
- `--tableIds`: Comma-separated list of table IDs (alias for `--collectionIds`)

### Configuration & Synchronization
- `--push`: Push local YAML/TypeScript config to Appwrite project
- `--sync`: Synchronize by pulling config from Appwrite project
- `--endpoint`: Set the Appwrite endpoint
- `--projectId`: Set the Appwrite project ID
- `--apiKey`: Set the Appwrite API key

### Transfer Operations
- `--transfer`: Transfer data between databases or collections with progress tracking
- `--transfer-users`: Transfer users between instances (will not overwrite)
- `--fromDbId` / `--sourceDbId`: Source database ID for transfer
- `--toDbId` / `--targetDbId`: Destination database ID for transfer
- `--fromCollectionId`: Source collection ID for transfer
- `--toCollectionId`: Destination collection ID for transfer
- `--fromBucketId`: Source bucket ID for transfer
- `--toBucketId`: Destination bucket ID for transfer
- `--remoteEndpoint`: Remote Appwrite endpoint for transfers
- `--remoteProjectId`: Remote Appwrite project ID for transfers
- `--remoteApiKey`: Remote Appwrite API key for transfers

### Function Management
- `--updateFunctionSpec`: Update function specifications
- `--functionId`: Function ID to update
- `--specification`: New function specification (s-0.5vcpu-512mb to s-8vcpu-8gb)

### Constants Generation
- `--generateConstants` / `--constants`: Generate cross-language constants files with database, collection, bucket, and function IDs
- `--constantsLanguages`: Comma-separated list of languages for constants generation (default: typescript)
  - Available languages: `typescript`, `javascript`, `python`, `php`, `dart`, `json`, `env`
- `--constantsOutput`: Output directory for generated constants files (default: config-folder/constants)

## Examples

### Generate Constants

Generate cross-language constants files for all your Appwrite resource IDs:

```bash
# Generate TypeScript constants (default)
npx appwrite-utils-cli appwrite-migrate --generateConstants

# Generate multiple language formats
npx appwrite-utils-cli appwrite-migrate --generateConstants --constantsLanguages="typescript,python,php,json"

# Generate all available formats
npx appwrite-utils-cli appwrite-migrate --generateConstants --constantsLanguages="typescript,javascript,python,php,dart,json,env"

# Generate with custom output directory
npx appwrite-utils-cli appwrite-migrate --generateConstants --constantsOutput="./my-constants"
```

This generates constants files in your configuration directory (e.g., `.appwrite/constants/`) containing:
- Database IDs
- Collection IDs  
- Bucket IDs
- Function IDs

**Example TypeScript output:**
```typescript
export const DATABASE_IDS = {
  MAIN_DATABASE: "main"
} as const;

export const COLLECTION_IDS = {
  USERS: "01JYDBQTB5W8SCBAYB654CCADQ",
  POSTS: "01JYDBQTB5W8SCBAYB654POSTS"
} as const;

// Type helpers and utility arrays included
export type DatabaseId = typeof DATABASE_IDS[keyof typeof DATABASE_IDS];
```

### Transfer Databases

Transfer databases within the same project or from a local to a remote project:

```bash
npx appwrite-utils-cli appwrite-migrate --transfer --fromDbId sourceDbId --toDbId targetDbId --remoteEndpoint https://appwrite.otherserver.com --remoteProjectId yourProjectId --remoteApiKey yourApiKey
```

### Transfer Specific Collections

Transfer specific collections from one place to another, with all of their data:

```bash
npx appwrite-utils-cli appwrite-migrate --transfer --fromDbId sourceDbId --toDbId targetDbId --fromCollectionId sourceCollectionId --toCollectionId targetCollectionId --remoteEndpoint https://appwrite.otherserver.com --remoteProjectId yourProjectId --remoteApiKey yourApiKey
```

### TablesDB Examples

#### Enable TablesDB Mode
```bash
# Force TablesDB API usage
npx appwrite-utils-cli appwrite-migrate --sync --use-tablesdb

# Generate schemas with TablesDB terminology
npx appwrite-utils-cli appwrite-migrate --generate --api-mode=tablesdb
```

#### Migration from Collections to TablesDB
```bash
# Convert Collections configuration to TablesDB format
npx appwrite-utils-cli appwrite-migrate --migrate-to-tablesdb

# Sync with TablesDB API (requires node-appwrite-tablesdb)
npx --package=node-appwrite-tablesdb appwrite-utils-cli appwrite-migrate --sync --use-tablesdb
```

#### Bulk Operations with TablesDB
```bash
# Import with bulk operations enabled
npx appwrite-utils-cli appwrite-migrate --import --bulk-operations

# Transfer tables with high-performance bulk operations
npx appwrite-utils-cli appwrite-migrate --transfer --fromDbId sourceDb --toDbId targetDb --tableIds table1,table2 --use-tablesdb
```

#### Working with Tables (TablesDB terminology)
```bash
# Wipe specific tables
npx appwrite-utils-cli appwrite-migrate --wipeTables --tableIds users,posts

# Generate constants with table terminology
npx appwrite-utils-cli appwrite-migrate --generateConstants --api-mode=tablesdb
```

### Transfer Buckets

Transfer files between buckets:

```bash
npx appwrite-utils-cli appwrite-migrate --transfer --fromBucketId sourceBucketId --toBucketId targetBucketId --remoteEndpoint https://appwrite.otherserver.com --remoteProjectId yourProjectId --remoteApiKey yourApiKey
```

### Update Function Specifications

**NOTE: IF IT DOES NOT WORK AND YOU ARE SELF-HOSTED, PLEASE SET MAX CPU_PER_FUNCTION AND RAM_PER_FUNCTION IN YOUR ENV VARS TO > 0, THIS IS A BUG IN 1.6.0 -- YOU SET THE FUNCTION RAM IN MB**

Update the CPU and RAM specifications for a function:

```bash
npx appwrite-utils-cli appwrite-migrate --updateFunctionSpec --functionId yourFunctionId --specification s-1vcpu-1gb
```

Available specifications:

- s-0.5vcpu-512mb: 0.5 vCPU, 512MB RAM
- s-1vcpu-1gb: 1 vCPU, 1GB RAM
- s-2vcpu-2gb: 2 vCPU, 2GB RAM
- s-2vcpu-4gb: 2 vCPU, 4GB RAM
- s-4vcpu-4gb: 4 vCPU, 4GB RAM
- s-4vcpu-8gb: 4 vCPU, 8GB RAM
- s-8vcpu-4gb: 8 vCPU, 4GB RAM
- s-8vcpu-8gb: 8 vCPU, 8GB RAM

## Additional Notes

- If you run out of RAM during large data imports, you can increase Node's memory allocation:

  ```bash
  export NODE_OPTIONS="--max-old-space-size=16384"
  ```

  This sets the allocation to 16GB. For most cases, 8GB (`8192`) should be sufficient.

- The CLI now supports OpenAPI generation for each attribute in the schema. Add a `description` to any attribute or collection, and it will export that schema to the `appwrite/openapi` folder.

This updated CLI ensures that developers have robust tools at their fingertips to manage complex Appwrite projects effectively from the command line, with both interactive and non-interactive modes available for flexibility.

## Changelog

### 1.5.0 - Recursive Zod v4 Schemas + One‑Way Relationship Support

Highlights
- Recursive getters: Generated schemas now use Zod v4 recursive getters (no `z.lazy`, no `BaseSchema` layering). Cleaner types and fully inferred mutual recursion.
- One‑way relationships: Relationship attributes are now included even when `twoWay: false`. Related imports resolve collection IDs to names.
- Required semantics: Relationship getters respect `required` and `array`:
  - required scalar → `RelatedSchema`
  - optional scalar → `RelatedSchema.nullish()`
  - required array → `RelatedSchema.array()`
  - optional array → `RelatedSchema.array().nullish()`
- JSON Schemas: `$ref` definitions use resolved collection names when YAML provides IDs.

Validation changes
- Relationship schema: `twoWayKey` and `side` are now required only when `twoWay` is `true`.
- Helpful errors: Keeps strong validation but removes false negatives for one‑way relationships.

Developer notes
- Imports: Schema generators import only `...Schema` from related collections (no type imports needed).
- Example YAML: `Posts.yaml` demonstrates a required `manyToMany` (`categories`) and a one‑way `manyToOne` (`author`).


### 1.3.0 - Zod v4 Upgrade & Collection Management Fixes

**🎉 Major Release - Zod v4 Compatibility & Reliability Improvements**

#### Breaking Changes
- **Zod v4 Upgrade**: Updated to Zod v4.0.0 for enhanced schema validation
  - ⚠️ **Potential Breaking Change**: Some schema validations may behave differently
  - Function event types are now more flexible to accommodate Zod v4 changes
  - Review your validation schemas if you experience issues

#### Major Bug Fixes
- **Fixed Duplicate Attribute Creation**: Resolved issue where attributes were being created multiple times
  - Implemented intelligent filtering to only process new or changed attributes
  - Enhanced status monitoring and error handling
  - Significantly improved sync performance and reliability

- **Fixed Index Creation Issues**: Resolved indexes not being created from collection configurations
  - Added proper null checks for index arrays from different collection sources
  - Enhanced index creation with comprehensive status monitoring
  - Improved error handling and retry logic for stuck indexes

#### Enhanced Collection Management
- **Smart Attribute Processing**: Attributes are now only created/updated when needed
  - Compares existing vs. config attributes before processing
  - Skips unchanged attributes with clear logging
  - Better handling of edge cases and error conditions

- **Improved Index Handling**: More robust index creation from collection configs
  - Proper fallback logic when indexes are undefined
  - Enhanced compatibility with different collection sources
  - Better error reporting and debugging information

#### Performance Improvements
- **Optimized Sync Operations**: Collections now process only necessary changes
- **Enhanced Status Monitoring**: Real-time feedback for attribute and index operations
- **Better Resource Management**: Reduced API calls through intelligent filtering

#### Developer Experience
- **Build Stability**: Resolved TypeScript compilation issues with function schemas
- **Type Safety**: Maintained strict typing while accommodating Zod v4 changes
- **Enhanced Logging**: Better progress reporting and error messages

#### Migration Guide
1. **Update Dependencies**: Ensure compatibility with Zod v4.0.0
2. **Test Collection Operations**: Verify that attribute and index creation works as expected
3. **Review Validations**: Check any custom validation schemas for potential breaking changes
4. **Function Events**: Function event arrays are now `string[]` for enhanced flexibility

**Integration Note**: This version significantly improves collection management reliability and provides full Zod v4 compatibility. The fixes address core synchronization issues that could cause duplicate resources or missing indexes.

---

### 1.1.0 - Enhanced Transfer System with Fault Tolerance

**🔧 Robust Transfer Operations with Status Monitoring**

#### Enhanced Attribute Creation with Fault Tolerance
- **Exponential Backoff**: Intelligent retry strategy starting at 2 seconds, doubling each retry (2s, 4s, 8s, 16s, 30s max)
- **Status Monitoring**: Real-time monitoring of attribute states ('available', 'processing', 'stuck', 'failed', 'deleting')
- **Retry Logic**: Collection deletion/recreation for stuck attributes with up to 5 retry attempts
- **Sequential Processing**: Attributes processed one-by-one to prevent overwhelming the server
- **Enhanced Logging**: Comprehensive console feedback with color-coded status messages

#### Enhanced Index Creation with Status Monitoring
- **Similar Fault Tolerance**: Index creation now includes the same robust monitoring and retry logic
- **Status Checking**: Real-time monitoring of index creation states with proper error handling
- **Collection Recreation**: Automatic collection deletion/recreation for stuck index operations
- **Sequential Processing**: Prevents rate limiting by processing indexes individually

#### Document Transfer Reliability Improvements
- **Enhanced Error Handling**: Improved document transfer with exponential backoff retry logic
- **Smaller Batch Sizes**: Reduced batch sizes (10 documents) to prevent server overload
- **Better Progress Reporting**: Enhanced progress tracking with success/failure counts
- **Fault Tolerance**: Graceful handling of duplicate documents and API errors

#### Remote Database Transfer Enhancements
- **Integrated Enhanced Methods**: Updated `transferDatabaseLocalToRemote` to use new attribute and index creation
- **Proper Wait Logic**: System now properly waits for attributes/indexes to be fully created before proceeding
- **Status Validation**: Comprehensive status checking throughout the transfer process
- **Continued Operation**: Transfer continues even if some attributes/indexes fail (with warnings)

#### AppwriteConfig Integration for Comprehensive Transfer
- **Smart Configuration Detection**: Automatically detects existing appwriteConfig for reuse
- **Source/Target Options**: Users can select their appwriteConfig for either source or target endpoints
- **Streamlined Setup**: Enhanced user experience with clear configuration prompts

#### Technical Implementation
- **Rate Limiting Respect**: Enhanced operations respect existing rate limiting while adding reliability
- **Memory Efficiency**: Optimized processing to handle large operations without overwhelming system resources
- **Error Resilience**: Comprehensive error handling with detailed user feedback and recovery options
- **Status Persistence**: Operations maintain state information for better debugging and monitoring

#### Usage Benefits
- **Reliability**: Transfer operations no longer fail due to timing issues or stuck operations
- **Visibility**: Clear progress indicators and status messages throughout all operations
- **Recovery**: Automatic retry and recovery mechanisms prevent data loss
- **Performance**: Optimized timing prevents API throttling while maintaining speed

**Breaking Change**: None - fully backward compatible with significantly enhanced reliability.

### 1.0.9 - Enhanced User Transfer with Password Preservation

**🔐 Complete Password Hash Preservation During User Transfers**

#### Password Hash Support
- **Universal Hash Support**: Support for all Appwrite password hash types:
  - **Argon2**: Modern default hashing (preserved)
  - **Bcrypt**: Industry standard (preserved)
  - **Scrypt**: Memory-hard function with custom parameters (preserved)
  - **Scrypt Modified**: Firebase-style with salt/separator/signer (preserved)
  - **MD5**: Legacy support (preserved)
  - **SHA variants**: SHA1, SHA256, SHA512 (preserved)
  - **PHPass**: WordPress-style hashing (preserved)
- **Dynamic Hash Detection**: Automatically detects and uses correct hash creation method
- **Parameter Preservation**: Maintains hash-specific parameters (salt, iterations, memory cost, etc.)

#### Enhanced User Transfer Logic
- **Smart Password Recreation**: Uses appropriate `create*User` method based on detected hash type
- **Fallback Mechanism**: Graceful fallback to temporary passwords if hash recreation fails
- **Hash Options Support**: Preserves algorithm-specific configuration from `hashOptions`
- **Detailed Logging**: Clear success/failure messages with hash type information

#### User Experience Improvements
- **Accurate Information**: Updated CLI messaging to reflect actual password preservation capabilities
- **Clear Expectations**: Distinguishes between users who keep passwords vs. those who need reset
- **Success Feedback**: Detailed reporting of password preservation success rate
- **Risk Assessment**: Proper warnings only for users who will lose passwords

#### Technical Implementation
- **Hash Type Detection**: `user.hash` field determines creation method
- **Configuration Parsing**: `user.hashOptions` provides algorithm parameters
- **Error Resilience**: Comprehensive try-catch with fallback to temporary passwords
- **Type Safety**: Proper handling of hash option types and parameters

#### Migration Benefits
- **Seamless Login**: Users with preserved hashes can immediately log in with original passwords
- **Reduced Support**: Dramatically fewer password reset requests after migration
- **Complete Fidelity**: Maintains original security posture and hash strength
- **Production Ready**: Safe for live user base migrations

#### Usage Examples
```bash
# Users will now preserve passwords during comprehensive transfer
npx appwrite-utils-cli@latest appwrite-migrate --it
# Select: 🚀 Comprehensive transfer (users → databases → buckets → functions)

# Example output:
# ✅ User 123 created with preserved argon2 password
# ✅ User 456 created with preserved bcrypt password  
# ⚠️  User 789 created with temporary password - password reset required
```

**Breaking Change**: None - fully backward compatible with enhanced capabilities.

### 1.0.8 - Comprehensive Transfer System with Enhanced Rate Limiting

**🚀 Complete Cross-Instance Transfer Solution**

#### Comprehensive Transfer System
- **New CLI Option**: `🚀 Comprehensive transfer (users → databases → buckets → functions)` in interactive mode
- **Orchestrated Transfer Flow**: Proper execution order (users → databases → buckets → functions) for dependency management
- **Cross-Instance Support**: Transfer entire Appwrite configurations between different instances/projects
- **Selective Transfer**: Choose which components to transfer (users, databases, buckets, functions)
- **Dry Run Mode**: Test transfers without making actual changes

#### Enhanced Rate Limiting Strategy
- **Configurable Limits**: 5 to 100 concurrent operations in steps of 5
- **Differentiated Rates**: Smart rate limiting based on operation type:
  - **General Operations**: Full rate (databases, functions)
  - **User Operations**: Half rate (more sensitive operations)
  - **File Operations**: Quarter rate (most bandwidth intensive)
- **Visual Feedback**: Real-time rate limit display during transfers
- **Intelligent Scaling**: Automatic calculation of optimal rates for different operations

#### File Transfer Enhancements
- **File Validation**: Comprehensive integrity checking (empty files, size limits)
- **Retry Logic**: Exponential backoff for failed file transfers
- **Error Handling**: Graceful handling of corrupt/invalid files
- **Progress Tracking**: Real-time progress for large file transfers

#### Function Transfer Integration
- **Automated Function Migration**: Download from source, redeploy to target
- **Temporary Management**: Automatic cleanup of downloaded function code
- **Existing Code Integration**: Leverages existing deployment infrastructure
- **Configuration Preservation**: Maintains function settings and variables

#### User Experience Improvements
- **Password Reset Warnings**: Clear notifications about Appwrite password limitations
- **Interactive Configuration**: Step-by-step prompts for source/target setup
- **Comprehensive Reporting**: Detailed transfer summaries with statistics
- **Smart Confirmations**: Risk-based confirmations for destructive operations

#### Technical Implementation
- **Rate Limiting**: Uses p-limit for concurrent operation control
- **Error Resilience**: Robust error handling with detailed user feedback
- **Memory Management**: Efficient processing of large datasets
- **Progress Tracking**: Real-time progress bars with ETA calculations

#### Usage Examples
```bash
# Interactive mode - select comprehensive transfer
npx appwrite-utils-cli@latest appwrite-migrate --it

# Example rate limiting at 20 concurrent:
# - General operations: 20 concurrent
# - User operations: 10 concurrent  
# - File operations: 5 concurrent
```

**Benefits**:
- Complete Appwrite instance migration capability
- Intelligent rate limiting prevents API throttling
- Enhanced file transfer reliability
- Comprehensive progress tracking and reporting
- Maintains data integrity across transfers

### 1.0.7 - Forgot to remove debug logs

### 1.0.6 - Cross-Language Constants Generation

**🚀 Enhanced Developer Experience with Multi-Language Constants**

#### Constants Generation System
- **Cross-Language Support**: Generate constants in 7 languages for seamless multi-platform development
  - **TypeScript**: Type-safe constants with `as const` and helper types
  - **JavaScript**: ES6 modules with utility arrays
  - **Python**: Class-based constants with snake_case dictionaries
  - **PHP**: Static class methods and associative arrays
  - **Dart**: Maps with individual getter methods (camelCase)
  - **JSON**: Structured data with metadata for cross-platform integration
  - **Environment Variables**: Prefixed environment variables for deployment
- **Smart Directory Structure**: Constants generated in `{config-folder}/constants/` by default
- **CLI Integration**: Both command-line and interactive mode support
- **Custom Output**: Support for custom output directories when needed

#### Resource ID Extraction
- **Database IDs**: Extract all configured database identifiers
- **Collection IDs**: Generate constants for all collection ULIDs/names
- **Bucket IDs**: Include storage bucket identifiers
- **Function IDs**: Support for Appwrite Function identifiers
- **Naming Conventions**: Language-appropriate naming (UPPER_CASE, camelCase, snake_case)

#### Developer Experience Improvements
- **Interactive Mode**: Checkbox selection for languages with smart defaults
- **CLI Commands**: Simple `--generateConstants` with language and output options
- **Type Safety**: Full TypeScript support with generated types and helpers
- **Cross-Platform Compatibility**: Enable seamless development across different tech stacks

#### Usage Examples
```bash
# Default TypeScript generation
npx appwrite-utils-cli appwrite-migrate --generateConstants

# Multi-language generation
npx appwrite-utils-cli appwrite-migrate --generateConstants --constantsLanguages="typescript,python,php,json"

# All formats with custom output
npx appwrite-utils-cli appwrite-migrate --generateConstants --constantsLanguages="typescript,javascript,python,php,dart,json,env" --constantsOutput="./constants"
```

**Migration Benefits**: 
- Eliminates hardcoded resource IDs across codebases
- Enables type-safe resource access in TypeScript projects
- Supports multi-language development workflows
- Maintains constants alongside configuration for easy maintenance

### 1.0.1 - Function Templates & Attribute Type Improvements

**🚀 Enhanced Function Management & Type System Fixes**

#### Function Template System Improvements
- **ES Module Compatibility**: Fixed `__dirname is not defined` error when creating function templates
- **Template-Specific Defaults**: Added pre-configured settings for all function templates:
  - `typescript-node`: Node.js 21.0, TypeScript build setup, 0.5vCPU/512MB
  - `uv`: Python 3.12, UV package manager, 0.5vCPU/512MB  
  - `count-docs-in-collection`: Node.js 21.0, specialized counting function, 1vCPU/512MB
- **YAML Config Integration**: Functions now automatically added to YAML config with generated ULIDs
- **Interactive Enhancement**: Improved template selection with descriptive names and smart defaults

#### YAML Collections Loading Fix
- **Path Resolution**: Fixed collections directory resolution for YAML configurations
- **Cross-Platform Support**: Collections now properly detected in `.appwrite/collections/` regardless of config location
- **Backwards Compatibility**: Both TypeScript and YAML collection structures fully supported

#### Attribute Type System Overhaul
- **Double/Float Compatibility**: Resolved Zod discriminated union errors with proper schema separation
- **Backwards Compatibility**: Full support for both "double" and "float" attribute types
- **Schema Architecture**: Clean separation with shared base schema for numeric attributes
- **Type Safety**: Enhanced TypeScript support with proper type inference

#### Developer Experience Improvements
- **Error Resolution**: Eliminated all TypeScript compilation errors
- **Template Updates**: Updated function template references from deprecated "poetry" to "uv"
- **Schema Validation**: Improved attribute validation with better error messages
- **Clean Architecture**: Removed deprecated relationship attributes for better performance

#### Bug Fixes
- Fixed YAML collection path resolution in nested directory structures
- Resolved ES module compatibility issues in function template creation
- Eliminated Zod discriminated union conflicts for numeric attribute types
- Updated outdated template references and improved template selection UX

**Migration Notes**: 
- Function templates now automatically integrate with YAML configs
- Existing collections continue to work; deprecated relationship attributes converted to manual references
- All numeric attributes now use consistent "double" type with "float" backwards compatibility

### 1.0.0 - YAML-First Architecture & Import System Revolution

**🎉 Major Release - Comprehensive Architecture Overhaul**

#### YAML-First Configuration System
- **Complete YAML support** with JSON Schema validation for IntelliSense
- **Automatic discovery**: `.appwrite/config.yaml`, `appwrite.yaml`, or `appwriteConfig.ts`
- **Cross-platform compatibility**: No TypeScript runner required
- **Better organization**: Clean `.appwrite` directory structure
- **Backward compatibility**: Existing TypeScript configs continue to work

#### Import System Revolution (50%+ Complexity Reduction)
- **Modular architecture**: Extracted DataLoader (1,688 lines) into focused services
  - `DataTransformationService` - Pure data transformation logic
  - `FileHandlerService` - URL downloads and local file handling
  - `UserMappingService` - Sophisticated email/phone deduplication
  - `ValidationService` - Centralized validation with detailed reporting
  - `RelationshipResolver` - Cross-collection ID mapping
  - `ImportOrchestrator` - High-level coordination
- **Enhanced rate limiting**: Configurable p-limit for different operations
  - `dataInsertion: 5` (concurrent document creation)
  - `fileUpload: 2` (conservative for large files)
  - `validation: 10` (pre-import validation)
  - `dataQuery: 25` (relationship resolution)

#### YAML Import Configuration System
- **Complete YAML import definitions** with JSON Schema support
- **Advanced file handling**: URL downloads with local file search fallback
- **Sophisticated validation**: Pre-import validation with detailed error reporting
- **Template generation**: Easy creation of new import configurations
- **Relationship mapping**: Intelligent cross-collection ID resolution
- **User deduplication**: Advanced email/phone matching with merge capabilities
- **AfterImportActions**: Complete post-import action system

#### Enhanced User Experience
- **Rich visual feedback**: Progress bars with ETA and speed indicators
- **Smart confirmation dialogs**: Risk-based confirmations for destructive operations
- **Operation summaries**: Detailed post-operation reports with statistics
- **Professional messaging**: Unified MessageFormatter with consistent styling
- **Configurable logging**: Disabled by default, full configuration support

#### Performance & Safety Improvements
- **Batch processing**: Enhanced with progress tracking and memory efficiency
- **Error resilience**: Improved error handling and recovery mechanisms
- **Transaction safety**: Better handling of partial failures
- **Memory optimization**: Efficient processing of large datasets

#### Developer Experience
- **JSON Schema generation**: Full IntelliSense support for YAML files
- **Example configurations**: Comprehensive templates and examples
- **Better error messages**: Clear validation and error reporting
- **Type safety**: Full TypeScript support for all new features

### Changelog

- 1.2.18: Fix users transfer comparison not counting email or phone validation as a reason
- 1.2.17: Fixed users transfer not keeping validation of email / phone, temporarily disable bulk transfer to see if permissions aren't being updated by it
- 1.2.15: Fixed various transfer and sync functionalities
- 1.0.5: Fixed `.` directories being ignored. Normally a good thing
- 1.0.4: Fixed `appwriteConfig.yaml` being the name for the converted config, instead of `config.yaml`
- 1.0.3: Fixed appwriteConfig detection for `--it` so it detects when you can migrate your config
- 1.0.2: Fixed migrations, sorry about that!

**Migration Note**: While fully backward compatible, we recommend migrating to YAML configuration for the best experience. Use `--setup` to generate new YAML configurations.

- 0.10.86: Fixed `selectCollections` not always filtering by `databaseId`
- 0.10.85: Added logging to `wipeCollection`
- 0.10.83: Actually fixed the import oops
- 0.10.82: Fixed the `lodash` import, replaced with `es-toolkit`
- 0.10.81: Fixed `wipeCollection` -- it wasn't properly deleting all files in a loop
- 0.10.80: Updated `appwrite-utils` req
- 0.10.78: Fixed `attributesSame` so it will properly update attributes that have changed
- 0.10.77: Added disclaimer to update function specifications for bug
- 0.10.76: Updated CLI commands to not require an `appwriteConfig.ts` if you set API Key, ProjectId, and Endpoint
- 0.10.75: Fixed slight issue writing ZOD Schema for collection without any attributes
- 0.10.74: Fix moving users, should update Labels now
- 0.10.73: Fix moving users, passwords should work now (from Appwrite, Argon2)
- 0.10.71: Fix create template function `__dirname`
- 0.10.70: Fixed `--transfer-users` phones
- 0.10.67: Added `--transfer-users` boolean flag to also transfer users between projects
- 0.10.66: Fixed `ignore` always being an empty array, if not set, so it properly ignores the defaults
- 0.10.65: Fixed the stupid local functions not caring about the ignore string, and added `__pycache__` and `.venv` to default ignores
- 0.10.64: Fixed `Deploy Function(s)` not ignoring things properly
- 0.10.63: My `collectLocalFunctions` function was failing to add the `scopes` and a few others to the function, accidentally, fixed now
- 0.10.62: Made `Deploy Function(s)` also update the functions, as not all things (timeout, specification, etc.) are updated via deploy
- 0.10.61: Fixed ignore haha, also added `ignore` to the `Functions` config, to specify what to ignore when creating the `.tar.gz` file
- 0.10.051: Added `node_modules` and a few others to the ignore
- 0.10.05: Made deploy function into deploy function(s) -- so you can do more than one at a time
- 0.10.04: Fixed stupid progress bar not updating -- also fixed double text
- 0.10.03: Fixed `syncDb` to push the configurations properly, accidentally hurt it during `synchronizeConfigurations`
- 0.10.02: Updated `wipeCollection` to handle errors gracefully
- 0.10.01: Fixed `predeployCommands` to work
- 0.10.001: Updated `deployFunction` to not updateConfig if it's already present
- 0.10.0: Fixed `synchronize configurations` for functions, now you do not need to deploy the function first
- 0.9.999: Fixed Functions, looks for `./functions` in addition to `appwriteConfigFolder/functions`
- 0.9.998: Fixed transfer finally, added `--targetDbId` and `--sourceDbId` as aliases
- 0.9.994: Added function deployment management, in BETA, and fixed document transfer between databases
- 0.9.993: Fixed `updateFunctionSpecifications` resetting functions to default with undefined values (oops)
- 0.9.992: Added `updateFunctionSpecifications` which lists functions and specifications to allow you to update your functions max CPU and RAM usage per-function
- 0.9.990: Fixed `transferFilesLocalToLocal` and `remote` if a document exists with that `$id`, also fixed wipe `"all"` option also wiping the associated buckets
- 0.9.983: Fixed `afterImportActions` not resolving
- 0.9.981: Try fixing `tryAwaitWithRetry` to catch `522` errors from Cloudflare, they were appearing for some users, also added a 1000ms delay to `tryAwaitWithRetry`
- 0.9.98: Fixing some import errors reported by users
- 0.9.95: Updated to include new version of `appwrite-utils`
- 0.9.93: Updated `selectDatabases` and `selectCollections` from the interactive CLI to prefer local collections or databases when synchronizing the databases
- 0.9.92: Fixed `createOrUpdateAttributes` so it deletes attributes that don't exist in local config when you are running `syncDb`. Also updated the database and collection selection, so it won't try and fetch the collections and databases that don't exist (ones you picked from local config) and error
- 0.9.91: Fixed another webpack error, screw you react (but you're supported now so I guess not-screw-you)
- 0.9.90: Fixed Webpack errors (why tf does webpack add an extra `default`...???)
- 0.9.80: Fixed collections not being unique between local and remote
- 0.9.79: Fixed local collections not being considered for the synchronization unless all de-selected
- 0.9.78: Added colored text! And also added a lot more customization options as to what to wipe, update, etc.
- 0.9.75: Fixed attribute bug
- 0.9.72: Fixed my own null bug
- 0.9.71: Reverted `node-appwrite` to 14, this seems to fix the xdefault error
- 0.9.70: I think I stopped it from deleting attributes, my bad on that
- 0.9.68: Temporarily disabled updating Attributes until `updateStringAttribute` is fixed -- it just deletes them now
- 0.9.65: Temporary fix for Appwrite's `updateStringAttribute` bug
- 0.9.64: Fixed string attribute requiring xdefault
- 0.9.61: Remove fileURLToPath -- should hopefully fix windows
- 0.9.60: Fix init command to repository URL
- 0.9.59: Fix to Windows path names for loading config
- 0.9.58: The same as before, I just missed it hah
- 0.9.57: Changed generated schema type of `$createdAt` and `$updatedAt` to string from `string | Date` cause honestly if you want a date just parse it
- 0.9.56: Changed the updateAttribute so it doesn't always update attributes and hopefully fixed the required error
- 0.9.55: Updated to use `node-appwrite@14` for appwrite 1.6.0
- 0.9.54: Added small delay (`100ms`) between collection deletions, reduced other delays from `1000` to `500/250ms`
- 0.9.53: Reduced delay, went too far
- 0.9.52: Add delay after creating indexes, attributes, and others to prevent `fetch failed` errors during large-scale collection creation
- 0.9.51: Fix transfer databases, remove "ensure duplicates" check
- 0.9.5: Fixed not checking for storage bucket for each database (checking the creation status) when importing data
- 0.9.4: Fixed migrations database ensuring it has the required collections
- 0.9.3: Fixed deployment error && fix lack of existing `appwriteConfig.ts` file from causing error (you want to be able to setup yeah? haha)
- 0.9.2: Added changelog back, whoops
- 0.0.90: Updated README with new command-line options and fixed alias issues
- 0.0.74: Added `--backup` support, even if only one database
- 0.0.73: Fixed weird `workspace` issue
- 0.0.72: Remove