@gftdcojp/gftd-cli/README.md

GFTD Schema Management CLI - Drizzle-like schema management for Kafka and ksqlDB

# GFTD CLI - Kafka & ksqlDB Schema Management

Drizzle-like schema management tool for Kafka topics and ksqlDB streams/tables with organization support.

## 🚀 Quick Start

### Installation

```bash
npm install -g @gftdcojp/gftd-cli
# or
pnpm add -g @gftdcojp/gftd-cli
```

### Initialize Project

```bash
gftd init --org-id=myorg --project=myproject
```

### Configure Credentials

Edit `gftd.config.yml`:

```yaml
orgId: "myorg"
project: "myproject" # optional

kafka:
  endpoint: ${KAFKA_ENDPOINT}
  apiKey: ${KAFKA_API_KEY}
  apiSecret: ${KAFKA_API_SECRET}

ksqlDB:
  endpoint: ${KSQLDB_ENDPOINT}
  apiKey: ${KSQLDB_API_KEY}
  apiSecret: ${KSQLDB_API_SECRET}
```

### Run Migrations

```bash
gftd migrate
```

## 📋 Commands

### Core Commands
| Command | Description |
|---------|-------------|
| `gftd init` | Initialize new project |
| `gftd migrate` | Run schema migrations |
| `gftd status` | Check current schema status |
| `gftd validate` | Validate schema definition |
| `gftd generate` | Generate SQL files only |
| `gftd info` | Show project information |

### Database Management (Supabase-like + Drizzle-style)
| Command | Description |
|---------|-------------|
| `gftd db:init` | Initialize database project with schema structure |
| `gftd migration:generate` | Generate migration from schema changes |
| `gftd migration:up` | Apply pending migrations |
| `gftd migration:down` | Rollback migrations |
| `gftd migration:list` | List migration status |
| `gftd generate:types` | Generate TypeScript types from schema |
| `gftd db:pull` | Pull schema from remote database |
| `gftd db:push` | Push schema changes to database |

### Development Tools
| Command | Description |
|---------|-------------|
| `gftd dev` | Start local development server with proxy |

## 🏗️ Schema Definition

Define your schemas in `gftd.schema.yml`:

```yaml
orgId: "myorg"
project: "myproject"
version: "1.0.0"

topics:
  users:
    partitions: 3
    replicationFactor: 3
    configEntries:
      - name: "cleanup.policy"
        value: "compact"
      - name: "retention.ms"
        value: "-1"

streams:
  users:
    valueFormat: "JSON"
    keyFormat: "KAFKA"
    columns:
      - name: "ID"
        type: "VARCHAR"
        key: true
      - name: "NAME"
        type: "VARCHAR"
      - name: "EMAIL"
        type: "VARCHAR"

tables:
  users:
    aggregation: |
      SELECT ID,
             LATEST_BY_OFFSET(NAME) AS NAME,
             LATEST_BY_OFFSET(EMAIL) AS EMAIL
      FROM {sourceStream}
      GROUP BY ID
```

## 📖 Naming Convention

Resources are created with the following naming pattern:

### Simple Environment-based Naming (env = orgId)

- **Topic**: `{topicName}`
  - Example: `users`
- **Stream**: `{ORGID}_{STREAM_NAME}`
  - Example: `MYORG_USERS`
- **Table**: `{ORGID}_{TABLE_NAME}`
  - Example: `MYORG_USERS`

This simplified approach treats the organization ID as the environment, making resource names cleaner and easier to manage.

## 🔧 Configuration

### Environment Variables

Create `.env` file:

```bash
KAFKA_ENDPOINT=your-kafka-endpoint
KAFKA_API_KEY=your-kafka-api-key
KAFKA_API_SECRET=your-kafka-api-secret

KSQLDB_ENDPOINT=your-ksqldb-endpoint
KSQLDB_API_KEY=your-ksqldb-api-key
KSQLDB_API_SECRET=your-ksqldb-api-secret
```

### Configuration File

`gftd.config.yml`:
- `orgId`: Organization identifier (required)
- `project`: Project name (optional)
- `kafka`: Kafka connection settings
- `ksqlDB`: ksqlDB connection settings
- `schemaPath`: Path to schema file (default: `./gftd.schema.yml`)
- `migrationPath`: Migration output directory (default: `./migrations`)

## 📊 Schema Validation

The CLI validates your schema for:
- Required fields
- Column type compatibility
- Topic/stream/table relationships
- Naming convention compliance

## 🎯 Example Workflow

```bash
# 1. Initialize project
gftd init --org-id=myorg --project=myapp

# 2. Edit gftd.schema.yml with your schema definition

# 3. Validate schema
gftd validate

# 4. Run migrations (dry run first)
gftd migrate --dry-run

# 5. Apply changes
gftd migrate

# 6. Check status
gftd status
```

## 🔗 Integration

Use with your favorite CI/CD:

```yaml
# GitHub Actions example
- name: Validate Schema
  run: gftd validate

- name: Run Migrations
  run: gftd migrate
  env:
    KAFKA_ENDPOINT: ${{ secrets.KAFKA_ENDPOINT }}
    KAFKA_API_KEY: ${{ secrets.KAFKA_API_KEY }}
    KAFKA_API_SECRET: ${{ secrets.KAFKA_API_SECRET }}
```

## 🚨 Migration Safety

- Always run with `--dry-run` first
- Migrations are executed in dependency order
- Use `--force` flag to continue on failures (use with caution)
- Generated SQL files are saved in migration directory

## 🗄️ Database Management (Supabase-like + Drizzle-style)

GFTD CLI provides a comprehensive database management system that combines the best of Supabase CLI and Drizzle's migration approach.

### Features

- 🏗️ **Supabase-like CLI**: Familiar commands for database operations
- 🔄 **Drizzle-style Migrations**: TypeScript-first schema management
- 📝 **Type Generation**: Automatic TypeScript types from database schema
- 🛠️ **Multi-database Support**: PostgreSQL, MySQL, SQLite, MongoDB
- 🔐 **Schema Validation**: Built-in schema validation and safety checks
- 🌱 **Seeding**: Database seeding capabilities

### Quick Start

```bash
# Initialize database project
gftd db:init --type postgresql --database myapp_dev

# Generate migration from schema changes
gftd migration:generate "add_user_table"

# Apply migrations
gftd migration:up

# Generate TypeScript types
gftd generate:types --output types/database.ts

# List migration status
gftd migration:list
```

### Database Configuration

GFTD uses `gftd.json` for database configuration:

```json
{
  "database": {
    "type": "postgresql",
    "host": "localhost",
    "port": 5432,
    "database": "myapp_dev",
    "username": "postgres",
    "password": "postgres",
    "migrationsPath": "migrations",
    "seedsPath": "seeds"
  },
  "migrations": {
    "path": "migrations",
    "tableName": "_gftd_migrations",
    "schemaName": "public",
    "createSchema": true
  }
}
```

### Schema-First Development

Define your database schema in TypeScript:

```typescript
// schemas/schema.ts
export interface User {
  id: number;
  email: string;
  name?: string;
  createdAt: Date;
  updatedAt: Date;
}

export interface Post {
  id: number;
  title: string;
  content: string;
  authorId: number;
  published: boolean;
  createdAt: Date;
  updatedAt: Date;
}

export type DatabaseSchema = {
  users: User;
  posts: Post;
};
```

### Migration Workflow

1. **Define Schema**: Update your TypeScript schema files
2. **Generate Migration**: `gftd migration:generate "description"`
3. **Review**: Check generated SQL files
4. **Apply**: `gftd migration:up`
5. **Generate Types**: `gftd generate:types`

### Advanced Usage

```bash
# Dry run migrations
gftd migration:up --dry

# Rollback migrations
gftd migration:down --steps 2

# Pull schema from existing database
gftd db:pull --force

# Push schema changes (dangerous!)
gftd db:push --drop --seed

# Watch for schema changes
gftd generate:types --watch
```

## 🚀 Local Development Proxy

GFTD CLI includes a powerful local development proxy system that allows you to access multiple projects via custom subdomains instead of remembering port numbers.

### Features

- 🌐 Access projects via `{id}.gftd.ai.dev` URLs
- 🔄 Automatic port allocation and management
- 🚀 Built-in project process management
- 📋 Easy project listing and control
- 🛑 Graceful shutdown and restart

### Quick Start

```bash
# Start development server in current directory
gftd dev

# Start with custom settings
gftd dev --name myproject --subdomain myapp --command "pnpm dev"

# Start proxy server only
gftd dev --proxy

# List all projects
gftd dev --list

# Stop a specific project
gftd dev --stop <project-id>

# Stop all projects
gftd dev --stop-all
```

### DNS Setup

To access projects via `*.gftd.ai.dev` domains, you need to set up DNS resolution:

#### Option 1: /etc/hosts (Simple)
```bash
# Add to /etc/hosts
127.0.0.1 *.gftd.ai.dev
```

#### Option 2: dnsmasq (Recommended)
```bash
# Install dnsmasq
brew install dnsmasq

# Configure wildcard DNS
echo "address=/.gftd.ai.dev/127.0.0.1" | sudo tee /etc/dnsmasq.d/gftd-dev.conf

# Start dnsmasq
sudo brew services start dnsmasq

# Configure DNS resolver
sudo mkdir -p /etc/resolver
echo "nameserver 127.0.0.1" | sudo tee /etc/resolver/gftd.ai.dev
```

#### Option 3: Built-in DNS Management (New!)
```bash
# GFTD CLI automatically manages your hosts file
gftd dev --name "My App" --subdomain "myapp"

# DNS entries are automatically added/removed
# No manual setup required!

# Check DNS status
gftd dns --status

# Initialize DNS management
gftd dns --init
```

### Example Workflow

```bash
# 1. Start your Next.js project
cd my-nextjs-app
gftd dev --name "My Next App" --subdomain "myapp"

# 2. Start another project
cd ../my-react-app
gftd dev --name "My React App" --subdomain "reactapp"

# 3. Access your projects
# - http://myapp.gftd.ai.dev
# - http://reactapp.gftd.ai.dev

# 4. List all running projects
gftd dev --list

# 5. Stop all projects when done
gftd dev --stop-all
```

### Configuration

The proxy system stores configuration in `gftd.json`:

```json
{
  "port": 8080,
  "domain": "gftd.ai.dev",
  "dnsEnabled": true,
  "dnsAutoManage": true,
  "projects": {
    "project-id": {
      "id": "project-id",
      "name": "My Project",
      "path": "/path/to/project",
      "port": 3000,
      "subdomain": "myproject",
      "command": "pnpm dev",
      "status": "running"
    }
  }
}
```

### DNS Management

GFTD CLI includes automatic DNS management that eliminates the need for manual DNS setup:

#### Features
- 🔧 **Automatic hosts file management**: No manual DNS configuration required
- 🛡️ **Safe backup & restore**: Automatic backup before any changes
- 🔄 **Auto cleanup**: DNS entries are removed when projects are deleted
- 🚀 **Zero configuration**: Works out of the box with proper permissions
- 🌐 **Cross-platform**: Works on macOS, Linux, and Windows

#### DNS Commands
| Command | Description |
|---------|-------------|
| `gftd dns --status` | Show DNS management status |
| `gftd dns --init` | Initialize DNS management |
| `gftd dns --cleanup` | Remove all GFTD DNS entries |
| `gftd dns --restore` | Restore DNS from backup |
| `gftd dns --add <hostname>` | Add DNS entry manually |
| `gftd dns --remove <hostname>` | Remove DNS entry manually |

#### How It Works
1. **Automatic Setup**: When you start a project, GFTD automatically adds a DNS entry to your hosts file
2. **Safe Management**: Creates backup before making any changes
3. **Clean Removal**: Removes DNS entries when projects are stopped or deleted
4. **Permission Handling**: Uses sudo when needed for hosts file access

#### Example Usage
```bash
# Start a project (DNS automatically managed)
gftd dev --name "My App" --subdomain "myapp"
# DNS entry added: 127.0.0.1 myapp.gftd.ai.dev

# Check DNS status
gftd dns --status

# Remove project (DNS automatically cleaned up)
gftd dev --remove <project-id>
# DNS entry removed automatically

# Manual DNS management
gftd dns --add custom.gftd.ai.dev
gftd dns --remove custom.gftd.ai.dev
```

## 📝 License

MIT

## 🤝 Contributing

1. Fork the repository
2. Create your feature branch
3. Commit your changes
4. Push to the branch
5. Create a Pull Request

## 🔬 統合設定オプション (NEW!)

圏論・写像の概念を活用した数理的に簡潔な統合設定が利用可能です。

### 統合設定の特徴

- **単一ファイル管理**: `gftd.yml` で設定とスキーマを統合
- **YAML参照**: `&` と `*` でDRY原則の徹底
- **コンパクト記法**: スキーマ定義が50%短縮
- **圏論的構造**:数学的一貫性のある設計

### 使用方法

```bash
# 統合設定でプロジェクト初期化
gftd init --org-id=myorg --project=myproject --unified

# 統合設定でマイグレーション
gftd migrate --unified

# 自動検出（gftd.ymlが存在する場合）
gftd migrate
```

### 統合設定例

```yaml
# gftd.yml - 設定とスキーマを統合
org: "myorg"
project: "myproject"

connections:
  kafka:
    endpoint: ${KAFKA_ENDPOINT}
    credentials: [${KAFKA_API_KEY}, ${KAFKA_API_SECRET}]
  ksqldb:
    endpoint: ${KSQLDB_ENDPOINT}
    credentials: [${KSQLDB_API_KEY}, ${KSQLDB_API_SECRET}]

schema:
  version: "1.0.0"
  topics:
    users:
      partitions: 3
      replication: 3
      config:
        cleanup.policy: compact
        retention.ms: -1

  streams:
    users:
      format: [JSON, KAFKA]
      schema:
        ID: VARCHAR(PK)    # コンパクト記法
        NAME: VARCHAR
        EMAIL: VARCHAR

  tables:
    users:
      source: users
      aggregation: |
        SELECT ID,
               LATEST_BY_OFFSET(NAME) AS NAME,
               LATEST_BY_OFFSET(EMAIL) AS EMAIL
        FROM {stream}
        GROUP BY ID

naming:
  topic: "{name}"
  stream: "{ORG}_{NAME}"
  table: "{ORG}_{NAME}"
```

### 詳細情報

統合設定の詳細は [CATEGORICAL.md](./CATEGORICAL.md) を参照してください。

## 🎯 クイックデモ

内蔵のDNS管理機能を試してみましょう：

```bash
# 1. テストプロジェクトに移動
cd test-project

# 2. 依存関係をインストール
npm install

# 3. GFTD CLI でプロジェクトを開始（DNS自動管理）
gftd dev --name "Test Project" --subdomain "test"

# 4. ブラウザで確認
# http://test.gftd.ai.dev でアクセス可能！

# 5. DNS状態を確認
gftd dns --status

# 6. プロジェクト一覧を確認
gftd dev --list

# 7. プロジェクトを停止
gftd dev --stop <project-id>
```

### 🚀 従来の方法との比較

**従来のやり方（dnsmasq使用）:**
```bash
# dnsmasqのインストール
brew install dnsmasq

# 設定ファイルの作成
echo "address=/.gftd.ai.dev/127.0.0.1" | sudo tee /etc/dnsmasq.d/gftd-dev.conf

# dnsmasqの起動
sudo brew services start dnsmasq

# DNS resolverの設定
sudo mkdir -p /etc/resolver
echo "nameserver 127.0.0.1" | sudo tee /etc/resolver/gftd.ai.dev

# プロジェクトの起動
npm run dev
```

**GFTD CLI（DNS自動管理）:**
```bash
# これだけ！
gftd dev --name "My Project" --subdomain "myproject"
```

### 🎉 特徴

- ✅ **dnsmasq不要**: 外部ツールのインストール不要
- ✅ **自動管理**: プロジェクトの追加・削除で自動的にDNSエントリを更新
- ✅ **安全**: 変更前に自動バックアップ、エラー時は復元可能
- ✅ **クロスプラットフォーム**: macOS、Linux、Windows対応
- ✅ **ゼロ設定**: 追加設定不要で即座に使用可能

---