@dataql/node/README.md

DataQL core SDK for unified data management with MongoDB and GraphQL - Production Multi-Cloud Ready

# DataQL Node SDK

## Installation

```sh
npm install @dataql/node
```

## Quick Start

```ts
import { Data } from "@dataql/node";

const data = new Data({
  appToken: "app_sometoken",
});

// Data operations
const users = data.collection("users", userSchema);
await users.create({ name: "Alice", email: "alice@example.com" });
```

## Features

- **🔌 Plugin Ecosystem**: Comprehensive plugin system for extending DataQL functionality
- **🔍 Database Introspection**: Automatically analyze existing databases and generate DataQL schemas
- **☁️ One-Click Cloud Migration**: Migrate entire databases to DataQL cloud infrastructure instantly
- **📱 Offline-First Architecture**: Work offline with automatic sync when reconnected
- **⚡ Real-Time Synchronization**: Live updates across all connected clients
- **🔄 Zero-API-Change Migration**: Drop-in replacement for existing ORMs and databases
- **🚀 Auto-Scaling Infrastructure**: Handles traffic spikes automatically
- **🗄️ Multi-Database Support**: MongoDB with expanding support for PostgreSQL, MySQL, SQLite
- **🔒 Type-Safe Operations**: Full TypeScript support with compile-time validation
- **🎯 Document-Scoped API**: Intuitive subdocument operations without MongoDB operators

## 🔌 Plugin Ecosystem

**NEW**: DataQL now features a revolutionary plugin system that transforms it into a extensible platform. Build database adapters, middleware, integrations, and extensions to create a powerful ecosystem around your data layer.

### Plugin Types

- **🔗 Database Adapters**: Support new database types (PostgreSQL, MySQL, SQLite, etc.)
- **⚙️ Middleware**: Process requests and responses (auth, rate limiting, logging)
- **🔧 Extensions**: Add new methods and functionality to DataQL
- **🪝 Hooks**: React to DataQL events and lifecycle
- **🔌 Integrations**: Connect with external services (analytics, notifications, etc.)
- **🎨 UI Components**: Build dashboards and management interfaces

### Quick Start

```ts
import { Data, Plugin } from "@dataql/node";

// Create a simple analytics plugin
class AnalyticsPlugin implements Plugin {
  id = "analytics";
  name = "Analytics Plugin";
  version = "1.0.0";
  type = "extension" as const;

  methods = {
    async trackEvent(eventName: string, properties?: any) {
      console.log(`Event: ${eventName}`, properties);
      // Send to analytics service
    },
  };

  async initialize(context) {
    context.logger.info("Analytics plugin initialized");
  }
}

// Register and use plugins
const data = new Data({ appToken: "your-token" });

await data.registerPlugin(new AnalyticsPlugin());
await data.initializePlugins();
await data.applyExtensions();

// Use extended functionality (added by plugin)
await (data as any).trackEvent("user_signup", { userId: "123" });
```

### Database Adapter Example

```ts
import { DatabaseAdapterPlugin } from "@dataql/node";

class PostgreSQLAdapter implements DatabaseAdapterPlugin {
  id = "postgresql-adapter";
  name = "PostgreSQL Adapter";
  version = "1.0.0";
  type = "adapter" as const;
  databaseType = "postgresql";

  async initialize(context) {
    // Setup PostgreSQL connection pool
  }

  async createConnection(connectionString: string) {
    // Create PostgreSQL connection
  }

  async introspect(connection, options) {
    // Analyze PostgreSQL database structure
  }

  async executeQuery(connection, query) {
    // Execute SQL queries
  }
}

// Register PostgreSQL support
await data.registerPlugin(new PostgreSQLAdapter(), {
  connectionString: "postgresql://user:pass@localhost:5432/db",
});
```

### Middleware Example

```ts
import { MiddlewarePlugin } from "@dataql/node";

class AuthMiddleware implements MiddlewarePlugin {
  id = "auth-middleware";
  name = "Authentication Middleware";
  type = "middleware" as const;
  order = 1; // Execute first

  async processRequest(request) {
    const token = request.headers.authorization;
    if (!token) throw new Error("Authentication required");

    // Verify JWT and add user to request
    request.user = await this.verifyToken(token);
    return request;
  }
}
```

### Integration Example

```ts
import { IntegrationPlugin } from "@dataql/node";

class SlackIntegration implements IntegrationPlugin {
  id = "slack-integration";
  name = "Slack Integration";
  type = "integration" as const;
  service = "slack";

  async syncTo(data, options) {
    // Send notification to Slack
    await this.sendToSlack({
      text: `DataQL Alert: ${data.message}`,
      channel: options?.channel || "#general",
    });
  }
}
```

### Hook System

React to DataQL events automatically:

```ts
import { HookPlugin } from "@dataql/node";

class CacheInvalidationHook implements HookPlugin {
  id = "cache-invalidation";
  type = "hook" as const;

  hooks = {
    afterCreate: async (data, context) => {
      await this.invalidateCache(data.collection);
    },
    afterUpdate: async (data, context) => {
      await this.invalidateCache(data.collection);
    },
  };
}
```

### Plugin Management

```ts
// Get plugin information
const plugins = data.getPlugins();
console.log(
  "Registered plugins:",
  plugins.map((p) => p.name)
);

// Plugin statistics
const stats = data.getPluginStats();
console.log("Plugin stats:", stats);

// Check for specific plugins
if (data.hasPlugin("postgres-adapter")) {
  console.log("PostgreSQL support available");
}

// Get plugins by type
const adapters = data.getPluginsByType("adapter");
const middleware = data.getPluginsByType("middleware");
```

### Plugin Discovery & Registry

```ts
import { PluginRegistry } from "@dataql/node";

const registry = new PluginRegistry();

// Search for plugins
const plugins = await registry.search("postgresql", "adapter");

// Install plugins
await registry.install("@yourorg/dataql-postgres-adapter");

// Publish your plugins
await registry.publish(myPlugin, manifest);
```

### Available Hooks

DataQL provides comprehensive hooks for plugin integration:

- **Request/Response**: `beforeRequest`, `afterRequest`
- **CRUD Operations**: `beforeCreate`, `afterCreate`, `beforeRead`, `afterRead`, `beforeUpdate`, `afterUpdate`, `beforeDelete`, `afterDelete`
- **Transactions**: `beforeTransaction`, `afterTransaction`
- **Database**: `beforeIntrospection`, `afterIntrospection`, `beforeMigration`, `afterMigration`
- **Lifecycle**: `onConnect`, `onDisconnect`, `onError`
- **Schema**: `onSchemaChange`, `onDataChange`

### Plugin Development

Create powerful plugins with full TypeScript support:

```ts
import { Plugin, PluginContext } from "@dataql/node";

export class MyPlugin implements Plugin {
  id = "my-plugin";
  name = "My Plugin";
  version = "1.0.0";
  description = "Custom plugin description";
  author = "Your Name";
  type = "extension" as const;

  configSchema = {
    apiKey: { type: "string", required: true },
    endpoint: { type: "string", required: false },
  };

  async initialize(context: PluginContext) {
    const { config, logger, events, utils } = context;

    // Setup plugin functionality
    logger.info("Plugin initialized");

    // Listen to events
    events.on("afterCreate", (data) => {
      // React to data creation
    });
  }

  async destroy() {
    // Cleanup resources
  }
}
```

### Plugin Package Structure

```json
{
  "name": "@yourorg/dataql-my-plugin",
  "version": "1.0.0",
  "dataql": {
    "plugin": {
      "type": "extension",
      "entry": "./dist/index.js",
      "name": "My Plugin",
      "config": {
        "apiKey": { "type": "string", "required": true }
      }
    }
  },
  "peerDependencies": {
    "@dataql/node": "^0.7.0"
  }
}
```

See the complete [Plugin Development Guide](./PLUGIN_DEVELOPMENT_GUIDE.md) for detailed documentation and examples.

## 🔍 Database Introspection

**NEW**: DataQL can automatically analyze your existing database and generate schemas, allowing you to migrate without changing your data structure. The introspection follows DataQL's core architecture: **SDK → Worker → Lambda → Database** (no direct database connections from SDK).

```ts
import { Data } from "@dataql/node";

const data = new Data({
  appToken: "your-app-token",
  env: "dev",
});

// Introspect an existing MongoDB database
const result = await data.introspect("mongodb://localhost:27017/ecommerce", {
  databaseName: "ecommerce",
  sampleSize: 100,
  includeIndexes: true,
  excludeCollections: ["logs", "temp"],
});

if (result.success) {
  console.log(`Found ${Object.keys(result.schemas).length} collections`);

  // Schemas are automatically registered - use them immediately!
  const users = data.collection("users", result.schemas.users);
  const products = data.collection("products", result.schemas.products);

  // All DataQL features now available on your existing data
  const allUsers = await users.find();
  await products.create({ name: "New Product", price: 99.99 });
}
```

### Introspection Benefits

- 🚀 **Zero-API-change migration**: Keep your existing database, gain DataQL benefits
- 🤖 **Smart type inference**: Automatically detects field types, formats, and constraints
- 📊 **Index preservation**: Maintains your existing database indexes and relationships
- 🔄 **Multi-database support**: MongoDB (available now), PostgreSQL, MySQL coming soon
- ⚡ **Instant migration**: From analysis to working DataQL code in seconds

### Advanced Introspection

```ts
// Fine-tune introspection for large databases
const result = await data.introspect(databaseUrl, {
  sampleSize: 50, // Documents to sample per collection
  maxDepth: 3, // Nested object analysis depth
  includeIndexes: true, // Analyze database indexes
  excludeCollections: ["logs", "analytics"], // Skip large collections
  includeCollections: ["users", "orders"], // Only analyze specific collections
});

// Use directly without Data instance
import { IntrospectionService } from "@dataql/node";

const result = await IntrospectionService.introspect(
  {
    url: "mongodb://localhost:27017/blog",
    name: "blog",
  },
  {
    sampleSize: 100,
    includeIndexes: true,
  }
);
```

See the [full introspection guide](./INTROSPECTION_GUIDE.md) for complete documentation and examples.

## ☁️ One-Click Cloud Migration

**REVOLUTIONARY**: Migrate your entire database to DataQL's cloud infrastructure in **ONE LINE OF CODE**! Get instant access to offline-first capabilities, real-time sync, and auto-scaling.

```ts
// Migrate your entire database to DataQL cloud in one line!
const result = await data.migrateToCloud("mongodb://localhost:27017/ecommerce");

if (result.success) {
  console.log(`✅ Migration completed!`);
  console.log(`☁️  Cloud database: ${result.cloudDatabase.name}`);
  console.log(`🔗 New connection: ${result.cloudDatabase.connectionString}`);
  console.log(
    `📊 Migrated: ${result.migration.collections} collections, ${result.migration.documents} documents`
  );

  // Your data is now available with DataQL superpowers!
  // Schemas are auto-registered and ready to use immediately
  const users = data.collection("users", result.schemas.users);
  const products = data.collection("products", result.schemas.products);

  // Everything works exactly the same, but now with:
  // ✅ Offline-first capabilities
  // ✅ Real-time synchronization
  // ✅ Auto-scaling infrastructure
  // ✅ Built-in caching and optimization
}
```

### Migration Features

- **One-Click Migration**: Entire database migration in a single method call
- **Zero Downtime**: Migrate without interrupting your application
- **Data Validation**: Automatic integrity checks during migration
- **Rollback Support**: 24-hour rollback window for peace of mind
- **Instant Benefits**: Immediate access to DataQL's powerful features
- **Smart Optimization**: Automatic performance tuning for your data patterns

### Advanced Migration Options

```ts
const result = await data.migrateToCloud(
  "mongodb://localhost:27017/ecommerce",
  {
    // Introspection options
    sampleSize: 500,
    includeIndexes: true,
    excludeCollections: ["temp_logs", "cache_data"],

    // Migration options
    batchSize: 2000,
    validateData: true,
    preserveIds: true,
    createBackup: true,
  }
);

// Real-world production migration
const prodResult = await data.migrateToCloud(
  "mongodb+srv://user:pass@cluster.mongodb.net/production",
  {
    excludeCollections: ["sessions", "temp_data", "audit_logs"],
    batchSize: 1000,
    validateData: true,
    preserveIds: true,
  }
);

if (prodResult.success) {
  console.log("🎉 Production migration successful!");
  console.log(`📊 Migrated ${prodResult.migration.collections} collections`);
  console.log(`⏱️  Duration: ${prodResult.migration.duration}ms`);

  // Update your environment variables:
  console.log(`DATABASE_URL=${prodResult.cloudDatabase.connectionString}`);
  console.log(`MIGRATION_ID=${prodResult.migrationId}`);
}
```

### Why Migrate to DataQL Cloud?

- 🚀 **Instant Performance**: Auto-scaling infrastructure handles any load
- 📱 **Offline-First**: Apps work perfectly without internet connection
- 🔄 **Real-Time Sync**: Live updates across all connected clients
- 🔒 **Built-in Security**: Enterprise-grade security and compliance
- 🌍 **Global CDN**: Fast access from anywhere in the world
- 🧠 **Smart Caching**: Intelligent caching reduces database load
- 🔧 **Zero Maintenance**: No server management or scaling concerns

### Example: E-commerce Migration

```ts
// Real-world e-commerce platform migration
const result = await data.migrateToCloud(
  "mongodb+srv://admin:pass@cluster.mongodb.net/ecommerce_prod",
  {
    excludeCollections: ["user_sessions", "audit_logs", "temp_cart_items"],
    batchSize: 2000,
    includeIndexes: true,
    validateData: true,
    preserveIds: true,
  }
);

if (result.success) {
  // Your e-commerce platform now has:
  // ✅ Offline-first shopping experience
  // ✅ Real-time inventory synchronization
  // ✅ Auto-scaling for Black Friday traffic
  // ✅ Global CDN for worldwide customers
  // ✅ Built-in conflict resolution for concurrent orders

  // Immediate usage of migrated data
  const products = data.collection("products", result.schemas.products);
  const orders = data.collection("orders", result.schemas.orders);

  // Create a new product to test the migrated system
  await products.create({
    name: "DataQL Migration Success T-Shirt",
    price: 29.99,
    category: "Apparel",
    description: "Celebrate your successful database migration!",
  });
}
```

For comprehensive migration examples, see `examples/cloud-migration.ts`.

## Example: Document-Scoped Subdocument Operations

```ts
const users = data.collection("users", userSchema);

// Create a user
const newUser = await users.create({
  name: "Alice",
  email: "alice@example.com",
});

// Document-scoped subdocument operations
const user = users({ id: newUser.insertedId });

// Work with subdocuments
await user.addresses.create({
  street: "123 Main St",
  city: "New York",
  zipCode: "10001",
});

await user.addresses.update({ city: "New York" }, { zipCode: "10002" });

const nyAddresses = await user.addresses.find({ city: "New York" });

await user.preferences.update({
  theme: "dark",
  notifications: true,
});
```

For more examples, see `examples/document-scoped-usage.ts`.

## Unique Document Creation

DataQL provides `createUnique()` method for both collections and subcollections to prevent duplicate documents based on comparable fields (excluding ID and subdocument fields).

### Collection-level createUnique

```ts
const users = data.collection("users", userSchema);

// Create a unique user (will create new document)
const result1 = await users.createUnique({
  name: "John Doe",
  email: "john@example.com",
  age: 30,
  preferences: {
    theme: "dark",
    notifications: true,
  },
});

console.log(result1.isExisting); // false
console.log(result1.insertedId); // "uuid-123"

// Try to create the same user again (will return existing)
// Comparison excludes 'id', 'createdAt', 'updatedAt', and subdocuments like 'preferences'
const result2 = await users.createUnique({
  name: "John Doe", // Same
  email: "john@example.com", // Same
  age: 30, // Same
  preferences: {
    theme: "light", // Different but ignored (subdocument)
    notifications: false,
  },
});

console.log(result2.isExisting); // true
console.log(result2.insertedId); // "uuid-123" (same as result1)
```

### Subcollection createUnique

```ts
const user = users({ id: userId });

// Create unique address
const address1 = await user.addresses.createUnique({
  street: "123 Main St",
  city: "New York",
  zipCode: "10001",
  country: "USA",
});

console.log(address1.isExisting); // false

// Try to create the same address (will return existing)
const address2 = await user.addresses.createUnique({
  street: "123 Main St",
  city: "New York",
  zipCode: "10001",
  country: "USA",
});

console.log(address2.isExisting); // true
```

### Excluded from Comparison

The `createUnique` method excludes these fields from uniqueness comparison:

1. **ID fields**: `id`, `_id`, and any field containing 'id'
2. **Auto-generated timestamps**: `createdAt`, `updatedAt`
3. **Subdocument objects**: Nested objects like `preferences`, `profile`
4. **Subdocument arrays**: Arrays of objects like `addresses`, `reviews`

### Compared Fields

Only these fields are compared for uniqueness:

1. **Primitive fields**: strings, numbers, booleans
2. **Enum fields**: category selections
3. **Simple arrays**: arrays of primitive values (if any)

### Return Value

```ts
type CreateUniqueResult = {
  insertedId?: string; // ID of document (new or existing)
  result?: any; // Full document data
  isExisting?: boolean; // true if existed, false if newly created
};
```

For complete examples, see `examples/unique-document-creation.ts`.

## Collection References

DataQL supports defining references between collections using the `Ref()` function, enabling relational data modeling with type safety.

### Quick Summary

To define references between collections in DataQL:

1. **Import `Ref`**: `import { Ref } from "@dataql/node"`
2. **Define reference fields**: `ownerId: Ref(userSchema)` or `assigneeId: Ref("User")`
3. **Arrays of references**: `memberIds: [Ref(userSchema)]`
4. **Use in operations**: Store actual IDs as values: `{ ownerId: "user-123" }`

### Defining References

```ts
import { Data, ID, String, Ref } from "@dataql/node";

// Define schemas
const userSchema = {
  id: ID,
  name: String,
  email: String,
} as const;

const projectSchema = {
  id: ID,
  name: String,
  description: String,
  ownerId: Ref(userSchema), // Reference to user schema
  teamMemberIds: [Ref(userSchema)], // Array of user references
} as const;

const taskSchema = {
  id: ID,
  title: String,
  completed: Boolean,
  userId: Ref(userSchema), // Reference to user
  projectId: Ref(projectSchema), // Reference to project
  assigneeId: Ref("User"), // Reference by string name
} as const;
```

### Reference Syntax Options

```ts
// 1. Schema Object Reference (Recommended)
ownerId: Ref(userSchema), // Direct schema reference

// 2. String Reference
assigneeId: Ref("User"), // Reference by collection name

// 3. Array of References
memberIds: [Ref(userSchema)], // Array of user references
tagIds: [Ref("Tag")], // Array of string references

// 4. Nested Reference Objects
participants: [
  {
    userId: Ref(userSchema),
    role: ["admin", "member", "viewer"],
    joinedAt: "Timestamp",
  }
],
```

### Working with Referenced Data

```ts
const data = new Data({ appToken: "your-token" });

const users = data.collection("users", userSchema);
const projects = data.collection("projects", projectSchema);
const tasks = data.collection("tasks", taskSchema);

// Create referenced documents
const user = await users.create({
  name: "Alice Johnson",
  email: "alice@example.com",
});

const project = await projects.create({
  name: "Website Redesign",
  description: "Complete redesign of company website",
  ownerId: user.insertedId, // Reference the user's ID
  teamMemberIds: [user.insertedId], // Array of user IDs
});

const task = await tasks.create({
  title: "Design homepage mockup",
  completed: false,
  userId: user.insertedId, // Reference to user
  projectId: project.insertedId, // Reference to project
});
```

### Cross-Collection Queries

```ts
// Find tasks for a specific user
const userTasks = await tasks.find({ userId: user.insertedId });

// Find projects owned by a user
const userProjects = await projects.find({ ownerId: user.insertedId });

// Find all team members for a project
const project = await projects.find({ id: projectId });
const teamMembers = await users.find({
  id: { $in: project[0].teamMemberIds },
});
```

### Reference Validation

DataQL automatically validates that referenced IDs exist when creating or updating documents:

```ts
// This will succeed if the user exists
await tasks.create({
  title: "New task",
  userId: "existing-user-id",
  projectId: "existing-project-id",
});

// This will fail with a validation error
await tasks.create({
  title: "Invalid task",
  userId: "non-existent-user-id", // Validation error
});
```

### Document-Scoped Operations with References

```ts
// Work with referenced data using document-scoped API
const project = projects({ id: projectId });

// Add team members to project
await project.update({
  teamMemberIds: [...existingMemberIds, newUserId],
});

// Find tasks for this project
const projectTasks = await tasks.find({ projectId: projectId });

// Update task assignments
const task = tasks({ id: taskId });
await task.update({ userId: newAssigneeId });
```

### Advanced Reference Patterns

```ts
// Self-referencing (e.g., user management hierarchy)
const userSchema = {
  id: ID,
  name: String,
  managerId: Ref("User"), // Self-reference
  directReports: [Ref("User")], // Array of self-references
} as const;

// Polymorphic references (reference to multiple collection types)
const commentSchema = {
  id: ID,
  content: String,
  authorId: Ref(userSchema),
  // Reference different entity types
  entityType: ["task", "project", "user"],
  entityId: String, // ID of the referenced entity
} as const;

// Many-to-many relationships through junction collections
const userProjectSchema = {
  id: ID,
  userId: Ref(userSchema),
  projectId: Ref(projectSchema),
  role: ["owner", "member", "viewer"],
  joinedAt: "Timestamp",
} as const;
```

### Reference Benefits

- **Type Safety**: Full TypeScript inference for referenced fields
- **Validation**: Automatic validation of reference integrity
- **Performance**: Optimized queries for related data
- **Consistency**: Maintains data relationships across collections
- **Flexibility**: Support for various reference patterns and relationships

For complete examples, see `examples/simple-references.ts` and `examples/collection-references.ts`.

### Schema Composition Benefits

Schema composition provides several advantages:

- **Reusability**: Define common structures once, use everywhere
- **Maintainability**: Update schema in one place, changes propagate
- **Type Safety**: Full TypeScript inference across composed schemas
- **Consistency**: Ensure uniform data structures across collections
- **Modularity**: Build complex schemas from simple building blocks

### Using Composed Schemas

```ts
// Create collections using composed schemas
const users = data.collection("users", userSchema);
const accounts = data.collection("accounts", accountSchema);
const organizations = data.collection("organizations", organizationSchema);

// Create documents with composed subdocuments
await accounts.create({
  accountName: "Enterprise Account",
  users: [
    { name: "John Doe", email: "john@example.com" },
    { name: "Jane Smith", email: "jane@example.com" },
  ],
  billingAddress: {
    street: "123 Business Ave",
    city: "Enterprise City",
    zipCode: "12345",
    country: "USA",
  },
  shippingAddresses: [
    {
      street: "456 Shipping St",
      city: "Warehouse City",
      zipCode: "67890",
      country: "USA",
    },
  ],
});
```

## Working with Subdocuments

### Creating Documents with Subdocuments

```ts
const users = data.collection("users", userSchema);

const newUser = await users.create({
  name: "John Doe",
  email: "john@example.com",
  preferences: {
    theme: "dark",
    notifications: true,
    language: "en",
  },
  addresses: [
    {
      street: "123 Main St",
      city: "New York",
      zipCode: "10001",
      country: "USA",
    },
    {
      street: "456 Oak Ave",
      city: "San Francisco",
      zipCode: "94102",
      country: "USA",
    },
  ],
  profile: {
    bio: "Software developer",
    social: {
      twitter: "@johndoe",
      github: "johndoe",
    },
    work: {
      company: "Tech Corp",
      position: "Senior Developer",
      address: {
        street: "789 Tech Blvd",
        city: "Silicon Valley",
        country: "USA",
      },
    },
  },
});
```

### Working with Composed Subdocuments

```ts
const accounts = data.collection("accounts", accountSchema);

// Document-scoped operations on composed subdocuments
const account = accounts({ id: accountId });

// Add new user to account
await account.users.create({
  name: "New User",
  email: "newuser@example.com",
});

// Update billing address (composed addressSchema)
await account.billingAddress.update({
  street: "Updated Street",
  city: "New City",
});

// Add new shipping address
await account.shippingAddresses.create({
  street: "Additional Shipping St",
  city: "Shipping City",
  zipCode: "99999",
  country: "USA",
});
```

### Querying Subdocuments

```ts
// Find users by nested field
const users = await users.find({
  "preferences.theme": "dark",
});

// Find users by deeply nested field
const techUsers = await users.find({
  "profile.work.company": "Tech Corp",
});

// Find users by array subdocument field
const nyUsers = await users.find({
  "addresses.city": "New York",
});
```

### Updating Subdocuments

```ts
const users = data.collection("users", userSchema);
const user = users({ id: userId });

// Update nested object fields directly
await user.preferences.update({
  theme: "light",
});

await user.profile.update({
  bio: "Senior Software Developer",
});

// Update entire nested object
await user.preferences.update({
  theme: "light",
  notifications: false,
  language: "es",
});

// Add to array of subdocuments (document-scoped API)
await user.addresses.create({
  street: "789 New St",
  city: "Boston",
  zipCode: "02101",
  country: "USA",
});

// Update specific subdocuments in array
await user.addresses.update({ city: "Boston" }, { zipCode: "02102" });
```

### Finding and Updating Subdocuments by Field

DataQL provides powerful document-scoped operations for working with subdocuments in an intuitive way.

#### Document-Scoped Subdocument API

```ts
const users = data.collection("users", userSchema);

// Document-scoped subdocument operations
const user = users({ id: userId });

// Create new subdocuments
await user.addresses.create({
  street: "123 Boston St",
  city: "Boston",
  zipCode: "02101",
  country: "USA",
});

// Find specific addresses within a user's document
const bostonAddresses = await user.addresses.find({ city: "Boston" });

// Update specific subdocuments by field
await user.addresses.update(
  { city: "Boston" }, // Find criteria
  { zipCode: "02102" } // Update data
);

// Delete specific subdocuments
await user.addresses.delete({ city: "Boston" });

// Batch operations on subdocuments
await user.addresses.updateMany({ country: "USA" }, { isActive: true });
await user.addresses.deleteMany({ isActive: false });

// Nested subdocument updates
await user.profile.update({ bio: "Updated bio text" });

// Variable-based approach for multiple operations
const userAddresses = user.addresses;
await userAddresses.create({ street: "456 Main St", city: "NYC" });
await userAddresses.update({ city: "NYC" }, { zipCode: "10001" });
const nycAddresses = await userAddresses.find({ city: "NYC" });
```

#### Benefits of Document-Scoped API

- **More Intuitive**: Natural object-oriented operations instead of MongoDB operators
- **Type Safe**: Full TypeScript support for subdocument fields
- **Document Context**: Clear parent-child relationship with automatic scoping
- **Cleaner Syntax**: No need for `$push`, `$pull`, `$` positional operators
- **Variable References**: Store document/subdocument instances for cleaner multi-operation code

## Schema Definition

### Basic Schema

```ts
const userSchema = {
  id: "ID",
  name: "String",
  email: "String",
  age: "Int",
  isActive: "Boolean",
  createdAt: "Date",
} as const;
```

### Auto-Generated Fields

DataQL automatically adds essential metadata fields to all schemas if they're not explicitly defined:

```ts
// You define this minimal schema
const userSchema = {
  name: "String",
  email: "String",
} as const;

// DataQL automatically expands it to include:
// {
//   id: "ID",           // Auto-added if not present
//   name: "String",
//   email: "String",
//   createdAt: "Timestamp", // Auto-added if not present
//   updatedAt: "Timestamp", // Auto-added if not present
// }
```

**Auto-generated fields:**

- **`id`**: Unique identifier (UUID v4 string) - added if not specified
- **`createdAt`**: Creation timestamp - added if not specified
- **`updatedAt`**: Last modification timestamp - added if not specified

This ensures all documents have consistent metadata without requiring boilerplate in every schema definition.

**When to explicitly define these fields:**

- **Custom ID format**: If you need a specific ID format (e.g., `orderId: "String"` for custom order numbers)
- **Different timestamp types**: If you prefer `Date` over `Timestamp` for certain use cases
- **Field constraints**: If you need validation rules or default values for these fields

```ts
// Explicit definition example
const customSchema = {
  orderId: "String", // Custom ID instead of auto-generated UUID
  name: "String",
  email: "String",
  createdAt: "Date", // Use Date instead of Timestamp
  // updatedAt will still be auto-added as Timestamp
} as const;
```

### Enum Field Definitions

DataQL supports multiple syntaxes for defining enum fields:

```ts
const userSchema = {
  name: "String",

  // Simple array syntax (recommended for clean schemas)
  role: ["admin", "user", "guest"],
  status: ["active", "inactive", "pending"],

  // Explicit enum object syntax (for additional options)
  priority: { enum: ["low", "medium", "high"] },
  theme: { enum: ["light", "dark"], required: true },

  // Mixed usage is perfectly valid
  department: ["engineering", "sales", "marketing"],
  level: { enum: ["junior", "mid", "senior"], required: true },
} as const;
```

**Benefits of simplified enum syntax:**

- **Cleaner code**: `["one", "two", "three"]` vs `{ enum: ["one", "two", "three"] }`
- **Less verbose**: Reduces boilerplate for simple enums
- **TypeScript friendly**: Full type inference and autocompletion
- **Backward compatible**: Existing `{ enum: [...] }` syntax still works

### Subdocuments (Nested Objects)

DataQL supports multiple ways to define subdocuments and nested data structures:

#### 1. Simple Nested Objects

```ts
const userSchema = {
  id: "ID",
  name: "String",
  email: "String",
  preferences: {
    theme: "String",
    notifications: "Boolean",
    language: "String",
  },
} as const;
```

#### 2. Arrays of Subdocuments

```ts
const userSchema = {
  id: "ID",
  name: "String",
  addresses: [
    {
      street: "String",
      city: "String",
      zipCode: "String",
      country: "String",
    },
  ],
} as const;
```

#### 3. Deep Nested Structures

```ts
const userSchema = {
  id: "ID",
  name: "String",
  profile: {
    bio: "String",
    social: {
      twitter: "String",
      github: "String",
      linkedin: "String",
    },
    work: {
      company: "String",
      position: "String",
      address: {
        street: "String",
        city: "String",
        country: "String",
      },
    },
  },
} as const;
```

#### 4. Schema Composition and Reusability

```ts
// Define reusable schemas (IDs auto-added by DataQL)
const addressSchema = {
  street: "String",
  city: "String",
  zipCode: "String",
  country: "String",
} as const;

const userSchema = {
  name: "String",
  email: "String",
} as const;

// Compose schemas using references (IDs auto-added)
const accountSchema = {
  accountName: "String",
  users: [userSchema], // Array of user subdocuments
  billingAddress: addressSchema, // Single address subdocument
  shippingAddresses: [addressSchema], // Array of address subdocuments
} as const;

const organizationSchema = {
  name: "String",
  accounts: [accountSchema], // Nested composition
  headquarters: addressSchema,
  employees: [userSchema],
} as const;
```

#### 5. Simplified Enum Syntax

```ts
// DataQL supports multiple enum syntaxes
const productSchema = {
  name: "String",
  category: ["electronics", "clothing", "books"], // Array shorthand
  status: { enum: ["active", "inactive"] }, // Explicit enum object
  priority: { enum: ["low", "medium", "high"], required: true }, // With options
} as const;
```

#### 6. Mixed Complex Schema

```ts
const orderSchema = {
  customerId: "String",
  status: ["pending", "completed", "cancelled"], // Simplified enum syntax
  items: [
    {
      productId: "String",
      name: "String",
      quantity: "Int",
      price: "Decimal",
      metadata: {
        category: "String",
        tags: ["String"],
      },
    },
  ],
  shipping: {
    address: {
      street: "String",
      city: "String",
      zipCode: "String",
    },
    method: "String",
    trackingNumber: "String",
  },
  payment: {
    method: "String",
    amount: "Decimal",
    currency: "String",
    processor: {
      name: "String",
      transactionId: "String",
      metadata: "Object", // For flexible JSON data
    },
  },
  createdAt: "Date",
  updatedAt: "Date",
} as const;
```

## Advanced Usage

- You can use `Data` and `BaseDataQLClient` classes directly for advanced scenarios.
- Supports transactions, bulk operations, and complex queries.
- Compatible with existing database adapters (Drizzle, Prisma, Mongoose, etc.).

## Dual ESM + CommonJS Support

DataQL Node SDK supports both ESM and CommonJS out of the box (Node.js 16+):

- **ESM Usage:**
  ```js
  import { Data } from "@dataql/node";
  const data = new Data({ appToken: "your-token" });
  ```
- **CommonJS Usage:**
  ```js
  const { Data } = require("@dataql/node");
  const data = new Data({ appToken: "your-token" });
  ```

The SDK uses the `exports` field in `package.json` to provide the correct entrypoint for both module types. No extra configuration is needed.

- ESM entry: `./dist/esm/index.js`
- CJS entry: `./dist/cjs/index.js`

**Note:** Do not add `type: "module"` to your own `package.json` unless you want your project to be ESM-only.

> **Note:** The SDK no longer accepts a `workerUrl` parameter. All requests are routed through the built-in DataQL worker URL (`https://edge.dataql.com`).
> If you need to override routing for advanced use cases, use the `customConnection` option.