miniml/README.md

A minimal, embeddable semantic data modeling language for generating SQL queries from YAML model definitions. Inspired by LookML.

# MiniML

A minimal, embeddable semantic data modeling language for generating SQL queries from YAML model definitions, inspired by LookML.

Ideal for connecting data to AI via MCP!

MiniML produces data-dictionary and SQL query output from dimensions and measures defined in a simple YAML model syntax.
Designed to be a stand-alone and light-weight library, embeddable in any server or app without requiring additional data platforming.
Supports BigQuery and Snowflake dialects.

## Table of Contents

- [Quick Start](#quick-start)
- [Installation](#installation)
- [Programmatic Usage](#programmatic-usage)
- [Model Definition](#model-definition)
- [Query Generation](#query-generation)
- [API Reference](#api-reference)
- [Examples](#examples)
- [Supported SQL Dialects](#supported-sql-dialects)
- [Contributing](#contributing)

## Quick Start

### 1. Define a Model

Create a YAML model file (`sales.yaml`) that describes your data structure:

```yaml
description: Sales fact table capturing transactional purchase data

dialect: snowflake
from: acme.public.sales
date_field: sale_date

dimensions:
  date:
    - The calendar date on which the sale occurred
    - DATE(sale_date)
  customer_id: Unique identifier for the customer making the purchase
  product_id: Unique identifier for the product sold

measures:
  total_amount:
    - Total sale amount generated by the transaction
    - SUM(total_amount)
  count:
    - The count of sales records
    - COUNT(*)
```

### 2. Produce a description of the model
```bash
npx miniml sales.yaml
```

**OUTPUT**
```
## DIMENSIONS
- `date` The calendar date on which the sale occurred.
- `sale_id` Unique identifier for each sale transaction.
- `customer_id` Unique identifier for the customer making the purchase.
- `customer_name` Display name or label associated with the customer.
- `product_id` Unique identifier for the product sold.
- `product_name` Display name or label associated with the product.
- `store_id` Unique identifier for the store where the sale took place.
- `store_name` Display name or label of the store location.

## MEASURES
- `total_amount` Total sale amount generated by the transaction.
- `price_avg` Average unit price across all sales.
- `count` The count of records for the row.
```

### 3. Generate a SQL query from the model
```bash
npx miniml sales.yaml --dimensions=date --measures=total_amount
```

**OUTPUT**
```sql
SELECT
  DATE(sale_date) AS date,
  SUM(total_amount) AS total_amount
FROM acme.public.sales
WHERE sale_date >= DATEADD(DAY, -30, CURRENT_DATE)
GROUP BY ALL
```

Add more options:

```bash
npx miniml sales.yaml \
  --dimensions=date,customer_name,product_name \
  --measures=total_amount,price_avg,count \
  --date-from=2024-01-01 \
  --date-to=2024-01-31 \
  --order-by=-date \
  --limit=10
```

**OUTPUT**
```
SELECT
  DATE(sale_date) AS date,
  customer_name AS customer_name,
  product_name AS product_name,
  SUM(total_amount) AS total_amount,
  AVG(unit_price) AS price_avg,
  COUNT(*) AS count
FROM acme.public.sales
JOIN acme.customers USING (customer_id)
JOIN acme.products USING (product_id)
WHERE sale_date BETWEEN '2024-01-01' AND '2024-01-31'
GROUP BY ALL
ORDER BY date DESC
LIMIT 10
```

## Installation

To use MiniML in your project:

```bash
npm install miniml
```

## Programmatic Usage

```typescript
import { loadModel, renderQuery } from 'miniml';

// Load the model
const model = await loadModel('./sales.yaml');

// Generate a SQL query
const sql = renderQuery(model, {
    dimensions: ['date', 'customer_name'],
    measures: ['total_amount', 'count'],
    date_from: '2024-01-01',
    date_to: '2024-01-31',
    order_by: ['-date'],
    limit: 10
});

console.log(sql);
```

**OUTPUT**
```sql
SELECT
  DATE(sale_date) AS date,
  customer_name AS customer_name,
  product_name AS product_name,
  SUM(total_amount) AS total_amount,
  AVG(unit_price) AS price_avg,
  COUNT(*) AS count
FROM acme.public.sales
JOIN acme.customers USING (customer_id)
JOIN acme.products USING (product_id)
WHERE sale_date BETWEEN '2024-01-01' AND '2024-01-31'
GROUP BY ALL
ORDER BY date DESC
LIMIT 10
```

## Model Definition

### Basic Structure

A MiniML model is a YAML file with the following structure:

```yaml
description: Human-readable description of the model
dialect: bigquery | snowflake
from: schema.table_name
date_field: primary_date_column
where: base_filter_conditions

join:
  join_name: JOIN clause definition

dimensions:
  dimension_name: definition

measures:
  measure_name: definition
```

### Dimensions

Dimensions are fields used for grouping data. They can be defined in several formats:

**Simple string (field name only):**
```yaml
dimensions:
  customer_id: Unique customer identifier
```

**Array format (description, SQL, join):**
```yaml
dimensions:
  customer_name:
    - Display name of the customer
    - customer_name
    - customer_join
```

**Object format:**
```yaml
dimensions:
  date:
    description: The transaction date
    sql: DATE(created_at)
    join: date_dimension_join
```

### Measures

Measures are aggregated fields. They default to `SUM()` if no SQL is provided:

**Simple string (defaults to SUM):**
```yaml
measures:
  revenue: Total revenue amount  # becomes SUM(revenue) AS revenue
```

**Custom aggregation:**
```yaml
measures:
  avg_price:
    - Average unit price
    - AVG(unit_price)
```

### Joins

Define reusable join clauses:

```yaml
join:
  customer_join: LEFT JOIN customers USING (customer_id)
  product_join: LEFT JOIN products USING (product_id)

dimensions:
  customer_name:
    - Customer display name
    - customer_name
    - customer_join  # References the join above
```

### SQL Dialect Detection

MiniML can automatically detect the SQL dialect from the file path:
- Files containing "bigquery" use BigQuery syntax
- Files containing "snowflake" use Snowflake syntax
- Otherwise, specify explicitly with the `dialect` field

**Examples:**
```
models/bigquery/sales.yaml     # Automatically uses BigQuery dialect
models/snowflake/users.yaml    # Automatically uses Snowflake dialect  
models/sales.yaml              # Requires explicit dialect: bigquery
```

This means you can omit the `dialect` field when your file paths include the database type.

## Query Generation

### Query Options

The `renderQuery` function accepts these options:

```typescript
interface MinimlQueryOptions {
    dimensions?: string[];        // Fields to group by
    measures?: string[];          // Aggregated fields to calculate
    date_from?: string;          // Start date filter (YYYY-MM-DD)
    date_to?: string;            // End date filter (YYYY-MM-DD)
    where?: string;              // Additional WHERE conditions
    having?: string;             // HAVING clause for aggregate filters
    order_by?: string[];         // Sort fields (prefix with - for DESC)
    limit?: number;              // Limit result count
    distinct?: boolean;          // Add DISTINCT to SELECT
    date_granularity?: string;   // Date truncation (day, week, month, year)
}
```

### Date Handling

**Automatic date filtering:**
```typescript
// Last 30 days (default when date_field is defined)
renderQuery(model, { measures: ['count'] });

// Specific date range
renderQuery(model, {
    measures: ['count'],
    date_from: '2024-01-01',
    date_to: '2024-01-31'
});
```

**Date granularity:**
```typescript
// Group by week
renderQuery(model, {
    dimensions: ['date'],
    measures: ['count'],
    date_granularity: 'week'
});
// Generates: DATE_TRUNC('WEEK', sale_date) AS date
```

### Filtering

**WHERE clause (for dimensions):**
```typescript
renderQuery(model, {
    dimensions: ['customer_id'],
    measures: ['total_amount'],
    where: "customer_type = 'premium'"
});
```

**HAVING clause (for measures):**
```typescript
renderQuery(model, {
    dimensions: ['customer_id'],
    measures: ['total_amount'],
    having: "total_amount > 1000"
});
```

### Ordering

```typescript
renderQuery(model, {
    dimensions: ['customer_id'],
    measures: ['total_amount'],
    order_by: ['customer_id', '-total_amount']  // ASC, then DESC
});
```

## API Reference

### Core Functions

#### `loadModel(file: string): Promise<MinimlModel>`
Asynchronously loads and processes a YAML model file.

#### `loadModelSync(file: string): MinimlModel`
Synchronously loads and processes a YAML model file.

#### `renderQuery(model: MinimlModel, options: MinimlQueryOptions): string`
Generates a SQL query from a model and query options.

### Interfaces

#### `MinimlModel`
```typescript
interface MinimlModel {
    description: string;
    dialect: string;
    from: string;
    join: Record<string, string>;
    where: string;
    date_field: string;
    dimensions: Record<string, MinimlDef>;
    measures: Record<string, MinimlDef>;
    info: string;  // Auto-generated documentation
}
```

#### `MinimlDef`
```typescript
interface MinimlDef {
    key: string;
    description: string;
    sql?: string;
    join?: string;
}
```

## Examples

### E-commerce Analytics

```yaml
# ecommerce.yaml
description: E-commerce sales and customer analytics

dialect: bigquery
from: ecommerce.orders
date_field: order_date

join:
  customer_join: LEFT JOIN ecommerce.customers USING (customer_id)
  product_join: LEFT JOIN ecommerce.products USING (product_id)

where: status = 'completed'

dimensions:
  date: Order date
  customer_id: Customer identifier
  customer_email:
    - Customer email address
    - email
    - customer_join
  product_category:
    - Product category
    - category
    - product_join

measures:
  revenue: Total revenue
  order_count: 
    - Number of orders
    - COUNT(DISTINCT order_id)
  avg_order_value:
    - Average order value
    - AVG(total_amount)
```

**Usage:**
```typescript
const model = await loadModel('./ecommerce.yaml');

// Revenue by category, last 7 days
const sql = renderQuery(model, {
    dimensions: ['product_category'],
    measures: ['revenue', 'order_count'],
    date_from: '2024-01-20',
    date_to: '2024-01-27',
    order_by: ['-revenue'],
    limit: 10
});
```

### Complex Joins and Calculations

```yaml
# advanced.yaml
description: Advanced analytics with multiple joins

dialect: snowflake
from: fact.sales
date_field: sale_date

join:
  customer_join: LEFT JOIN dim.customers c ON c.id = customer_id
  product_join: LEFT JOIN dim.products p ON p.id = product_id
  region_join: LEFT JOIN dim.regions r ON r.code = c.region_code

dimensions:
  customer_segment:
    - Customer segment classification
    - c.segment
    - customer_join
  product_brand:
    - Product brand name
    - p.brand
    - product_join
  region_name:
    - Geographic region
    - r.name
    - region_join

measures:
  revenue_per_customer:
    - Average revenue per customer
    - SUM(amount) / COUNT(DISTINCT customer_id)
  profit_margin:
    - Profit margin percentage
    - (SUM(amount) - SUM(cost)) / SUM(amount) * 100
```

## Supported SQL Dialects

### BigQuery
- Supports `GROUP BY ALL` syntax
- Uses `DATE_TRUNC` for date granularity
- Compatible with BigQuery standard SQL functions

### Snowflake
- Supports Snowflake-specific functions
- Uses `DATE_TRUNC` for date granularity
- Compatible with Snowflake SQL syntax

More dialects coming soon!

## Advanced Features

### Template Variables in Info

Models support Jinja-style templating in the `info` section:

```yaml
info: |
  ## Available Dimensions
  {%- for dimension in dimensions %}
  - `{{ dimension.key }}`: {{ dimension.description }}
  {%- endfor %}

  ## Available Measures  
  {%- for measure in measures %}
  - `{{ measure.key }}`: {{ measure.description }}
  {%- endfor %}
```

### Filter Reference Expansion

MiniML automatically expands dimension and measure references in filter clauses:

```typescript
// If you have a measure: conversion_rate: "clicks / impressions"
// And use it in a HAVING clause:
renderQuery(model, {
    having: "conversion_rate > 0.05"
});
// MiniML expands this to: HAVING (clicks / impressions) > 0.05
```

### Validation

MiniML validates query parameters against model definitions and provides helpful error messages:

```typescript
// This will throw an error with specific invalid keys
renderQuery(model, {
    dimensions: ['invalid_dimension'],
    measures: ['invalid_measure']
});
// Error: The following keys are invalid:
// - dimensions: invalid_dimension  
// - measures: invalid_measure
```

## Security

Because MiniML allows flexible `where` and `having` clauses to be specified in native SQL format, comprehensive SQL validation is implemented to guard against SQL injection attacks for user-provided query parameters:

- **AST-based validation**: All user-provided SQL expressions are parsed and validated using node-sql-parser
- **Allowlist approach**: Only safe SQL constructs are permitted (comparisons, logical operators, basic functions)
- **Model-aware validation**: Column references are validated against your model schema
- **Dialect-specific rules**: Validation adapts to BigQuery/Snowflake syntax differences
- **Date input validation**: Date parameters (`date_from`, `date_to`) are strictly validated using regex patterns

### Security Model

**Trusted Inputs**: YAML model files are considered trusted input and are not validated for SQL injection. Model files should be:
- Created and maintained by trusted developers
- Stored securely with appropriate access controls  
- Reviewed before deployment to production environments
- Treated as code artifacts subject to version control and security policies

**Untrusted Inputs**: All query parameters provided at runtime (`where`, `having`, `date_from`, `date_to`, etc.) are considered untrusted and undergo strict validation.

### Safe Expression Examples

**WHERE clauses:**
```typescript
renderQuery(model, {
    where: "account_name = 'Acme Corp'"
});

renderQuery(model, {
    where: "date >= '2024-01-01' AND category_name LIKE 'Electronics%'"
});

renderQuery(model, {
    where: "revenue > 1000 OR quantity IS NOT NULL"
});
```

**HAVING clauses:**
```typescript
renderQuery(model, {
    having: "total_amount > 1000 AND order_count >= 5"
});
```

### Blocked Constructs

For security, the following are automatically blocked:
- **Subqueries**: `SELECT`, `EXISTS`, `ANY`, `ALL`
- **DDL/DML statements**: `DROP`, `ALTER`, `CREATE`, `INSERT`, `UPDATE`, `DELETE`
- **System functions**: Potentially dangerous database functions
- **Comments**: SQL comments that could hide malicious code
- **Unauthorized columns**: References to columns not defined in your model

### Validation Errors

When unsafe expressions are detected, MiniML provides helpful error messages:

```typescript
// Error: "Subqueries are not allowed in WHERE clauses. Use simple comparisons instead."
renderQuery(model, {
    where: "customer_id IN (SELECT id FROM users WHERE active = 1)"
});

// Error: "Column 'user_password' not found. Available dimensions: account_name, date, category_name"
renderQuery(model, {
    where: "user_password = 'secret'"
});
```

## Building and Development

```bash
# Install dependencies
npm install

# Build TypeScript
npm run build

# Run tests (requires Mocha setup)
npx mocha test/**/*.test.js
```

## Contributing

1. Fork the repository
2. Create a feature branch
3. Add tests for new functionality
4. Ensure all tests pass
5. Submit a pull request

## License

MIT License - see LICENSE file for details.