digitaltwin-core/README.md

Minimalist framework to collect and handle data in a Digital Twin project

# Digital Twin Core

Digital Twin Core is a minimalist TypeScript framework used to collect and process data for Digital Twin projects. It provides building blocks to create scheduled collectors, harvesters and HTTP handlers while abstracting storage and database access.

## Features

- **Collectors** - fetch regular data from APIs (typically JSON) based on a Buffer schedule, store it and expose it via GET endpoints.
- **Harvesters** – transform data collected by collectors, store the results and expose them via GET endpoints.
- **Handlers** – expose GET endpoints that directly return the result of the method defined in the decorator.
- **Assets Manager** – upload, store and manage file assets with metadata, providing RESTful endpoints for CRUD operations.
- **Storage adapters** – currently local filesystem and OVH Object Storage via S3 API.
- **Database adapter** – implemented with [Knex](https://knexjs.org/) to index metadata.
- **Engine** – orchestrates components, schedules jobs with BullMQ and exposes endpoints via Express.

## Installation

```bash
npm install
```

The project requires Node.js 18 or later.

## Building

Compile the TypeScript sources to `dist/`:

```bash
npm run build
```

During development you can use the watcher:

```bash
npm run dev
```

## Running tests

The test suite uses [Japa](https://github.com/japa/runner). Run all tests with:

```bash
npm test
```

## Example usage

Below is a very small example showing how the engine may be instantiated. Storage and database implementations are selected through the provided factories.

```ts
import { DigitalTwinEngine } from './src/engine/digital_twin_engine.js';
import { StorageServiceFactory } from './src/storage/storage_factory.js';
import { KnexDatabaseAdapter } from './src/database/adapters/knex_database_adapter.js';
import { Env } from './src/env/env.js';

// Validate environment variables and bootstrap services
const env = Env.validate({
  STORAGE_CONFIG: Env.schema.enum(['local', 'ovh'])
});

const storage = StorageServiceFactory.create();
const database = new KnexDatabaseAdapter({ client: 'sqlite3', connection: ':memory:' }, storage);

const engine = new DigitalTwinEngine({ storage, database });
engine.start();
```

## Components

### Collectors

Collectors are scheduled components that fetch data from external sources at regular intervals. They implement a `collect()` method that returns a Buffer, which is then stored and exposed via HTTP endpoints.

**Key features:**
- Cron-based scheduling
- Automatic storage and metadata indexing
- HTTP GET endpoint for retrieving latest data
- Event emission on successful collection

### Assets Manager

The Assets Manager provides a complete solution for file asset management with metadata support. It's an abstract base class that can be extended for specific asset types.

**Key features:**
- File upload with metadata (description, source URL, owner, filename)
- RESTful CRUD operations via HTTP endpoints
- Content-type aware storage and retrieval
- Separate display and download endpoints
- Source URL validation for data provenance
- Component isolation (each manager handles its own asset type)

**Available endpoints:**
- `GET /{assetType}` - List all assets with metadata
- `POST /{assetType}/upload` - Upload new asset with metadata
- `GET /{assetType}/{id}` - Retrieve asset content for display
- `GET /{assetType}/{id}/download` - Download asset with attachment headers
- `PUT /{assetType}/{id}` - Update asset metadata
- `DELETE /{assetType}/{id}` - Delete asset

**Example usage:**
```typescript
class GLTFAssetsManager extends AssetsManager {
  getConfiguration() {
    return {
      name: 'gltf',
      description: 'GLTF 3D models manager',
      contentType: 'model/gltf-binary',
      tags: ['assets', '3d', 'gltf']
    }
  }
}
```

## Project Scaffolding

Use [create-digitaltwin](https://github.com/CePseudoBE/create-digitaltwin) to quickly bootstrap new projects:

```bash
npm init digitaltwin my-project
cd my-project
npm install
npm run dev
```

Generated projects include [digitaltwin-cli](https://github.com/CePseudoBE/digitaltwin-cli) for component generation:

```bash
node dt make:collector WeatherCollector --description "Weather data collector"
node dt make:handler ApiHandler --method post
node dt make:harvester DataProcessor --source weather-collector
```

## Folder structure

- `src/` – framework sources
    - `components/` – base classes for collectors, harvesters, handlers and assets manager
    - `engine/` – orchestration logic
    - `storage/` – storage service abstractions and adapters
    - `database/` – metadata database adapter
    - `env/` – environment configuration helper
- `tests/` – unit tests

---

This project is licensed under the MIT License.