@verdnatura/myt/README.md

MySQL version control

# Myt - MariaDB and MySQL version control

Utilities to ease the maintenance of MariaDB or MySQL database versioning using
a Git repository.

This project is just to bring an idea to life and is still in an early stage of
development, so any help is welcomed! Feel free to contribute.

## Requirements

* Git
* Docker >= 23 with BuildKit (Optional, used to manage local server)

## Installation

Required libraries to build with *node-gyp*.
```text
# apt install libkrb5-dev libssl-dev
```

It's recommended to install the package globally.

```text
# npm install -g @verdnatura/myt
$ myt <command>
```

You can also install locally and use the *npx* command to execute it.

```text
$ npm install @verdnatura/myt
$ npx myt <command>
```

## How to use

Execute *myt* with the desired command.

```text
$ [npx] myt [-w|--workspace <string>] [-r|--remote <string>] [-d|--debug]
[-h|--help] <command> [<args>]
```

The default workspace directory is the current working directory and unless 
otherwise indicated, the default remote is *local*.

Database versioning commands:

 * **init**: Initialize an empty workspace.
 * **pull**: Incorporate database routine changes into workspace.
 * **push**: Apply changes into database.
 * **version**: Creates a new version.
 * **create**: Creates a new routine file.
 * **clean**: Cleans old versions.

Local server management commands:

 * **dump**: Export database structure and fixtures.
 * **fixtures**: Export local database fixtures.
 * **build**: Builds local database server image.
 * **apply**: Applies structure, fixtures and changes to database.
 * **run**: Build and start local database server container.
 * **start**: Start local database server container.

Each command can have its own specific commandline options, you can see them by
running the command along with the _--help_ parameter, only the most commonly
used ones are documented here.

```text
$ myt <command> [-h|--help]
```

### Usage as library

It is also possible to use myt as a JavaScript library, not only from the
command line. Below is a reference example.

```js
const Myt = require('@verdnatura/myt/myt');
const Run = require('@verdnatura/myt/myt-run');

async function main() {
   const myt = new Myt();
   await myt.init({
      workspace: path.join(__dirname, '..')
   });
   const server = await myt.run(Run, {
      random: true,
      persist: process.platform != 'linux'
   });
   console.log(server.dbConfig);
   await myt.deinit();
}
```

## Basic information

First of all you have to initialize the workspace.

```text
$ myt init
```

Now you can configure Myt using *myt.config.yml* file, located at the root of
your workspace. This file should include the project codename and schemas/tables
which are exported when you use *pull* or *dump* commands.

Don't forget to initialize git (if it isn't initialized yet).

```text
$ git init
```

### Remotes

Create database connection configuration for each environment at *remotes*
folder using standard MySQL *ini* configuration files. The convention remote
names are *local*, *production* and *test*.

```text
remotes/[remote].ini
```
### Startup

Once the basic configuration is done, routines can be imported from the
database into the project, it is recommended to use the *production* remote.

```text
$ myt pull production
```

From now on, you can use the project as if it were a standard git repository
(since it is). To apply changes to the database run the *push* command on the
desired remote.

```text
$ myt push [<remote>] [--commit]
```

### Routines

Routines are placed inside *routines* folder. All objects that have PL/SQL code 
are considered routines. It includes events, functions, procedures, triggers 
and views with the following structure.

```text
  routines
  `- schema
     |- events
     |  `- eventName.sql
     |- functions
     |  `- functionName.sql
     |- procedures
     |  `- procedureName.sql
     |- triggers
     |  `- triggerName.sql
     `- views
        `- viewName.sql
```

### Versions

Versions are placed inside *versions* folder with the following structure.
Don't place your PL/SQL objects here, use the routines folder!

```text
  versions
  |- 00001-firstVersionCodeName
  |  |- 00-firstExecutedScript.sql
  |  |- 01-secondScript.sql
  |  `- 99-lastScript.sql
  `- 00002-secondVersion
     |- 00-firstExecutedScript.sql
     `- 00-sameNumbers.sql
```

### Local server

The local server is created as a MariaDB Docker container using the base dump 
created with the *dump* command plus pushing local versions and changed 
routines.

### Dumps

You can create your local fixture and structure files.

* *structure/before.sql*
* *structure/after.sql*
* *fixtures/before.sql*
* *fixtures/after.sql*
* *fixtures/local.sql*

### Realms

Realms allows users to apply version-specific changes based on their configured
environment, or "realm". This ensures that certain changes are only applied
when the user is operating within a specific realm, providing an additional
layer of customization and control.

Realm folders structure.

```text
   realms
   |- marvel
   |  `- 00-spiderman.sql
   |  `- 01-hulk.sql
   `- dc
      `- 00-superman.sql
      `- 01-batman.sql
      `- 02-wonder_woman.sql
      `- 03-flash.sql
```

#### Configuration

To make use of this feature, you need to configure your realm in the
`versionConfig` table. Set your realm by inserting or updating a record in
this table. The `realm` field should contain the identifier of your
environment.

#### Realm-specific version files

To designate a version file as realm-specific, include a `.your_realm.` segment
before the file extension. For example, if your realm is set to 'foo', the file 
should be named like this: `filename.foo.sql`.

## Versioning commands

### init

Initializes an empty workspace.

```text
$ myt init [<args>]
```

### pull

Incorporates database routine changes into workspace.

```text
$ myt pull [<args>] [-f|--force] [<remote>]
```

When *--checkout* option is provided, it does the following before export:

1. Get the last database push commit (saved in versioning tables).
2. Creates and checkout to a new branch based in database commit.

### push

Applies versions and routine changes into database.

```text
$ myt push [<args>] [-f|--force] [-c|--commit] [<remote>]
```

Commit is saved into database only if *--commit* option is provided, it
prevents from accidentally saving local commits into shared servers, causing 
subsequent pushes from other clients to fail because they can't get that 
commit from the git tree in order to get differences.

### version

Creates a new version folder, when name is not specified it generates a random 
name mixing a color with a plant name.

```text
$ myt version [<args>] [<name>]
```

### create

Creates a new routine file with a default template.

```text
$ myt create [<args>] [-t <type>] <schema>.<name>
```

### clean

Cleans all already applied versions older than  *maxOldVersions*.

```text
$ myt clean [<args>] [-p|--purge]
```

## Local server commands

### dump

Exports database structure and fixtures from remote into hidden files located
in *dump* folder. If no remote is specified *production* is used.

```text
$ myt dump [<args>] [-l|--lock] [<remote>]
```

### fixtures

Exports local database fixtures into *dump/fixtures.sql* files. If no remote is
specified *local* is used.

```text
$ myt fixtures [<args>] [<remote>]
```

### build

Builds local database server image. It only rebuilds the image when sources
have been modified.

```text
$ myt build [<args>] [-n|--name] [-t|--tag] [-f|--force] [<realm>]
```

### apply

Applies structure, fixtures and changes to database. Primarily used by the
build command to rebuild the local database.

```text
$ myt apply [<args>] [<realm>]
```

### run

Builds and starts local database server container. It only rebuilds the image
when dump have been modified.

```text
$ myt run [<args>] [-c|--ci] [-r|--random] [<realm>]
```

### start

Starts local database server container. It does the minium effort, if it 
doesn't exists calls the run command, if it is started does nothing. Keep in 
mind that when you do not rebuild the docker you may be using an outdated 
version of it.

```text
$ myt start [<args>]
```

## Why

The main reason for starting this project is because there are no fully free 
and open source migration tools available that allow versioning database 
routines with a standard CVS system as if they were normal application code.

Also, the existing tools are too complex and require too much knowledge to 
initiate a small project.

## To-Do

* Undo changes when there is an error applying a version using "undo" files.
* Console logging via events.
* Lock version table row when pushing.
* Preserve all characteristics on pull: SQL mode, character set, algorithm...

## Built With

* [Git](https://git-scm.com/)
* [Node.js](https://nodejs.org/)
* [NodeGit](https://www.nodegit.org/)
* [Docker](https://www.docker.com/)