peertube-plugin-transposer-connector/DEVELOPMENT.md

Transposer connector is a PeerTube language tool plugin to transcribe and translate with Whisper

# DEVELOPMENT DOC

## HOW TO DEPLOY

### Deploy Locally

To deploy the plugin locally, follow these steps:

1. Install:

```bash
npm install
```

2. Build the plugin:

```bash
npm run build
```

3. Install the plugin into your PeerTube instance using peertube-cli:

```bash
peertube-cli plugins install --path "$(pwd)" --url <PEERTUBE_URL> --username <ADMIN_USERNAME> --password <ADMIN_PASSWORD>
```

or

```bash
 peertube-cli plugins install --path /Users/monz/Git/fairkom/peertube-plugin-transposer-connector -u  <PEERTUBE_URL> -U <ADMIN_USERNAME> --password <ADMIN_PASSWORD>
```

### Deploy via Docker

1. Make sure you have peertube-cli installed globally:

```bash
npm install -g @peertube/peertube-cli
```

2. Copy the plugin into the Docker container:

```bash
docker cp "$(pwd)" <DOCKER_CONTAINER>:/tmp/peertube-plugin-transposer-connector
```

3. Install the plugin inside the Docker container:

```bash
docker exec -it <DOCKER_CONTAINER> bash -c "cd /tmp/peertube-plugin-transposer-connector && peertube-cli plugins install --path /tmp/peertube-plugin-transposer-connector --url <PEERTUBE_URL> --username <ADMIN_USERNAME> --password <ADMIN_PASSWORD>"
```

4. Alternatively, run the command directly from within the container:

```bash
peertube-cli plugins install --path /tmp/peertube-plugin-transposer-connector --url <PEERTUBE_URL> --username <ADMIN_USERNAME> --password <ADMIN_PASSWORD>
```

### Deploy on Server

To deploy on a server:

1. Publish the new version of the plugin using npm publish:

```bash
npm publish
```

3. It may take some time for the update. To manually update the database, you can run the following SQL command in the PeerTube database:

```bash
psql -U <DATABASE_USER>
<DATABASE_NAME># UPDATE plugin
SET "latestVersion" = '<PLUGIN_VERSION>'
WHERE name = '<PLUGIN_NAME>';
```

IMPORTANT!

When updating to a new version, make sure to adapt the proxy endpoint so it continues working correctly.

For version changes, follow these steps:

- For Development Environment:

  Go to the Kong Admin page for the development environment and edit to the new version

  https://kong.transposer.eu/#/services/edit/88531e8e-8a7e-4ab4-885b-4704d4dc7427

- For Production Environment:

  Go to the Kong Admin page for the production environment and edit to the new version:

  https://kong.transposer.eu/#/services/e43fbf1d-0260-4278-92f8-437d6d493de3

## HOW TO DEBUG

To debug the plugin and view logs:

1. You can check the PeerTube logs or other related tools for errors and events.
2. To access the debugging tools, set up an SSH tunnel to the transposer server using:

```bash
ssh -L8001:127.0.0.1:8001 -p 3345 transposer@transposer.eu
```

This will allow you to access services:

### Explain Services for Transposer

The Transposer system consists of several key services that work together to provide transcription, translation, and metadata functionality. Below are the services and how they interconnect:
Key Services in the Transposer System:

- Kong: Acts as an API gateway that routes requests to the appropriate services. It handles load balancing and ensures the proper communication between services.

- Node-RED: A flow-based development tool used for wiring together various services. It coordinates tasks like triggering transcription and translation processes and managing workflows.

- Kafka: A distributed event streaming platform that ensures asynchronous communication between services. Kafka queues messages that need to be processed by other services (e.g., transcription or translation).

- Whisper: The transcription engine (powered by OpenAI's Whisper) that processes video audio and generates transcriptions.

#### How to Debug Transposer

When you encounter an issue with the Transposer system, follow these steps to identify and resolve errors:

1.  Access Node-RED

- Go to the Node-RED UI at: Node-RED Transposer
- Log in with the following credentials:

  - User: admin
  - Password: \*\*\*

Once logged in, inspect the flow and look for any issues in the process, particularly with triggering transcription and translation tasks.

2.  Check Kafka Logs

- Kafka is where events are queued for processing. To view Kafka logs, do the following:
- Go to: Kafka Transposer
  Inspect the message queues for any errors or unexpected delays in processing.

3. Check Individual Services

Here are some of the key services to check in order: (Inspect the newest messages to see if any errors occurred during the transcription or translation process.)

- peertube_transcribe_and_translate:
  Look at the messages in the Kafka queue related to transcription and translation.

- peertube_transcribe_and_translate_response:
  After processing the transcription or translation, check the responses to ensure successful handling.

- whisper:
  Check the whisper service for any errors or issues with generating transcriptions.

- whisper_response:
  Inspect the responses to see if there are any issues with the transcriptions produced.

- generic_translate and generic_translate_response:
  If you're facing issues with translations, check these services.

4. View Logs in Transposer

- To get more detailed logs, you can SSH into the Transposer server and check for issues:
- ssh -p 3345 transposer@transposer.eu
- Once logged in, navigate to the adapter directories, for example, whisper:
- cd whisper/
- You can use reverse-i-search to find relevant logs:

```bash
ctrl + R # Reverse search through logs
```

To view the latest logs, use the following command:

```bash
tail -f -n 100 service-connector/var/log/connector.log
```

5. Restart Services if Needed

If you identify an issue with a specific service or adapter, you may need to restart it. For example, to restart the peertube adapter, use the following command:

```bash
systemctl --user restart chagai_peertube-transcribe-and-translate_1.service
```

6. Observe Service Connection Logs

Look for logs that indicate successful connections or errors. For example, the following log entry shows the start of a new HTTPS connection:

Starting new HTTPS connection (1): service.cba.media:443

This log entry indicates that the process has started.