mstf-kit/SERVER_INTEGRATION.en.md

一个现代化的 JavaScript/TypeScript 工具库，提供了丰富的常用工具函数

# Audio Tools Server Integration Guide

[简体中文](./SERVER_INTEGRATION.md) | English

This document provides detailed guidance and code examples for integrating the mstf-kit audio utilities with server-side applications, particularly useful for real-time audio processing and speech recognition scenarios.

## Table of Contents

- [Integration Overview](#integration-overview)
- [REST API Mode](#rest-api-mode)
- [WebSocket Mode](#websocket-mode)
- [Server-Side Implementation](#server-side-implementation)
- [Best Practices](#best-practices)
- [Common Issues](#common-issues)

## Integration Overview

Integrating recording functionality with servers enables advanced features like real-time speech recognition and audio processing. We support two main integration patterns, each with its own advantages and disadvantages:

### REST API Mode vs WebSocket Mode

| Feature | REST API | WebSocket |
|---------|----------|-----------|
| Real-time Capability | Medium (depends on request frequency) | High (continuous connection) |
| Implementation Complexity | Low | Medium |
| Server Load | Each request requires a new connection | Maintains long connections, lower connection overhead |
| Firewall Friendliness | High | Medium (may require special configuration) |
| Bidirectional Communication | Limited | Fully supported |
| Suitable Scenarios | Short phrases, non-strict real-time scenarios | Long audio streams, immediate feedback needed |

## REST API Mode

The REST API mode sends audio data chunks via periodic HTTP requests, making it ideal for integration with existing RESTful architectures.

### Client Implementation

```typescript
import { createRecorder, audioBlobToBase64 } from 'mstf-kit';

async function recordAndSendToServer() {
  const API_ENDPOINT = 'https://api.example.com/speech-recognition';
  let sessionId = null;
  
  // Create streaming recorder
  const recorder = await createRecorder({
    format: 'wav',
    sampleRate: 16000,  // 16kHz, suitable for speech recognition
    isStreaming: true,
    timeslice: 500,     // Send data every 500ms
    
    // Streaming data callback
    onDataAvailable: async (blob) => {
      try {
        // Create form data
        const formData = new FormData();
        formData.append('audio', blob, 'chunk.wav');
        
        // Add session ID to the request if available
        if (sessionId) {
          formData.append('sessionId', sessionId);
        }
        
        // Send data to server
        const response = await fetch(API_ENDPOINT, {
          method: 'POST',
          body: formData
        });
        
        if (!response.ok) {
          throw new Error(`Server error: ${response.status}`);
        }
        
        const result = await response.json();
        
        // Save session ID for subsequent requests
        if (!sessionId && result.sessionId) {
          sessionId = result.sessionId;
        }
        
        // Process server response
        if (result.text) {
          console.log('Recognition result:', result.text);
          updateTranscription(result.text);
        }
      } catch (error) {
        console.error('Failed to send audio data:', error);
      }
    }
  });
  
  // Start recording
  await recorder.start();
  
  // UI update function
  function updateTranscription(text) {
    const transcriptionElement = document.getElementById('transcription');
    if (transcriptionElement) {
      transcriptionElement.textContent = text;
    }
  }
  
  // Stop button event handler
  document.getElementById('stopButton').addEventListener('click', async () => {
    // Stop recording
    const finalBlob = await recorder.stop();
    
    // Send the final data chunk, marked as final
    const formData = new FormData();
    formData.append('audio', finalBlob, 'final.wav');
    formData.append('sessionId', sessionId);
    formData.append('isFinal', 'true');
    
    const response = await fetch(API_ENDPOINT, {
      method: 'POST',
      body: formData
    });
    
    if (response.ok) {
      const result = await response.json();
      updateTranscription(result.text);
      console.log('Final recognition result:', result.text);
    }
    
    // Release resources
    recorder.release();
  });
}
```

### Implementation Details

1. **Session Management**: Using `sessionId` to track continuous audio chunks, ensuring the server can properly combine them
2. **Data Packaging**: Using `FormData` to encapsulate audio data for easy server processing
3. **Error Handling**: Implementing error catching and recovery mechanisms
4. **Final Markers**: Sending special flags at the end of recording to notify the server to process the complete audio

## WebSocket Mode

WebSocket mode uses persistent connections to enable true real-time audio streaming, particularly suitable for low-latency applications.

### Client Implementation

```typescript
import { createRecorder, audioBlobToBase64 } from 'mstf-kit';

async function webSocketAudioStream() {
  // WebSocket connection
  const socket = new WebSocket('wss://api.example.com/audio-stream');
  let isConnected = false;
  
  // Start recording after connection is established
  socket.addEventListener('open', async () => {
    console.log('WebSocket connection established');
    isConnected = true;
    
    // Send initialization message
    socket.send(JSON.stringify({
      type: 'init',
      format: 'wav',
      sampleRate: 16000,
      language: 'en-US'
    }));
    
    // Create streaming recorder
    const recorder = await createRecorder({
      format: 'wav',
      sampleRate: 16000,
      isStreaming: true,
      timeslice: 200,  // Smaller interval for more real-time experience
      
      // Streaming data callback
      onDataAvailable: async (blob) => {
        if (isConnected) {
          try {
            // Convert Blob to Base64 and send
            const base64 = await audioBlobToBase64(blob);
            
            // Send audio data
            socket.send(JSON.stringify({
              type: 'audio',
              data: base64
            }));
          } catch (error) {
            console.error('Failed to send audio data:', error);
          }
        }
      }
    });
    
    // Start recording
    await recorder.start();
    
    // Handle messages from the server
    socket.addEventListener('message', (event) => {
      try {
        const message = JSON.parse(event.data);
        
        // Handle different types of messages
        switch (message.type) {
          case 'result':
            // Interim recognition result
            updatePartialResult(message.text);
            break;
          case 'finalResult':
            // Final recognition result
            updateFinalResult(message.text);
            break;
          case 'error':
            console.error('Server error:', message.error);
            break;
        }
      } catch (error) {
        console.error('Failed to process server message:', error);
      }
    });
    
    // Stop button event handler
    document.getElementById('stopButton').addEventListener('click', async () => {
      // Send end message
      socket.send(JSON.stringify({
        type: 'end'
      }));
      
      // Stop recording and release resources
      await recorder.stop();
      recorder.release();
      
      // Close WebSocket connection
      socket.close();
      isConnected = false;
    });
    
    // Handle WebSocket errors and closure
    socket.addEventListener('error', (error) => {
      console.error('WebSocket error:', error);
      isConnected = false;
    });
    
    socket.addEventListener('close', () => {
      console.log('WebSocket connection closed');
      isConnected = false;
      recorder.stop().then(() => recorder.release());
    });
  });
  
  // UI update functions
  function updatePartialResult(text) {
    const resultElement = document.getElementById('partialResult');
    if (resultElement) {
      resultElement.textContent = text;
    }
  }
  
  function updateFinalResult(text) {
    const resultElement = document.getElementById('finalResult');
    if (resultElement) {
      const previousText = resultElement.textContent;
      resultElement.textContent = previousText 
        ? `${previousText} ${text}` 
        : text;
    }
  }
}
```

### Implementation Details

1. **Protocol Design**: Using JSON messages to transfer commands and data, including initialization, audio data, and end signals
2. **Connection Management**: Handling WebSocket lifecycle, including open, error, and close events
3. **Data Conversion**: Converting binary audio data to Base64 for transmission
4. **Bidirectional Communication**: Receiving and processing real-time recognition results from the server

## Server-Side Implementation

Below are server-side examples implemented in different languages and frameworks that support both integration modes mentioned above.

### Node.js Implementation (Using Baidu Speech Recognition)

```javascript
// Express server example
const express = require('express');
const multer = require('multer');
const { Readable } = require('stream');
const axios = require('axios');
const fs = require('fs').promises;
const path = require('path');

const app = express();
const upload = multer({ storage: multer.memoryStorage() });

// For storing session data
const sessions = new Map();

// Baidu Speech Recognition API configuration
const baiduConfig = {
  apiKey: 'YOUR_BAIDU_API_KEY',
  secretKey: 'YOUR_BAIDU_SECRET_KEY',
  appId: 'YOUR_BAIDU_APP_ID',
  accessToken: null,
  expiresAt: 0
};

// Get Baidu API access token
async function getBaiduAccessToken() {
  const now = Date.now();
  
  // If token is valid, return it directly
  if (baiduConfig.accessToken && now < baiduConfig.expiresAt) {
    return baiduConfig.accessToken;
  }
  
  // Get new token
  try {
    const response = await axios.get(
      'https://aip.baidubce.com/oauth/2.0/token',
      {
        params: {
          grant_type: 'client_credentials',
          client_id: baiduConfig.apiKey,
          client_secret: baiduConfig.secretKey
        }
      }
    );
    
    baiduConfig.accessToken = response.data.access_token;
    // Set expiration time (expire 5 minutes early to ensure safety)
    baiduConfig.expiresAt = now + (response.data.expires_in - 300) * 1000;
    
    return baiduConfig.accessToken;
  } catch (error) {
    console.error('Failed to get Baidu access token:', error);
    throw new Error('Cannot get speech recognition authorization');
  }
}

// Call Baidu Speech Recognition API
async function recognizeSpeech(audioBuffer, options = {}) {
  try {
    const accessToken = await getBaiduAccessToken();
    
    // Convert audio to Base64
    const base64Audio = audioBuffer.toString('base64');
    
    // Set default parameters
    const params = {
      format: options.format || 'wav',
      rate: options.sampleRate || 16000,
      channel: options.channelCount || 1,
      dev_pid: 1537,  // Mandarin recognition model, can be changed as needed
      cuid: 'mstf-kit',
      speech: base64Audio,
      len: audioBuffer.length
    };
    
    // Call API
    const response = await axios.post(
      `https://vop.baidu.com/server_api?access_token=${accessToken}`,
      params
    );
    
    if (response.data.err_no !== 0) {
      throw new Error(`Baidu speech recognition error: ${response.data.err_msg}`);
    }
    
    return response.data.result[0];
  } catch (error) {
    console.error('Speech recognition failed:', error);
    throw error;
  }
}

// Endpoint for handling audio data
app.post('/speech-recognition', upload.single('audio'), async (req, res) => {
  try {
    const audioBuffer = req.file.buffer;
    const sessionId = req.body.sessionId || Date.now().toString();
    const isFinal = req.body.isFinal === 'true';
    
    // Get or create session
    let session = sessions.get(sessionId);
    if (!session) {
      session = {
        audioChunks: [],
        partialText: '',
      };
      sessions.set(sessionId, session);
    }
    
    // Add new audio data
    session.audioChunks.push(audioBuffer);
    
    if (!isFinal) {
      // If not the final chunk, directly recognize the current audio chunk
      try {
        const recognitionResult = await recognizeSpeech(audioBuffer, {
          format: 'wav',
          sampleRate: 16000
        });
        
        session.partialText = recognitionResult;
        
        // Return current recognized text and session ID
        res.json({
          sessionId,
          text: session.partialText,
        });
      } catch (error) {
        console.error('Partial recognition failed:', error);
        // Return session ID even on failure, so frontend can continue sending
        res.json({
          sessionId,
          text: session.partialText,
          error: 'Partial recognition failed'
        });
      }
    } else {
      // Final chunk, combine all audio and perform complete recognition
      // Note: Since Baidu API has limits on audio length, we may need to segment longer audio
      // This is a simplified implementation; real applications should consider segmenting long audio
      const completeAudio = Buffer.concat(session.audioChunks);
      
      try {
        const finalResult = await recognizeSpeech(completeAudio, {
          format: 'wav',
          sampleRate: 16000
        });
        
        // Clean up session
        sessions.delete(sessionId);
        
        // Return final text
        res.json({
          sessionId,
          text: finalResult,
          isFinal: true,
        });
      } catch (error) {
        console.error('Final recognition failed:', error);
        res.status(500).json({ 
          error: 'Final recognition failed',
          details: error.message
        });
      }
    }
  } catch (error) {
    console.error('Failed to process audio:', error);
    res.status(500).json({ error: 'Failed to process audio' });
  }
});

// WebSocket handling
const WebSocket = require('ws');
const server = app.listen(3000);
const wss = new WebSocket.Server({ server });

wss.on('connection', (ws) => {
  console.log('WebSocket connection established');
  
  // State for each connection
  const state = {
    format: 'wav',
    sampleRate: 16000,
    language: 'en-US',
    audioChunks: [],
  };
  
  // Handle messages
  ws.on('message', async (message) => {
    try {
      const msg = JSON.parse(message);
      
      switch (msg.type) {
        case 'init':
          // Initialize connection
          state.format = msg.format || 'wav';
          state.sampleRate = msg.sampleRate || 16000;
          state.language = msg.language || 'en-US';
          break;
          
        case 'audio':
          // Receive audio data
          const audioBuffer = Buffer.from(msg.data, 'base64');
          state.audioChunks.push(audioBuffer);
          
          // Perform real-time speech recognition
          try {
            const recognitionResult = await recognizeSpeech(audioBuffer, {
              format: state.format,
              sampleRate: state.sampleRate
            });
            
            // Send recognition result back to the client
            ws.send(JSON.stringify({
              type: 'result',
              text: recognitionResult,
            }));
          } catch (error) {
            console.error('Real-time recognition failed:', error);
            ws.send(JSON.stringify({
              type: 'error',
              error: 'Real-time recognition failed',
              details: error.message
            }));
          }
          break;
          
        case 'end':
          // Client ends recording
          const completeAudio = Buffer.concat(state.audioChunks);
          
          try {
            // Perform final recognition
            const finalResult = await recognizeSpeech(completeAudio, {
              format: state.format,
              sampleRate: state.sampleRate
            });
            
            // Send final result
            ws.send(JSON.stringify({
              type: 'finalResult',
              text: finalResult,
            }));
          } catch (error) {
            console.error('Final recognition failed:', error);
            ws.send(JSON.stringify({
              type: 'error',
              error: 'Final recognition failed',
              details: error.message
            }));
          }
          break;
      }
    } catch (error) {
      console.error('Failed to process WebSocket message:', error);
      ws.send(JSON.stringify({
        type: 'error',
        error: 'Failed to process message',
      }));
    }
  });
  
  ws.on('close', () => {
    console.log('WebSocket connection closed');
    // Clean up resources
  });
});

console.log('Server running at http://localhost:3000');
```

### Python Implementation (Using FastAPI)

```python
from fastapi import FastAPI, UploadFile, File, Form, WebSocket, WebSocketDisconnect
from fastapi.middleware.cors import CORSMiddleware
import uvicorn
import asyncio
import base64
import json
import io
import os
import uuid
import httpx
import time
from typing import List, Dict, Optional, Any
from pydantic import BaseModel

app = FastAPI(title="Speech Recognition Server")

# Enable CORS
app.add_middleware(
    CORSMiddleware,
    allow_origins=["*"],  # In production, should restrict to specific domains
    allow_credentials=True,
    allow_methods=["*"],
    allow_headers=["*"],
)

# Session storage
sessions: Dict[str, Dict[str, Any]] = {}

# Baidu Speech Recognition API configuration
BAIDU_API_KEY = "YOUR_BAIDU_API_KEY"
BAIDU_SECRET_KEY = "YOUR_BAIDU_SECRET_KEY"
BAIDU_APP_ID = "YOUR_BAIDU_APP_ID"
access_token = None
token_expires_at = 0

# Get Baidu API access token
async def get_baidu_access_token():
    global access_token, token_expires_at
    
    # If token is valid, return it directly
    if access_token and time.time() < token_expires_at:
        return access_token
    
    # Get new token
    try:
        async with httpx.AsyncClient() as client:
            response = await client.get(
                "https://aip.baidubce.com/oauth/2.0/token",
                params={
                    "grant_type": "client_credentials",
                    "client_id": BAIDU_API_KEY,
                    "client_secret": BAIDU_SECRET_KEY
                }
            )
            
            data = response.json()
            access_token = data["access_token"]
            # Set expiration time (expire 5 minutes early to ensure safety)
            token_expires_at = time.time() + data["expires_in"] - 300
            
            return access_token
    except Exception as e:
        print(f"Failed to get Baidu access token: {e}")
        raise Exception("Cannot get speech recognition authorization")

# Call Baidu Speech Recognition API
async def recognize_speech(audio_data: bytes, options: Dict = None):
    if options is None:
        options = {}
    
    try:
        token = await get_baidu_access_token()
        
        # Convert audio to Base64
        base64_audio = base64.b64encode(audio_data).decode('utf-8')
        
        # Set default parameters
        params = {
            "format": options.get("format", "wav"),
            "rate": options.get("sampleRate", 16000),
            "channel": options.get("channelCount", 1),
            "dev_pid": 1537,  # Mandarin recognition model, can be changed as needed
            "cuid": "mstf-kit-python",
            "speech": base64_audio,
            "len": len(audio_data)
        }
        
        # Call API
        async with httpx.AsyncClient() as client:
            response = await client.post(
                f"https://vop.baidu.com/server_api?access_token={token}",
                json=params
            )
            
            result = response.json()
            
            if result["err_no"] != 0:
                raise Exception(f"Baidu speech recognition error: {result['err_msg']}")
            
            return result["result"][0]
    except Exception as e:
        print(f"Speech recognition failed: {e}")
        raise e

# REST API route
@app.post("/speech-recognition")
async def speech_recognition(
    audio: UploadFile = File(...),
    sessionId: Optional[str] = Form(None),
    isFinal: bool = Form(False)
):
    try:
        # Read audio data
        audio_data = await audio.read()
        
        # Create or get session
        session_id = sessionId or str(uuid.uuid4())
        if session_id not in sessions:
            sessions[session_id] = {
                "audio_chunks": [],
                "partial_text": ""
            }
        
        session = sessions[session_id]
        
        # Add new audio data
        session["audio_chunks"].append(audio_data)
        
        if not isFinal:
            # If not final chunk, directly recognize current audio chunk
            try:
                recognition_result = await recognize_speech(audio_data, {
                    "format": "wav",
                    "sampleRate": 16000
                })
                
                session["partial_text"] = recognition_result
                
                # Return current recognized text and session ID
                return {
                    "sessionId": session_id,
                    "text": session["partial_text"]
                }
            except Exception as e:
                # Return session ID even on failure, so frontend can continue sending
                return {
                    "sessionId": session_id,
                    "text": session["partial_text"],
                    "error": f"Partial recognition failed: {str(e)}"
                }
        else:
            # Final chunk, combine all audio and perform complete recognition
            complete_audio = b''.join(session["audio_chunks"])
            
            try:
                final_result = await recognize_speech(complete_audio, {
                    "format": "wav",
                    "sampleRate": 16000
                })
                
                # Clean up session
                del sessions[session_id]
                
                # Return final text
                return {
                    "sessionId": session_id,
                    "text": final_result,
                    "isFinal": True
                }
            except Exception as e:
                return {
                    "error": "Final recognition failed",
                    "details": str(e)
                }
    except Exception as e:
        return {"error": f"Failed to process audio: {str(e)}"}

# WebSocket connection management
class ConnectionManager:
    def __init__(self):
        self.active_connections: Dict[WebSocket, Dict] = {}

    async def connect(self, websocket: WebSocket):
        await websocket.accept()
        self.active_connections[websocket] = {
            "format": "wav",
            "sampleRate": 16000,
            "language": "en-US",
            "audio_chunks": []
        }

    def disconnect(self, websocket: WebSocket):
        if websocket in self.active_connections:
            del self.active_connections[websocket]

    async def send_message(self, websocket: WebSocket, message: Dict):
        await websocket.send_text(json.dumps(message))

manager = ConnectionManager()

# WebSocket route
@app.websocket("/ws/audio-stream")
async def websocket_audio_stream(websocket: WebSocket):
    await manager.connect(websocket)
    
    try:
        while True:
            data = await websocket.receive_text()
            message = json.loads(data)
            
            state = manager.active_connections[websocket]
            
            if message["type"] == "init":
                # Initialize connection
                state["format"] = message.get("format", "wav")
                state["sampleRate"] = message.get("sampleRate", 16000)
                state["language"] = message.get("language", "en-US")
                await manager.send_message(websocket, {
                    "type": "info",
                    "message": "Connection initialized successfully"
                })
            
            elif message["type"] == "audio":
                # Receive audio data
                audio_data = base64.b64decode(message["data"])
                state["audio_chunks"].append(audio_data)
                
                # Perform real-time speech recognition
                try:
                    recognition_result = await recognize_speech(audio_data, {
                        "format": state["format"],
                        "sampleRate": state["sampleRate"]
                    })
                    
                    # Send recognition result back to the client
                    await manager.send_message(websocket, {
                        "type": "result",
                        "text": recognition_result
                    })
                except Exception as e:
                    await manager.send_message(websocket, {
                        "type": "error",
                        "error": "Real-time recognition failed",
                        "details": str(e)
                    })
            
            elif message["type"] == "end":
                # Client ends recording
                if state["audio_chunks"]:
                    complete_audio = b''.join(state["audio_chunks"])
                    
                    try:
                        # Perform final recognition
                        final_result = await recognize_speech(complete_audio, {
                            "format": state["format"],
                            "sampleRate": state["sampleRate"]
                        })
                        
                        # Send final result
                        await manager.send_message(websocket, {
                            "type": "finalResult",
                            "text": final_result
                        })
                    except Exception as e:
                        await manager.send_message(websocket, {
                            "type": "error",
                            "error": "Final recognition failed",
                            "details": str(e)
                        })
    
    except WebSocketDisconnect:
        manager.disconnect(websocket)
    except Exception as e:
        await manager.send_message(websocket, {
            "type": "error",
            "error": f"Failed to process message: {str(e)}"
        })
        manager.disconnect(websocket)

if __name__ == "__main__":
    uvicorn.run("app:app", host="0.0.0.0", port=8000, reload=True)
```

### Key Server Features

1. **Session Management**: Using Map/Dict to store session state, including accumulated audio data
2. **API Integration**: Integration with Baidu Speech Recognition API, including token management and API calls
3. **Streaming Processing**: Support for real-time audio stream processing and result return
4. **Multi-platform Support**: Provides Node.js and Python implementations to support different development environments

## Best Practices

### Audio Quality vs Performance Balance

- For speech recognition, 16kHz, mono, 16-bit audio format is typically the optimal choice
- Setting the `timeslice` parameter too small can lead to excessive network requests, while too large can affect real-time performance
- Consider implementing simple noise reduction on the client side to improve recognition accuracy

### Security Considerations

- Implement proper authentication and authorization mechanisms to prevent unauthorized access
- Consider encrypting audio data, especially when containing sensitive information
- Implement rate limiting to prevent DoS attacks

### Scalability Design

- Use message queues to handle audio processing requests in high-traffic scenarios
- Consider using microservice architecture to separate recording and recognition functions
- Implement retry mechanisms and resumable uploads to improve system stability

## Common Issues

### 1. Recording Not Working

- Ensure you're running in an HTTPS environment or localhost, as modern browsers require secure contexts for microphone access
- Check if microphone permissions have been granted
- Verify that recording parameters (format, sample rate, etc.) are correctly set

### 2. Server Connection Issues

- Check network connectivity and CORS configuration
- WebSocket connections may require special proxy settings, especially in corporate network environments
- Implement connection retry and auto-reconnect mechanisms

### 3. Low Speech Recognition Accuracy

- Improve audio quality using better microphones or noise reduction settings
- Adjust sample rate and bit depth to ensure compatibility with the recognition service
- Consider professional speech recognition services like Google Speech-to-Text, Azure, or Amazon Transcribe

### 4. Poor Real-time Performance

- Reduce audio chunk size (timeslice parameter)
- Use WebSocket instead of REST API for lower latency
- Optimize server processing logic to reduce unnecessary computation

---

This guide provides a basic framework that you can adjust and extend according to your specific needs. For specific questions, please refer to the official documentation or contact our support team.