betfair-node

Version:
A comprehensive Node.js TypeScript library for the Betfair Exchange API, providing both JSON-RPC API integration and real-time Exchange Stream API support for automated betting and trading applications.
github.com/felixmccuaig/betfair-node
felixmccuaig/betfair-node
616 lines (483 loc) • 22.9 kB
Markdown
View Raw
# betfair-node

A comprehensive Node.js TypeScript library for the Betfair Exchange API, providing both JSON-RPC API integration and real-time Exchange Stream API support for automated betting and trading applications. Features advanced market recording capabilities for capturing raw TLS stream data and structured market summaries.

[![npm version](https://badge.fury.io/js/betfair-node.svg)](https://badge.fury.io/js/betfair-node)
[![TypeScript](https://img.shields.io/badge/TypeScript-Ready-blue.svg)](https://www.typescriptlang.org/)
[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)

## Features

### 🚀 **Core API Integration**
- **Authentication**: Interactive login with username/password (certificate auth coming soon)
- **Market Data**: List markets, get market books, runner details, and pricing information
- **Order Management**: Place, cancel, replace, and update bets with comprehensive validation
- **Account Operations**: Get account details, balance, statement, and currency rates

### 📡 **Real-time Exchange Stream API**
- **Live Market Data**: Subscribe to real-time market updates with low latency
- **Order Stream**: Real-time order updates, unmatched orders, and matched positions
- **Message Segmentation**: Automatic handling of segmented messages for optimal performance
- **Stream Decoding**: Automatic deserialization and caching of market/order deltas
- **Connection Management**: Robust connection handling with automatic reconnection
- **Heartbeat Monitoring**: Built-in connection health monitoring

### 🛠️ **Developer Experience**
- **Full TypeScript Support**: Complete type definitions for all API responses
- **Functional API**: Modern functional programming approach with immutable state
- **Error Handling**: Comprehensive error handling and validation
- **Flexible Configuration**: Support for different currencies, locales, and regions

### 📊 **Professional Trading Features**
- **Price Ladder Display**: Real-time price ladder visualization
- **Price Ladder Utilities**: Complete Betfair price ladder generation and tick navigation
- **Market Caching**: Efficient delta-based market state management  
- **Request Conflation**: Batch multiple market subscriptions for efficiency
- **Currency Conversion**: Automatic currency rate handling and conversion

### 📹 **Market Recording & Analysis**
- **Raw Data Recording**: Capture pure TLS stream transmissions before any processing
- **Structured Market Summaries**: Automatic BSP, winner, and trading volume extraction
- **Market-Specific Files**: Individual files per market for organized data storage
- **Production-Ready Monitoring**: Robust heartbeat and error handling for long recordings

## Installation

```bash
npm install betfair-node
```

## Quick Start

### 1. Set up your credentials

Create a `.env` file in your project root:

```env
BETFAIR_APP_KEY=your_application_key_here
BETFAIR_USERNAME=your_username_here
BETFAIR_PASSWORD=your_password_here
```

### 2. Basic API Usage

```typescript
import { createBetfairApiState, login, listMarketCatalogue } from 'betfair-node';

async function main() {
  // Create API state
  const apiState = createBetfairApiState(
    'en',    // locale
    'AUD',   // currency
    500,     // conflateMs
    5000,    // heartbeatMs
    () => {} // marketChangeCallback (not used in this example)
  );
  
  // Login to Betfair
  const authenticatedState = await login(
    apiState,
    process.env.BETFAIR_APP_KEY!,
    process.env.BETFAIR_USERNAME!,
    process.env.BETFAIR_PASSWORD!
  );
  
  // List horse racing markets
  const marketFilter = {
    eventTypeIds: ['7'], // Horse Racing
    marketCountries: ['AU'],
    marketTypeCodes: ['WIN']
  };
  
  const markets = await listMarketCatalogue(
    authenticatedState,
    marketFilter,
    ['MARKET_START_TIME', 'RUNNER_DESCRIPTION'],
    'FIRST_TO_START',
    10
  );
  
  console.log(`Found ${markets.data.result.length} markets`);
}
```

### 3. Real-time Market Data

```typescript
import { 
  createBetfairApiState, 
  login,
  createAndConnectStream,
  subscribeToMarkets 
} from 'betfair-node';

async function streamMarketData() {
  // Setup and authenticate
  const apiState = createBetfairApiState(
    'en',    // locale
    'AUD',   // currency
    500,     // conflateMs
    5000,    // heartbeatMs
    (marketCache, deltas) => {
      // Handle real-time market updates
      console.log('Market update received:', Object.keys(marketCache));
    }
  );
  
  const authenticatedState = await login(apiState, appKey, username, password);
  
  // Create stream connection
  const streamState = await createAndConnectStream(
    authenticatedState.sessionKey!,
    authenticatedState.appKey!,
    false,   // segmentationEnabled
    500,     // conflateMs
    5000,    // heartbeatMs
    { currencyCode: 'AUD', rate: 1.0 }, // audCurrencyRate
    (marketCache, deltas) => {
      // Handle real-time market updates
      console.log('Market update received:', Object.keys(marketCache));
    }
  );
  
  // Subscribe to specific markets
  await subscribeToMarkets(
    streamState,
    ['1.234567890'], // market IDs
    ['EX_BEST_OFFERS'] // price data fields
  );
}
```

### 4. Place a Bet

```typescript
import { 
  createBetfairApiState,
  login,
  placeOrders,
  validateOrderParameters 
} from 'betfair-node';

async function placeBet() {
  const apiState = createBetfairApiState(
    'en',    // locale
    'AUD',   // currency
    500,     // conflateMs
    5000,    // heartbeatMs
    () => {} // marketChangeCallback (not used in this example)
  );
  
  const authenticatedState = await login(apiState, appKey, username, password);
  
  const marketId = '1.234567890';
  const selectionId = 123456;
  const price = 2.50;
  const size = 10.00;
  
  // Validate before placing
  const validation = validateOrderParameters(marketId, selectionId, price, size);
  if (validation.isValid) {
    const placeInstruction = {
      orderType: 'LIMIT' as const,
      selectionId: selectionId,
      side: 'BACK' as const,
      limitOrder: {
        size: size,
        price: price,
        persistenceType: 'LAPSE' as const
      }
    };
    
    const result = await placeOrders(
      authenticatedState,
      marketId,
      [placeInstruction],
      'my-bet-ref',
      undefined, // marketVersion
      undefined, // customerStrategyRef
      false      // async
    );
    
    console.log('Bet placed:', result);
  }
}
```

### 5. Price Ladder Navigation

```typescript
import { 
  getTickSize,
  getNextTick,
  getPreviousTick,
  getNearestValidPrice,
  generatePriceLadder 
} from 'betfair-node';

// Get tick sizes for different price ranges
console.log(getTickSize(1.5));   // 0.01 (1.01-2 range)
console.log(getTickSize(3.35));  // 0.05 (3-4 range)  
console.log(getTickSize(120));   // 10 (100-1000 range)

// Navigate price ladder
console.log(getNextTick(120));     // 130
console.log(getNextTick(3.35));    // 3.4
console.log(getPreviousTick(130)); // 120
console.log(getPreviousTick(3.4)); // 3.35

// Round invalid prices to valid ladder prices
console.log(getNearestValidPrice(1.234));           // 1.23 (nearest)
console.log(getNearestValidPrice(3.123, 'up'));     // 3.15 (round up)
console.log(getNearestValidPrice(3.149, 'down'));   // 3.1 (round down)

// Generate complete price ladder (memoized for performance)
const ladder = generatePriceLadder(1.90, 2.10);
console.log(ladder); 
// [1.9, 1.91, 1.92, ..., 1.99, 2, 2.02, 2.04, ..., 2.1]

// Betfair price ranges and increments:
// 1.01→2: 0.01 | 2→3: 0.02 | 3→4: 0.05 | 4→6: 0.1 | 6→10: 0.2
// 10→20: 0.5 | 20→30: 1 | 30→50: 2 | 50→100: 5 | 100→1000: 10
```

### 6. Market Recording

```typescript
import { 
  createBetfairApiState,
  login,
  listMarketCatalogue,
  createAndConnectRecordingStream,
  subscribeToMarkets,
  createMarketRecorderState,
  startRecording,
  createRecordingMarketChangeCallback,
  createRawDataCallback
} from 'betfair-node';

async function recordMarketData() {
  // Setup authentication
  const apiState = createBetfairApiState('en', 'AUD', 250, 5000, () => {});
  const authenticatedState = await login(apiState, appKey, username, password);

  // Configure recording (both raw and structured data)
  const recorderState = createMarketRecorderState({
    outputDirectory: './recordings',
    enableBasicRecording: true,    // Structured summaries
    enableRawRecording: true,      // Pure TLS stream data
    rawFilePrefix: '',             // Files: {marketId}.txt
    basicFilePrefix: 'basic_',     // Files: basic_{marketId}.json
  });

  // Find markets and start recording
  const markets = await listMarketCatalogue(/* ... */);
  const marketIds = markets.data.result.map(m => m.marketId);
  
  startRecording(recorderState, marketIds);

  // Connect with recording-optimized stream (30s heartbeat)
  const streamState = await createAndConnectRecordingStream(
    authenticatedState.sessionKey!,
    appKey,
    false,                        // segmentationEnabled
    250,                          // conflateMs
    { currencyCode: 'AUD', rate: 1.0 },
    createRecordingMarketChangeCallback(recorderState), // Basic recording
    createRawDataCallback(recorderState)                // Raw recording
  );

  subscribeToMarkets(streamState, marketIds);
  
  // Records automatically to:
  // - {marketId}.txt (raw TLS transmissions)
  // - basic_{marketId}.json (market summaries with BSP, winners, etc.)
}
```

## API Reference

### Core Functions

#### Authentication
- `createBetfairApiState(locale, currencyCode, conflateMs, heartbeatMs, marketChangeCallback)` - Create initial API state
- `login(state, appKey, username, password)` - Authenticate with Betfair and return updated state
- `logout(sessionKey)` - End session

#### Market Data
- `listEventTypes(state, filter)` - Get available sports/event types
- `listMarketCatalogue(state, filter, projections, sort, maxResults)` - Search for markets
- `listMarketBook(state, marketIds, priceProjection)` - Get market pricing data
- `listCurrentOrders(state, betIds?, marketIds?)` - Get current orders

#### Order Management
- `placeOrders(state, marketId, instructions, customerRef?)` - Place new bets
- `cancelOrders(state, marketId, instructions, customerRef?)` - Cancel existing bets
- `replaceOrders(state, marketId, instructions, customerRef?)` - Replace existing bets
- `updateOrders(state, marketId, instructions, customerRef?)` - Update existing bets

#### Account Operations
- `getAccountFunds(state, wallet?)` - Get account balance
- `getAccountDetails(state)` - Get account information
- `getAccountStatement(state, options)` - Get account statement

#### Market Recording
- `createMarketRecorderState(config)` - Initialize market recorder with configuration
- `startRecording(state, marketIds)` - Start recording for specified markets
- `stopRecording(state)` - Stop recording and save all data to files
- `createRecordingMarketChangeCallback(state, originalCallback?)` - Create callback for structured recording
- `createRawDataCallback(state)` - Create callback for raw TLS stream recording
- `getRecordingStatus(state, marketId)` - Get current recording status for a market
- `loadBasicRecord(config, marketId)` - Load previously saved market summary
- `listRecordedMarkets(config)` - List all recorded markets in output directory

### Streaming API

#### Connection Management
- `createAndConnectStream(authToken, appKey, segmentationEnabled, conflateMs, heartbeatMs, audCurrencyRate, marketChangeCallback, rawDataCallback?)` - Connect to stream
- `createAndConnectRecordingStream(authToken, appKey, segmentationEnabled, conflateMs, audCurrencyRate, marketChangeCallback, rawDataCallback?)` - Connect with recording optimization (30s heartbeat)
- `closeStream(state)` - Disconnect from stream

#### Market Subscriptions
- `subscribeToMarkets(state, marketIds, fields, segmentationEnabled?)` - Subscribe to markets
- `resubscribeToMarkets(state, marketIds, fields?)` - Update subscription
- `unsubscribeFromMarkets(state, marketIds?)` - Unsubscribe from markets

#### Order Stream Subscriptions
- `subscribeToOrders(state, orderFilter?)` - Subscribe to order updates with optional filtering
- `getOrderStreamCache(state)` - Get current order cache from stream

### Message Segmentation

Betfair's Exchange Stream API supports **message segmentation** to handle large data payloads efficiently. When enabled, large messages are broken into smaller segments that are automatically reassembled by the library.

#### How Segmentation Works

When `segmentationEnabled: true` is set:

1. **Large messages are segmented** into multiple parts:
   - `SEG_START` - First segment of a message
   - Middle segments (no segmentationType) - Additional data segments  
   - `SEG_END` - Final segment containing remaining data

2. **Automatic reassembly** - The library automatically:
   - Buffers segments until complete message received
   - Merges all market/order changes from all segments
   - Processes the complete message only after reassembly
   - Maintains separate buffers for market and order messages

3. **Performance benefits**:
   - Improved end-to-end performance and latency
   - Faster time to first and last byte
   - Reduced memory pressure for large messages

#### Usage Example

```typescript
// Enable segmentation for better performance with large messages
const streamState = await createAndConnectStream(
  authToken,
  appKey,
  true,    // segmentationEnabled - IMPORTANT for order streams
  500,     // conflateMs
  5000,    // heartbeatMs
  currencyRate,
  marketChangeCallback,
  orderChangeCallback  // Order callback for order stream functionality
);

// Subscribe to order updates (requires segmentation for full functionality)
const orderFilter = {
  includeOverallPosition: true,  // Include all position data
  customerStrategyRefs: ['MyStrategy'], // Optional: filter by strategy
  partitionMatchedByStrategyRef: false  // Optional: partition data
};

const updatedState = subscribeToOrders(streamState, orderFilter);
```

#### Important Notes

- **Order streams require segmentation enabled** for full functionality
- Segmentation is automatically handled - no manual intervention needed
- Non-segmented messages are processed immediately
- Segment buffers are automatically cleaned up after processing
- Multiple concurrent segmented message streams are supported

### Utility Functions

#### General Utilities
- `validateOrderParameters(marketId, selectionId, price, size)` - Validate bet parameters
- `calculateBackProfit(stake, odds)` - Calculate back bet profit
- `calculateLayLiability(stake, odds)` - Calculate lay bet liability
- `findCurrencyRate(rates, currency)` - Find currency conversion rate

#### Price Ladder Functions
- `getTickSize(price)` - Get the appropriate tick increment for any price
- `generatePriceLadder(minPrice?, maxPrice?)` - Generate complete price ladder with memoization
- `getNextTick(price)` - Get the next valid tick price (or null if at maximum)
- `getPreviousTick(price)` - Get the previous valid tick price (or null if at minimum)
- `getNearestValidPrice(price, direction?)` - Round to nearest valid price ('up', 'down', or 'nearest')

## Examples

The `examples/` directory contains comprehensive examples:

### 🐕 Greyhound Markets (`npm run example:greyhounds`)
Lists Australian greyhound racing markets with detailed information including venues, race times, and runner details.

### 💰 Place Bet (`npm run example:bet`)  
Demonstrates placing a real back bet with validation, error handling, and success reporting.

### 📋 Order Stream (`npm run example:orders`)
Real-time order stream monitoring showing your live betting positions and order updates. Features:
- **Complete Order View**: Shows all current orders via JSON-RPC API for comparison
- **Real-time Updates**: Live order status changes, matches, and position updates  
- **Segmentation Support**: Demonstrates proper message segmentation handling
- **Strategy Filtering**: Examples of filtering orders by customer strategy references

### 📺 Live Trading View (`npm run example:live`)
Professional real-time trading interface showing live price ladders, similar to trading software. Features:
- Real-time price updates (500ms refresh)
- Color-coded price changes
- Best back/lay prices and volumes
- Clean, professional display

### 📹 Market Recorder (`npm run example:recorder`)
Comprehensive market data recording system that captures both raw TLS transmissions and structured market summaries. Features:
- **Dual Recording**: Raw TLS stream data + structured market summaries
- **Production Ready**: 30-second heartbeat, graceful error handling
- **Market-Specific Files**: Individual files per market for organized storage
- **Complete Lifecycle**: Records from market open through settlement

### 📹 Simple Market Recorder (`npm run example:recorder-simple`)
Streamlined recording example with two modes:
- **Finite mode**: Records until all markets complete (default)
- **Timeout mode**: Records for 30 seconds for quick testing (`--timeout` flag)

### 🔄 Perpetual Greyhound Recorder (`npm run example:recorder-perpetual`)
Advanced perpetual recording system that continuously records ALL greyhound markets:
- **Auto-discovery**: Finds new markets every 5 minutes
- **Dynamic recording**: Adds new markets without interrupting existing recordings
- **Multi-country**: Records from AU, GB, IE, US greyhound tracks
- **Production ready**: Runs indefinitely until manually stopped

### 📚 Historical Data Backtesting
Analyze historical Betfair Exchange data files to generate market and runner-level analytics for research and strategy testing.

• Simple Backtest (`npm run example:backtest`)
  - Easy to run, processes a historical file from `examples/`
  - Outputs JSON records and a summary report in `backtest_results/`

• Advanced Backtest (`npm run example:backtest-verbose`)
  - Programmatic API for custom analysis with verbose logging options

Input format: line-delimited JSON with Betfair Exchange Stream `mcm` messages. Place a file (e.g., `examples/1.216777904`) in the examples directory.

Outputs: JSON records per market and an aggregated `backtest_summary.json` in `backtest_results/`.

See detailed docs: `examples/BACKTEST_README.md`.

> **📖 Detailed Documentation**: See [`examples/MARKET_RECORDER_README.md`](examples/MARKET_RECORDER_README.md) for comprehensive market recorder documentation including architecture, data flow, and advanced usage.

### Usage
```bash
# List greyhound markets
npm run example:greyhounds

# Place a bet (uses real money!)
npm run example:bet

# Live market view (specify market ID)
npm run example:live -- marketid=1.234567890

# Record market data (stops when all markets complete)
npm run example:recorder

# Quick recording test - finite mode (stops when markets complete)
npm run example:recorder-simple

# Quick recording test - timeout mode (stops after 30 seconds)
npm run example:recorder-simple --timeout

# Perpetual greyhound recording (runs forever, auto-discovers new markets)
npm run example:recorder-perpetual

# Simple historical backtest (reads file from examples/)
npm run example:backtest

# Advanced/verbose backtest
npm run example:backtest-verbose
```

## Architecture

### Core Components

- **BetfairApi**: Main API client for JSON-RPC endpoints
- **BetfairExchangeStreamApi**: Real-time streaming client  
- **BetfairStreamDecoder**: Handles stream data deserialization and caching
- **MarketRecorder**: Dual-layer recording system for raw TLS data and structured summaries
- **Heartbeat**: Connection monitoring and health checking
- **Utils**: Helper functions for validation and calculations

### Design Principles

- **Functional Programming**: Immutable state management with pure functions
- **Type Safety**: Comprehensive TypeScript definitions for all API responses
- **Error Handling**: Explicit error handling with detailed error messages
- **Performance**: Efficient caching and delta processing for real-time data
- **Flexibility**: Support for different regions, currencies, and market types

## Environment Variables

| Variable | Required | Default | Description |
|----------|----------|---------|-------------|
| `BETFAIR_APP_KEY` | ✅ Yes | - | Your Betfair application key |
| `BETFAIR_USERNAME` | ✅ Yes | - | Your Betfair account username |
| `BETFAIR_PASSWORD` | ✅ Yes | - | Your Betfair account password |
| `TIMEZONE` | ❌ No | `Australia/Sydney` | Timezone for time-based operations |
| `LOCALE` | ❌ No | `en` | Locale for API responses |
| `CURRENCY` | ❌ No | `AUD` | Default currency for amounts |

## Getting Betfair API Access

1. **Create a Betfair Account**: Sign up at [betfair.com](https://www.betfair.com)
2. **Get Developer Access**: Apply for API access at [developer.betfair.com](https://developer.betfair.com)
3. **Create an Application**: Generate your App Key from the developer portal
4. **Fund Your Account**: Add funds to place real bets (required for full API access)

## Development

```bash
# Install dependencies
npm install

# Run tests
npm test

# Build the project
npm run build

# Run examples
npm run example:greyhounds
npm run example:bet
npm run example:live -- marketid=1.234567890
npm run example:recorder
npm run example:recorder-simple
npm run example:recorder-perpetual

# Development mode
npm run dev
```

## Testing

```bash
# Run all tests
npm test

# Run tests in watch mode
npm run test:watch

# Run tests with coverage
npm run test:coverage
```

## Contributing

Contributions are welcome! Please feel free to submit a Pull Request. For major changes, please open an issue first to discuss what you would like to change.

## Roadmap

- 🔐 Certificate-based authentication support
- 📊 Advanced market analysis utilities  
- 🎯 Strategy backtesting framework using recorded market data
- 📈 Historical data integration and replay functionality
- 🔄 Enhanced request conflation
- 📱 React/Vue.js integration examples
- 🗂️ Market recording database integration (PostgreSQL/MongoDB)
- 📊 Real-time recording dashboard and monitoring

## License

This project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details.

## Disclaimer

This software is for educational and development purposes. Users are responsible for complying with Betfair's terms of service and applicable gambling regulations. The authors are not responsible for any financial losses incurred through the use of this software.

---

**Author**: Felix McCuaig  
**Email**: felixmccuaig@gmail.com  
**GitHub**: [betfair-node](https://github.com/felixmccuaig/betfair-node)

For questions, issues, or feature requests, please use the GitHub issues page.