node-multi-curl/README.md

A performant multi-curl class in Node.js that supports proxies and concurrent requests

# node-multi-curl

A performant multi-curl class in Node.js that supports working with proxies and concurrent requests.

## Features

- Parallel HTTP requests with concurrency control
- Proxy support with automatic rotation
- Docker integration for HTTP/2 support
- Axios-like API for ease of use
- Request/response interceptors
- Automatic retry mechanism
- HTTP/2 support** via Docker curl integration
- Cloudflare capabilities with advanced header management
- Brotli, Gzip and Deflate** compression support

## Installation

```bash
npm install node-multi-curl
```

## Quick Start

```javascript
const { MultiCurl } = require('node-multi-curl');

// Simple GET request
const response = await MultiCurl.get('https://example.com/api');

// Multiple concurrent requests
const client = new MultiCurl({ concurrency: 5 });
const responses = await client.execute([
  { url: 'https://example.com/api/users' },
  { url: 'https://example.com/api/products' },
  { url: 'https://example.com/api/orders' }
]);
```

## Advanced Features

### HTTP/2 and Compression Support

```javascript
const client = new MultiCurl({
  curlConfig: {
    useDocker: true,
    dockerImage: 'badouralix/curl-http2:latest'
  }
});

// This request will use HTTP/2 if the server supports it
const response = await client.get('https://http2.pro/api/v1');

// Brotli, Gzip and other compression methods are automatically handled
```

### Cloudflare Protection Bypass

```javascript
// Configure with specialized headers
const client = new MultiCurl();

client.interceptors.request.use(request => {
  request.headers = {
    ...request.headers,
    'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.36',
    'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8',
    'Accept-Language': 'en-US,en;q=0.5',
    'Accept-Encoding': 'gzip, deflate, br',
    'Connection': 'keep-alive',
    'Upgrade-Insecure-Requests': '1',
    'Sec-Fetch-Dest': 'document',
    'Sec-Fetch-Mode': 'navigate',
    'Sec-Fetch-Site': 'none',
    'Sec-Fetch-User': '?1'
  };
  return request;
});

// Now you can access Cloudflare-protected sites
const response = await client.get('https://cloudflare-protected-site.com');
```

### Query Parameters

```javascript
// Using query parameters
const response = await MultiCurl.get('https://httpbin.org/get', {
  queryParams: {
    param1: 'value1',
    param2: 123,
    param3: true,
    nullParam: null, // This will be skipped
  }
});
// Makes a request to: https://httpbin.org/get?param1=value1&param2=123&param3=true

// Or with the execute method:
const client = new MultiCurl();
const responses = await client.execute([
  { 
    url: 'https://httpbin.org/get', 
    queryParams: { search: 'test', page: 1 } 
  }
]);
```

## Documentation

For detailed documentation and examples, check the [examples](./examples) directory.

## License

MIT