@bsv/overlay-express/README.md

BSV Blockchain Overlay Express

# Overlay Express

BSV BLOCKCHAIN | Overlay Express

An opinionated but configurable Overlay Services deployment system:
- Uses express for HTTP on a given local port
- Easy setup with just a private key and a hosting URL
- Import and configure your topic managers the way you want
- Add lookup services, with easy factories for Mongo and Knex
- Implements a configurable web UI to show your custom overlay service docs to the world
- Uses common Knex/SQL and Mongo databases across all services for efficiency
- Supports SHIP, SLAP, and GASP sync out of the box (or it can be disabled)
- Supports Arc callbacks natively for production (or disable it for simplicity during local development)
- Supports Arcade-first broadcast/proof lookup, Arc fallback broadcast,
  Chaintracks header validation, BASM reorg streaming, and active
  monitor-driven maintenance

## Example Usage

Here's a quick example:

```typescript
import OverlayExpress from '@bsv/overlay-express'
import dotenv from 'dotenv'
dotenv.config()

// Hi there! Let's configure Overlay Express!
const main = async () => {

    // We'll make a new server for our overlay node.
    const server = new OverlayExpress(

        // Name your overlay node with a one-word lowercase string
        `testnode`,

        // Provide the private key that gives your node its identity
        process.env.SERVER_PRIVATE_KEY!,

        // Provide the HTTPS URL where your node is available on the internet
        process.env.HOSTING_URL!
    )

    // Decide what port you want the server to listen on.
    server.configurePort(8080)

    // Connect to your SQL database with Knex
    await server.configureKnex(process.env.KNEX_URL!)

    // Also, be sure to connect to MongoDB
    await server.configureMongo(process.env.MONGO_URL!)

    // Here, you will configure the overlay topic managers and lookup services you want.
    // - Topic managers decide what outputs can go in your overlay
    // - Lookup services help people find things in your overlay
    // - Make use of functions like `configureTopicManager` and `configureLookupServiceWithMongo`
    // ADD YOUR OVERLAY SERVICES HERE

    // For simple local deployments, sync can be disabled.
    server.configureEnableGASPSync(false)

    // Production deployments should configure at least one transaction
    // propagation provider. Arcade can be used as the primary provider with Arc
    // as a fallback. The callback token is optional, but recommended when
    // exposing /arc-ingest publicly.
    server.configureArcade(process.env.ARCADE_URL!, {
      apiKey: process.env.ARCADE_API_KEY,
      deploymentId: process.env.ARCADE_DEPLOYMENT_ID,
      chaintracksApiPrefix: '/chaintracks/v2'
    })
    if (process.env.ARC_API_KEY) {
      server.configureArcApiKey(process.env.ARC_API_KEY)
    }
    if (process.env.ARC_CALLBACK_TOKEN) {
      server.configureArcCallbackToken(process.env.ARC_CALLBACK_TOKEN)
    }

    // Chaintracks-compatible services provide block headers for BASM and can
    // stream reorg notifications. Arcade exposes go-chaintracks under
    // /chaintracks/v2.
    server.configureChaintracks(process.env.CHAINTRACKS_URL ?? process.env.ARCADE_URL!, {
      apiPrefix: '/chaintracks/v2',
      reorgStream: true,
      scanDepth: 3
    })

    // Lastly, configure the engine and start the server!
    await server.configureEngine()
    await server.start()
}

// Happy hacking :)
main()
```

## Full API Docs

Check out [API.md](./API.md) for the API docs.

## Additional Features

### Advanced Engine Configuration

We've introduced a new method, `configureEngineParams`, that allows you to pass advanced configuration options to the underlying Overlay Engine. Here's an example usage:

```typescript
server.configureEngineParams({
  logTime: true,
  throwOnBroadcastFailure: true,
  overlayBroadcastFacilitator: new MyCustomFacilitator()
})
```

`throwOnBroadcastFailure` should remain `true` for most production overlays. With
provider-chain broadcast configured, this means a transaction is not committed to
overlay state unless at least one provider accepts it or returns an already-known
success. Set it to `false` only for deliberate offline/dev workflows where local
overlay admission may proceed without current network propagation.

### Transaction Propagation Providers

Overlay Express can compose multiple transaction propagation and proof sources:

```typescript
server.configureArcade('https://arcade-v2-us-1.bsvblockchain.tech', {
  apiKey: process.env.ARCADE_API_KEY,
  deploymentId: 'my-overlay-node',
  chaintracksApiPrefix: '/chaintracks/v2'
})

server.configureArcApiKey(process.env.ARC_API_KEY!)
server.configureArcCallbackToken(process.env.ARC_CALLBACK_TOKEN!)

server.configureChaintracks('https://arcade-v2-us-1.bsvblockchain.tech', {
  apiPrefix: '/chaintracks/v2',
  reorgStream: true,
  scanDepth: 3
})
```

- `configureArcade` registers Arcade as the first-choice broadcaster and proof
  lookup provider.
- `configureArcApiKey` registers the standard Arc broadcaster as fallback.
- `configureArcCallbackToken` requires inbound `/arc-ingest` callbacks to present
  the expected token.
- `configureChaintracks` configures a go-chaintracks compatible service for block
  header lookup, BASM anchor header resolution, and optional reorg SSE.

Provider callbacks posted to `/arc-ingest` are classified as successful proof,
terminal invalidation, double spend, or transient status. Double-spend and other
terminal invalid statuses remove the affected transaction from the admitted
overlay state so the lookup layer does not keep serving data that the network has
rejected.

### BASM And Unproven Maintenance

BASM is opt-in because it requires direct proofs and block header resolution:

```typescript
server.configureEnableBASMSync(true)
server.configureBASMBlockPollInterval(10 * 60 * 1000)
server.configureUnprovenMaintenance({
  intervalMs: 60 * 60 * 1000,
  thresholdBlocks: 144
})
```

When configured, unproven maintenance first tries the configured proof providers
for transactions that remain unproven past the threshold. Only transactions that
still cannot be proven are evicted. Operators can also run this manually through
the admin endpoints documented below.

### Admin-Protected Endpoints

We also provide admin-protected endpoints for advanced operations like manually
syncing advertisements, triggering GASP/BASM sync, running unproven maintenance,
evicting specific outpoints, and running the janitor. These endpoints require a
Bearer token. You can supply a custom token in the constructor of
`OverlayExpress`, or retrieve the auto-generated token by calling
`server.getAdminToken()`.

Common admin endpoints:

- `POST /admin/syncAdvertisements`
- `POST /admin/startGASPSync`
- `POST /admin/startBASMSync`
- `POST /admin/refreshUnprovenProofs`
- `POST /admin/evictUnproven`
- `POST /admin/maintainUnproven`
- `POST /admin/evictOutpoint`
- `POST /admin/janitor`

### Health Endpoints

Overlay Express now exposes three health endpoints:

- `GET /health/live`: liveness-only status for process-level checks.
- `GET /health/ready`: readiness status for critical dependencies such as the Overlay engine, Knex, and MongoDB.
- `GET /health`: full component report combining liveness, readiness, service metadata, and any registered custom checks.

You can attach additional application-aware checks and context:

```typescript
server.configureHealth({
  contextProvider: async () => ({
    deployment: 'cars-project-backend',
    network: process.env.NETWORK
  })
})

server.registerHealthCheck({
  name: 'custom-cache',
  critical: false,
  handler: async () => ({
    status: 'ok',
    details: { warmed: true }
  })
})
```

The janitor service also understands the richer `/health` response format, so existing SHIP/SLAP health validation remains compatible.

### Overlay Monitor

`OverlayMonitor` provides a reusable worker for monitoring Overlay Express lookup
behavior and, when configured with an admin token, running maintenance actions.
It posts configured `/lookup` probes to any Overlay Express deployment, measures
response size, parses returned BEEF, and reports whether responsive output
transactions have direct Merkle proofs or are being served with deeper proof
ancestry.

This is intended to be run by deployments as a long-running monitor process or by cluster scheduling. It is not tied to a specific deployment platform.

```typescript
import { OverlayMonitor } from '@bsv/overlay-express'

const monitor = new OverlayMonitor({
  intervalMs: 24 * 60 * 60 * 1000,
  targets: [
    {
      name: 'example-overlay',
      baseUrl: 'https://overlay.example',
      probes: [
        {
          name: 'example-topic',
          service: 'ls_example',
          query: { topic: 'tm_example' },
          maxOutputs: 20
        }
      ],
      adminToken: process.env.OVERLAY_ADMIN_TOKEN,
      maintenance: {
        startBASMSync: true,
        maintainUnproven: {
          thresholdBlocks: 144
        },
        janitor: true
      }
    }
  ],
  onReport: async report => {
    console.log(JSON.stringify(report.summary))
  }
})

monitor.start()
```

Maintenance requests are reported alongside lookup probes so operators can alert
on failed maintenance separately from lookup/proof-shape warnings.

## License

The license for the code in this repository is the Open BSV License. Refer to [LICENSE.txt](./LICENSE.txt) for the license text.

Thank you for being a part of the BSV Blockchain Overlay Express Project. Let's build the future of BSV Blockchain together!