autotel/skills/tune-sampling/SKILL.md

Write Once, Observe Anywhere

---
name: tune-sampling
description: >
  Choose a sampling strategy for an autotel-instrumented service. Covers
  head sampling (per-span-kind rates, parent-based, ratio), tail sampling
  (keep errors, slow, AI-aware, debug-headers), cost vs cardinality
  tradeoffs, and the math for picking rates that hit a target spans/second
  budget. Includes recipes for low-volume admin services, high-volume APIs,
  AI agents, and Cloudflare Workers.
license: MIT
---

# Tune sampling

Untuned tracing is either expensive (100 % at scale costs money + drowns dashboards) or unhelpful (1 % loses the failure modes you need to see). The right answer is almost always **head sample most of the boring traffic, tail keep all the interesting traffic**, with explicit overrides for AI calls and customer escalations.

## When to use

- Hitting your observability budget
- Dashboards too sparse to spot anomalies
- "We have the trace IDs but the spans are gone" complaints
- New service launching at scale
- Long-running AI agents producing 50+ spans per request

## The mental model

```
Total cost = (spans/sec × $/span) + (storage_GB × $/GB-month)
                   ↑
Head sampling reduces this directly.
```

Head sampling makes a decision **at span start**: fast but coarse, since it cannot know whether the span will fail.
Tail sampling makes the decision **at span end**: slower and heavier on storage, but precise.

The right mix:

- **Head sample at the entry point** to keep volume tractable.
- **Tail keep** the high-value subset (errors, slow, AI, debug-headered).
- **Don't sample audit spans**: separate processor, see [`build-audit-trails`](../build-audit-trails/SKILL.md).

## Head sampling recipes

### Default for a typical web service

```typescript
init({
  service: 'my-app',
  sampling: {
    rates: {
      server: 25, // server entry spans — sample ¼
      client: 5, // outbound HTTP — sample 1/20
      internal: 5, // internal sub-spans — sample 1/20
    },
  },
});
```

Children of a sampled root are **all** kept (parent-based propagation is the default). So `server: 25` means 25 % of _user requests_, complete trace each.

### High-volume API (>1 k req/s)

```typescript
sampling: {
  rates: { server: 5, client: 1, internal: 1 }, // 5 % → tail keeps errors anyway
  tail: keepInterestingTraces,
},
```

### Low-volume admin / internal service (<10 req/s)

100 % is fine. Don't penalise yourself for a service that produces 1 GB of traces a week.

### Cloudflare Workers (per-colo budget)

Workers run distributed. Head sampling is your friend because there's no central queue:

```typescript
defineWorkerFetch(
  {
    service: { name: 'edge' },
    sampling: { rates: { server: 10 } }, // 10 % per colo, scales naturally
  },
  handler,
);
```

## Tail sampling: keep interesting traces

Tail sampling looks at the full trace (root span + children) before deciding. autotel ships `TailSamplingProcessor`:

```typescript
import { TailSamplingProcessor } from 'autotel/processors';
import { SpanStatusCode } from '@opentelemetry/api';

const tail = new TailSamplingProcessor({
  keep: (trace) => {
    // 1. Always keep errors
    if (trace.localRootSpan.status?.code === SpanStatusCode.ERROR) return true;
    if (trace.spans.some((s) => s.status?.code === SpanStatusCode.ERROR))
      return true;

    // 2. Always keep slow traces (configurable threshold)
    if (durationMs(trace.localRootSpan) > 1_000) return true;

    // 3. Always keep customer-marked traces
    if (trace.localRootSpan.attributes['debug.trace'] === true) return true;

    // 4. Always keep AI traces (rare + expensive — full visibility helps)
    if (
      trace.spans.some(
        (s) =>
          typeof s.attributes['gen_ai.provider.name'] === 'string' ||
          // legacy: third-party instrumentations may still emit gen_ai.system
          typeof s.attributes['gen_ai.system'] === 'string',
      )
    )
      return true;

    // 5. Otherwise: respect head sampling decision
    return false;
  },
});
```

### Combining with multi-backend

```typescript
spanProcessors: composeSpanProcessors([
  // Drop nothing here — we want the tail processor to see the full trace
  new BatchSpanProcessor(localExporter),
  tail, // filters before remote export
  new BatchSpanProcessor(expensiveRemoteExporter),
]);
```

## AI / LLM-aware sampling

LLM calls produce 5–50 spans per request and are 100× more expensive than a typical handler call. Tradeoffs:

- **Don't head-sample AI handlers below 50 %**: debugging "why did the model loop" requires the full chain.
- **Always tail-keep AI traces**: the `gen_ai.*` attributes flag them.
- **Cost-aware sampling**: keep all calls above a $ threshold:

```typescript
keep: (trace) => {
  const cost = trace.spans.reduce(
    (acc, s) =>
      acc +
      (typeof s.attributes['gen_ai.usage.cost.usd'] === 'number'
        ? (s.attributes['gen_ai.usage.cost.usd'] as number)
        : 0),
    0,
  );
  if (cost > 0.1) return true; // any trace > $0.10 → keep
  if (cost > 0.01) return Math.random() < 0.5; // > $0.01 → 50 %
  return Math.random() < 0.1; // < $0.01 → 10 %
};
```

## Customer-driven sampling (debug header)

Let support flip on full tracing per request:

```typescript
const tail = new TailSamplingProcessor({
  keep: (trace) => trace.localRootSpan.attributes['x-debug-trace'] === '1' || /* … */,
})
```

In your middleware:

```typescript
if (request.headers.get('x-debug-trace') === '1') {
  useLogger().set({ 'x-debug-trace': '1' });
}
```

Now any user can mark a request as "trace this fully" by sending the header. Invaluable for reproducing customer reports.

## Keeping the trace whole across services

`RandomSampler` rolls the dice in each process. The API can keep a trace its
worker drops, which leaves a waterfall with holes and no way to tell a dropped
span from a fast one. `DeterministicSampler` hashes a key that travels with the
request, so every service reaches the same verdict:

```typescript
import { DeterministicSampler } from 'autotel/sampling';
import { trace } from '@opentelemetry/api';

init({
  service: 'checkout',
  sampler: new DeterministicSampler({
    sampleRate: 0.1,
    key: () => trace.getActiveSpan()?.spanContext().traceId,
  }),
});
```

Give every service the same rate and the same key. Different rates break the
agreement again.

## Per-key target rates

A single rate cannot fit skewed traffic. At 1 % your busiest endpoint still
floods storage while the rare tenant whose failures you need disappears.
`KeyTargetRateSampler` counts traffic per key over a rolling window and gives
each key its own rate, so every key contributes about the same number of
events:

```typescript
import { KeyTargetRateSampler } from 'autotel/sampling';

new KeyTargetRateSampler({
  key: (context) => context.operationName,
  targetPerKey: 10, // ~10 events per key per window
  windowMs: 30_000,
  maxKeys: 1000, // keys past this share one overflow bucket
});
```

The first window keeps everything, because no traffic history exists yet. Rates
apply from the second window on.

## Recording the rate

Sampled counts lie unless you record what you discarded. A query over a 1 %
sample reports a hundredth of your traffic, and the number looks plausible
enough to act on.

autotel writes `autotel.sampling.rate` on the span as "1 in N", so
`COUNT * rate` estimates the population. It appears only when N exceeds 1, so
fully captured spans stay clean. A custom sampler joins in by implementing
`sampleRate()`:

```typescript
import type { Sampler } from 'autotel/sampling';

class TenantSampler implements Sampler {
  shouldSample(context) {
    /* … */
  }
  sampleRate() {
    return 20; // each kept event stands for 20
  }
}
```

Reach for `hashUnitInterval` from `autotel/sampling` when you need the same
spread for your own keys.

## Sizing the rate

Target volume:

```
spans/sec ≈ requests/sec × spans_per_request × head_rate × tail_keep_rate
```

Worked example for a 100 req/s API with 8 spans/req:

| Head rate | Tail keep                        | Result                                      |
| --------- | -------------------------------- | ------------------------------------------- |
| 100 %     | 100 %                            | 800 spans/sec: expensive                    |
| 10 %      | 100 % (errors + slow + AI ≈ 5 %) | ≈ 110 spans/sec: sweet spot                 |
| 1 %       | 100 %                            | ≈ 18 spans/sec: too sparse for p99 alerting |

For per-vendor pricing:

- **Honeycomb**: $0.000005 / event for paid plans. 110 spans/sec × 86 400 s = 9.5 M events/day = $48/day.
- **Datadog APM**: ~$1.27/M spans ingested (varies by region). Same volume → ~$12/day.
- **Grafana Cloud**: 100 GB free tier; 110 spans/sec ≈ 5 GB/day.

## Anti-patterns

| Anti-pattern                                 | Fix                                                           |
| -------------------------------------------- | ------------------------------------------------------------- |
| 100 % sampling at scale "to be safe"         | You're paying 10–100× without proportional value              |
| 1 % sampling with no tail keep               | You'll miss every interesting failure                         |
| Forgetting to tail-keep errors               | Sampled traces with errors → silent customer pain             |
| Same rate for `server` and `internal`        | Internal sub-spans are 5–20× more numerous; sample harder     |
| Ratio-based sampling on service entry point  | Use parent-based: children of a sampled trace stay together   |
| Head-sampling AI calls below 50 %            | Debugging tool loops requires the full chain                  |
| Audit spans subject to sampling              | Route them to a separate processor (see `build-audit-trails`) |
| Tail processor before exporter (loses spans) | Tail processor goes between head sampler and remote exporter  |
| Rate-by-route hand-coded in handlers         | Use head sampler + tail keep: declarative, one place          |
| Counting sampled spans as the population     | Multiply by `autotel.sampling.rate`                           |
| `RandomSampler` on a multi-service trace     | `DeterministicSampler` keyed on the trace id keeps it whole   |
| One flat rate over skewed traffic            | `KeyTargetRateSampler` gives rare keys their own rate         |