UNPKG

axiom

Version:

Axiom AI SDK provides - an API to wrap your AI calls with observability instrumentation. - offline evals - online evals

github.com/axiomhq/ai

axiomhq/ai

98 lines (50 loc) • 1.82 kB
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
[**axiom v0.51.1**](../../README.md) *** [axiom](../../README.md) / [evals](../README.md) / EvalParams # Type Alias: EvalParams\<TInput, TExpected, TOutput\> > **EvalParams**\<`TInput`, `TExpected`, `TOutput`\> = `object` Configuration parameters for running an evaluation. Used with [Eval](../functions/Eval.md) to define how an evaluation should be executed. Results are captured in EvalReport format. ## Type Parameters ### TInput `TInput` ### TExpected `TExpected` ### TOutput `TOutput` ## Properties ### capability > **capability**: `string` *** ### configFlags? > `optional` **configFlags**: `string`[] Optional reduction of flag namespace *** ### data > **data**: readonly `CollectionRecord`\<`TInput`, `TExpected`\>[] \| `Promise`\<readonly `CollectionRecord`\<`TInput`, `TExpected`\>[]\> \| () => readonly `CollectionRecord`\<`TInput`, `TExpected`\>[] \| `Promise`\<readonly `CollectionRecord`\<`TInput`, `TExpected`\>[]\> Dataset with input/expected pairs for evaluation, or a function that returns one *** ### metadata? > `optional` **metadata**: `Record`\<`string`, `unknown`\> Optional metadata for the evaluation *** ### scorers > **scorers**: `ReadonlyArray`\<[`ScorerLike`](../../scorers/scorers/type-aliases/ScorerLike.md)\<`TInput`, `TExpected`, `TOutput`\>\> Array of scoring functions to evaluate the task output *** ### step? > `optional` **step**: `string` *** ### task > **task**: [`EvalTask`](EvalTask.md)\<`TInput`, `TExpected`, `TOutput`\> The task function to evaluate *** ### timeout? > `optional` **timeout**: `number` Optional timeout in milliseconds for task execution *** ### trials? > `optional` **trials**: `number` Number of times to run each case. Defaults to 1. Each trial runs the task independently, and scores are aggregated per scorer.