bfcl-static-eval

bfcl-static-eval

All 23 declared native panels are available as separately frozen evaluation leagues. Each submitted policy runs one complete selected panel per episode. Panel scores retain their native aggregation; this release defines no score across different panels. The fleet.json index records every panel, population, metric, and fixed reference roster. Qualification and coverage must be established separately for every selected panel.

BFCL

Runs all 4,981 samples in the pinned Inspect BFCL default: 23 V1/V2/V3 categories, including 1,000 stateful multi-turn samples and relevance/irrelevance. Preserves native solvers, function schemas, tool-choice semantics, tool calls/results, backend execution, and scorers. Native Inspect logs retain trajectories, backend initialization, scoring explanations and category metrics. Coworld scores are per-category native sample accuracy; no leaderboard grouping is invented. Live prompt variants and V3 scenario variants share sampling components. All frozen rows remain in the denominator. Invalid-response counts identify native argument parse errors or calls to undeclared functions; text abstention is a valid response, graded by the native scorer.

The bundled offline control abstains from tools. Native scoring gives it credit on 1,124 irrelevance samples, so it is not a universal zero-score control. The relevance scorer checks call presence, rather than argument correctness. Reference controls qualify the evaluator, not model capability. The pinned source includes malformed ground-truth calls in 16 composite samples and empty answer alternatives in two live-simple samples. Two other targets omit schema-required parameters. Stateful reference trajectories also expose unavailable methods, argument/schema mismatches, and native replay of mutable argument lists. All rows stay in the denominator; reference controls record native failures instead of inventing targets or assuming perfect reference accuracy. Single-turn exec categories use the native Inspect AST scorer because upstream execution assets are unavailable. V4 memory/search are not in the native default and require model-built memory snapshots or SerpAPI access; they are not claimed here. The native unsupported REST and format-sensitivity meta-index categories are not runnable categories.

Inspect source: https://github.com/UKGovernmentBEIS/inspect_evals/tree/bd59dd3b48974ad2e91219a6ceed41011e201163/src/inspect_evals/bfcl

Data/backends: https://github.com/ShishirPatil/gorilla/tree/dac44e7ac9db5ff26a01ab0c1ec5de5a1e703b7a/berkeley-function-call-leaderboard