Skip to content

Metrics

Registry and contracts

A metric scores one task output; an aggregator combines scores across a run. Metrics declare name and compatible_output_types; compute_async(...) returns a number, None, or named values.

Pass metric instances to evaluate_async(...). Loader default_metrics lists their registry names; the Python API does not select them automatically or filter incompatible output types. Check compatible_output_types against the agent's output family.

metric_registry module-attribute 

metric_registry = ClassRegistry[MetricProtocol]('metric')

MetricProtocol

Bases: Protocol

Task-level evaluation metric registered by name.

name class-attribute 

name: str

compatible_output_types class-attribute 

compatible_output_types: list[str]

compute_async async 

compute_async(
    task: NL2QTaskOutput,
    db_connector: DataConnector | None = None,
) -> NumericOrNull | dict[str, NumericOrNull]

MetricAggregatorProtocol

Bases: Protocol

Aggregation policy over the task outputs in one run.

aggregate 

aggregate(result: NL2QRunResult) -> dict[str, Any]

NumericOrNull module-attribute 

NumericOrNull: TypeAlias = Union[float, int, None]

Execution comparison

See Evaluation and analysis for benchmark metrics.

SimpleEx 

SimpleEx(
    abs_tol: float = 0.01, ignore_repetitions: bool = True
)

A simple execution accuracy metric. An improved version of Spider2's EX metric.

Features that are different from Spider2's EX implementation: - For boolean values, we consider True == 1 == "1.0" == 1.0 - For convertible string values, we consider "3.0" == 3.0 == 3 - We consider "nan" == None == math.nan == np.nan - We fixed the [-2, 0] != [-2, -0.000001] bug - Row order does not matter by default - Repetitions are ignored by default (like bird_sql_ex)

Features that are the same as Spider2's EX implementation: - Column order does not matter - Additional columns are allowed

name class-attribute 

name: str = 'simple_ex'

compatible_output_types class-attribute 

compatible_output_types: list[str] = [
    "simple",
    "ambig-simple",
    "ambig-flat",
    "ambig-structured",
]

abs_tol instance-attribute 

abs_tol = abs_tol

ignore_repetitions instance-attribute 

ignore_repetitions = ignore_repetitions

compute_async async 

compute_async(
    task: NL2QTaskOutput,
    db_connector: DataConnector | None = None,
) -> float

BirdSQLEx

name class-attribute 

name: str = 'bird_sql_ex'

compatible_output_types class-attribute 

compatible_output_types: list[str] = [
    "simple",
    "ambig-simple",
    "ambig-flat",
    "ambig-structured",
]

compute_async async 

compute_async(
    task: NL2QTaskOutput,
    db_connector: DataConnector | None = None,
) -> float

BirdSQLExSoft

name class-attribute 

name: str = 'bird_sql_ex_soft'

compatible_output_types class-attribute 

compatible_output_types: list[str] = [
    "simple",
    "ambig-simple",
    "ambig-flat",
    "ambig-structured",
]

compute_async async 

compute_async(
    task: NL2QTaskOutput,
    db_connector: DataConnector | None = None,
) -> float

Spider2Ex

name class-attribute 

name: str = 'spider2_ex'

compatible_output_types class-attribute 

compatible_output_types: list[str] = [
    "simple",
    "ambig-simple",
    "ambig-flat",
    "ambig-structured",
]

compute_async async 

compute_async(
    task: NL2QTaskOutput,
    db_connector: DataConnector | None = None,
) -> float

Spider2DuckdbMatch

name class-attribute 

name: str = 'spider2_duckdb_match'

compatible_output_types class-attribute 

compatible_output_types: list[str] = ['dbt']

compute_async async 

compute_async(
    task: NL2QTaskOutput,
    db_connector: DataConnector | None = None,
) -> NumericOrNull

CypherBenchEx

CypherBench execution accuracy (EX).

Faithful port of the official metric: column-permutation search, multiset row comparison, and ORDER BY-aware ordering (inferred from the gold Cypher query).

name class-attribute 

name: str = 'cypherbench_ex'

compatible_output_types class-attribute 

compatible_output_types: list[str] = ['simple']

compute_async async 

compute_async(
    task: NL2QTaskOutput,
    db_connector: DataConnector | None = None,
) -> float

Prediction and execution diagnostics

PredSuccess checks whether a query was produced; Executable checks whether it executed successfully. Gold diagnostics check reference-query execution and nonempty results. Raw-prediction metrics evaluate queries from before postprocessing.

PredSuccess

name class-attribute 

name: str = 'pred_success'

compatible_output_types class-attribute 

compatible_output_types: list[str] = [
    "simple",
    "ambig-simple",
    "ambig-flat",
    "ambig-structured",
]

compute_async async 

compute_async(
    task: NL2QTaskOutput,
    db_connector: DataConnector | None = None,
) -> float

Executable

name class-attribute 

name: str = 'executable'

compatible_output_types class-attribute 

compatible_output_types: list[str] = [
    "simple",
    "ambig-simple",
    "ambig-flat",
    "ambig-structured",
]

compute_async async 

compute_async(
    task: NL2QTaskOutput,
    db_connector: DataConnector | None = None,
) -> float

GoldExecutable

name class-attribute 

name: str = 'gold_executable'

compatible_output_types class-attribute 

compatible_output_types: list[str] = [
    "simple",
    "ambig-simple",
    "ambig-flat",
    "ambig-structured",
]

compute_async async 

compute_async(
    task: NL2QTaskOutput,
    db_connector: DataConnector | None = None,
) -> float

GoldResultNotEmpty

name class-attribute 

name: str = 'gold_result_not_empty'

compatible_output_types class-attribute 

compatible_output_types: list[str] = [
    "simple",
    "ambig-simple",
    "ambig-flat",
    "ambig-structured",
]

compute_async async 

compute_async(
    task: NL2QTaskOutput,
    db_connector: DataConnector | None = None,
) -> float

RawPredSimpleEx 

RawPredSimpleEx(
    abs_tol: float = 0.01, ignore_repetitions: bool = True
)

name class-attribute 

name: str = 'raw_pred_simple_ex'

compatible_output_types class-attribute 

compatible_output_types: list[str] = ['simple']

abs_tol instance-attribute 

abs_tol = abs_tol

ignore_repetitions instance-attribute 

ignore_repetitions = ignore_repetitions

compute_async async 

compute_async(
    task: NL2QTaskOutput,
    db_connector: DataConnector | None = None,
) -> NumericOrNull

RawPredBirdSQLEx

name class-attribute 

name: str = 'raw_pred_bird_sql_ex'

compatible_output_types class-attribute 

compatible_output_types: list[str] = ['simple']

compute_async async 

compute_async(
    task: NL2QTaskOutput,
    db_connector: DataConnector | None = None,
) -> NumericOrNull

SchemaLinkingStats

name class-attribute 

name: str = 'schema_linking_stats'

compatible_output_types class-attribute 

compatible_output_types: list[str] = [
    "simple",
    "ambig-simple",
    "ambig-flat",
    "ambig-structured",
]

compute_async async 

compute_async(
    task: NL2QTaskOutput,
    db_connector: DataConnector | None = None,
) -> dict[str, NumericOrNull]

Ambiguity metrics

SimpleEx compares the final prediction with the intended reference; FoundOne accepts any valid reference interpretation. Ambiguity-point metrics measure which phrases and interpretations the agent identified.

AmbigPointStats 

AmbigPointStats(llm: str = 'openai:gpt-4.1-2025-04-14')

name class-attribute 

name: str = 'ambig_point_stats'

compatible_output_types class-attribute 

compatible_output_types: list[str] = [
    "ambig-simple",
    "ambig-flat",
    "ambig-structured",
]

llm instance-attribute 

llm = llm

compute_async async 

compute_async(
    task: NL2QTaskOutput,
    db_connector: DataConnector | None = None,
) -> dict[str, NumericOrNull]

GoldAmbigPointStats

name class-attribute 

name: str = 'gold_ambig_point_stats'

compatible_output_types class-attribute 

compatible_output_types: list[str] = [
    "ambig-simple",
    "ambig-flat",
    "ambig-structured",
]

compute_async async 

compute_async(
    task: NL2QTaskOutput,
    db_connector: DataConnector | None = None,
) -> dict[str, NumericOrNull]

FoundOne 

FoundOne(
    abs_tol: float = 0.01, ignore_repetitions: bool = True
)

Whether the predicted query match the execution result of one of the gold queries. Only applicable to ambig tasks.

name class-attribute 

name: str = 'found_one'

compatible_output_types class-attribute 

compatible_output_types: list[str] = [
    "ambig-simple",
    "ambig-flat",
    "ambig-structured",
]

abs_tol instance-attribute 

abs_tol = abs_tol

ignore_repetitions instance-attribute 

ignore_repetitions = ignore_repetitions

simple_ex instance-attribute 

simple_ex = SimpleEx(self.abs_tol, self.ignore_repetitions)

compute_async async 

compute_async(
    task: NL2QTaskOutput,
    db_connector: DataConnector | None = None,
) -> NumericOrNull

PSJS

Provenance Subgraph Jaccard Similarity (PSJS) from CypherBench.

For each query, extracts the MATCH pattern, runs it against Neo4j to collect node element IDs, then computes Jaccard similarity between the gold and predicted provenance node sets.

name class-attribute 

name: str = 'psjs'

compatible_output_types class-attribute 

compatible_output_types: list[str] = ['simple']

compute_async async 

compute_async(
    task: NL2QTaskOutput,
    db_connector: DataConnector | None = None,
) -> NumericOrNull

Aggregators

Pass aggregators to evaluate_async(..., metric_aggregators=[...]). The default is SimpleAverageAggregator(); pass [] to skip aggregation.

SimpleAverageAggregator includes zeros and excludes None from the average. OfficialSplitScoreAggregator divides by the configured full split size, treating missing tasks as zero. Other aggregators group scores by database, difficulty, or ambiguity type. Report the task count and missing values with scores.

For paired comparisons, match QIDs and compare task scores. The optional comparison script also summarizes usage and latency.

SimpleAverageAggregator 

SimpleAverageAggregator(
    ops: Sequence[AggregationOp] = ("avg",),
)

ops instance-attribute 

ops = tuple(ops)

aggregate 

aggregate(result: NL2QRunResult) -> dict[str, Any]

OfficialSplitScoreAggregator

Divide by the official split size, treating missing tasks as zero.

Unlike SimpleAverageAggregator which divides by the number of evaluated tasks, this divides by the known split size so that unevaluated or missing predictions are implicitly counted as failures.

DATASET_CONFIGS class-attribute instance-attribute 

DATASET_CONFIGS: dict[tuple[str, str], tuple[int, str]] = {
    ("bird-sql", "dev"): (1534, "bird_sql_ex"),
    ("bird-sql", "dev_20251106"): (1534, "bird_sql_ex"),
    ("bird-sql", "train"): (9428, "bird_sql_ex"),
    ("spider2-snow", "test"): (547, "spider2_ex"),
    ("spider2-lite", "test"): (547, "spider2_ex"),
    ("spider2-dbt", "test"): (68, "spider2_duckdb_match"),
    ("beaver", "test"): (209, "simple_ex"),
    ("arcs", "test"): (331, "simple_ex"),
    ("arcs", "test_unsampled"): (101, "simple_ex"),
    ("ambrosia-s", "test"): (1149, "simple_ex"),
    ("ambrosia-s", "few_shot_examples"): (128, "simple_ex"),
}

aggregate 

aggregate(result: NL2QRunResult) -> dict[str, Any]

SimpleInferenceMetricsAggregator 

SimpleInferenceMetricsAggregator(
    ops: Sequence[AggregationOp] = ("avg", "sum", "max"),
)

ops instance-attribute 

ops = tuple(ops)

aggregate 

aggregate(result: NL2QRunResult) -> dict[str, Any]

ByDBAggregator 

ByDBAggregator(
    ops: Sequence[AggregationOp] = ("avg",),
    metric_keys: Sequence[str] = (
        "simple_ex",
        "perfect_linked_schema_r",
    ),
    max_dbs: int = 200,
)

ops instance-attribute 

ops = tuple(ops)

metric_keys instance-attribute 

metric_keys = tuple(metric_keys)

max_dbs instance-attribute 

max_dbs = max_dbs

aggregate 

aggregate(result: NL2QRunResult) -> dict[str, Any]

ByAmbigPointNumAggregator 

ByAmbigPointNumAggregator(
    ops: Sequence[AggregationOp] = ("avg",),
    metric_keys: Sequence[str] = ("simple_ex",),
)

ops instance-attribute 

ops = tuple(ops)

metric_keys instance-attribute 

metric_keys = tuple(metric_keys)

aggregate 

aggregate(result: NL2QRunResult) -> dict[str, Any]

ByAmbrosiaTaxonomyTypeAggregator 

ByAmbrosiaTaxonomyTypeAggregator(
    ops: Sequence[AggregationOp] = ("avg",),
    metric_keys: Sequence[str] = ("simple_ex",),
)

ops instance-attribute 

ops = tuple(ops)

metric_keys instance-attribute 

metric_keys = tuple(metric_keys)

aggregate 

aggregate(result: NL2QRunResult) -> dict[str, Any]

ByBirdSQLDifficultyAggregator 

ByBirdSQLDifficultyAggregator(
    ops: Sequence[AggregationOp] = ("avg",),
    metric_keys: Sequence[str] = (
        "bird_sql_ex",
        "simple_ex",
        "perfect_linked_schema_r",
    ),
)

ops instance-attribute 

ops = tuple(ops)

metric_keys instance-attribute 

metric_keys = tuple(metric_keys)

aggregate 

aggregate(result: NL2QRunResult) -> dict[str, Any]

Aggregation values and operations

aggregate_metrics combines scalar or consistently nested values without requiring an NL2QRunResult.

aggregate_metrics 

aggregate_metrics(
    metrics: Sequence[MetricValue],
    ops: Sequence[AggregationOp] = (
        "avg",
        "sum",
        "max",
        "min",
    ),
    decimals: int = 4,
) -> dict[str, Any]

Aggregate scalar or consistently nested metric values.

AggregationOp module-attribute 

AggregationOp: TypeAlias = Literal[
    "avg", "sum", "max", "min"
]

MetricValue module-attribute 

MetricValue: TypeAlias = NumericOrNull | dict[str, Any]

Custom metrics and aggregators

See Add a custom metric for a complete implementation. Metrics can also return a dictionary whose keys become task metric names. Register with metric_registry.register(YourMetric) for name-based lookup.

A custom aggregator implements aggregate(result: NL2QRunResult) and returns named run-level values. Pass it in metric_aggregators, including SimpleAverageAggregator() if you also want averages. See aggregation and the metric contracts.