Skip to content

Tasks and runs

Queries

GoldQuery stores a reference query and accepted result variants. PredQuery stores an agent prediction. Both can carry an ExecResult.

GoldQuery

Bases: BaseModel

A reference query and accepted result variants for benchmark evaluation.

id class-attribute instance-attribute 

id: str = 'GQRY'

query instance-attribute 

query: str | None

In Spider2, some gold queries are not available, so we allow it to be None

parameter_names class-attribute instance-attribute 

parameter_names: list[str] = Field(default_factory=list)

parameter_values class-attribute instance-attribute 

parameter_values: dict[str, Any] = Field(
    default_factory=dict
)

If parameter_names is not empty and parameter_values is empty, the query is parameterized.

exec_result class-attribute instance-attribute 

exec_result: ExecResult | None = None

required_columns class-attribute instance-attribute 

required_columns: list[int] | None = None

Columns that must be present in the result, None means all columns must be present

required_sorted class-attribute instance-attribute 

required_sorted: bool = False

True if row order matters

alternative_results class-attribute instance-attribute 

alternative_results: list[ExecResult] = Field(
    default_factory=list
)

Alternative correct results, used in spider2-snow

extra_info class-attribute instance-attribute 

extra_info: dict[str, Any] = Field(default_factory=dict)

to_directory 

to_directory(directory: str) -> None

to_markdown 

to_markdown(heading_level: int = 2) -> str

PredQuery

Bases: BaseModel

A query predicted by a research agent, optionally with its execution result.

id class-attribute instance-attribute 

id: str = 'PQRY'

query instance-attribute 

query: str

parameter_names class-attribute instance-attribute 

parameter_names: list[str] = Field(default_factory=list)

parameter_values class-attribute instance-attribute 

parameter_values: dict[str, Any] = Field(
    default_factory=dict
)

exec_result class-attribute instance-attribute 

exec_result: ExecResult | None = None

from_execution classmethod 

from_execution(execution: QueryExecution) -> PredQuery

to_directory 

to_directory(directory: str) -> None

to_markdown 

to_markdown(heading_level: int = 2) -> str

Simple tasks

SimpleNL2QTask

Bases: BaseModel

An unambiguous natural-language-to-query benchmark task.

task_type class-attribute instance-attribute 

task_type: Literal['simple'] = 'simple'

qid instance-attribute 

qid: str

db instance-attribute 

db: str

question instance-attribute 

question: str

question_instructions class-attribute instance-attribute 

question_instructions: str | None = None

Instructions that apply to this question only.

dataset_instructions class-attribute instance-attribute 

dataset_instructions: str | None = None

Instructions (e.g. for formatting) that apply to all questions in the dataset.

document class-attribute instance-attribute 

document: str | None = None

gold_query instance-attribute 

gold_query: GoldQuery

extra_info class-attribute instance-attribute 

extra_info: dict[str, Any] = Field(default_factory=dict)

to_directory 

to_directory(directory: str) -> None

to_markdown 

to_markdown(heading_level: int = 1) -> str

SimpleNL2QTaskOutput

Bases: SimpleNL2QTask

Prediction and run metadata for a simple task.

output_type class-attribute instance-attribute 

output_type: Literal['simple'] = 'simple'

pred_query instance-attribute 

pred_query: PredQuery | None

trajectory class-attribute instance-attribute 

trajectory: Trajectory | list[Trajectory] | None = None

usage class-attribute instance-attribute 

usage: Usage | None = None

inference_metrics class-attribute instance-attribute 

inference_metrics: dict[str, Any] = Field(
    default_factory=dict
)

Metrics produced during agent prediction, e.g. latency, API costs, etc.

eval_metrics class-attribute instance-attribute 

eval_metrics: dict[str, Any] = Field(default_factory=dict)

Metrics produced during evaluation, e.g. accuracy, etc.

extra_pred_info class-attribute instance-attribute 

extra_pred_info: ExtraPredInfo = Field(
    default_factory=ExtraPredInfo
)

task_type class-attribute instance-attribute 

task_type: Literal['simple'] = 'simple'

qid instance-attribute 

qid: str

db instance-attribute 

db: str

question instance-attribute 

question: str

question_instructions class-attribute instance-attribute 

question_instructions: str | None = None

Instructions that apply to this question only.

dataset_instructions class-attribute instance-attribute 

dataset_instructions: str | None = None

Instructions (e.g. for formatting) that apply to all questions in the dataset.

document class-attribute instance-attribute 

document: str | None = None

gold_query instance-attribute 

gold_query: GoldQuery

extra_info class-attribute instance-attribute 

extra_info: dict[str, Any] = Field(default_factory=dict)

to_directory 

to_directory(directory: str) -> None

to_markdown 

to_markdown(heading_level: int = 1) -> str

to_summary 

to_summary(
    eval_metrics: Sequence[str] = (),
) -> CSVSummaryRow

ExtraPredInfo

Bases: BaseModel

Optional intermediate artifacts produced while predicting a query.

linked_schema class-attribute instance-attribute 

linked_schema: list[ColumnRef] | None = None

raw_pred_query class-attribute instance-attribute 

raw_pred_query: PredQuery | None = None

If your method includes a postprocessing step, this field can store the raw predicted query before postprocessing to analyze its impact. The raw_pred_*_ex metrics evaluate these raw predictions.

other class-attribute instance-attribute 

other: dict[str, Any] = Field(default_factory=dict)

Ambiguous tasks

Ambiguous tasks specify interpretation choices or open-ended parameters. The three output families represent an intended query, a flat collection of interpretations, or explicitly structured ambiguity points.

AmbigNL2QTask

Bases: BaseModel

A benchmark task with explicit ambiguity points and resolution queries.

qid instance-attribute 

qid: str

task_type class-attribute instance-attribute 

task_type: Literal['ambig'] = 'ambig'

has_intended_resolution instance-attribute 

has_intended_resolution: bool

db instance-attribute 

db: str

question instance-attribute 

question: str

dataset_instructions class-attribute instance-attribute 

dataset_instructions: str | None = None

Instructions (e.g. for formatting) that apply to all questions in the dataset

gold_ambiguity_points instance-attribute 

gold_ambiguity_points: Annotated[
    list[GoldAmbiguityPoint], AfterValidator(is_id_unique)
]

gold_queries instance-attribute 

gold_queries: Annotated[
    list[GoldQuery], AfterValidator(is_id_unique)
]

gold_intended_query_id instance-attribute 

gold_intended_query_id: str | None

Ground-truth query intended by the user

extra_info class-attribute instance-attribute 

extra_info: dict[str, Any] = Field(default_factory=dict)

gold_intended_query property 

gold_intended_query: GoldQuery | None

gold_finite_ambiguity_points property 

gold_finite_ambiguity_points: list[GoldAmbiguityPointFinite]

gold_infinite_ambiguity_points property 

gold_infinite_ambiguity_points: list[
    GoldAmbiguityPointInfinite
]

gold_num_interpretation_comb property 

gold_num_interpretation_comb: int

to_directory 

to_directory(directory: str) -> None

to_markdown 

to_markdown(heading_level: int = 1) -> str

SimpleAmbigNL2QTaskOutput

Bases: AmbigNL2QTask

An ambiguity-aware prediction containing only the resolved final query.

output_type class-attribute instance-attribute 

output_type: Literal['ambig-simple'] = 'ambig-simple'

pred_intended_query instance-attribute 

pred_intended_query: PredQuery | None

trajectory class-attribute instance-attribute 

trajectory: Trajectory | list[Trajectory] | None = None

usage class-attribute instance-attribute 

usage: Usage | None = None

user_simulator_usage class-attribute instance-attribute 

user_simulator_usage: Usage | None = None

inference_metrics class-attribute instance-attribute 

inference_metrics: dict[str, Any] = Field(
    default_factory=dict
)

Metrics produced during agent prediction, e.g. latency, API costs, etc.

eval_metrics class-attribute instance-attribute 

eval_metrics: dict[str, Any] = Field(default_factory=dict)

Metrics produced during evaluation, e.g. accuracy, etc.

extra_pred_info class-attribute instance-attribute 

extra_pred_info: ExtraPredInfo = Field(
    default_factory=ExtraPredInfo
)

qid instance-attribute 

qid: str

task_type class-attribute instance-attribute 

task_type: Literal['ambig'] = 'ambig'

has_intended_resolution instance-attribute 

has_intended_resolution: bool

db instance-attribute 

db: str

question instance-attribute 

question: str

dataset_instructions class-attribute instance-attribute 

dataset_instructions: str | None = None

Instructions (e.g. for formatting) that apply to all questions in the dataset

gold_ambiguity_points instance-attribute 

gold_ambiguity_points: Annotated[
    list[GoldAmbiguityPoint], AfterValidator(is_id_unique)
]

gold_queries instance-attribute 

gold_queries: Annotated[
    list[GoldQuery], AfterValidator(is_id_unique)
]

gold_intended_query_id instance-attribute 

gold_intended_query_id: str | None

Ground-truth query intended by the user

extra_info class-attribute instance-attribute 

extra_info: dict[str, Any] = Field(default_factory=dict)

gold_intended_query property 

gold_intended_query: GoldQuery | None

gold_finite_ambiguity_points property 

gold_finite_ambiguity_points: list[GoldAmbiguityPointFinite]

gold_infinite_ambiguity_points property 

gold_infinite_ambiguity_points: list[
    GoldAmbiguityPointInfinite
]

gold_num_interpretation_comb property 

gold_num_interpretation_comb: int

to_directory 

to_directory(directory: str) -> None

to_markdown 

to_markdown(heading_level: int = 1) -> str

to_summary 

to_summary(
    eval_metrics: Sequence[str] = (),
) -> CSVSummaryRow

FlatAmbigNL2QTaskOutput

Bases: AmbigNL2QTask

Flat interpretations, their queries, and the resolved final query.

output_type class-attribute instance-attribute 

output_type: Literal['ambig-flat'] = 'ambig-flat'

interpretations instance-attribute 

interpretations: list[str]

parameters instance-attribute 

parameters: list[PredAmbiguityPointInfinite]

pred_queries instance-attribute 

pred_queries: Annotated[
    list[PredQuery], AfterValidator(is_id_unique)
]

pred_intended_query_id instance-attribute 

pred_intended_query_id: str | None

trajectory class-attribute instance-attribute 

trajectory: Trajectory | list[Trajectory] | None = None

usage class-attribute instance-attribute 

usage: Usage | None = None

user_simulator_usage class-attribute instance-attribute 

user_simulator_usage: Usage | None = None

inference_metrics class-attribute instance-attribute 

inference_metrics: dict[str, Any] = Field(
    default_factory=dict
)

Metrics produced during agent prediction, e.g. latency, API costs, etc.

eval_metrics class-attribute instance-attribute 

eval_metrics: dict[str, Any] = Field(default_factory=dict)

Metrics produced during evaluation, e.g. accuracy, etc.

extra_pred_info class-attribute instance-attribute 

extra_pred_info: ExtraPredInfo = Field(
    default_factory=ExtraPredInfo
)

pred_intended_query property 

pred_intended_query: PredQuery | None

qid instance-attribute 

qid: str

task_type class-attribute instance-attribute 

task_type: Literal['ambig'] = 'ambig'

has_intended_resolution instance-attribute 

has_intended_resolution: bool

db instance-attribute 

db: str

question instance-attribute 

question: str

dataset_instructions class-attribute instance-attribute 

dataset_instructions: str | None = None

Instructions (e.g. for formatting) that apply to all questions in the dataset

gold_ambiguity_points instance-attribute 

gold_ambiguity_points: Annotated[
    list[GoldAmbiguityPoint], AfterValidator(is_id_unique)
]

gold_queries instance-attribute 

gold_queries: Annotated[
    list[GoldQuery], AfterValidator(is_id_unique)
]

gold_intended_query_id instance-attribute 

gold_intended_query_id: str | None

Ground-truth query intended by the user

extra_info class-attribute instance-attribute 

extra_info: dict[str, Any] = Field(default_factory=dict)

gold_intended_query property 

gold_intended_query: GoldQuery | None

gold_finite_ambiguity_points property 

gold_finite_ambiguity_points: list[GoldAmbiguityPointFinite]

gold_infinite_ambiguity_points property 

gold_infinite_ambiguity_points: list[
    GoldAmbiguityPointInfinite
]

gold_num_interpretation_comb property 

gold_num_interpretation_comb: int

to_directory 

to_directory(directory: str) -> None

to_markdown 

to_markdown(heading_level: int = 1) -> str

to_summary 

to_summary(
    eval_metrics: Sequence[str] = (),
) -> CSVSummaryRow

StructuredAmbigNL2QTaskOutput

Bases: AmbigNL2QTask

Predicted ambiguity structure, interpretation queries, and final resolution.

output_type class-attribute instance-attribute 

output_type: Literal["ambig-structured"] = (
    "ambig-structured"
)

pred_ambiguity_points instance-attribute 

pred_ambiguity_points: Annotated[
    list[PredAmbiguityPoint], AfterValidator(is_id_unique)
]

pred_queries instance-attribute 

pred_queries: Annotated[
    list[PredQuery], AfterValidator(is_id_unique)
]

pred_intended_query_id instance-attribute 

pred_intended_query_id: str | None

trajectory class-attribute instance-attribute 

trajectory: Trajectory | list[Trajectory] | None = None

usage class-attribute instance-attribute 

usage: Usage | None = None

user_simulator_usage class-attribute instance-attribute 

user_simulator_usage: Usage | None = None

inference_metrics class-attribute instance-attribute 

inference_metrics: dict[str, Any] = Field(
    default_factory=dict
)

Metrics produced during agent prediction, e.g. latency, API costs, etc.

eval_metrics class-attribute instance-attribute 

eval_metrics: dict[str, Any] = Field(default_factory=dict)

Metrics produced during evaluation, e.g. accuracy, etc.

extra_pred_info class-attribute instance-attribute 

extra_pred_info: ExtraPredInfo = Field(
    default_factory=ExtraPredInfo
)

pred_intended_query property 

pred_intended_query: PredQuery | None

pred_finite_ambiguity_points property 

pred_finite_ambiguity_points: list[PredAmbiguityPointFinite]

pred_infinite_ambiguity_points property 

pred_infinite_ambiguity_points: list[
    PredAmbiguityPointInfinite
]

pred_num_interpretation_comb property 

pred_num_interpretation_comb: int

qid instance-attribute 

qid: str

task_type class-attribute instance-attribute 

task_type: Literal['ambig'] = 'ambig'

has_intended_resolution instance-attribute 

has_intended_resolution: bool

db instance-attribute 

db: str

question instance-attribute 

question: str

dataset_instructions class-attribute instance-attribute 

dataset_instructions: str | None = None

Instructions (e.g. for formatting) that apply to all questions in the dataset

gold_ambiguity_points instance-attribute 

gold_ambiguity_points: Annotated[
    list[GoldAmbiguityPoint], AfterValidator(is_id_unique)
]

gold_queries instance-attribute 

gold_queries: Annotated[
    list[GoldQuery], AfterValidator(is_id_unique)
]

gold_intended_query_id instance-attribute 

gold_intended_query_id: str | None

Ground-truth query intended by the user

extra_info class-attribute instance-attribute 

extra_info: dict[str, Any] = Field(default_factory=dict)

gold_intended_query property 

gold_intended_query: GoldQuery | None

gold_finite_ambiguity_points property 

gold_finite_ambiguity_points: list[GoldAmbiguityPointFinite]

gold_infinite_ambiguity_points property 

gold_infinite_ambiguity_points: list[
    GoldAmbiguityPointInfinite
]

gold_num_interpretation_comb property 

gold_num_interpretation_comb: int

to_directory 

to_directory(directory: str) -> None

to_markdown 

to_markdown(heading_level: int = 1) -> str

to_summary 

to_summary(
    eval_metrics: Sequence[str] = (),
) -> CSVSummaryRow

ARCSAmbiguityType

Bases: str, Enum

ARCS ambiguity taxonomy labels.

semantic_column class-attribute instance-attribute 

semantic_column = 'semantic_column'

semantic_table class-attribute instance-attribute 

semantic_table = 'semantic_table'

semantic_value class-attribute instance-attribute 

semantic_value = 'semantic_value'

semantic_computation class-attribute instance-attribute 

semantic_computation = 'semantic_computation'

syntactic_column class-attribute instance-attribute 

syntactic_column = 'syntactic_column'

syntactic_table class-attribute instance-attribute 

syntactic_table = 'syntactic_table'

syntactic_value class-attribute instance-attribute 

syntactic_value = 'syntactic_value'

syntactic_computation class-attribute instance-attribute 

syntactic_computation = 'syntactic_computation'

GoldAmbiguityPointFinite

Bases: BaseModel

A finite ambiguity with an enumerated set of interpretations.

id instance-attribute 

id: Annotated[str, StringConstraints(pattern='^[A-Z]$')]

A, B, C, etc.

phrase instance-attribute 

phrase: str

type class-attribute instance-attribute 

type: Literal['finite'] = 'finite'

ambiguity_type instance-attribute 

ambiguity_type: ARCSAmbiguityType

interpretations instance-attribute 

interpretations: list[str]

intended_interpretation_idx instance-attribute 

intended_interpretation_idx: int | None

GoldAmbiguityPointInfinite

Bases: BaseModel

An open-ended ambiguity represented by a typed query parameter.

id instance-attribute 

id: Annotated[str, StringConstraints(pattern='^[A-Z]+$')]

A, B, C, etc.

phrase instance-attribute 

phrase: str

type class-attribute instance-attribute 

type: Literal['infinite'] = 'infinite'

ambiguity_type instance-attribute 

ambiguity_type: ARCSAmbiguityType

parameter_name instance-attribute 

parameter_name: str

parameter_dtype instance-attribute 

parameter_dtype: Literal['int', 'float', 'str']

parameter_sample_operators instance-attribute 

parameter_sample_operators: list[
    Literal["<", ">", "<=", ">=", "=", "<>"]
]

parameter_sample_values instance-attribute 

parameter_sample_values: list[Any]

intended_parameter_operator instance-attribute 

intended_parameter_operator: Literal[
    "<", ">", "<=", ">=", "=", "<>"
]

intended_parameter_value instance-attribute 

intended_parameter_value: Any | None

GoldAmbiguityPoint module-attribute 

GoldAmbiguityPoint = Annotated[
    Union[
        GoldAmbiguityPointFinite, GoldAmbiguityPointInfinite
    ],
    Field(discriminator="type"),
]

PredAmbiguityPointFinite

Bases: BaseModel

A predicted finite ambiguity and its candidate interpretations.

id instance-attribute 

id: Annotated[str, StringConstraints(pattern='^[A-Z]+$')]

A, B, C, etc.

phrase instance-attribute 

phrase: str

type class-attribute instance-attribute 

type: Literal['finite'] = 'finite'

interpretations instance-attribute 

interpretations: list[str]

intended_interpretation_idx class-attribute instance-attribute 

intended_interpretation_idx: int | None = None

rejected_by_user class-attribute instance-attribute 

rejected_by_user: bool = False

PredAmbiguityPointInfinite

Bases: BaseModel

A predicted open-ended ambiguity represented by a typed parameter.

id instance-attribute 

id: Annotated[str, StringConstraints(pattern='^[A-Z]+$')]

A, B, C, etc.

phrase instance-attribute 

phrase: str

type class-attribute instance-attribute 

type: Literal['infinite'] = 'infinite'

parameter_name instance-attribute 

parameter_name: str

parameter_dtype instance-attribute 

parameter_dtype: Literal['int', 'float', 'str']

parameter_description class-attribute instance-attribute 

parameter_description: str | None = None

parameter_sample_operators instance-attribute 

parameter_sample_operators: list[
    Literal["<", ">", "<=", ">=", "=", "<>"]
]

parameter_sample_values instance-attribute 

parameter_sample_values: list[Any] | list[list[Any]]

intended_parameter_operator class-attribute instance-attribute 

intended_parameter_operator: (
    Literal["<", ">", "<=", ">=", "=", "<>"] | None
) = None

intended_parameter_value class-attribute instance-attribute 

intended_parameter_value: Any | None = None

rejected_by_user class-attribute instance-attribute 

rejected_by_user: bool = False

PredAmbiguityPoint module-attribute 

PredAmbiguityPoint = Annotated[
    Union[
        PredAmbiguityPointFinite, PredAmbiguityPointInfinite
    ],
    Field(discriminator="type"),
]

dbt tasks

DbtTask

Bases: BaseModel

A dbt data-transformation task.

task_type class-attribute instance-attribute 

task_type: Literal['dbt'] = 'dbt'

qid instance-attribute 

qid: str

db instance-attribute 

db: str

Instance ID (e.g. "zuora001"), maps to a DuckDB connector for the project's source database.

question instance-attribute 

question: str

Natural-language instruction describing the transformation to build.

question_instructions class-attribute instance-attribute 

question_instructions: str | None = None

Instructions that apply to this question only.

dataset_instructions class-attribute instance-attribute 

dataset_instructions: str | None = None

Instructions (e.g. for formatting) that apply to all questions in the dataset.

project_dir instance-attribute 

project_dir: str

Relative path to the original dbt project directory (e.g. "data/Spider2/spider2-dbt/examples/zuora001").

working_dir class-attribute instance-attribute 

working_dir: str | None = None

Path to the loader-created working copy of the project (e.g. "runs/exp123/work/zuora001").

gold_db_path class-attribute instance-attribute 

gold_db_path: str | None = None

Relative path to the gold .duckdb file for evaluation.

gold_tables instance-attribute 

gold_tables: list[DbtGoldTable]

Tables to compare in evaluation, from the evaluation spec.

extra_info class-attribute instance-attribute 

extra_info: dict[str, Any] = Field(default_factory=dict)

to_directory 

to_directory(directory: str) -> None

to_markdown 

to_markdown(heading_level: int = 1) -> str

DbtTaskOutput

Bases: DbtTask

Output of a dbt agent.

output_type class-attribute instance-attribute 

output_type: Literal['dbt'] = 'dbt'

pred_db_path class-attribute instance-attribute 

pred_db_path: str | None = None

Path to the predicted DuckDB file produced by the agent (e.g. "runs/exp123/work/zuora001/zuora.duckdb").

pred_db_schema class-attribute instance-attribute 

pred_db_schema: SQLSchema | None = None

Schema of the predicted database after dbt run, including any tables/views created by the agent.

pred_model_files class-attribute instance-attribute 

pred_model_files: dict[str, str] = Field(
    default_factory=dict
)

Maps path relative to working_dir (e.g. "models/my_model.sql") to file content.

dbt_run_success class-attribute instance-attribute 

dbt_run_success: bool | None = None

dbt_run_log class-attribute instance-attribute 

dbt_run_log: str | None = None

trajectory class-attribute instance-attribute 

trajectory: Trajectory | list[Trajectory] | None = None

usage class-attribute instance-attribute 

usage: Usage | None = None

inference_metrics class-attribute instance-attribute 

inference_metrics: dict[str, Any] = Field(
    default_factory=dict
)

Metrics produced during agent prediction, e.g. latency, API costs, etc.

eval_metrics class-attribute instance-attribute 

eval_metrics: dict[str, Any] = Field(default_factory=dict)

Metrics produced during evaluation.

extra_pred_info class-attribute instance-attribute 

extra_pred_info: ExtraPredInfo = Field(
    default_factory=ExtraPredInfo
)

Not used for dbt tasks. Present for compatibility with the NL2QTaskOutput union.

task_type class-attribute instance-attribute 

task_type: Literal['dbt'] = 'dbt'

qid instance-attribute 

qid: str

db instance-attribute 

db: str

Instance ID (e.g. "zuora001"), maps to a DuckDB connector for the project's source database.

question instance-attribute 

question: str

Natural-language instruction describing the transformation to build.

question_instructions class-attribute instance-attribute 

question_instructions: str | None = None

Instructions that apply to this question only.

dataset_instructions class-attribute instance-attribute 

dataset_instructions: str | None = None

Instructions (e.g. for formatting) that apply to all questions in the dataset.

project_dir instance-attribute 

project_dir: str

Relative path to the original dbt project directory (e.g. "data/Spider2/spider2-dbt/examples/zuora001").

working_dir class-attribute instance-attribute 

working_dir: str | None = None

Path to the loader-created working copy of the project (e.g. "runs/exp123/work/zuora001").

gold_db_path class-attribute instance-attribute 

gold_db_path: str | None = None

Relative path to the gold .duckdb file for evaluation.

gold_tables instance-attribute 

gold_tables: list[DbtGoldTable]

Tables to compare in evaluation, from the evaluation spec.

extra_info class-attribute instance-attribute 

extra_info: dict[str, Any] = Field(default_factory=dict)

to_directory 

to_directory(directory: str) -> None

to_markdown 

to_markdown(heading_level: int = 1) -> str

to_summary 

to_summary(
    eval_metrics: Sequence[str] = (),
) -> CSVSummaryRow

DbtGoldTable

Bases: BaseModel

An expected output table for dbt evaluation.

table_name instance-attribute 

table_name: str

required_columns class-attribute instance-attribute 

required_columns: list[int] = Field(default_factory=list)

Column indices to compare. Empty means all columns.

required_sorted class-attribute instance-attribute 

required_sorted: bool = False

True if row order matters.

Datasets and runs

NL2QDataset holds live connectors keyed by task database names. Close those connectors when your program finishes. NL2QRunResult stores experiment data and supports JSON serialization, directory reports, and CSV summaries.

to_directory(...) exports the current run, including available query-result DataFrames as CSVs. Reusing a directory updates its reports.

total_usage records agent usage; total_user_simulator_usage records clarification usage. aggregated_inference_metrics contains available inference statistics, including task latency. These depend on fields returned by each agent. Costs depend on available model pricing; missing usage does not mean zero cost. Report preprocessing costs separately.

Runs record task QIDs and agent configuration. For reproducibility, set the schema formatter explicitly, pin the TabulaFlow version, and record runtime settings separately. Reload matching tasks with the original database snapshot, paths, and credentials before continuing execution or evaluation.

NL2QDataset

Bases: BaseModel

A benchmark split with tasks and connectors keyed by database name.

name instance-attribute 

name: str

split instance-attribute 

split: str

databases class-attribute instance-attribute 

databases: list[str] | None = None

subsample_size class-attribute instance-attribute 

subsample_size: int | None = None

dataset_extra_kwargs class-attribute instance-attribute 

dataset_extra_kwargs: dict[str, Any] = Field(
    default_factory=dict
)

tasks instance-attribute 

tasks: list[NL2QTask]

db_connectors instance-attribute 

db_connectors: dict[str, Any]

NL2QRunResult

Bases: BaseModel

Configuration, outputs, usage, and aggregate metrics for one experiment run.

start_time instance-attribute 

start_time: datetime

end_time instance-attribute 

end_time: datetime

dataset instance-attribute 

dataset: str

split instance-attribute 

split: str

databases instance-attribute 

databases: list[str] | None

subsample_size instance-attribute 

subsample_size: int | None

dataset_extra_kwargs class-attribute instance-attribute 

dataset_extra_kwargs: dict[str, Any] = Field(
    default_factory=dict
)

agent instance-attribute 

agent: str

agent_config instance-attribute 

agent_config: dict[str, Any]

total_usage class-attribute instance-attribute 

total_usage: Usage | None = None

Total usage of the agent, does not include user simulator usage

total_user_simulator_usage class-attribute instance-attribute 

total_user_simulator_usage: Usage | None = None

aggregated_inference_metrics class-attribute instance-attribute 

aggregated_inference_metrics: dict[str, Any] = Field(
    default_factory=dict
)

aggregated_eval_metrics class-attribute instance-attribute 

aggregated_eval_metrics: dict[str, Any] = Field(
    default_factory=dict
)

tasks instance-attribute 

tasks: list[NL2QTaskOutput]

to_directory 

to_directory(
    directory: str,
    eval_metrics_in_summary: Sequence[str] | None = None,
) -> None

Save the run, summary CSV, and readable task reports.

Parameters:

Name Type Description Default
directory str

Destination directory.

 required
eval_metrics_in_summary Sequence[str] | None

Metric columns in the summary. None includes all recorded task metrics in first-seen order; an empty sequence omits metrics. An explicit sequence sets the column order.

 None

to_csv 

to_csv(
    path: str, eval_metrics: Sequence[str] | None = None
) -> None

Save one summary row per task.

Parameters:

Name Type Description Default
path str

Destination CSV file.

 required
eval_metrics Sequence[str] | None

Metric columns to include. None includes all recorded task metrics in first-seen order; an empty sequence omits metrics. An explicit sequence sets the column order.

 None

CSVSummaryRow

Bases: BaseModel

One flattened task row in an experiment summary CSV.

qid instance-attribute 

qid: str

db instance-attribute 

db: str

question instance-attribute 

question: str

question_instructions class-attribute instance-attribute 

question_instructions: str | None = None

gold_query class-attribute instance-attribute 

gold_query: str | None = None

pred_query class-attribute instance-attribute 

pred_query: str | None = None

gold_exec_result class-attribute instance-attribute 

gold_exec_result: str | None = None

pred_exec_result class-attribute instance-attribute 

pred_exec_result: str | None = None

metrics class-attribute instance-attribute 

metrics: dict[str, Any] = Field(default_factory=dict)

fields 

fields() -> list[str]

data 

data() -> list[Any]

NL2QTask module-attribute 

NL2QTask = Annotated[
    Union[SimpleNL2QTask, AmbigNL2QTask, DbtTask],
    Field(discriminator="task_type"),
]

NL2QTaskOutput module-attribute 

NL2QTaskOutput = Annotated[
    Union[
        SimpleNL2QTaskOutput,
        SimpleAmbigNL2QTaskOutput,
        FlatAmbigNL2QTaskOutput,
        StructuredAmbigNL2QTaskOutput,
        DbtTaskOutput,
    ],
    Field(discriminator="output_type"),
]

User interaction

Ambiguity-aware agents use UserSimulatorProtocol to request clarification and track its usage and effort.

UserSimulatorProtocol

Bases: Protocol

Interface used by ambiguity-aware agents to request clarifications.

ask_async async 

ask_async(
    question: UserFreeTextQuestion,
) -> UserFreeTextAnswer | None

ask_async(
    question: UserMultipleChoiceQuestion,
) -> UserMultipleChoiceAnswer | None

ask_async(
    question: UserValueQuestion,
) -> UserValueAnswer | None

ask_async(question: UserQuestion) -> UserAnswer | None

usage 

usage() -> Usage

trajectory 

trajectory() -> Trajectory

user_effort 

user_effort() -> float

UserFreeTextQuestion

Bases: BaseModel

A clarification question answered with free text.

type class-attribute instance-attribute 

type: Literal['free_text'] = 'free_text'

question instance-attribute 

question: str

UserFreeTextAnswer

Bases: BaseModel

Free-text clarification answer.

answer_free_text instance-attribute 

answer_free_text: str

UserMultipleChoiceQuestion

Bases: BaseModel

A clarification question answered by selecting one option.

type class-attribute instance-attribute 

type: Literal['multiple_choice'] = 'multiple_choice'

question instance-attribute 

question: str

options instance-attribute 

options: list[str]

UserMultipleChoiceAnswer

Bases: BaseModel

Selected option index for a multiple-choice clarification.

answer_index instance-attribute 

answer_index: int

UserValueQuestion

Bases: BaseModel

A clarification question answered with a typed comparison value.

type class-attribute instance-attribute 

type: Literal['value'] = 'value'

question instance-attribute 

question: str

value_dtype instance-attribute 

value_dtype: Literal['int', 'float', 'str']

value_operator_options instance-attribute 

value_operator_options: list[
    Literal["<", ">", "<=", ">=", "=", "<>"]
]

UserValueAnswer

Bases: BaseModel

Comparison operator and value supplied for a clarification.

operator instance-attribute 

operator: Literal['<', '>', '<=', '>=', '=', '<>']

value instance-attribute 

value: int | float | str

UserQuestion module-attribute 

UserQuestion: TypeAlias = Annotated[
    Union[
        UserFreeTextQuestion,
        UserMultipleChoiceQuestion,
        UserValueQuestion,
    ],
    Field(discriminator="type"),
]

UserAnswer module-attribute 

UserAnswer: TypeAlias = Union[
    UserFreeTextAnswer,
    UserMultipleChoiceAnswer,
    UserValueAnswer,
]