GeoSQL-Eval Leaderboard

Metric Definitions

FUNCTION_PURPOSE: Function Purpose Identification

PARAMETER_CHECK: Parameter Order and Type Verification

RETURN_TYPE: Return Type Recognition

GENERAL_KNOWLEDGE: General Spatial Knowledge

Rank	Model	Category	FUNCTION_PURPOSE	PARAMETER_CHECK	RETURN_TYPE	GENERAL_KNOWLEDGE	Average

Metric Definitions

Execution Pass Rate: Query execution success rate (Level 2)

Syntactic Validity: SQL syntax correctness (Level 2)

FNR: Function Name Hit Rate (Level 3)

AMA: Argument Match Accuracy (Level 3)

ACC_ALL: Overall Accuracy (Level 4)

ACC_Geo: Geometry Query Accuracy (Level 4)

ACC_Other: Non-Geometry Query Accuracy (Level 4)

pass@k: Success rate at k attempts (Level 5)

CV: Coefficient of Variation (Level 5)

SA: Semantic Alignment Score (Level 5)

Rank	Model	Category	Exec Rate	Syntax Valid	FNR	AMA	ACC_ALL	ACC_Geo	ACC_Other	pass@1	pass@3	pass@5	CV	SA	ACC_EXPL	ACC_UNDER	Time	Tokens

Metric Definitions

THR: Table Hit Rate (Level 3)

FHR: Field Hit Rate (Level 3)

FNR: Function Name Hit Rate (Level 3)

ACC_ALL: Overall Accuracy (Level 4)

ACC_Geo: Geometry Query Accuracy (Level 4)

Resource Consumption: Time (s) and Tokens

Rank	Model	Category	Exec Rate	Syntax Valid	THR	FHR	FNR	ACC_ALL	ACC_Geo	pass@1	pass@3	pass@5	CV	SA	ACC_EXPL	ACC_UNDER	Time	Tokens

In recent years, large language models (LLMs) have achieved remarkable progress in natural language understanding and structured query generation (NL2SQL). However, extending these advances to GeoSQL tasks in the PostGIS environment remains challenging due to the complexity of spatial functions, geometric data types, and execution semantics. Existing evaluations primarily focus on general relational databases or Google Earth Engine code generation, leaving a lack of systematic benchmarks tailored to spatial databases. To address this gap, this study introduces GeoSQL-Eval, the first end-to-end automated evaluation framework for PostGIS query generation. Built upon Webb’s Depth of Knowledge (DOK) model, the framework encompasses four cognitive dimensions, five proficiency levels, and twenty task categories,......