gobi is a geospatial dataframe library for Go, built on top of
Apache Arrow. Think of it as a GeoPandas-shaped
API with Polars-shaped internals: columnar, chunk-slice fast paths, and
built around a strongly-typed schema.
Status: early. The API is settled enough to build a small pipeline on; semver stability begins with v1.0. GeoParquet v1.1 output has been verified against GeoPandas v1.1.1 and QGIS v4.0.2.
- Arrow-native. A
Frameis a set of ArrowColumns.Head,Tail, and row selection are zero-copy where possible. - Full 2D + optional XYZ geometry.
Point,LineString,Polygon(with holes),MultiPoint,MultiLineString,MultiPolygon,GeometryCollection. WKB and WKT round-trip per OGC/ISO SFA 1.2 (type codes 1..7 for 2D, 1001..1007 for XYZ). - Real spatial operations. Area, length, centroid (every geometry
type), convex hull, containment,
Simplify(Douglas-Peucker),Bufferwith rounded OR square joins/caps,EstimateUTMCRSon every type, full topological predicates (Intersects/Contains/Within/Touches/Overlaps/Crosses/Disjoint). - 3D geodesic and volumetric geometry (v0.4.7).
Point.Distance3Ddispatches on CRS — projected → 3D Cartesian, geographic → ECEF-Euclidean via WGS84 (through-Earth chord distance in meters) — closing a gap that PostGISST_3DDistance, Shapely, polars-st, and DuckDB Spatial all leave open (each is projected- Cartesian only and silently mixes degrees with meters on geographic input).ExtrudedPolygon(2D footprint ×[MinZ, MaxZ]) models buildings, airspace volumes, and bathymetric prisms as a first-class shape withContains3D/Intersects3D/ SoAPointsInPrismFromXYZ.SphereandCapsulebackBuffer3DPoint/Buffer3DLineString;ConvexHull3DFromXYZreturns the extruded-prism approximation. Series-level ops (GeomDistance3D,GeomLength3D,GeomZ,GeomIntersects3D,GeomForce2D,GeomForceZ) and matchingExprwire-in compose 3D operations into LazyFrame pipelines. Every primitive is SoA-first — batch ECEF conversion, vectorized Z band-pass PIP, scalar-target distance kernels — so a 1M-rowGeomDistance3Dcall never materializes[]Pointor allocates broadcast slabs for the target coordinate. - Polygon boolean ops. Pure-Go Martinez-Rueda sweepline:
Clip/Union/Difference/SymDifferenceand aDissolve(geoms)collection reducer using spatially-sorted divide-and-conquer merge (like shapely'sunary_union). Bit-exact parity with geopandas verified on a 500-polygon benchmark; a Sutherland-Hodgman fast path takes over when both operands are convex (4× faster than the general sweep). Linestring-vs-polygon clipping has its own Cyrus-Beck primitive:LineString.Clip(p)andSplitBy(p)(same onMultiLineString), targeting the Overture road-vs-h3-cell partitioning workload — 100-vertex hex clip in ~890 ns on M3 Pro. - Antimeridian handling.
CrossesAntimeridian(g)detects ±180°-crossing geographic-CRS input;SplitAtAntimeridian(g)splits it into per-side components via linear-lon interpolation.EstimateUTMCRSrefuses crossing input with a clear error rather than silently returning the wrong zone (a Fiji-shaped bbox[178, -178]used to pick UTM 31N over Africa). - Geometry constructors from columns.
PointsFromXY(x, y, crs)andPointsFromXYZ(x, y, z, crs)build a WKB geometry Series directly from numeric coordinate columns — modeled ongeopandas.points_from_xy. Mixed numeric types (Float64, Float32, Int64, Int32) auto-promote; nulls in either input yield a null geometry. - Reprojection engine. WGS84 ↔ Web Mercator ↔ all 120 UTM zones, using the ellipsoidal Redfearn/Snyder formulas. Sub-cm round-trip accuracy verified against reference cities worldwide.
- Spatial index and join. Static Sort-Tile-Recursive R-tree with
bounding-box and k-nearest queries. Internal storage is struct-of-
arrays (parallel
[]float64bbox arrays) — measured −27% wall time on bbox-intersect Search (100k-item tree, 1k queries) compared to the AoS shape it replaces. Public API unchanged.Frame.SJoin(right, ..., pred)withSPIntersects/SPContains/SPWithinpredicates, multi-threaded across left rows, tunable viaWorkers(n). - SoA geometry kernels (v0.4.1). For hot-path callers that touch
a geometry's coordinates without needing the full
[]Point/Polygonobject graph — a common shape on the parquetio write path, spatial-sort pipelines, and per-polygon many-candidate refine loops.PointsView{Xs, Ys []float64, …}materializes parallel coordinate slabs viaLineString.View(),Polygon.RingViews(), and friends. Zero-allocation WKB byte-stream scanners (BoundsFromWKB,CentroidFromWKB,CentroidAndBoundsFromWKB,PIPFromWKB,PlanarLengthFromWKB,PlanarAreaFromWKB) walk WKB bytes tracking running accumulators — semantics matchParseWKB(data).<op>()exactly on projected CRSes.PIPRingFromXY/PIPPolygonFromRingsdeliver a measured 3× held-view speedup vs.Polygon.Contains(pt)on the SJoin-refine shape (64-vertex polygon × 100 candidates). Wired intogeoparquet.computeBboxColumns(−31% wall time, −47% memory on 100k-row GeoParquet writes), both Hilbert sort paths (−24% wall time, essentially zero-alloc WKB pass on 100k-row sorts), andSeries.GeomArea/Series.GeomLengthon projected CRSes (−48% / −67% wall time on 1M-vertex WKB blobs, zero allocations per row). Iterative Douglas-Peucker on parallel slabs (SimplifyDPFromXY,PointsView.SimplifyDP) replaces the recursive AoS[]Pointsplice+append pattern — wired into everySimplifyentry point transparently. −81% wall time and 24k× fewer allocations on a 1M-vertex polyline (3.85 s → 0.75 s, 260k allocs → 11). WKB → PointsView direct-parse (PolygonRingViewsFromWKB,MultiPolygonRingViewsFromWKB,LineStringViewFromWKB,PrepareFromWKB) skips the[]Pointintermediate for polygon-family types — 2.3× faster and 5× less memory thanParseWKB(data).RingViews()on a 1M-vertex polygon. SoA min-distance kernels (PointToSegmentDistanceSqXY,PointToPolylineMinDistanceSq) power an internal rewrite ofplanarMinDistance— everyGeomDistance/WithinDistancecaller gets −80% wall time on a 64-vertex Polygon×Polygon and −82% at 1024 vertices, from deferringmath.Sqrtto a single call at the end instead of paying one per (vertex, segment) pair. Andrew's monotone-chain convex hull on slabs (ConvexHullFromXY,PointsView.ConvexHull) replaces the Graham-scan polar-angle sort — everyPolygon.ConvexHull/geometry.ConvexHull(g)caller sees −30 to −54% wall time at n≥1024 and 3-4× lower memory (index sort on[]intinstead of[]Pointstructs). WKB-directSeries.GeomDistancefast path (PlanarMinDistanceFromWKB) skips the AoSParseWKBon bbox-disjoint rows — −73% memory on 10k rows of 64-vertex polygons vs a fixed target (65 MB → 17.6 MB per call, −12% wall time). Slice 14 extends the same bbox-reject pattern toSeries.GeomIntersects/GeomContains/GeomWithinand the LazyFrameCol('geom').GeomIntersects(x)filter expression — −74% wall time and 98.5% less memory on a 5k-row × fixed-AOI benchmark (802 μs → 205 μs, 2.37 MB → 36 KB).Series.GeomBoundsandSeries.GeomCentroidalso wired to the byte-stream scanners. Slice 15 sweeps the last non-clip Series entry points:Series.GeomDWithin(bbox-min-distance reject: −66% wall, −93% memory) andSeries.GeomType(WKB-header peek: −88% wall, 358× fewer allocs — was pure ParseWKB waste). Slice 16 adds boundary-inclusivePIPInclusiveFromWKBand wires an SJoinPoints × Polygonsfast path that skips bothdecodeGeometryColumnpasses — −62% wall, −70% memory, 33× fewer allocs on the 10k-points × 100-large-polygons bench. Slice 17 adds Series-level bbox reject forGeomClip(empty on disjoint intersection) andGeomDifference(row unchanged on disjoint difference) — −81% / −93% wall time and 334× / 530× fewer allocs on 5k rows × small mask. Slice 18 adds convex-containment fast paths insidegeometry.Boolean()itself — when either operand is a convex polygon that fully contains the other, intersection returns the contained operand unchanged. −50% wall and −96% memory on the isolated containment case; −26% wall, −30% memory on the M-cell loop (100 cells × user disc). Slice 19 extends the Sutherland-Hodgman fast path from convex×convex to convex clipper × any single-ring subject when the intersection is guaranteed simply connected (transition-count gate on vertex containment) — −83% wall time on the concave-L-shape-clipped-by-AOI shape. Slice 20 extends the SJoin fast path from Points-only to LineString × Polygon and single-ring Polygon × single-ring Polygon (bilateral vertex-inside + AoS fallback for edge crossings) — −36% wall and −72% memory on 1k lines × 100 large polygons. Plus Union/Difference containment fast paths ingeometry.Boolean()(union with a convex containing operand → operand unchanged;a ⊆ convex bdifference → empty without the sweep). Slice 21 finishes clip coverage:Series.GeomUnionandSeries.GeomSymDifferencenow build the combined MultiPolygon inline for bbox-disjoint rows (−89% wall, 22× fewer allocs on 5k rows × small mask), and SJoin gains SPWithin / SPContains fast paths for Polygon×Polygon and LineString×Polygon (convex-container gate; convexity precomputed during extract). Slice 22 pushes the same shape 10BC8 into the LazyFrame executor: Series scalar-comparison ops (GtScalar/LtScalar/ etc.) now dispatch tocompute.CmpF64*SIMD kernels, and the expression executor recognizes AND-chained range and bbox filters, dispatching to fused kernels (compute.AndChainF64Range/AndChainF64BBox) that skip intermediate boolean-column materialization — −49% wall and 71% fewer allocs on a 100k-row 4-comparison bbox filter. Slice 23 adds Int64 comparison kernels (compute.CmpI64Ge/Le/Gt/Lt) with matching Series wire-in, a foundationalcompute.CountTruebool-reduce +Series.CountTrue()wrap, and a lane-count gate on the SIMD compare kernels that fixes a pre-existing Apple 2-lane NEON regression the Slice 22a dispatch had exposed (SIMD build now matches scalar build on Apple, wins expected on amd64 AVX2 / AVX-512). - Prepared-geometry predicate API.
PreparedGeometry+Prepare(g)+TestPrepared(pred, a, b)amortize the AoS→SoA materialization tax across many predicate calls — the shape spatial-index-free refine loops need. Fast paths for Point×Polygon and Point×MultiPolygon; other pair shapes fall through toTest()transparently. Not used by gobi's built-in SJoin (measurement showed R-tree pre-filter drives candidate count too low for amortization to pay off); designed for advanced callers with dense overlapping polygons or many candidates per prepared geometry. - DataFrame ops.
Filter,Take,Head,Tail,SortBy(multi-key stable, nulls-last),WithColumn,DropColumn,SelectCols,Rename,Explode(also as aLazyFramestreaming step),Join(inner / left / right / full / semi / anti with coalesced keys).GroupBy(...).Agg(...)with built-in kinds:Count,Sum,Mean,Min,Max,First,Last,NUnique,Std,Var,Median,Mode. Aggregations can carry a per-aggregationFilter ExprforSUM(x) FILTER (WHERE …)-style reductions. Series arithmetic, comparisons, aggregations — all with single-chunk bulk fast paths and Int64-preserving scalar arithmetic. - Built-in collect-set aggregators.
NewStringSetAggregator(),NewInt64SetAggregator(),NewUint64SetAggregator(),NewInt32SetAggregator(),NewUint32SetAggregator()— distinct- value roll-ups that emitList<T>per group and stream through the aggregate executor viaIncrementalAggregator. - User-defined aggregations.
type Aggregator interface { ... }plugs directly intoGroupBy.Aggalongside the built-ins. Opt into the streaming aggregate executor by additionally implementingIncrementalAggregator(Clone/Update/Finalize) — per- batch state updates instead of a materialize-then-reduce pass. - Expression IR.
gobi.Col("price").Mul(gobi.Lit(1.08)).Gt(gobi.Lit(100))builds a data tree, not a chain of already-executed calls.Frame.FilterExprandFrame.WithColumnExprevaluate it. The built-in vocabulary covers arithmetic (Add/Sub/Mul/Div), bitwise (BitAnd/BitOr/BitXor), comparisons, logical (And/Or/Not),IsNull/IsNotNull,Cast(dtype)(numeric- to-numeric + Timestamp source),If/Coalesce,LitNull(dtype),LitEmptyList(elem),ListLen,ListUnion,Shift(n), window functions (.Sum()/.Mean()/.MinAgg()/.MaxAgg()/.Count()/.Median()/ .Mode()/.BitOrAgg()/.BitAndAgg()/.BitXorAgg().Over(cols...)for scalar-agg-and-broadcast; shape-preserving inners likeShift(1).Over(K)for prev-row-within-partition patterns),Round()/Floor()/Ceil()/Trunc(),UnixNano()(Timestamp → Int64 ns),IcebergBucket(n)(Iceberg'sbucket(n)partition transform), andHaversineExpr(lat1, lon1, lat2, lon2, unit)for great-circle distance between two point columns. ACustom(node ExprNode)escape hatch lets sibling packages (H3, hashes, ML inference) plug in their own expression types alongside the built-ins. - Alignment-aware fast paths.
PartitionMetadataclaims (attached viaLazyFrame.WithPartitionAssertionor produced bycontrib/athenaioon Iceberg CTAS output) letGroupBy,Over, andJoinskip the hash-shuffle and linear-scan partition boundaries directly. Runs 30-70% faster than the general path when applicable; falls through automatically otherwise. - List and Struct columns. First-class support end-to-end:
Explode expands list rows,
ListUnionmerges per-row, aggregations can emitList<T>(see set aggregators above), and Struct-typed builders round-trip through Frame → LazyFrame → Frame. - LazyFrame + rule-based optimizer.
df.Lazy()andparquetio.ScanFile(path)build plan trees that don't execute until.Collect(). Nine rewrite rules run to a fixed point: constant folding, dead-filter removal, adjacent-filter combining, push-filter-below-project, push-filter-below-sort, column projection pushdown, predicate pushdown (into row-group stats), and cascade-empty (short-circuitsLit(false)-derived subtrees). Projection pushdown routes intoparquetio.ReadOptions.Columns— 2.4× faster reads on partial-column queries, matching the eager baseline. Optimizer overhead is ~8 µs on a five-node plan; always-on. - Parallel streaming executor.
LazyFrame.Collect()compiles the optimized plan to a tree ofExecOperators that pull one record batch at a time — bounded memory regardless of source size. Filter, Project, WithColumn, Drop, Rename, Select, Explode, Limit, ScanFrame, and ScanFile all stream natively. Adjacent batch-transform ops are fused into a single per-batch pass by the compiler (~22% fewer allocations on typical Filter → Project → WithColumn chains). Aggregate (all built-in kinds + customIncrementalAggregators) and hash-join (Inner/Left/Semi/Anti) run as native streaming operators — no materialization step. Parquet scan parallelizes across row-groups; the streaming hash aggregate partitions rows across workers by key hash. Both scale toGOMAXPROCSout of the box.LazyFrame.ExplainPhysical()prints what strategy each node compiles to (worker counts included). - Datetime + timezone-aware ops.
Timestamp[ns]columns with optional IANA tz label. Component extractors,AddDuration/DiffDuration, comparisons, sub-day + calendar truncation.ResampleEvery(timeCol, interval)for downsampling andRollingBy(timeCol, period)for trailing time windows; plus fixed-sizeSeries.RollingSum/Mean/Min/Max/Count. - Multi-format I/O. Every format has
ReadFile/WriteFileat Frame level plus aScanFileLazyFrame entry point where the underlying source supports it, and every format also has struct-directReadStructs[T]/WriteStructs[T]wrappers with a per-format struct-tag namespace —parquet:/csv:/geojson:/gpkg:/shp:/kml:/pgio:— so the same Go type can carry different column names per format (shp's 10-char DBF alias,parquet:"-"to omit from parquet only, etc.). Formats: CSV read/write (.gz/.zstdetected from the filename both ways,.bz2read-only), Parquet with proper GeoParquet 1.1 metadata (snappy / gzip / brotli / zstd / lz4, canonical PROJJSON for EPSG:3857 + all 120 UTM zones), full RFC 7946 GeoJSON (every geometry type + XYZ,.geojsonlstreaming), OGC GeoPackage 1.3 (SQLite, RTree spatial index, spec-compliant metadata), PostgreSQL / PostGIS (viapgx/v5, nativeCopyFrombulk load), KML + KMZ read/write (zipped KML auto-detected by.kmzextension), and Shapefile read/write (.shp+.shx+.dbf+ optional.prj). - Streaming readers.
csvio.ReadFileChunksFuncandparquetio.ReadFileChunksFuncyield one Frame per record batch (~64k rows), releasing arrow buffers after each callback. Peak memory is bounded regardless of source-file size — good for ETL over multi-GB inputs. - Column projection.
parquetio.ReadOptions{Columns: ...}skips fetch, decompress, and arrow materialization for the columns you don't need. Composes with streaming. - Parquet write tuning.
parquetio.WriteOptionsexposesRowGroupRows(predicate-pushdown-friendly small groups vs. compression-friendly large ones),BloomFilterColumns+BloomFilterFPP(equality-filter skipping in DuckDB / Spark / Polars / pyarrow; gobi's reader doesn't use them yet).parquetio.NewWriterwrites a file incrementally with caller-chosen row-group boundaries and returns footer statistics on Close. - Parallelism controls. Package-level
SetMaxParallelism(n)or per-opWorkers(n). - Pure Go, no cgo. No GDAL, no GEOS, no libproj. Cross-compiles cleanly to every architecture Go supports.
go get github.com/zoobst/gobiRequires Go 1.27 or newer (needed for the portable simd stdlib
under GOEXPERIMENT=simd; the scalar build still works on any
Go the module supports).
Full API reference is on
pkg.go.dev —
auto-generated from source doc comments. Every subpackage
(parquetio, csvio, geometry, geojsonio, gpkgio, pgio,
kmlio, shpio) has its own page; use the same base URL with
the package path appended.
package main
import (
"github.com/zoobst/gobi/csvio"
"github.com/zoobst/gobi/parquetio"
)
type city struct {
Name string `csv:"name"`
Population int64 `csv:"population"`
Geom string `csv:"geometry" geom:"true"`
}
func main() {
// Fields find their columns by header name. .gz / .zst / .bz2 are
// auto-detected from the filename; explicit ReadOptions.Compression
// overrides.
df, err := csvio.ReadFile[city]("cities.csv.gz", &csvio.ReadOptions{CRSHint: 4326})
if err != nil { panic(err) }
defer df.Release()
// The output file carries a spec-compliant GeoParquet 1.1 metadata
// blob and reads cleanly in GeoPandas / QGIS. nil = Snappy + arrow's
// default row-group sizing.
_ = parquetio.WriteFile(df, "cities.parquet", nil)
}// Turn two numeric columns into a WKB geometry column. Argument
// order is x, y — i.e. longitude, latitude — matching WKB / GeoJSON /
// geopandas.points_from_xy. Nulls on either side yield a null point.
lng, _ := df.Column("lng")
lat, _ := df.Column("lat")
points, _ := gobi.PointsFromXY(lng, lat, 4326)
df, _ = df.WithColumn("geometry", points)
// df["geometry"] is now a proper WKB column, ready for SJoin,
// GeoParquet write, etc.cities, _ := parquetio.ReadFile("cities.parquet", nil) // 1M points
regions, _ := parquetio.ReadFile("regions.parquet", nil) // 5k polygons
// Which region contains each city?
joined, err := cities.SJoin(regions, "geometry", "geometry", gobi.SPWithin)
// Cap parallelism per-op (see "Parallelism" below):
joined, err = cities.SJoin(regions, "geometry", "geometry", gobi.SPWithin, gobi.Workers(4))gb, _ := df.GroupBy("region")
totals, _ := gb.Agg(
gobi.Aggregation{Column: "population", Kind: gobi.AggSum},
gobi.Aggregation{Column: "population", Kind: gobi.AggMean, Alias: "avg_pop"},
)// Multi-key, stable, nulls-last. Earlier keys have priority;
// later keys break ties.
sorted, _ := df.SortBy(
gobi.SortKey{Column: "date"}, // ascending
gobi.SortKey{Column: "revenue", Descending: true},
)// Small row groups + bloom filters on high-cardinality equality columns.
// DuckDB / Spark / Polars / pyarrow readers all use the bloom filters
// for predicate pushdown on equality filters.
err := parquetio.WriteFile(df, "events.parquet", &parquetio.WriteOptions{
Codec: parquetio.CodecZstd,
RowGroupRows: 128_000, // 0 = arrow default cap (64Mi rows)
BloomFilterColumns: []string{"user_id", "session_id"}, // sized per row group
BloomFilterFPP: 0.01, // 0 = arrow default (0.01)
})// One row group per key range, written batch by batch; Close returns
// row count, file size and per-column min/max/null counts from the
// footer (what a table format needs to register the file).
w, err := parquetio.NewWriter(out, schema, &parquetio.WriteOptions{Codec: parquetio.CodecZstd})
for _, batch := range batchesSortedByCell {
if err := w.Write(batch.Frame); err != nil { return err }
if batch.LastOfCell {
w.EndRowGroup() // row groups never straddle a cell
}
}
stats, err := w.Close()// lon / lat already bound each point, so declare them as the
// GeoParquet covering instead of generating four float64 bbox
// columns. Readers (gobi, DuckDB, GDAL) prune row groups from the
// lon / lat min/max stats; every row is checked against its geometry.
err := parquetio.WriteFile(df, "points.parquet", &parquetio.WriteOptions{
Coverings: map[string]parquetio.Covering{"geometry": parquetio.PointCovering("lon", "lat")},
KeyValueMetadata: map[string]string{"writer": "my-etl/v3"},
})// Reads a 5 GB parquet file at ~15 MB peak memory. Only two columns are
// fetched off disk; the rest are never decompressed.
err := parquetio.ReadFileChunksFunc(
"events.parquet",
&parquetio.ReadOptions{Columns: []string{"user_id", "ts"}, ChunkRows: 64_000},
func(batch *gobi.Frame) error {
// Process ~64k rows at a time. The batch is released after
// return; call batch.Retain() to keep it past this callback.
return sink.Write(batch)
},
)CSV has the same shape: csvio.ReadFileChunksFunc[Row](path, opts, fn).
Two shapes. WithColumn accepts any Series the caller built by hand:
// A user-space helper produces the derived Series any way it likes
// (external library call, vectorized loop, whatever). WithColumn wires
// it back into the Frame — appending, or replacing an existing column
// of the same name.
lat, _ := df.Column("lat")
lng, _ := df.Column("lng")
cells, _ := h3x.Encode(lat, lng, 9)
df, _ = df.WithColumn("h3", cells)
df, _ = df.DropColumn("raw_geometry")WithColumnExpr accepts an expression tree, so pipelines composed from
built-in ops read left-to-right:
// Same shape via the expression IR — no intermediate Series to name.
df, _ = df.WithColumnExpr("usd_price",
gobi.Col("eur_price").Mul(gobi.Lit(1.08)),
)
df, _ = df.WithColumnExpr("margin",
gobi.Col("revenue").Sub(gobi.Col("cost")),
)// Compute the 95th percentile of a numeric column per group.
type P95 struct{}
func (P95) Aggregate(s gobi.Series, rows []int) (any, error) {
arr := s.Column().Data().Chunks()[0].(*array.Float64)
vals := make([]float64, 0, len(rows))
for _, r := range rows {
if !arr.IsNull(r) { vals = append(vals, arr.Value(r)) }
}
if len(vals) == 0 { return nil, nil }
sort.Float64s(vals)
return vals[int(float64(len(vals)-1)*0.95)], nil
}
func (P95) Type() arrow.DataType { return arrow.PrimitiveTypes.Float64 }
func (P95) Name() string { return "p95" }
// Mix custom + built-in aggregations in one call.
gb, _ := df.GroupBy("h3") // Uint64 keys are hashable
out, _ := gb.Agg(
gobi.Aggregation{Column: "latency_ms", Kind: gobi.AggMean},
gobi.Aggregation{Column: "latency_ms", Fn: P95{}},
)Two shapes. Filter takes an already-computed Boolean Series (mask):
pops, _ := df.Column("population")
mask, _ := pops.GtScalar(1_000_000)
big, _ := df.Filter(mask)FilterExpr takes an expression tree — no intermediate Series to name:
big, _ := df.FilterExpr(
gobi.Col("population").Gt(gobi.Lit(1_000_000)).
And(gobi.Col("country").Eq(gobi.Lit("US"))),
)Over(partitionCols...) runs either an aggregate (broadcast to every
row in the partition) or a shape-preserving inner like Shift(1)
(prev-row-within-partition). Both compose with the alignment
fast paths when PartitionMetadata proves the frame is
partition-contiguous.
// Per-region total (broadcast). Same shape works for Mean / Min /
// Max / Count / Median / Mode.
df, _ := df.WithColumnExpr("region_total",
gobi.Col("sales").Sum().Over("region"),
)
// Previous row's timestamp within (eid) partition. First ping of
// each entity emits null.
df, _ := df.WithColumnExpr("prev_ts",
gobi.Col("ts").Shift(1).Over("eid"),
)
// Great-circle distance between successive pings, entirely in the
// expression tree — no Custom ExprNode needed.
df, _ := df.WithColumnExpr("step_km", gobi.HaversineExpr(
gobi.Col("lat"),
gobi.Col("lon"),
gobi.Col("lat").Shift(1).Over("eid"),
gobi.Col("lon").Shift(1).Over("eid"),
geometry.UnitKilometers,
))For flag-unpacking a packed Int64 into per-bit indicator columns:
const FlagBogonIP = 1 << 3
df, _ := df.WithColumnExpr("has_bogon_ip",
gobi.Col("flags").
BitAnd(gobi.Lit(int64(FlagBogonIP))).
Ne(gobi.Lit(int64(0))),
)p := geometry.Point{X: -73.9857, Y: 40.7484, CRSValue: geometry.WGS84}
utm, _ := p.EstimateUTMCRS() // EPSG:32618 (WGS 84 / UTM zone 18N)
proj, _ := p.ToCRS(utm) // coordinates now in meterspoly := geometry.SimplePolygon(points, geometry.PseudoMercator)
// Rounded (semicircle caps, rounded joins) — the default.
rounded, _ := geometry.Buffer(poly, 100, geometry.BufferOptions{Segments: 32})
// Square-style buffer: flat/extended caps, mitre joins. Faster and
// produces fewer vertices (5 instead of 33 for a Point).
square, _ := geometry.Buffer(poly, 100, geometry.BufferOptions{
Style: geometry.BufferSquare,
})
simpler := poly.Simplify(5.0) // Douglas-Peucker at 5-unit tolerance// 3D distance dispatches on CRS. Projected → 3D Cartesian;
// geographic → ECEF-Euclidean via WGS84 (straight-line through-
// Earth chord distance in meters). Pre-v0.4.7 the geographic path
// errored; now it produces the honest answer.
jfk := geometry.Point{X: -73.7789, Y: 40.6413, Z: 0, HasZ: true, CRSValue: geometry.WGS84}
lax := geometry.Point{X: -118.4085, Y: 33.9425, Z: 0, HasZ: true, CRSValue: geometry.WGS84}
chord, _ := jfk.Distance3D(lax, geometry.UnitKilometers) // ~3880 km (arc is ~3944)
// Column-level: 3D distance from every row to a fixed target.
d3, _ := points.GeomDistance3D(jfk, geometry.UnitMeters)
// ExtrudedPolygon = 2D footprint × [MinZ, MaxZ] — the practical
// 90% of 3D GIS (buildings, airspace, bathymetry) without a mesh
// library. Point-in-prism composes 2D PIP with a Z band-pass; the
// SoA kernel runs a whole column in one pass, no []Point.
building, _ := geometry.NewExtrudedPolygon(footprint, 0, 120) // 120m tall
inside, _ := points.GeomIntersects3D(building)
// Buffer shapes for Point/LineString are Sphere and Capsule
// respectively — Minkowski sum with a ball, projected-only today.
sphere, _ := geometry.Buffer3DPoint(pt, 50)
capsules, _ := geometry.Buffer3DLineString(ls, 10)
// Coordinate-dimension promote/demote per PostGIS ST_Force2D /
// ST_Force3DZ. Every branch defensively copies point slabs so the
// returned geometry never aliases the caller's backing arrays.
flat, _ := points.GeomForce2D()
lifted, _ := flat.GeomForceZ(100)subject := geometry.SimplePolygon(subjectPts, geometry.PseudoMercator)
mask := geometry.SimplePolygon(maskPts, geometry.PseudoMercator)
// Row-scalar ops on a whole geometry Series.
gs, _ := df.Column("geom")
clipped, _ := gs.GeomClip(mask) // intersection per row
unioned, _ := gs.GeomUnion(mask)
diffed, _ := gs.GeomDifference(mask)
// Dissolve every row's polygon into a single geometry (like
// shapely's unary_union — divide-and-conquer merge, bit-exact
// against geopandas on the bundled 500-polygon bench).
merged, _ := gs.GeomDissolve()
// Or reach the raw ops directly on a Geometry.
inter, _ := geometry.Clip(subject, mask)
uni, _ := geometry.Union(subject, mask)Boolean ops handle Polygon-vs-Polygon; linestring-vs-polygon has its own primitive that skips the general polygon sweep entirely.
line := geometry.NewLineString([]geometry.Point{
{X: -5, Y: 5}, {X: 15, Y: 5}, {X: 15, Y: -5}, {X: -5, Y: -5},
}, geometry.PseudoMercator)
cell := geometry.SimplePolygon(hexPts, geometry.PseudoMercator)
inside := line.Clip(cell) // fragments inside cell
inside, outside := line.SplitBy(cell) // both sides, one pass
// MultiLineString version — flat []LineString result, walk order
// preserved across components.
ml := geometry.NewMultiLineString([]geometry.LineString{line, other}, geometry.PseudoMercator)
frags := ml.Clip(cell)Convex polygons (including h3 cells) take a Cyrus-Beck fast path with
per-segment AABB reject; concave / multi-ring polygons fall back to
sort-and-march. Coordinate system is planar — densify first if you
need geodesic accuracy, and use SplitAtAntimeridian before clipping
if your linestring crosses ±180°.
// A geographic-CRS polygon spanning ±180° confuses every naive
// cartesian primitive. Detect + split it into per-side components
// before feeding it downstream.
poly := geometry.SimplePolygon([]geometry.Point{
{X: 170, Y: -10}, {X: -170, Y: -10},
{X: -170, Y: 10}, {X: 170, Y: 10}, {X: 170, Y: -10},
}, geometry.WGS84)
if geometry.CrossesAntimeridian(poly) {
split, _ := geometry.SplitAtAntimeridian(poly)
// split is a MultiPolygon with one component in [170,180] and
// another in [-180,-170]; each has valid, non-crossing bounds.
}Every io package has a struct-direct wrapper that uses per-format
struct tags — parquet: for parquet, shp: for shapefile,
geojson: for GeoJSON, etc. Resolution falls back to a universal
gobi: tag when the format-specific one is absent, then to csv:
(legacy), then to the Go field name. format:"-" skips a field.
type Feature struct {
ID int64 `parquet:"id" shp:"OBJECTID"` // per-format aliases
Name string `gobi:"name"` // shared across formats
Population float64 `shp:"POP10"` // 10-char DBF alias
Notes string `parquet:"-"` // omit from parquet
Geometry []byte `geom:"true"` // geometry marker
}
// Read a slice of structs directly — no Frame boilerplate.
rows, err := parquetio.ReadStructs[Feature]("in.parquet", nil)
// Write the same slice out as a shapefile — column names come from
// the shp: tags automatically.
err = shpio.WriteStructs(rows, "out", nil)The same shape works for csvio.ReadStructs / WriteStructs,
geojsonio.ReadStructs / WriteStructs, gpkgio.ReadStructs /
WriteStructs, kmlio.ReadStructs / WriteStructs, and
pgio.ReadStructsQuery / ReadStructsTable / WriteStructsTable.
tree := geometry.NewRTree(bboxes)
hits := tree.Search(query) // IDs whose bounds intersect query
nearest := tree.Nearest(x, y, k) // k closest bounds, sortedInternal storage is struct-of-arrays: parallel []float64 bbox
arrays for both item and node bboxes. Search sees a measured
27% wall-time reduction on 100k-item / 1k-query workloads vs. the
AoS shape it replaces. Public API is unchanged from earlier
releases.
type event struct {
Name string `csv:"name"`
When time.Time `csv:"when" time:"2006-01-02 15:04:05"`
}
df, _ := csvio.ReadFile[event]("events.csv", nil)
when, _ := df.Column("when")
// Render the same instants in New York local time.
nyWhen, _ := when.WithTimezone("America/New_York")
// Component extractors honor the tz.
hourNY, _ := nyWhen.Hour() // Int64 series with local hours
// Truncate to the top of each local day.
dayStart, _ := nyWhen.TruncateToCalendar(gobi.CalendarDay)// Downsample to hourly buckets (Unix-epoch aligned).
r, _ := df.ResampleEvery("when", time.Hour)
hourly, _ := r.Agg(
gobi.Aggregation{Column: "value", Kind: gobi.AggSum},
gobi.Aggregation{Column: "value", Kind: gobi.AggMean, Alias: "avg"},
)
// Trailing 5-minute rolling sum keyed by timestamp.
tr, _ := df.RollingBy("when", 5*time.Minute)
rollSum, _ := tr.Agg("value", gobi.AggSum)
// Fixed-window rolling on a plain Series.
val, _ := df.Column("value")
m7, _ := val.RollingMean(7) // 7-row moving average// KML → Frame (auto-parses ExtendedData into columns)
places, _ := kmlio.ReadFile("places.kml", nil)
_ = kmlio.WriteFile(places, "out.kml", nil)
// KMZ works the same way — extension picks the format automatically.
// Set kmlio.WriteOptions{Format: FormatKMZ} explicitly for Writer flows.
_ = kmlio.WriteFile(places, "out.kmz", nil) // writes zip with doc.kml
// Shapefile → Frame (reads .shp + .shx + .dbf + optional .prj)
counties, _ := shpio.ReadFile("counties", nil) // no .shp suffix needed
_ = shpio.WriteFile(counties, "counties_out", nil) // writes all four files| Package | What it does |
|---|---|
github.com/zoobst/gobi |
Frame, Series, GroupBy, Join, SJoin, Explode, datetime + rolling + resample, options |
.../gobi/geometry |
2D + 3D primitives (Point/LineString/Polygon + ExtrudedPolygon/Sphere/Capsule), WKB / WKT, CRS + reprojection (incl. WGS84 ECEF), predicates, R-tree, Buffer / Simplify / Centroid, geodesic 3D distance |
.../gobi/csvio |
Typed CSV read + streaming (ReadFileChunksFunc), gzip / zstd / bzip2 auto-detect; ReadStrings for CSVs with no struct (every column String); WriteFile / streaming NewWriter (gzip / zstd) |
.../gobi/jsonio |
JSON array or NDJSON records → Frame with lossless type inference (int64 IDs above 2^53 survive), or AllStrings |
.../gobi/parquetio |
Parquet read/write + streaming + column projection + row-group + bloom-filter tuning; snappy/gzip/brotli/zstd/lz4 + GeoParquet 1.1 |
.../gobi/geojsonio |
Full RFC 7946 GeoJSON (all geometry types + XYZ) — Frame-level ReadFile/WriteFile/ScanFile, .geojsonl streaming |
.../gobi/gpkgio |
Read / write OGC GeoPackage 1.3 (SQLite) with RTree spatial index + LazyFrame ScanFile + SQL predicate pushdown |
.../gobi/pgio |
Beta. PostgreSQL / PostGIS via pgx/v5 — ReadQuery/ReadTable/ScanTable + WriteTable with CopyFrom bulk load. Integration tests are //go:build integration-gated; set PGIO_TEST_DSN and run against a live PostGIS to exercise them. |
.../gobi/kmlio |
Read / write KML (OGC 12-007r2) + KMZ (zipped KML). Placemarks + ExtendedData. .kmz extension auto-detected. |
.../gobi/shpio |
Read / write ESRI Shapefile (.shp + .shx + .dbf + optional .prj) |
.../gobi/contrib/athenaio |
AWS Athena CTAS integration — UnloadAndRead, UnloadAndReadBuckets, RawCTAS, RawCTASBuckets return LazyFrames with PartitionMetadata claims that flow into gobi's aligned fast paths. Own go.mod; versions independently. |
gobi intentionally does not use an Arrow custom-extension type for
geometry. Geometries are Arrow Binary columns holding WKB, with the
column marked in schema metadata:
"gobi:geometry_type" = "WKB"
"gobi:crs_epsg" = "4326"
When writing Parquet, gobi additionally emits a proper GeoParquet 1.1
geo blob at the file level with primary_column, geometry_types,
crs, and bbox. That is what makes gobi-produced files interoperate
with GeoPandas and QGIS out of the box.
Use gobi.GeometryField(name, epsg) to construct a tagged field manually.
Three extension points cover most add-on work today, without forking:
1. Derived columns via a helper package + Frame.WithColumn.
Write a sibling package (e.g. h3x, hashcol) whose functions take one or
more gobi.Series and return a gobi.Series. Users compose:
lat, _ := df.Column("lat")
lng, _ := df.Column("lng")
cells, _ := h3x.Encode(lat, lng, 9)
df, _ = df.WithColumn("h3", cells)Because the helper controls the whole loop, it can dispatch to native
libraries (H3, MurmurHash, whatever) once per row without the DataFrame
having to know anything about the operation. WithColumn appends or
replaces, DropColumn removes.
2. Custom aggregations via the Aggregator interface.
type Aggregator interface {
// Reduce s[rows...] to a single scalar. Return nil for a null.
Aggregate(s Series, rows []int) (any, error)
// Declares the arrow type of Aggregate's return values. Supports
// Float32/64, Int32/64, Uint32/64, Bool, String, Binary, Timestamp.
Type() arrow.DataType
// Suffix for the default output column name.
Name() string
}Set Aggregation{Column: "col", Fn: myAgg} and call GroupBy.Agg as
usual. Mix custom + built-in aggregations in a single call. If the
returned dynamic type doesn't match the declared Type(), Agg
returns an error naming the offending aggregation rather than
panicking.
Opt into the streaming aggregate executor by additionally
implementing IncrementalAggregator:
type IncrementalAggregator interface {
Aggregator
// Clone returns a fresh instance with empty per-group state.
// Called once per group at first-touch; each group runs on its
// own clone, no cross-group state sharing.
Clone() IncrementalAggregator
// Update adds col[rows] to this instance's state. Called at
// most once per input batch per group. Additive — do not reset.
Update(col Series, rows []int) error
// Finalize returns the group's aggregated value. Called once
// per group after all Updates.
Finalize() any
}Plain Aggregators still work — they route through the materializing
fallback exactly as before. IncrementalAggregators skip the
materialize-then-reduce and process per batch, matching the built-in
aggregators' streaming shape. The bundled set aggregators
(NewStringSetAggregator, etc.) are the reference implementations.
3. Custom expression nodes via the ExprNode interface.
type ExprNode interface {
// Evaluate against a Frame. Return a Series with input.NumRows() rows.
Eval(input *Frame) (Series, error)
// Declared output arrow type given the input schema. Used by
// FilterExpr / WithColumnExpr for validation.
Type(schema *arrow.Schema) (arrow.DataType, error)
// Sub-expressions, for tree walkers.
Children() []Expr
// Pretty-printer for logs and debug output.
String() string
}Wrap your node with gobi.Custom(node) and it composes with the
built-in Col, Lit, and operator methods:
// h3x.Encode returns a gobi.Expr backed by a custom node.
cellExpr := h3x.Encode(gobi.Col("lat"), gobi.Col("lng"), 9)
df, _ = df.WithColumnExpr("h3", cellExpr)
df, _ = df.FilterExpr(cellExpr.Eq(gobi.Lit(uint64(0xdead))))Because expressions are data, not function calls, the tree can be
inspected before evaluation (e.String(), e.Node().Children()),
type-checked without touching the buffers (e.Node().Type(schema)),
and — in a future release — rewritten by an optimizer that pushes
predicates into scans and prunes unused columns. Extension points
that implement ExprNode will benefit from those passes automatically.
Group-by key types. Hashable key columns: String, Bool,
Int32, Int64, Uint32, Uint64, Float64, Timestamp.
Uint64 is what makes H3-cell grouping ergonomic.
Two layers of parallel execution work together in a LazyFrame collect: the parquet scan splits row-groups across workers, and the streaming aggregate partitions rows by key hash across workers.
- Parallel scan.
parquetio.ScanFile(path, &parquetio.ReadOptions{ScanWorkers: N})splits row-groups across N goroutines. Each worker reads a disjoint subset of row-groups; batches fan-in through a bounded channel.ScanWorkers: 0(the default) auto-picksGOMAXPROCS, capped at the file's row-group count.ScanWorkers: 1forces serial for reproducibility. Files with a single row-group skip parallel scan automatically (no benefit possible). - Parallel aggregate. The streaming hash aggregate
(
GroupBy(...).Agg(...)on a LazyFrame with built-in Kinds) partitions rows acrossGOMAXPROCSworkers by key hash — no cross-worker key overlap, no locks, no value-level combine at merge. Kicks in for any aggregate where everyAggregationuses a built-inKind; customFnaggregators still route through the materializing fallback.
Both layers respect the package-level SetMaxParallelism(n) /
per-op Workers(n) overrides, in this priority:
- Per-op
gobi.Workers(n)option - Package default via
gobi.SetMaxParallelism(n) GOMAXPROCS
gobi.SetMaxParallelism(4) // process-wide default
df.SJoin(..., gobi.Workers(8)) // override for one call
df.SJoin(..., gobi.Workers(1)) // force sequentialLazyFrame.ExplainPhysical() shows the resolved worker count for
each parallel node — useful when debugging why a query didn't get
the parallelism you expected.
- Pure Go, no cgo. GDAL, GEOS, libproj, and other C libraries are
intentionally off the table. This keeps
go buildclean across every platform Go targets and avoids the LGPL/toolchain overhead. The trade: no File Geodatabase support and no PROJ-grade reprojection beyond WGS84 / Web Mercator / UTM. Polygon boolean ops (Clip / Union / Difference / SymDifference / Dissolve) DO ship — as a pure-Go Martinez-Rueda sweepline — but complex-projection pairs like Albers ↔ Lambert still require an external tool.
All numbers Apple M3 Pro, warm cache, 10–20 iterations per op. Fixtures
and scripts live under benchmarks/ — regenerate with
go run generate_fixture.go, go run generate_csv_fixture.go, and
go run generate_spatial_fixture.go.
| Op | gobi (default) | gobi (SIMD) | pandas 2.3 | Polars 1T | Polars all |
|---|---|---|---|---|---|
Sum(value_a) |
0.98 ms | 0.37 ms | 0.31 ms | 0.10 ms | 0.09 ms |
value_a + value_b |
1.31 ms | 1.21 ms | 1.03 ms | 0.82 ms | 0.90 ms |
Filter(value_a > 500k) |
15.87 ms | 15.40 ms | 6.83 ms | 1.96 ms | 1.60 ms |
GroupBy(key).Agg(Sum,Mean) |
45.86 ms | 45.25 ms | 19.73 ms | 9.89 ms | 2.42 ms |
Polars 1T = POLARS_MAX_THREADS=1; Polars all = default (all cores).
gobi (SIMD) = compiled with GOEXPERIMENT=simd on Go 1.27+ arm64/amd64;
gobi (default) = normal build with the scalar compute path. Sum gained
a 2.4× speedup under SIMD (compute.SumF64 via simd.Float64s.Add
lane-parallel reduce). The other ops changed less because SIMD helps
compute-bound loops most and pandas/Polars' remaining lead here is
architectural (NumPy's arena allocator, Polars' cache-blocked kernels)
rather than kernel-level SIMD. See compute/ for the full
per-op arrow-go vs gobi kernel positioning.
Cross-arch SIMD note (v0.4.1): the SIMD wins in this table are on
Apple M-series (deep out-of-order execution engines). Measured on
AWS Graviton 2 / Ampere Altra (Neoverse-N-class server ARM64), the
compute.BoundsF64 kernel actually regresses ~48% because
Neoverse's NEON Float64.Min/Max instructions cost more per lane
than scalar compare-branch. Other kernels (SumF64, and the
new-in-v0.4.1 PolygonCentroidShoelace) win on both architectures
— shoelace measures −28% wall time on Ampere at every size ≥64,
while flat on Apple silicon. The rule: GOEXPERIMENT=simd is safe
to enable on server ARM64, but call compute.BoundsF64 explicitly
only when targeting Apple silicon. geometry.BoundsFromXY stays
portable scalar so it's fast everywhere. See the "Slice 6" section
in .vscode/CLAUDE.md for the arch-vs-kernel
decision table + measured numbers.
| Reader | per-read | notes |
|---|---|---|
| Polars 1.42, all threads | 11.5 ms | multi-threaded Rust tokenizer + SIMD (typed schema) |
| Polars 1.42, 1 thread | 13.5 ms | SIMD numeric parse, single core |
pandas 2.3, engine="pyarrow" |
43.2 ms | pyarrow C++ tokenizer |
| pandas 2.3, default (C engine) | 149.4 ms | pandas' native C tokenizer |
gobi csvio.Read |
224.3 ms | arrow-go's CSV wraps stdlib encoding/csv |
The gap is entirely in stdlib encoding/csv allocating a []string per
row + per-cell strconv — 99.5% of gobi's CSV allocations show up
there in a pprof run. Closing it means replacing that layer with a
byte-level tokenizer that writes straight into Arrow buffers; not on
the roadmap yet. Maybe the arrow-go folks will pick that up.
| Op | gobi | geopandas 1.1 | result |
|---|---|---|---|
| Read points.parquet (100k) | 4.04 ms | 37.74 ms | 9.3× faster |
| Read polygons.parquet (100) | 0.26 ms | 0.83 ms | 3.2× faster |
Area(polygons) |
0.02 ms | 0.13 ms | 6.5× faster |
Centroid(polygons) |
0.02 ms | 0.16 ms | 8× faster |
SJoin(100k pts, 100 polys) |
3.49 ms | 2.62 ms | 1.3× slower |
Gobi wins on read and per-row bulk ops because it doesn't have to
construct Shapely Python objects per row on load. The one gap is
sjoin: geopandas uses Shapely 2's GEOS-backed STRtree in C++; gobi's
Sort-Tile-Recursive R-tree is pure Go. Landing within 40% of a
GEOS-C++ index while staying cgo-free is the intended trade.
polars-st is polars' geometry extension, backed by geo-rust — pure Rust, cgo-free, the closer analog to gobi than shapely's GEOS-cgo path. Small / medium / large polygon fixtures at 600 / 300 / 100 rows respectively, plus a 10k LineString fixture and a 1k-polygon Dissolve fixture.
| Op | gobi | polars-st | result |
|---|---|---|---|
Area (600 small polys) |
~0 ms | 0.35 ms | tied — both instant |
Centroid (600 small polys) |
0.1 ms | 0.53 ms | gobi ~5× |
Envelope (600 small polys) |
0.2 ms | 0.74 ms | gobi ~4× |
Buffer(0.1) (600 small polys) |
1.1 ms | 8.3 ms | gobi ~8× |
Simplify(0.01) (600 small) |
0.3 ms | 3.7 ms | gobi ~12× |
ConvexHull (600 small polys) |
0.5 ms | 1.4 ms | gobi ~3× |
Area (300 medium polys) |
0.1 ms | 0.32 ms | gobi ~3× |
Buffer(0.1) (300 medium) |
3.5 ms | 54.5 ms | gobi ~16× |
Simplify(0.01) (300 medium) |
0.9 ms | 10.8 ms | gobi ~12× |
ConvexHull (300 medium polys) |
1.5 ms | 2.4 ms | gobi ~1.6× |
Buffer(0.1) (100 large polys) |
19 ms | 1389 ms | gobi ~72× |
Simplify(0.01) (100 large) |
7.1 ms | 99.8 ms | gobi ~14× |
ConvexHull (100 large polys) |
11.1 ms | 6.6 ms | polars-st ~1.7× |
Length (10k LineStrings) |
1.1 ms | 3.1 ms | gobi ~3× |
Simplify (10k LineStrings) |
12.2 ms | 161.9 ms | gobi ~13× |
union_all (1k polys) |
17 s | 271 s | gobi ~16× |
The wins cluster around Buffer, Simplify, and union_all — the
ops where v0.4.1's SoA push moved the whole hot loop off per-row
[]Point allocation. Buffer scales especially well on large
polygons (72× at the large-polygon fixture). The one loss —
ConvexHull on the large-polygon fixture — flips at 100 polys ×
large vertex count: gobi's Andrew's monotone chain wins at ≤ 300
polys but trails polars-st on the largest shape. Length and
Simplify on 10k LineStrings show the SoA slab kernels
(SimplifyDPFromXY, LineStringViewFromWKB) working end-to-end
without a []Point intermediate. union_all at 1k polys — 17 s
vs 271 s — reflects the parallel divide-and-conquer dissolve merge
composed with Martinez-Rueda's convex-containment fast paths
cascading through the merge tree.
Same 1M-row parquet fixture, Select(id, value_a) — reading 2 of 4
columns. Measures the projection-pushdown rule's effect on I/O and
decode cost.
| Path | per-op | vs. baseline |
|---|---|---|
ReadFile(path, Options{Columns:[id,value_a]}) (eager) |
6.21 ms | 1.0× (baseline) |
ScanFile(path).Select(id,value_a).Collect() (optimized) |
8.12 ms | ~1.31× — close to eager |
ScanFile(path).Select(id,value_a).CollectRaw() (no rules) |
14.58 ms | 2.3× slower — reads all 4 cols |
The optimizer's ProjectionPushdown rule turns the lazy pipeline
into the equivalent of an eager Options.Columns — 2.0× faster than
the same pipeline with optimization disabled. Optimizer overhead
itself is 8 µs per plan (measured on a 5-node pipeline), or
0.14% of the collect time. Always-on optimization is effectively
free.
The 1BRC fixture is 1 billion
weather-station rows in Snappy-compressed parquet (~4 GB on disk).
Query: min / mean / max of temperature grouped by station. Apple
M3 Pro, 11 GOMAXPROCS.
| Engine | wall | user CPU | peak RSS |
|---|---|---|---|
| gobi streaming (parallel scan + agg) | 14.0s | ~93s | 643 MB |
| Polars 1.42 streaming (reference) | 3.0 s | ~15s | 4.42 GB |
| Polars 1.42 eager (reference) | 12.0 s | ~120s | 20.96 GB |
Streaming end-to-end — no LazyFrame.CollectRaw() materialization,
no disk spill. Getting here took six complementary changes:
partitioning the parquet scan across row-groups; sharding the
streaming hash aggregate by key hash across workers; eliminating
per-row key allocations (reusable scratch buffers +
single-string-key fast path that reads the arrow value zero-copy);
eliminating the per-row map[*aggGroup][]int bucket lookup that
sat between groups[key] and the accumulator Update calls
(v0.3.9); pooling the parallel dispatcher's per-worker []int
row-index payloads via sync.Pool (v0.3.9); and threading the
pooled payloads as *[]int through the channel path so
sync.Pool.Put doesn't box the slice header into an interface
per call (v0.3.9). gobi's own allocations across a full 1BRC run
are ~540 MB total — down from ~13 GB pre-pool.
Peak RSS is 6.9× lower than Polars streaming and 33× lower than
Polars eager because gobi keeps at most one batch per worker in
memory + recycles per-batch scratch through sync.Pool — Polars
buffers larger working sets by design.
When the caller can prove input partitioning + sort order via
WithPartitionMeta(...), Over and GroupBy skip the general
hash-map path and linear-scan group boundaries directly. Same
bench_alignment.parquet fixture (1M rows, 10 regions × 100 ids,
pre-sorted by (region, id)) across all three rows; Polars doesn't
expose alignment metadata, so it picks its own path.
| Op | gobi (unaligned) | gobi (aligned) | Polars all |
|---|---|---|---|
Over(region, id).Sum(v) |
47.8 ms | 29.9 ms (37% faster) | 7.11 ms |
GroupBy(region, id).Sum(v) |
94.6 ms | 22.1 ms (77% faster) | 4.65 ms |
The aligned GroupBy number improved from 32.4 ms → 22.1 ms in v0.3.9
(vs the v0.2.0 landing) via two complementary changes: the streaming-
aggregate refactor (buckets-map elimination + sync.Pool for per-worker
row-index payloads, -32% overall) and the aligned-path SIMD reduce
wire-in (compute.SumF64 on contiguous per-group slices under
GOEXPERIMENT=simd).
Sort-merge Inner join (also alignment-gated) is measured in-tree
rather than in this fixture bench — a fixture-scale self-join is
dominated by 100M-row cross-product construction rather than the
join algorithm itself. BenchmarkJoin_MergeAligned on 10k×10k Int64
keys shows the aligned sort-merge path is 31% faster than the hash
join it replaces, and BenchmarkJoin_HashMultiProbeBatch shows the
streaming-hash-join build-side cache fix that landed with the
alignment work is 49% faster on multi-batch Inner joins.
Polars still wins in absolute terms — the aligned fast paths close a 1.6× gap on Over and cut a 3.3× gap on GroupBy to 4.8× overall. If the workload can carry alignment metadata, taking it is nearly free.
Two write-path measurements on a 100k-row 9-vertex-polygon Frame,
before/after the Slice 2 + Slice 3 wire-ins (BoundsFromWKB,
CentroidAndBoundsFromWKB). No config change required — the
scanners are the default path on every build.
| Path | before (v0.4.0) | after (v0.4.1) | delta |
|---|---|---|---|
WithBboxCoveringColumns (write) |
31.7 ms / 154 MB / 600k allocs | 21.9 ms / 81 MB / 300k allocs | −31% wall, −47% mem |
SortByHilbert |
33.6 ms / 115.6 MB / 300k allocs | 25.5 ms / 42.9 MB / 66 allocs | −24% wall, −63% mem, essentially zero-alloc pass |
HilbertSortWithCovering |
51.1 ms / 203.5 MB / 600k allocs | 41.8 ms / 131 MB / 300k allocs | −18% wall, −36% mem |
The wins come from skipping the ParseWKB full-geometry
allocation on every row when the caller immediately discards the
geometry after .Bounds() / .Centroid(). Byte-stream scanners
walk the WKB blob tracking running accumulators — zero allocation
per row on well-formed input.
Slice 7 extends the same shape to Series.GeomArea /
Series.GeomLength on projected CRSes via PlanarAreaFromWKB /
PlanarLengthFromWKB. Microbench on Apple M3 Pro:
| Path | n | ParseWKB + AoS | Scanner | delta wall | allocs/op |
|---|---|---|---|---|---|
PlanarLengthFromWKB |
1M | 12.2 ms | 4.0 ms | −67% | 2 → 0 |
PlanarLengthFromWKB |
1K | 14.5 μs | 3.4 μs | −76% | 2 → 0 |
PlanarAreaFromWKB |
1M | 6.6 ms | 3.5 ms | −48% | 3 → 0 |
PlanarAreaFromWKB |
1K | 8.9 μs | 3.5 μs | −61% | 3 → 0 |
Geographic CRSes still take the AoS haversine / spherical-excess path (the scanner has no CRS context from the WKB blob alone).
Slice 9 replaces the recursive AoS douglasPeucker with an
iterative SoA form (SimplifyDPFromXY /
PointsView.SimplifyDP). Every Simplify entry point routes
through the shared internal helper, so LineString.Simplify /
Polygon.Simplify / MultiLineString.Simplify /
MultiPolygon.Simplify / series.GeomSimplify all benefit
transparently. Wiggly-sine polyline benchmark, Apple M3 Pro:
Path (Simplify) |
n | AoS recursive | SoA iterative | delta wall | allocs/op |
|---|---|---|---|---|---|
SimplifyDP |
64 | 1.80 μs / 17 allocs | 302 ns / 3 allocs | −83% | 17 → 3 |
SimplifyDP |
1K | 126 μs / 259 allocs | 25 μs / 4 allocs | −80% | 259 → 4 |
SimplifyDP |
64K | 81.5 ms / 17k / 122 MB | 16.7 ms / 8 / 242 KB | −80% | 17k → 8 |
SimplifyDP |
1M | 3.85 s / 260k / 5.75 GB | 0.75 s / 11 / 3.2 MB | −81% | 260k → 11 |
The 24k× allocation reduction on 1M-vertex input is the killer
metric — the AoS recursion appends O(log n) intermediate []Point
slices at every split, which the SoA form skips entirely with a
retain-bitmap + argmax-of-cross-squared inner loop that defers
the per-split sqrt to a single compare.
Slice 10 closes the remaining AoS gap in the PreparedGeometry
build path. Before: Prepare(ParseWKB(data)) walked the WKB
bytes to []Point, then walked []Point to []float64 slabs —
two full O(n) passes and two allocations per ring. After:
PolygonRingViewsFromWKB / PrepareFromWKB walk the WKB bytes
once into slabs. On a 1M-vertex polygon (Apple M3 Pro):
| Path | ParseWKB + RingViews |
direct parse | delta wall | delta mem |
|---|---|---|---|---|
PolygonRingViewsFromWKB |
8.4 ms / 80 MB / 6a | 3.7 ms / 16 MB / 3a | −56% | −80% |
PrepareFromWKB (retains AoS G) |
9.5 ms / 80 MB / 6a | 7.4 ms / 80 MB / 6a | −22% | flat |
The PrepareFromWKB gap is smaller because it still materializes
the AoS Polygon for TestPrepared's non-fast-path fall-through
contract — but that materialization is a cheap slab-to-Point
copy, not a second WKB byte walk. Advanced callers wanting the
full slab-only win can build PreparedGeometry directly with
PolygonRingViewsFromWKB.
Slice 11 targets a different hot loop: planarMinDistance, the
internal driver behind GeomDistance and WithinDistance. The
pre-Slice-11 shape walked geometries via forEachVertex /
forEachSegment closures and called math.Hypot per (vertex,
segment) pair — for a Polygon×Polygon distance at n=1024 that's
~1M Hypot calls (each doing a sqrt) per row. The rewrite
extracts each input's polylines into slab form once, then tracks
running-min squared distance on flat coordinate arrays; the
final sqrt runs exactly once at the end.
| n | Legacy AoS (closure + per-pair Hypot) | Slab-form SoA | delta wall |
|---|---|---|---|
| 16 | 6.9 μs | 1.7 μs | −76% |
| 64 | 106 μs | 21 μs | −80% |
| 256 | 1.60 ms | 272 μs | −83% |
| 1024 | 28.8 ms | 5.3 ms | −82% |
A geospatial analytics workload: bbox-filter 2M (lat, lon, eid, dt, h3_res8) rows down to ~500K survivors, group by H3 cell (~39K
groups), and aggregate count, n_unique(eid), min(dt),
max(dt). The h3-agg bench harness lives at
benchmarks/gobi/gobi_h3_agg_lazy_bench.go
benchmarks/python/polars_h3_agg_bench.py— precomputed h3 cells so every engine measures the same aggregation throughput, not h3 encoding.
| Engine | per-iter wall | peak RSS |
|---|---|---|
| Polars 1.42 streaming | 15 ms | 341 MB |
| gobi (v0.3.9) | 86 ms | 501 MB |
| pandas 2.3 | 86 ms | 792 MB |
| gobi (v0.3.8, pre-session) | 152 ms | ~500 MB |
gobi (v0.3.9) matches pandas within measurement noise on wall
time and uses 36% less peak RSS than pandas. Getting here took
five stacked changes that each earn their entry in
CHANGELOG.md:
- Filter fusion —
Col(a) OP lit AND Col(b) OP lit AND …evaluates in a single row-loop with short-circuit on first false; no intermediate Boolean Series per comparison. - Scalar-cmp fast path for
Ne/Le/Ge— the pre-existingbinOpNode.Evalshortcut only fired onEq/Lt/Gt. - Null-check hoisting in the fused filter — when every leaf's
column has
NullN() == 0(typical for numeric parquet), the per-row × per-leafIsNull()calls vanish. aggFast(fast-path GroupBy) coversAggNUnique+ Timestamp Min/Max — previously "one bad agg disabled the whole call."- Streaming
nUniqueAcctype-specialization — LazyFrame groupby-with-nunique dispatches to nativemap[string]/[int64]/ [float64]instead of byte-encoded keys.
Polars is still 5.7× faster wall-time on this shape. That gap is architectural (cache-blocked kernels, arena allocator, compiled expression graph) rather than kernel-level and won't close without a rewrite of the same magnitude. gobi's differentiator here is matching pandas on throughput while using less memory than pandas — on this specific workload Polars beats gobi on both wall and RSS, because Polars streams the parquet decode straight into its grouper without our intermediate LazyFrame batches. The RSS story inverts at the 1BRC scale below.
Sum / Add are already memory-bandwidth-bound. The remaining
gap on Sum is SIMD reduction (Polars and numpy both use
parallel-lane accumulators). Go 1.27 shipped the portable simd
stdlib package with arm64 NEON + amd64 AVX2/AVX-512 backends under
GOEXPERIMENT=simd, and the compute/ subpackage plus
series_ops_simd.go (v0.4.0) ship SIMD kernels behind
//go:build goexperiment.simd && (arm64 || amd64). Slices 22–23
extended the wire-in past the raw kernels: Series.GtScalar /
LtScalar / Ge / Le now dispatch to compute.CmpF64* /
CmpI64* directly, and the expression executor recognizes
AND-chained range and bbox filters and dispatches to fused
kernels (compute.AndChainF64Range / AndChainF64BBox) that
skip intermediate boolean-column materialization — measured
−49% wall, −71% allocs on a 100k-row 4-comparison bbox
filter. Slice 23 also added a lane-count gate on the compare
kernels that fixed a pre-existing Apple 2-lane NEON regression
(SIMD build now matches scalar build on Apple, wins expected on
amd64 AVX2 / AVX-512).
The compute-ops table above shows where that leaves single-op
performance today: Sum in the SIMD build is 2.6× faster than
the scalar build; Add is 8% faster; Filter and GroupBy are
within 3%, because those two are aggregate-consume-bound rather
than kernel-bound and the streaming aggregate's Update methods
run per-group scalar loops that don't compose cleanly with the
lane-parallel reducers (see the negative-result note below).
The remaining ~4.7× 1BRC gap vs Polars streaming (14.0s vs 3.0s) breaks down as follows, from a fresh CPU profile against current HEAD (Apple M3 Pro, 11 GOMAXPROCS, 84.85s total across workers):
- ~42% runtime scheduling / worker parking
(
pthread_cond_wait20.6%,usleep11.6%,pthread_cond_signal9.6%). Workers finish batches faster than the parquet decoder can feed them — a scheduling-shape issue, not raw compute cost. Cutting either the map probe or the parquet decode below shifts this proportionally. - ~18% arrow-go parquet decode (
pqarrow.NextBatchcum;s2Decodesnappy 2.1%). Not our code; a custom pooledmemory.Allocatorwrapping arrow-go's decoder is a scoped-out future win. - ~15% string-keyed map probe (
mapaccess2_faststr5.9% flat +memHashAES4.8% +dispatchBatch3.0% +fnvHashString11.5%). Fundamental hash-by-string cost on 413 distinct station keys × 1B rows; addressable with a specialized Robin-Hood / Swiss-table impl but the generalmap[string]is fast enough that the win would be modest. - ~10-13% GC (
scanObjectsSmall3.9%,tryDeferToSpanScan2.1%). Down from ~14% in the v0.3.9-era profile, reflecting the SoA push's allocation cuts. - ~10% aggregate consume (
minMaxAcc.updateFloat641.6%,meanAcc.Update1.6%,minMaxAcc.update1.6%). Tight typed- slice loops; the SIMD-reduce wire-in that helps the aligned path ingroupby_aligned.gowas tried here and regressed (see the negative result below). - ~10% syscalls / memory ops (
rawsyscalln6.2%,memmove2.4%,madvise1.2%). Kernel accounting for arrow buffer allocation.
Wall-time gains from here fall into two levers:
- Custom decoder-buffer allocator to cut arrow-go's parquet
decode share. Pooled
memory.Allocatorwrapping the decoder path; needs upstream changes to arrow-go, so timeline depends on that landing first. - Specialized string-hash map for the groupby probe. Swiss-table variant tuned to gobi's arena-scoped key lifetime. ~5-8% wall on 1BRC-shaped workloads.
An earlier version of this section proposed a third lever — wiring
compute.SumF64 / MinF64 / MaxF64 into the streaming
aggregate's Update methods, mirroring what groupby_aligned.go
already does. That was attempted and measured: on Apple M3 Pro
(2-lane NEON) it regressed 1BRC wall time by ~10%. The
streaming aggregate's Update receives a scatter (group's row
indices within the batch, non-contiguous), so using the SIMD
reduce requires a gather step to materialize a contiguous slab.
At the typical 1BRC per-group-per-batch cardinality (~220 rows /
group), the gather cost plus the fixed per-call overhead of
compute.SumF64 exceeded the 2-lane NEON reduce speedup.
The pattern may still pay on 8-lane AVX-512 amd64, but the code
complexity isn't worth carrying for a hardware-conditional win —
so the streaming aggregate stays on its scalar per-row loop, and
this section documents the negative result so future readers
don't retry the same shape.
None are on a specific timeline; the v0.3.9 wins already put
gobi solidly ahead of pandas on wall time (matched exactly on
h3-agg, substantially ahead on memory), and the Slice 22–23
compare-kernel wire-in landed a substantial fused-filter gain
on Filter-heavy shapes. gobi's peak RSS at 1BRC scale is
still 6.9× lower than Polars streaming and 32× lower than
Polars eager — the memory story remains the differentiator
even where wall time trails.
go test -race ./...should pass before you push.- Please keep dependencies minimal — Arrow is the one big one on purpose.
MIT. See LICENSE.