Diffing and pivot/aggregation utilities for CSV-shaped data in Go, built entirely
on the standard library's encoding/csv. Zero third-party dependencies.
go-csvkit does not parse CSV itself -- encoding/csv already does that well.
Instead it operates on the [][]string shape returned by csv.Reader.ReadAll,
adding two things the standard library doesn't give you:
DiffRows-- compare two CSV-shaped datasets (e.g. yesterday's export vs. today's) and get back which rows were added, removed, or changed, including exactly which columns changed.Pivot-- group rows by one column and sum a numeric column, e.g. total sales amount per region.
go get github.com/kasapdev/go-csvkit
package main
import (
"encoding/csv"
"fmt"
"strings"
"github.com/kasapdev/go-csvkit"
)
func mustReadCSV(s string) [][]string {
rows, err := csv.NewReader(strings.NewReader(s)).ReadAll()
if err != nil {
panic(err)
}
return rows
}
func main() {
before := mustReadCSV(`id,name,department,salary
1,Alice,Engineering,90000
2,Bob,Sales,60000
3,Carol,Marketing,70000
`)
after := mustReadCSV(`id,name,department,salary
1,Alice,Engineering,95000
2,Bob,Sales,60000
4,Dave,Support,50000
`)
// DiffRows expects both datasets to include a header row as element 0,
// and keyColumn is the index of the column that uniquely identifies a
// row (here, "id" at index 0).
diff, err := csvkit.DiffRows(before, after, 0)
if err != nil {
panic(err)
}
fmt.Println("Added:")
for _, row := range diff.Added {
fmt.Println(" ", row)
}
fmt.Println("Removed:")
for _, row := range diff.Removed {
fmt.Println(" ", row)
}
fmt.Println("Changed:")
for _, c := range diff.Changed {
fmt.Printf(" key=%s\n", c.Key)
for _, col := range c.ChangedColumns {
fmt.Printf(" %s: %q -> %q\n", col.Column, col.Old, col.New)
}
}
// Pivot groups sales rows by region and sums the amount column.
sales := mustReadCSV(`region,amount
North,100
South,50
North,25
East,75
South,10
`)
totals, err := csvkit.Pivot(sales[1:], sales[0], "region", "amount")
if err != nil {
panic(err)
}
fmt.Println("Totals by region:")
fmt.Printf(" North: %.2f\n", totals["North"])
fmt.Printf(" South: %.2f\n", totals["South"])
fmt.Printf(" East: %.2f\n", totals["East"])
}Expected output:
Added:
[4 Dave Support 50000]
Removed:
[3 Carol Marketing 70000]
Changed:
key=1
salary: "90000" -> "95000"
Totals by region:
North: 125.00
South: 60.00
East: 75.00
Compares two CSV-shaped datasets, a ("before") and b ("after"), and reports
which rows were added, removed, or changed.
Both a and b must include a header row as element 0, followed by zero or
more data rows. keyColumn is the index (into each data row) of the column
that uniquely identifies a row (e.g. an id or sku column). The header row
itself is not compared as data; the header of a (or, if a has no header,
the header of b) is used only to label ChangedColumns by name.
If a dataset contains multiple rows with the same key, the last such row (in input order) wins.
Returns an error if keyColumn is negative, or if any data row is too short
to contain keyColumn.
type ColumnChange struct {
Column string // column name, from the header row
Index int // zero-based column index
Old string
New string
}
type RowChange struct {
Key string
Old []string
New []string
ChangedColumns []ColumnChange
}
type RowDiff struct {
Added [][]string // rows whose key is only in b
Removed [][]string // rows whose key is only in a
Changed []RowChange // rows whose key is in both, with differing values
}Groups rows by the value in the groupBy column and sums the numeric values
in the valueCol column, returning a map from each distinct groupBy value to
the total sum of valueCol across all rows in that group.
rows holds data rows only (no header row). header supplies the column
names for rows, positionally; groupBy and valueCol are column names
looked up within header.
Returns an error, rather than silently skipping bad input, if:
groupByorvalueColis not present inheader;- a row does not have enough columns to contain
groupByorvalueCol; - a row's
valueColvalue cannot be parsed as afloat64viastrconv.ParseFloat(surrounding whitespace is trimmed first).
go test ./...
MIT -- see LICENSE.