8000 GitHub - kasapdev/go-csvkit: CSV diffing and pivot/aggregation utilities built on Go's encoding/csv. Zero-dependency Go stdlib-only library. · GitHub
[go: up one dir, main page]

Skip to content
kasapdevPublic

About

CSV diffing and pivot/aggregation utilities built on Go's encoding/csv. Zero-dependency Go stdlib-only library.

Topics

Resources

Stars

0 stars

Watchers

0 watching

Forks

Latest commit

 

History

1 Commit

Folders and files

Repository files navigation

go-csvkit

Diffing and pivot/aggregation utilities for CSV-shaped data in Go, built entirely on the standard library's encoding/csv. Zero third-party dependencies.

go-csvkit does not parse CSV itself -- encoding/csv already does that well. Instead it operates on the [][]string shape returned by csv.Reader.ReadAll, adding two things the standard library doesn't give you:

  • DiffRows -- compare two CSV-shaped datasets (e.g. yesterday's export vs. today's) and get back which rows were added, removed, or changed, including exactly which columns changed.
  • Pivot -- group rows by one column and sum a numeric column, e.g. total sales amount per region.

Installation

go get github.com/kasapdev/go-csvkit

Usage

package main

import (
	"encoding/csv"
	"fmt"
	"strings"

	"github.com/kasapdev/go-csvkit"
)

func mustReadCSV(s string) [][]string {
	rows, err := csv.NewReader(strings.NewReader(s)).ReadAll()
	if err != nil {
		panic(err)
	}
	return rows
}

func main() {
	before := mustReadCSV(`id,name,department,salary
1,Alice,Engineering,90000
2,Bob,Sales,60000
3,Carol,Marketing,70000
`)

	after := mustReadCSV(`id,name,department,salary
1,Alice,Engineering,95000
2,Bob,Sales,60000
4,Dave,Support,50000
`)

	// DiffRows expects both datasets to include a header row as element 0,
	// and keyColumn is the index of the column that uniquely identifies a
	// row (here, "id" at index 0).
	diff, err := csvkit.DiffRows(before, after, 0)
	if err != nil {
		panic(err)
	}

	fmt.Println("Added:")
	for _, row := range diff.Added {
		fmt.Println(" ", row)
	}

	fmt.Println("Removed:")
	for _, row := range diff.Removed {
		fmt.Println(" ", row)
	}

	fmt.Println("Changed:")
	for _, c := range diff.Changed {
		fmt.Printf("  key=%s\n", c.Key)
		for _, col := range c.ChangedColumns {
			fmt.Printf("    %s: %q -> %q\n", col.Column, col.Old, col.New)
		}
	}

	// Pivot groups sales rows by region and sums the amount column.
	sales := mustReadCSV(`region,amount
North,100
South,50
North,25
East,75
South,10
`)

	totals, err := csvkit.Pivot(sales[1:], sales[0], "region", "amount")
	if err != nil {
		panic(err)
	}

	fmt.Println("Totals by region:")
	fmt.Printf("  North: %.2f\n", totals["North"])
	fmt.Printf("  South: %.2f\n", totals["South"])
	fmt.Printf("  East: %.2f\n", totals["East"])
}

Expected output:

Added:
  [4 Dave Support 50000]
Removed:
  [3 Carol Marketing 70000]
Changed:
  key=1
    salary: "90000" -> "95000"
Totals by region:
  North: 125.00
  South: 60.00
  East: 75.00

API

func DiffRows(a, b [][]string, keyColumn int) (RowDiff, error)

Compares two CSV-shaped datasets, a ("before") and b ("after"), and reports which rows were added, removed, or changed.

Both a and b must include a header row as element 0, followed by zero or more data rows. keyColumn is the index (into each data row) of the column that uniquely identifies a row (e.g. an id or sku column). The header row itself is not compared as data; the header of a (or, if a has no header, the header of b) is used only to label ChangedColumns by name.

If a dataset contains multiple rows with the same key, the last such row (in input order) wins.

Returns an error if keyColumn is negative, or if any data row is too short to contain keyColumn.

type ColumnChange struct {
	Column string // column name, from the header row
	Index  int    // zero-based column index
	Old    string
	New    string
}

type RowChange struct {
	Key            string
	Old            []string
	New            []string
	ChangedColumns []ColumnChange
}

type RowDiff struct {
	Added   [][]string  // rows whose key is only in b
	Removed [][]string  // rows whose key is only in a
	Changed []RowChange // rows whose key is in both, with differing values
}

func Pivot(rows [][]string, header []string, groupBy, valueCol string) (map[string]float64, error)

Groups rows by the value in the groupBy column and sums the numeric values in the valueCol column, returning a map from each distinct groupBy value to the total sum of valueCol across all rows in that group.

rows holds data rows only (no header row). header supplies the column names for rows, positionally; groupBy and valueCol are column names looked up within header.

Returns an error, rather than silently skipping bad input, if:

  • groupBy or valueCol is not present in header;
  • a row does not have enough columns to contain groupBy or valueCol;
  • a row's valueCol value cannot be parsed as a float64 via strconv.ParseFloat (surrounding whitespace is trimmed first).

Testing

go test ./...

License

MIT -- see LICENSE.

About

CSV diffing and pivot/aggregation utilities built on Go's encoding/csv. Zero-dependency Go stdlib-only library.

Topics

Resources

Stars

0 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages

0