A Blazingly fast diff tool for comparing csv files.
Csvdiff is a difftool to compute changes between two csv files.
- It is not a traditional diff tool. It is most suitable for comparing csv files dumped from database tables. GNU diff tool is orders of magnitude faster on comparing line by line.
- Supports specifying group of columns as primary-key.
- Supports selective comparison of fields in a row.
- Compares csvs of million records csv in under 2 seconds. Comparisons and benchmarks here.
$ csvdiff base.csv delta.csv
# Additions: 1
# Modifications: 20
# Rows:
...# binary will be $GOPATH/bin/csvdiff
curl -sfL https://raw.githubusercontent.com/aswinkarthik/csvdiff/master/install.sh | sh -s -- -b $GOPATH/bin
# or install it into ./bin/
curl -sfL https://raw.githubusercontent.com/aswinkarthik/csvdiff/master/install.sh | sh -s
# In alpine linux (as it does not come with curl by default)
wget -O - -q https://raw.githubusercontent.com/aswinkarthik/csvdiff/master/install.sh | sh -sgo get -u github.com/aswinkarthik/csvdiff- Cases where you have a base database dump as csv. If you receive the changes as another database dump as csv, this tool can be used to figure out what are the additions and modifications to the original database dump. The
additions.csvcan be used to create aninsert.sqland with themodifications.csvanupdate.sqldata migration. - The delta file can either contain just the changes or the entire table dump along with the changes.
- Additions
- Modifications
- Deletions
- Non comma separators
- Cannot be used as a generic difftool. Requires a column to be used as a primary key from the csv.
- By default, it marks the row as ADDED or MODIFIED by introducing a new column at last.
% csvdiff examples/base-small.csv examples/delta-small.csv
Additions 1
Modifications 1
Rows:
24564,907,completely-newsite.com,com,19827,32902,completely-newsite.com,com,1621,909,19787,32822,ADDED
69,1048,aol.com,com,97543,225532,aol.com,com,70,49,97328,224491,MODIFIED- The
--primary-keyin an integer array. Specify comma separated positions if the table has a compound key. Using this primary key, it can figure out modifications. If the primary key changes, it is an addition.
% csvdiff base.csv delta.csv --primary-key 0,1- If you want to compare only few columns in the csv when computing hash,
% csvdiff base.csv delta.csv --primary-key 0,1 --columns 2- Supports JSON format for post processing
% csvdiff examples/base-small.csv examples/delta-small.csv --format json
{
"Additions": [
"24564,907,completely-newsite.com,com,19827,32902,completely-newsite.com,com,1621,909,19787,32822"
],
"Modifications": [
"69,1048,aol.com,com,97543,225532,aol.com,com,70,49,97328,224491"
]
}$ git clone https://github.com/aswinkarthik/csvdiff
$ go get ./...
$ go build
# To run tests
$ go get github.com/stretchr/testify/assert
$ go test -v ./...- Creates a map of <uint64, uint64> for both base and delta file
keyis a hash of the primary key values as csvvalueis a hash of the entire row
- Two maps as initial processing output
- base-map
- delta-map
- The delta map is compared with the base map. As long as primary key is unchanged, they row will have same
key. An entry in delta map is a- Addition, if the base-map's does not have a
value. - Modification, if the base-map's
valueis different.
- Addition, if the base-map's does not have a
- Uses 64 bit xxHash algorithm, an extremely fast non-cryptographic hash algorithm, for creating the hash. Implementations from cespare
- Used Majestic million data for demo.
Benchmark tests can be found here.