jq-101

JQ-101: parsing Twitter data

Using data often requires parsing json format. JQ makes this easy.

This tutorial is stored in a github repository so we'll need to complete the follownig prerequisites:

Please login to github.com/login, visit this repo, and click "Fork" in the upper right hand conner to continue this tutorial.

Now open the github application on your local machine and clone data_manipulation on to your desktop:

Choose the location:

You now have the tutorial on your desktop.

note: Github has a great keygen explanation. You should eventually learn about such things if you want to access the many unitiles available on github.

The easiest way that I've found to install jq is via homebrew. So open a terminal and run the following:

ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"

Now we can install jq with this simple command.

brew install jq

Example results from Gnip's search api are included in this directory under armour.json.

First make sure that you cd into the location with the data.

cd ~/Desktop/data_manipulation/jq-101

Let's simply run the jq statments below and see what we can learn about this dataset.

run jq . armour.json to see all of the data parsed.
run jq .results[] armour.json to see all of the results from Gnip's search.
run jq .results[1] armour.json to see the first result from Gnip's search.
run jq .results[].actor armour.json to see the actor field from each result.
run jq .results[].actor.id armour.json to see each actor id field as a string.
run jq .results[].actor.id[15:] armour.json to see all characters after the 15th character of each id field.
run jq .results[].actor.id[15:] armour.json | sed 's/"//g' | sed 's/^/from:/g' to see a list of rules that you could place into Gnip's powertrack.
run jq .results[].actor.id[15:] armour.json | sed 's/"//g' | sort | uniq -c to see counts of the user ids.
run jq .results[].actor.id[15:] armour.json | sed 's/"//g' | sort | uniq -c | sort -nr to see counts of the user ids sorted numerically in reverse order.
run jq .results[].actor.id[15:] armour.json | sed 's/"//g' | sort | uniq -c | sort -nr | sed 's/ *$//g' | sed 's/^ *//g' | sed 's/ /,/g' to see counts of the user ids sorted numerically in reverse order as a valid csv.
run jq .results[].actor.id[15:] armour.json | sed 's/"//g' | sort | uniq -c | sort -nr | sed 's/ *$//g' | sed 's/^ *//g' | sed 's/ /,/g' > my_results.csv to save the csv.

Now, let's open my_restuls.csv:

open my_results.csv

What can these results tell you about this dataset?