A way to retrieve large Hive results from Qubole Data Service (QDS), made for clients of Interactive Query.
To get Python, we recommend either the Anaconda Python Shell or Cygwin. If using Cygwin, you may first need to run the command easy_install pip before installing this project.
Windows-only Anaconda Python Shell step by step directions:
- Please download the Anaconda Python Shell.
- Next, copy iq-get from your local Git (typically in your …\GitHub\iq-get\bin) folder and save it to the Anaconda directory (this is typically in C:\\Users\<username>\AppData\Local\Continuum\Anaconda, with <username> replaced with your username. NOTE: This folder will not exist until Anaconda has been installed).
- Open up the Anac 6B27 onda shell and set the shell working directory to the Anacondo directory containing the iq-get script that you placed in step 2. Set the shell working directory by entering: cd C:\\Users\<username>\AppData\Local\Continuum\Anaconda (or the appropriate path locating your Anconda directory).
- Enter the command
pip install qds_sdkand wait for the process to finish. - To run the Python program, enter:
python iq-get <qubole_api_token> <query_ID>
IQ-Get is available on PyPI.
pip install iq-get
Download the source code:
git clone git@github.com:MediaMath/iq-get.gitRun the following commands (installing the project may require root)
$ cd iq-get $ python setup.py install
This is a command-line utility and can be used as follows:
$ iq-get -h
| usage: iq-get [-h] [-o OUTPUT] [-d DELIMITER] Token Query_ID
|
| positional arguments:
| Token API token of the account from which this query ran
| Query_ID The ID of the query whose results you wish to download
|
| optional arguments:
| -h, --help show this help message and exit
| -o OUTPUT, --output OUTPUT
| The name of the file you wish to write to test
| -d DELIMITER, --delimiter DELIMITER
| Custom delimiter you would like to use
$ iq-get QDS_API_TOKEN 1234567 # Will write a comma-delimited file with results of query with ID 1234567 to ~/Desktop/full_result_1234567.csv
$ iq-get QDS_API_TOKEN 1234567 -o my_filename.csv # Will write a comma-delimited file with results of query with ID 1234567 to ~/Desktop/my_filename.csv
$ iq-get QDS_API_TOKEN 1234567 -d $'\t' # Will write a tab-delimited file with results of query with ID 1234567 to ~/Desktop/full_result_1234567.tsv