Introduction | Data Sources | Sample Images | Code | Filter Responses | Learning Curve | Acknowledgements

This repository accompanies our publication "Multi-Task Deep Learning for Predicting Poverty From Satellite Images" (AAAI Conference on Innovative Applications of Artificial Intelligence: IAAI 2018), and contains the code and model weights for two prediction tasks:

1. Predict, using a multi-task fully convolutional deep neural network (PNG, JSON, Weights), three developmental parameters -- the main material of the roof, source of lighting and source of drinking water -- from satellite imagery.

   Following are the categories for each of the three tasks in the multi-task model. The multi-task model outputs 24 values (9 for roof type, 6 for source of lighting and 9 for source of drinking water) as three probability distributions, one distribution per task.

   #	Roof Type	Lighting Source	Drinking Water Source
   1	Grass/thatch/bamboo/wood/mud	Electricity	Treated tap-water
   2	Plastic/polythene	Kerosene	Untreated tap-water
   3	Hand made tiles	Solar energy	Covered well
   4	Machine made tiles	Other oil	Uncovered well
   5	Burnt brick	Any other	Handpump
   6	Stone/slate	No lighting	Tubewell/borehole
   7	G.I./metal/asbestos		River/canal
   8	Concrete		Tank/pond/lake
   9	Any other material		Other source

2. Predict, using a simple four-layer fully-connected neural network (PNG, JSON, Weights), the income levels (a direct indicator of poverty) using the predicted developmental parameter outputs of the first (multi-task) model -- model P.D., trained on predicted data.

3. Predict, using a simple four-layer fully-connected neural network (PNG, JSON, Weights), the income levels using the actual developmental parameter values -- model C.D., trained on census data.

We obtained the Census of India (2011) data from these websites:

1. Primary Census Abstract Data Tables: http://censusindia.gov.in/pca/pcadata/pca.html
2. Percentage of Households to Total Households by Amenities and Assets: http://www.censusindia.gov.in/2011census/HLO/HL_PCA/Houselisting-housing-HLPCA.html
3. Socio-Economic and Caste Census: http://www.secc.gov.in/statewiseEmploymentAndIncomeReport?reportType=Employment%20and%20Income

We utilized Google's Geocoding API (https://developers.google.com/maps/documentation/geocoding/) to obtain coordinates of the center of a region from its address in the census data.

Further, we used Google Static Maps API (https://developers.google.com/maps/documentation/static-maps/) to extract 1920x1920 satellite images for the villages at the "16" zoom level.

We provide a small subset of our dataset in this repository to test all three models.

Sample satellite images of parts of six different regions are shown below. Labels indicate the percentage of households with a particular roof type, source of lighting or source of drinking water. The satellite images in the paper are scaled down to fit the paper width. Zooming to true size makes, roads, roof types and farms distinctly visible.

Developmental parameters are not only related to “direct” features such as roofs and settlements, but also to “indirect” features such as roads and farmlands. For instance, the presence of roads is correlated with high electrification.

Use these commands to generate predictions from the three models mentioned earlier and calculate correlation, accuracy, precision and recall on the dataset provided in this repository.

First, clone the repository or download it as a zip file. To clone:

git clone "https://github.com/agarwalt/satimage.git"
cd satimage

Next, install the required python packages:

pip install -r requirements.txt

1. The file data/region_info.csv contains the centre latitudes and longitudes for some regions from our complete dataset. Use Google's Static Maps API (https://developers.google.com/maps/documentation/static-maps/) to download 1920x1920 satellite images for these regions at the 16 zoom level. The name of each image should be <region_code>.png. For example, for the region with region_code = 12345, the name of the corresponding image file should be 12345.png.

2. Download the multi-task model's weights and place the downloaded file in the models/developmental folder.

3. Change the working directory (cd code) and launch an interactive python shell (e.g. ipython).

4. Load the weights from the downloaded file:

   import util
   model = util.restore_model('../models/developmental/best_model_architecture.json', '../models/developmental/best_model_weights.h5')

5. Next, generate and save predictions of developmental parameters from the downloaded images.

   import satimage
   satimage.generate_predictions(model, '../images', '../data/predicted_developmental.csv')

   Here, ../images is the directory where the satellite images are placed.

   The predictions of developmental parameters for regions whose images were downloaded will be written to data/predicted_developmental.csv.

   Information about headers of the data files is available in data/developmental_header_detail.txt.

Load the multi-task model using steps 1-4, as necessary, in the previous section. To see and save filter responses for a particular region and convolutional layer, execute:

import satimage
layer_index = 19
filter_index = None
input_img_path = '../images/12345.png'
save_dir = '../images'
satimage.show_filter_responses(model, layer_index, input_img_path, save_dir, filter_index)

A copy of the filter responses will be saved in save_dir.

Vary the layer_index, filter_index and input_img_path variables to see responses of filters in different layers. For layer and filter indices, refer to the PNG file depicting the model's architecture in the models/developmental folder.

1. Download model weights, model P.D. and model C.D., and place the downloaded files in models/income_poverty_pd and models/income_poverty_cd respectively.

2. Change the working directory (cd code) and launch an interactive python shell (e.g. ipython).

3. Load the models's weights:

   import util
   model_pd = util.restore_model('../models/income_poverty_pd/best_model_architecture.json', '../models/income_poverty_pd/best_model_weights.h5')
   model_cd = util.restore_model('../models/income_poverty_cd/best_model_architecture.json', '../models/income_poverty_cd/best_model_weights.h5')

4. Generate and save predictions of income levels from the developmental parameters (generated in the previous section) using:

   import secc
   secc.generate_predictions(model_pd, '../data/predicted_developmental.csv', '../data/region_info.csv', '../data/pd_subdistrict_income.csv')
   secc.generate_predictions(model_cd, '../data/data_developmental.csv', '../data/region_info.csv', '../data/cd_subdistrict_income.csv')

   The predictions of income levels using predicted developmental parameters will be written to data/pd_subdistrict_income.csv. The predictions of income levels using actual values of the developmental parameters will be written to data/cd_subdistrict_income.csv.

   Information about headers of the data files is available in data/income_header_detail.txt.

5. To compare the predicted income levels above against ground truth values, and to calculate the accuracy of povery prediction, use:

   secc.compare_income_predictions('../data/data_subdistrict_income.csv', '../data/pd_subdistrict_income.csv')  # For model trained on predicted data, model P.D.
   secc.compare_income_predictions('../data/data_subdistrict_income.csv', '../data/cd_subdistrict_income.csv')  # For model trained on census data, model C.D.

The figure below shows the average training and validation loss for the three tasks in the multi-task model. The minimum validation loss occurs at the last step in the graph. Further training leads to overfitting.

We provide filter responses for our first (multi-task) model.

In the multi-task model's first block, consistent with the observations reported in the literature, filters learn edges with different orientations. The figure below shows differently oriented edges for different filters for a particular region.

The following figures illustrate the filter responses for the task specific branches for each of the three tasks in the multi-task model. In each figure, the larger images show different areas of two regions (A and B). The smaller images show filter activations for:

1. Roof type
2. Source of lighting, and,
3. Source of drinking water.

In contrast to the activations for shared layers' filters, the activations for the task-specific branches in the multi-task model highlight objects of relevance to the respective tasks.

The authors are grateful to NVIDIA Corporation for donating the TITAN X GPUs used for this research.