Running GitLab Runner behind a proxy

This guide aims specifically to making GitLab Runner with Docker executor work behind a proxy.

Before continuing, ensure that you’ve already installed Docker and GitLab Runner on the same machine.

Configuring cntlm

cntlm is a Linux proxy which can be used as a local proxy and has 2 major advantages compared to adding the proxy details everywhere manually:

• One single source where you need to change your credentials
• The credentials can not be accessed from the Docker runners

Assuming you have installed cntlm, you need to first configure it.

Make cntlm listen to the docker0 interface

For added security and protection from the internet, bind cntlm to listen on docker0 interface, which has an IP address that containers can reach. If you tell cntlm on the Docker host to bind only to this address, Docker containers can reach it, but the outside world can’t.

1. Find the IP that Docker is using:

   ip -4 -oneline addr show dev docker0

   The IP address is usually 172.17.0.1, let’s call it docker0_interface_ip.

2. Open the configuration file for cntlm (/etc/cntlm.conf). Enter your username, password, domain and proxy hosts, and configure the Listen IP address which you found from the previous step. It should look like this:

   Username     testuser
   Domain       corp-uk
   Password     password
   Proxy        10.0.0.41:8080
   Proxy        10.0.0.42:8080
   Listen       172.17.0.1:3128 # Change to your docker0 interface IP

3. Save the changes and restart its service:

   sudo systemctl restart cntlm

Configuring Docker for downloading images

For information about how to use proxy, see Docker documentation.

The service file should look like this:

[Service]
Environment="HTTP_PROXY=http://docker0_interface_ip:3128/"
Environment="HTTPS_PROXY=http://docker0_interface_ip:3128/"

Adding Proxy variables to the GitLab Runner configuration

The proxy variables need to also be added to the GitLab Runner configuration, so that it can connect to GitLab.com from behind the proxy.

This action is the same as adding the proxy to the Docker service above:

1. Create a systemd drop-in directory for the gitlab-runner service:

   mkdir /etc/systemd/system/gitlab-runner.service.d

2. Create a file called /etc/systemd/system/gitlab-runner.service.d/http-proxy.conf that adds the HTTP_PROXY environment variables:

   [Service]
   Environment="HTTP_PROXY=http://docker0_interface_ip:3128/"
   Environment="HTTPS_PROXY=http://docker0_interface_ip:3128/"

   To connect GitLab Runner to any internal URLs, like a GitLab Self-Managed instance, set a value for the NO_PROXY environment variable.

   [Service]
   Environment="HTTP_PROXY=http://docker0_interface_ip:3128/"
   Environment="HTTPS_PROXY=http://docker0_interface_ip:3128/"
   Environment="NO_PROXY=gitlab.example.com"

3. Save the file and flush changes:

   systemctl daemon-reload

4. Restart GitLab Runner:

   sudo systemctl restart gitlab-runner

5. Verify that the configuration has been loaded:

   systemctl show --property=Environment gitlab-runner

   You should see:

   Environment=HTTP_PROXY=http://docker0_interface_ip:3128/ HTTPS_PROXY=http://docker0_interface_ip:3128/

Adding the Proxy to the Docker containers

After you register your runner, you may want to propagate your proxy settings to the Docker containers (for example, for git clone).

To do this, you need to edit /etc/gitlab-runner/config.toml and add the following to the [[runners]] section:

pre_get_sources_script = "git config --global http.proxy $HTTP_PROXY; git config --global https.proxy $HTTPS_PROXY"
environment = ["https_proxy=http://docker0_interface_ip:3128", "http_proxy=http://docker0_interface_ip:3128", "HTTPS_PROXY=docker0_interface_ip:3128", "HTTP_PROXY=docker0_interface_ip:3128"]

Where docker0_interface_ip is the IP address of the docker0 interface.

Proxy settings when using dind service

When using the Docker-in-Docker executor (dind), it may be necessary to specify docker:2375,docker:2376 in the NO_PROXY environment variable. The ports are required, otherwise docker push is blocked.

Communication between dockerd from dind and the local docker client (as described here: https://hub.docker.com/_/docker/) uses proxy variables held in root’s Docker configuration.

To configure this, you need to edit /root/.docker/config.json to include your complete proxy configuration, for example:

{
    "proxies": {
        "default": {
            "httpProxy": "http://proxy:8080",
            "httpsProxy": "http://proxy:8080",
            "noProxy": "docker:2375,docker:2376"
        }
    }
}

To pass on the settings to the container of the Docker executor, a $HOME/.docker/config.json also needs to be created inside the container. This may be scripted as a before_script in the .gitlab-ci.yml, for example:

before_script:
  - mkdir -p $HOME/.docker/
  - 'echo "{ \"proxies\": { \"default\": { \"httpProxy\": \"$HTTP_PROXY\", \"httpsProxy\": \"$HTTPS_PROXY\", \"noProxy\": \"$NO_PROXY\" } } }" > $HOME/.docker/config.json'

Or alternatively, in the configuration of the gitlab-runner (/etc/gitlab-runner/config.toml) that is affected:

[[runners]]
  pre_build_script = "mkdir -p $HOME/.docker/ && echo \"{ \\\"proxies\\\": { \\\"default\\\": { \\\"httpProxy\\\": \\\"$HTTP_PROXY\\\", \\\"httpsProxy\\\": \\\"$HTTPS_PROXY\\\", \\\"noProxy\\\": \\\"$NO_PROXY\\\" } } }\" > $HOME/.docker/config.json"

If the NO_PROXY list needs to be extended, wildcards * only work for suffixes, but not for prefixes or CIDR notation. For more information, see https://github.com/moby/moby/issues/9145 and https://unix.stackexchange.com/questions/23452/set-a-network-range-in-the-no-proxy-environment-variable.

Handling rate limited requests

A GitLab instance may be behind a reverse proxy that has rate-limiting on API requests to prevent abuse. GitLab Runner sends multiple requests to the API and could go over these rate limits.

As a result, GitLab Runner handles rate limited scenarios by using the following retry logic:

Retry logic

When GitLab Runner receives a 429 Too Many Requests response, it follows this retry sequence:

1. The runner checks the response headers for a RateLimit-ResetTime header.
   • The RateLimit-ResetTime header should have a value which is a valid HTTP date (RFC1123), like Wed, 21 Oct 2015 07:28:00 GMT.
   • If the header is present and has a valid value, the runner waits until the specified time and issues another request.
2. If the RateLimit-ResetTime header is invalid or missing, the runner checks the response headers for a Retry-After header.
   • The Retry-After header should have a value in seconds format, like Retry-After: 30.
   • If the header format is present and has a valid value, the runner waits until the specified time and issues another request.
3. If both headers are missing or invalid, the runner waits for the default interval and issues another request.

The runner retries failed requests up to 5 times. If all retries fail, the runner logs the error from the final response.

Supported header formats