Repository navigation
Downloads ignore the session (and therefore custom proxies) passed to the client #162
Description
Summary
Note: with the new API tokens,
cdsapi.Clientdelegates to
ecmwf.datastores.legacy_client.LegacyClient, so the root cause lives in
ecmwf-datastores-client.
Upstream issue/PR: ecmwf/ecmwf-datastores-client#73 / ecmwf/ecmwf-datastores-client#74.
When a custom requests.Session is passed to the client (e.g. to route traffic
through a corporate proxy), API requests (submit, status polling) correctly use
that session, but the final file download does not. The download is
performed by multiurl.download() without the session, so it creates a fresh
default session that bypasses the proxy. On networks that require a proxy for
outbound traffic, the download fails with a DNS resolution error:
Recovering from connection error [HTTPSConnectionPool(host='object-store.os-api.cci2.ecmwf.int', port=443):
Max retries exceeded with url: /cci2-prod-cache-2/2026-03-18/....grib
(Caused by NameResolutionError("HTTPSConnection(host='object-store.os-api.cci2.ecmwf.int', port=443):
Failed to resolve 'object-store.os-api.cci2.ecmwf.int' ([Errno 11001] getaddrinfo failed)"))], attempt 1 of 500
Environment
ecmwf-datastores-client0.5.3cdsapi0.7.7 (which delegates toecmwf.datastores.legacy_client.LegacyClient)multiurl0.3.9- Python 3.14, Windows
Reproduction
import requests
import cdsapi
session = requests.Session()
session.proxies = {
"http": "http://user:pass@proxy.example.com:3128",
"https": "http://user:pass@proxy.example.com:3128",
}
client
6862
= cdsapi.Client(session=session)
client.retrieve(
"reanalysis-era5-pressure-levels",
{
"product_type": ["reanalysis"],
"variable": ["geopotential"],
"year": ["2024"],
"month": ["03"],
"day": ["01"],
"time": ["13:00"],
"pressure_level": ["1000"],
"data_format": "grib",
},
"download.grib",
)On a machine where direct outbound traffic is blocked (corporate network), the
request is submitted and polled successfully (these go through the proxied
session), but the download of the resulting object-store URL fails because it
does not go through the proxy.
Root cause
In ecmwf/datastores/processing.py, Results._download() calls:
multiurl.download(
url,
target=target,
maximum_retries=0,
**self.request_options,
**download_options,
)self.request_options only contains timeout and verify (see
Client._request_options in ecmwf/datastores/client.py). The session is
never forwarded, so multiurl's HTTPDownloaderBase.__init__ falls back to
the module-level requests object:
self.session = requests if session is None else sessionNote that multiurl.download() already accepts a session keyword argument —
it is simply never passed by this client.
Proposed fix
Forward the session to multiurl.download() in Results._download():
multiurl.download(
url,
target=target,
maximum_retries=0,
session=self.session,
**self.request_options,
**download_options,
)(self.session is already available on ApiResponse.)
Workaround
Monkey-patch multiurl.download before calling retrieve() (session is the
same session passed to cdsapi.Client):
import multiurl
original = multiurl.download
def download_with_session(url, target, **kwargs):
kwargs.setdefault("session", session)
return original(url, target, **kwargs)
multiurl.download = download_with_sessionThis works because processing.py resolves multiurl.download at call time.
Related issues
- Using cdsapi on Python Anywhere #81 — using cdsapi behind the PythonAnywhere proxy; no documentation on proxy setup.
- Caused by ProxyError('Unable to connect to proxy' #96 —
ProxyError('Unable to connect to proxy')with endless retries.
Both are likely affected by this: even with a proxied session (or proxy settings),
the final download does not go through it.
Activity