xarray backend for the waves.met.no Zarr data store, with a Rust core (HTTP range reads, Zarr v3 shard indexes, zstd, hyperslab assembly; GIL released, parallel requests). The store format is optimized for both Python access (this package) and wasm/web access (browser clients reading the same objects over HTTP).
The Rust core is built with maturin when installing from git, so a Rust toolchain is needed. Python >= 3.10; numpy and xarray are installed as dependencies.
With mamba (creates a new environment with Rust and maturin):
mamba create -n x3d -c conda-forge python rust maturin pip
mamba activate x3d
pip install "git+https://github.com/gauteh/x3d"
Or with rustup, in an existing Python environment:
pip install maturin
pip install "git+https://github.com/gauteh/x3d"
import xarray as xr
ds = xr.open_dataset("x3d+https://rgw.met.no/5e4f32366fc74342866b4c67dbd4713d:waves-ww3/ww3_4km") # the x3d+ prefix selects the engine
# NorKyst-800 v3 surface layer: "x3d+https://rgw.met.no/5e4f32366fc74342866b4c67dbd4713d:norkyst800m-v3/norkyst800"
ds.hs.isel(rlat=500, rlon=300).values # point series: served from ts.zarr
ds.hs.isel(time=10).values # field: served from map.zarr
xr.open_dataset("x3d+https://rgw.met.no/5e4f32366fc74342866b4c67dbd4713d:waves-ww3/ww3_4km", layout="ov") # coarse 1/8 grid (block means), lossyOpening reads latest.json, dataset.json and the consolidated metadata of ts.zarr / map.zarr
(a few small requests); data is lazy.
URL: x3d+<http(s)|file URL or path>[?run=<id>&layout=auto|ts|map|ov] (kwargs run=, layout= win).
The x3d+ prefix selects the engine; without it (plain path or URL) pass engine="x3d".
- Layout choice (
layout="auto"): per read, the Rust core plans requests and bytes fortsandmapfrom the chunk geometry andx3d.layout.choosepicks the cheapest. Both are exact, so the choice is invisible.ovis lossy (block means) and never chosen implicitly.logging.getLogger("x3d")at DEBUG shows the choice. - Data decodes to float32 (int16 +
scale_factor+_FillValuevia normal CF decoding);latitude/longitudeare coordinates. - Absent shards/chunks (land) read as fill without data transfer.
Prototype limits / next steps: only the latest run (no best-forecast aggregate across runs yet); no write
support; only the codecs the pipeline writes (sharding_indexed, bytes, zstd, crc32c); no read coalescing,
caching of decoded chunks or overview-to-field refinement.
Regenerate (needs network; overwrites the image, then commit it):
pytest --benchmark-enable --benchmark-only --benchmark-columns=min,median,mean,rounds
The store layout is specified in docs/format.md; the producer is x3d-publisher.
Create the conda env from environment.yml (Rust, maturin, pytest, xarray, zarr, ...):
mamba env create -f environment.yml
conda activate x3d
maturin develop --release # builds src/ into python/x3d/_core*.so
cargo test # Rust unit tests (hand-encoded shards, corruption, missing chunks)
pytest # everything below
pytest -m "not rgw and not store" # offline
pytest --benchmark-enable --benchmark-only # THREDDS vs x3d benchmark -> docs/thredds_vs_x3d.png
Tests (tests/):
- Synthetic store written with zarr-python (same layout/codecs as the published stores, both shard index locations) under
tmp; values are checked against numpy and against zarr-python itself. - A local HTTP server with Range support that logs requests: asserts open cost and requests per access pattern.
-m rgw: integration against the public RGW bucket (X3D_RGW_URLoverrides; skipped if unreachable).-m store: against a local store (X3D_TEST_STORE, defaultstore/ww3_4km), e.g. one built by x3d-publisher.tests/test_benchmark.py(markedrgw): pytest-benchmark of the same data through up to four backends, one plot panel per dataset (a: ww3_4kmhs, all four backends; b: NorKyst-800 surface currentu_eastward, THREDDSfou-hi/norkystv3_800m_m00_beat depth 0 (X3D_THREDDS_NK800_URL) and x3d on RGW (X3D_RGW_NK800_URL)): THREDDS OPeNDAP (X3D_THREDDS_URL), the NetCDF file on/lustreread with netCDF4 (newestww3_4km_<date>T00Z.ncbelowX3D_LUSTRE_ROOT, orX3D_LUSTRE_FILE), x3d on RGW and x3d on the local store (x3d+file://,X3D_TEST_STORE); unavailable ones are skipped. Scenarios: open, point series (fixed and random point), field, box, several fields. Each round opens the dataset anew (not timed). The two file-based backends are only cold in the first round (later rounds hit the OS page cache), so read them as best case. The /lustre file is uncompressed float32 (8.8 GB, one field per chunk), so netCDF4 does no decoding and is not a like-for-like comparison with the int16 + zstd store (~0.2 GB); the plot legend says so. The local x3d reads are CPU-bound (zstd) on a thread pool, and the pause between rounds roughly doubles them (a 100x100x24 box: 1.0 ms back to back, 2.2 ms after a 0.3 s pause; 7 ms on one thread), so the plot is the idle-start figure. Benchmarking is disabled by default (addopts = --benchmark-disable: each case runs once as a plain test);--benchmark-enablemeasures,--benchmark-onlyruns only the benchmarks, and the bar plot goes todocs/(X3D_BENCH_OUT; rounds viaX3D_BENCH_ROUNDS). Separate tests check THREDDS and the lustre file (when from the same model run) return the same numbers as x3d.tests/test_microbench.py(-m "rgw or store" -s): where a read spends its time, fromArray.read_profiled(index requests, store fetches, zstd decode, copy into the output; thread-time summed, next to wall time), against RGW and the local store. On RGW the wall time is fetching; decode + copy are a few ms even for six full fields, so decoding is not the bottleneck.X3D_THREADSsets the size of the I/O + decode pool (default max(16, cores)).