A small systems lab for understanding how storage data reaches NVIDIA GPU memory.
The project is intentionally broader than a single GPUDirect Storage benchmark. It walks through the host-staged path first, then removes bottlenecks step by step:
File / NVMe
|
+--> POSIX + pageable RAM + synchronous H2D
|
+--> POSIX + pinned RAM + asynchronous H2D
|
+--> O_DIRECT + pinned RAM + asynchronous H2D
|
+--> double-buffered read / copy / GPU compute overlap
|
+--> cuFile compatibility path
|
`--> true GDS direct path [supported platforms only]
cuFile is an application-facing I/O API. nvidia-fs is a kernel-side component used by the traditional GDS path. They are not competing technologies and they live at different layers of the stack.
On an unsupported GPU or filesystem, cuFile can use compatibility mode and stage I/O through host memory. The lab therefore follows three rules:
- Measured numbers are reported as measured.
- Simulated paths are explicitly labelled as simulated.
- Unsupported direct-GDS tests print
SKIPPEDinstead of publishing host-staged throughput as GDS throughput.
The initial development target is Ubuntu on GeForce RTX 5080 / RTX 5090. These machines are useful for studying pageable vs pinned memory, O_DIRECT, asynchronous H2D copies, overlap, DataLoader tuning, cuFile APIs, and compatibility mode.
Do not interpret a successful cuFile call on a GeForce system as proof that storage DMA bypassed host memory. Confirm the active path with gdscheck, cuFile statistics/logging, system topology, and the current NVIDIA support matrix.
A later run on a supported data-center / professional GPU can fill in the final true-GDS comparison without changing the overall experiment model.
make inspectBy default this writes a 4 GiB file of deterministic finite float32 values to /data/gds-lab.bin (xfs on the local NVMe). /dev/zero is not used: a zero file makes experiment 08's checksum identically 0.0, which hides wrong-offset reads. Override DATASET only if you intend to measure a different filesystem.
make dataset SIZE_GIB=4make inspect compares the dataset against MemTotal and warns when it is small enough to live entirely in the page cache. On a large-memory host, treat --backend direct as the storage-path measurement and the buffered backends as cache-path measurements.
A toolkit is selected by capability, not by path: scripts/cuda_env.sh rejects an nvcc that cannot target the installed GPU. This matters because Ubuntu 24.04's /usr/bin/nvcc is CUDA 12.0 and cannot emit code for sm_120 (RTX 5080/5090). If no usable toolkit is present, install a user-local one (compiler, runtime and cuFile; no driver):
./scripts/install_cuda_redist.sh
source scripts/cuda_env.sh
make buildThe redist install also provides cufile.h, libcufile.so and NVIDIA's GDS tools (gdscheck.py, gds_stats, gdsio). source scripts/cuda_env.sh puts those tools on PATH. It does not install nvidia-fs, so cuFile still runs in compatibility mode.
pip install -r requirements.txtThe default PyPI torch wheel is CPU-only and makes experiment 08 meaningless; requirements.txt pins the CUDA build.
./build/gds_lab --backend pageable --file /data/gds-lab.bin --bytes 1G
./build/gds_lab --backend pinned --file /data/gds-lab.bin --bytes 1G
./build/gds_lab --backend direct --file /data/gds-lab.bin --bytes 1G
./build/gds_lab --backend overlap --file /data/gds-lab.bin --bytes 1G --chunk-bytes 64MOr make bench, which runs all of them and then checks that they agree.
Each result line carries the facts needed to interpret it:
backend=pinned-async offset=0 bytes=1073741824 seconds=0.1183
effective_GBps=9.07
read_seconds=... read_GBps=... storage -> host buffer
copy_seconds=... h2d_GBps=... host buffer -> device
checksum=...
read_* and copy_* are reported separately because a single total cannot say
whether a change moved the storage path or the transfer path; pipelined backends
(overlap, cufile) omit the split rather than double-count overlapped time.
The checksum is the GPU-side sum of every byte in the region. It is identical
across backends, block sizes and access patterns whenever the transfer is
correct, so a mismatch is a bug in the run and not a performance result.
make bench fails if the backends disagree.
Variables the experiments sweep:
| Flag | Experiment | Meaning |
|---|---|---|
--read-chunk SIZE |
04 | storage read granularity (default 16M) |
--chunk-bytes SIZE |
03, 07 | overlap pipeline buffer size (default 64M) |
--random-access, --seed N |
05 | shuffle the block visit order |
--direct |
03, 07 | run the overlap pipeline on O_DIRECT |
--no-buf-register |
06 | skip cuFileBufRegister |
--offset SIZE, --fixed-offset |
all | which region to read, and whether iterations advance |
--iterations N advances the offset by --bytes each time by default, so a
repeat run reads new extents instead of re-reading the page cache. Pass
--fixed-offset when a hot-cache measurement is what you actually want; the run
labels itself either way.
CUFILE_ENV_PATH_JSON=$PWD/configs/cufile.json \
CUFILE_FORCE_COMPAT_MODE=true \
./build/gds_lab --backend cufile --file /data/gds-lab.bin --bytes 1GThis command is intentionally labelled cuFile compatibility, not true GDS.
python3 python/dataloader_bench.py \
--file /data/gds-lab.bin \
--workers 0,2,4,8 \
--batches 100| ID | Experiment | Main question |
|---|---|---|
| 00 | Capability | What storage/GPU/GDS paths can this host actually use? |
| 01 | Pageable vs pinned | What does page-locked host memory change? |
| 02 | Sync vs async | When does non-blocking H2D help? |
| 03 | Double buffering | Can read/copy/compute overlap hide latency? |
| 04 | Block size | Where is the throughput knee for this NVMe/filesystem? |
| 05 | Sequential vs random | How much does access pattern matter? |
| 06 | cuFile compatibility | What does the same application-facing API look like without direct GDS? |
| 07 | I/O + compute overlap | Does the GPU remain fed while work executes? |
| 08 | DataLoader | How do workers, prefetch, pinned memory and persistent workers affect GPU starvation? |
See experiments/README.md for the run plan.
Buffered POSIX and mmap benchmarks can turn into RAM/page-cache benchmarks after the first iteration. This repository keeps buffered and O_DIRECT paths separate for that reason.
Treat results as one of:
hot cache page cache -> application -> GPU
buffered/cold NVMe -> page cache -> application -> GPU
O_DIRECT NVMe -> aligned host buffer -> GPU
cuFile compat cuFile -> POSIX/host staging -> GPU
true GDS storage DMA -> GPU memory [supported hosts]
Do not compare those labels as if they represented the same data path.
scripts/profile_nsys.sh /data/gds-lab.bin overlap
scripts/collect_system_metrics.shUseful external tools include:
gdscheck.py -pnvidia-smi topo -mnvidia-smi dmoniostat -xz 1pidstat -dru 1- Nsight Systems
- cuFile statistics / TRACE logging when diagnosing the actual cuFile path
gds-lab/
├── src/ C++/CUDA benchmark backends
├── experiments/ experiment plans and interpretation notes
├── python/ PyTorch/DataLoader experiments
├── scripts/ inspection, dataset, profiling and metric helpers
├── configs/ per-workload cuFile configuration
├── docs/ architecture notes
└── results/ local result artifacts (ignored except README)
- NVIDIA GPUDirect Storage Overview Guide: https://docs.nvidia.com/gpudirect-storage/overview-guide/
- NVIDIA cuFile API Reference: https://docs.nvidia.com/gpudirect-storage/api-reference-guide/
- NVIDIA GDS Benchmarking and Configuration Guide: https://docs.nvidia.com/gpudirect-storage/configuration-guide/
- NVIDIA GDS O_DIRECT Requirements Guide: https://docs.nvidia.com/gpudirect-storage/o-direct-guide/
- Inspiration for staged storage / preprocessing experiments: https://github.com/cfregly/ai-performance-engineering/tree/main/code/ch05
Initial scaffold. The first milestone is reproducible single-GPU storage-path characterization on Ubuntu + RTX 5080/5090. True direct-GDS validation is deliberately deferred to a supported host.