A high-performance GPU-accelerated HTTP server and packet processing framework using NVIDIA DOCA SDK and GPUNetIO.
GAZSI enables zero-copy processing of network packets directly on the GPU for minimal latency. The system implements:
- GPU-Direct HTTP Server: HTTP request/response handling entirely on GPU
- TensorRT Inference Integration: ML inference via lock-free CPU-GPU ring buffer
- Dynamic Batching: Automatic request batching for improved throughput
The current implementation runs GPUNetIO networking and TensorRT inference on one physical GPU. They use separate CUDA streams on the same CUDA device; the design does not require MIG or another GPU-partitioning mechanism.
┌─────────────────────────────────────────────────────────────────────┐
│ Network Interface │
│ SmartNIC Rx/Tx Queues │
└──────────────────────────────┬──────────────────────────────────────┘
│ GPUNetIO direct DMA
▼
┌─────────────────────────────────────────────────────────────────────┐
│ GPU Memory │
│ │
│ Networking Stream Inference Stream │
│ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ │
│ │ TCP Receive │─►│ HTTP Parser │────────►│ GPU │ │
│ │ / Transmit │ │ │ │ Tokenizer │ │
│ └──────┬──────┘ └──────┬──────┘ └──────┬──────┘ │
│ │ │ ▼ │
│ │ ┌──────────▼─────────────────────────────┐ │
│ └────►│ Device-Resident Request/Response Ring │◄────┐ │
│ └────────────────────────────────────────┘ │ │
│ ┌────────▼───┐ │ │
│ │ TensorRT │ │ │
│ └────────┬───┘ │ │
│ ┌────────▼─────┴──┐ │
│ │ GPU Formatter │ │
│ └─────────────────┘ │
└──────────────────────────────┬──────────────────────────────────────┘
│ Compact control metadata only
▼
┌────────────────────────┐
│ CPU Metadata Dispatcher│
│ Batch Formation/Launch │
└────────────────────────┘
Request and response bytes remain in device memory throughout this path. A small host-mapped, pinned control structure contains only slot state and record descriptors. The CPU uses that metadata to form batches and invoke TensorRT, but it does not copy, tokenize, or format request and response data. The zero-copy mechanism therefore remains intact while the CPU performs the execution control required to launch GPU work.
- Zero-Copy Packet Processing: Direct NIC-to-GPU data path
- Split-Plane Lock-Free Ring Buffer: 256 metadata slots with device-only payload storage
- Semaphore Signaling: Event-driven CPU-GPU synchronization
- Dynamic Batch Inference: Up to 8 requests per batch
- HTTP/1.1 Support: GET/POST request handling on GPU
- NVIDIA GPU with Compute Capability 7.0+
- NVIDIA ConnectX-6 or newer SmartNIC
- 8GB+ System Memory
- Linux
- NVIDIA DOCA SDK with GPUNetIO support
- CUDA Toolkit
- TensorRT
- GCC
The project builds as part of the DOCA SDK application framework:
# Create symlink in DOCA applications directory
sudo ln -s /path/to/GAZSI \
/opt/mellanox/doca/applications/GAZSI
# Build
cd /opt/mellanox/doca/applications
sudo meson build -Denable_GAZSI=true
sudo ninja -C buildThe host-mediated comparison used in the paper is published in
baseline/baseline_trt.cpp. It uses the Linux
kernel TCP stack and a nonblocking epoll server while retaining the following
GPU-side optimizations:
- pinned host input and output buffers;
- asynchronous H2D and D2H copies;
- a nonblocking CUDA stream;
- TensorRT with batch size one; and
- a captured batch-one CUDA Graph enabled with
-Ofor the reported configuration.
Request parsing, the workload-specific hash transformation, and response
formatting remain on the CPU. See baseline/README.md for
build and run commands. The VMA/XLIO experiment uses this same executable with
the kernel-bypass library preloaded, so it remains a separate control rather
than part of the kernel TCP configuration.
tests/ab_perf.sh is a source-tree regression check. It no longer assumes an
author-specific directory: pass the comparison checkout with --base or set
GAZSI_BASE_DIR.
The platform, workload, run-policy, and measurement details used by the paper
are collected in REPRODUCIBILITY.md.
# Basic usage
sudo ./GAZSI -g <GPU_PCI> -n <NIC_PCI> -q <NUM_QUEUES>
# With HTTP server
sudo ./GAZSI -g E6:00.0 -n c1:00.0 -q 2 -s
# Options:
# -g GPU PCIe address
# -n NIC PCIe address
# -q Number of receive queues (1-4)
# -s Enable HTTP server mode
# -e Path to TensorRT engine file| Endpoint | Method | Description |
|---|---|---|
/index. |
GET | Static index page |
/inference?d=<text> |
GET | TensorRT inference endpoint |
# Send inference request
curl "http://10.0.0.6/inference?d=hello%20world"
# Response
{
"input": "hello world",
"tokens": 3,
"embedding_sample": [0.123, -0.456, 0.789],
"inference_time_us": 1234,
"batch_size": 1
}BSD-3-Clause. See individual source files for details.
Built on NVIDIA DOCA SDK and GPUNetIO technology.