Blazing-fast parallel S3 client & 100% drop-in
s5cmdreplacement in Rust: ~1.5 ms cold start, ~4.5 MiB idle memory footprint, pre-allocatedpwrite/preadzero-lock NVMe I/O, and sister project to RustStack.
t3 (Turbo 3) is an ultra-fast, memory-safe S3 CLI and sync engine written in pure Rust. Built as the sister project to RustStack, t3 is engineered to outperform both peak/s5cmd (Go) and the official aws s3 CLI (Python) across startup latency, small-object IOPS, large multipart transfers, and peak RSS memory usage.
Measured end-to-end transfer time and sustained throughput across dataset sizes from 4 MiB up to 512 MiB against RustStack:
| Dataset & Direction | ⚡ t3 v0.1.0 Bandwidth (Time) |
s5cmd v2.3.0 Bandwidth (Time) |
t3 Bandwidth & Time Advantage |
|---|---|---|---|
Upload 512 MiB (64 × 8 MiB shards) |
2,453.0 MiB/s (208.7 ms) |
1,947.4 MiB/s (262.9 ms) |
1.26x faster (+505.6 MiB/s) |
Upload 256 MiB (64 × 4 MiB shards) |
2,366.9 MiB/s (108.2 ms) |
1,895.2 MiB/s (135.1 ms) |
1.25x faster (+471.7 MiB/s) |
Download 256 MiB (64 × 4 MiB shards) |
3,505.8 MiB/s (73.0 ms) |
3,278.5 MiB/s (78.1 ms) |
1.07x faster (+227.3 MiB/s) |
Upload 128 MiB (256 × 512 KiB shards) |
1,910.5 MiB/s (67.0 ms) |
1,255.0 MiB/s (102.0 ms) |
1.52x faster (+655.5 MiB/s) |
Download 128 MiB (256 × 512 KiB shards) |
2,418.5 MiB/s (52.9 ms) |
1,815.7 MiB/s (70.5 ms) |
1.33x faster (+602.8 MiB/s) |
Upload 64 MiB (1,000 × 64 KiB files) |
1,613.2 MiB/s (38.7 ms) |
511.8 MiB/s (122.1 ms) |
3.15x faster (+1,101.4 MiB/s) |
Download 64 MiB (1,000 × 64 KiB files) |
1,954.7 MiB/s (32.0 ms) |
471.4 MiB/s (132.6 ms) |
4.15x faster (+1,483.3 MiB/s) |
Upload 4 MiB (1,000 × 4 KiB files) |
221.7 MiB/s (17.6 ms) |
44.7 MiB/s (87.4 ms) |
4.96x faster (+177.0 MiB/s) |
Download 4 MiB (1,000 × 4 KiB files) |
155.6 MiB/s (25.1 ms) |
36.2 MiB/s (107.8 ms) |
4.29x faster (+119.4 MiB/s) |
Measured end-to-end wall-clock execution time and peak resident memory (RSS) across 1,000 × 4 KiB objects against RustStack:
| Workload (1,000 × 4 KiB Objects) | ⚡ t3 v0.1.0 (Rust) |
s5cmd v2.3.0 (Go) |
t3 Speedup |
t3 Memory Reduction |
|---|---|---|---|---|
Cold Startup (version) |
3.1 ms (6.2 MiB) | 4.8 ms (13.2 MiB) | 1.55x faster | 2.12x less RAM |
Parallel Upload (cp 1,000 files → S3) |
17.6 ms (14.3 MiB) | 87.4 ms (33.6 MiB) | 4.96x faster | 2.35x less RAM |
Parallel Download (cp S3 → 1,000 files) |
25.1 ms (12.9 MiB) | 107.8 ms (43.3 MiB) | 4.29x faster | 3.34x less RAM |
Object Listing (ls 1,000 objects) |
7.3 ms (8.3 MiB) | 26.5 ms (20.1 MiB) | 3.61x faster | 2.42x less RAM |
Disk Usage Summary (du 1,000 objects) |
6.5 ms (8.4 MiB) | 28.3 ms (20.6 MiB) | 4.36x faster | 2.44x less RAM |
Incremental Sync (sync --size-only) |
8.2 ms (7.9 MiB) | 58.6 ms (31.1 MiB) | 7.18x faster | 3.93x less RAM |
Pipelined Batch Delete (rm 1,000 keys) |
8.6 ms (8.7 MiB) | 43.7 ms (23.5 MiB) | 5.07x faster | 2.71x less RAM |
| Metric / Capability | ⚡ t3 (Rust) |
s5cmd v2.3.0 (Go) |
aws s3 CLI v2 (Python) |
|---|---|---|---|
| Process Cold-Start Latency | ~1.5 ms | ~20 ms | ~450 ms |
| Idle / Small-Transfer RSS Memory | ~4.5 MiB | ~35–300+ MiB | ~85+ MiB |
| Multipart Memory Backpressure | Bounded Slab BufferPool |
Unbounded per-worker 5 MiB heap allocs | Sequential / Python GIL bound |
| Local NVMe File I/O | set_len + concurrent pwrite/pread + DashSet mkdir cache |
Standard os.File syscalls |
Standard Python file I/O |
| SigV4 Request Signing | Cached daily HMAC key (~220k+ ops/s/core) | Re-derived per request | botocore Python signing |
| S3-to-S3 Copy > 5 GiB | Automatic Parallel Multipart Copy | Fails (#29) | Supported (slow) |
Path Traversal Guard (../) |
Enforced (sanitize_relative_path) |
Vulnerable (#872) | Enforced |
sync --delete with --exclude |
Honors --exclude / --include DFA |
Deletes excluded files (#815) | Honors filters |
~/.aws/config endpoint_url |
Supported natively + multi-endpoint round-robin | Unsupported | Supported |
Automated benchmark rating measured via crates/t3-benchmarks/:
| Component | Operation | Payload / Batch | Throughput | Bandwidth | p50 Latency | p95 Latency | Grade |
|---|---|---|---|---|---|---|---|
| t3-storage | PathTraversalGuard + WildcardSplit |
Key Validation | 413,000+ ops/s | N/A | 1.4 µs | 1.5 µs | A+ (Ultra Fast) |
| t3-core | GlobFilterDFA (Include+Exclude) |
4 Globs | 409,000+ ops/s | N/A | 1.4 µs | 1.5 µs | A+ (Ultra Fast) |
| t3-core | SelectQueryExecute (NDJSON) |
4 Rows/Doc | 298,000+ ops/s | 33.9 MB/s | 2.4 µs | 2.4 µs | A+ (Ultra Fast) |
| t3-s3-client | SigV4SignRequest (Cached Key) |
Auth Header | 221,000+ ops/s | N/A | 3.4 µs | 3.8 µs | A+ (Ultra Fast) |
| t3-s3-client | SigV4PresignUrl |
Query Auth | 167,000+ ops/s | N/A | 4.8 µs | 5.3 µs | A+ (Ultra Fast) |
| t3-storage | PositionalPread (read_exact_at) |
64 KB | 118,000+ ops/s | 7,384.1 MB/s | 7.3 µs | 9.4 µs | A+ (Ultra Fast) |
| t3-engine | BatchDeleteObjects (1000/batch) |
200 Keys | 62,500+ ops/s | N/A | 16.0 µs | 16.0 µs | A+ (Ultra Fast) |
| t3-storage | PreallocatedPwrite (write_all_at) |
64 KB | 49,100+ ops/s | 3,070.2 MB/s | 17.3 µs | 23.3 µs | A+ (Ultra Fast) |
| t3-engine | ParallelDownload (pwrite + DirCache) |
200 x 4 KB | 37,800+ ops/s | 147.9 MB/s | 26.4 µs | 26.4 µs | A+ (Ultra Fast) |
| t3-engine | ParallelRangeDownload (pwrite) |
32 MiB (8x ranges) | 42 ops/s | 1,343.9 MB/s | 23.8 ms | 23.8 ms | A+ (Ultra Fast) |
| t3-engine | MultipartUpload (pread + BufferPool) |
32 MiB (8x parts) | 24 ops/s | 781.1 MB/s | 40.9 ms | 40.9 ms | A+ (Ultra Fast) |
t3 supports s5cmd's exact CLI syntax (global flags before or after the subcommand):
cp [options] <src> <dst>:- Directions:
Local → S3,S3 → Local,S3 → S3(including > 5 GiB parallel multipart copy),Local → Local. - Flags:
-c/--concurrency(default 5),-p/--part-size(default 50 MiB),-f/--flatten,-n/--no-clobber,-s/--if-size-differ,-u/--if-source-newer,-sp/--show-progress,--exclude,--include,--raw,--no-follow-symlinks,--storage-class,--sse,--sse-kms-key-id,--acl,--cache-control,--expires,--content-type,--content-encoding,--content-disposition,--metadata Key=Value,--metadata-directive,--source-region,--destination-region,--version-id.
- Directions:
mv [options] <src> <dst>: Parallel copy followed by source deletion.sync [options] <src> <dst>: Concurrent source & destination listing + streaming diff with--delete(respecting--exclude/--include),--size-only, and--exact-timestamps.ls [options] [target]: Lists buckets or objects/prefixes (-e/--etag,-H/--humanize,-s/--show-fullpath,--all-versions).rm [options] <target>...: Pipelined 1,000-keyDeleteObjectsbatcher across wildcards (--raw,--exclude,--include,--version-id,--all-versions).run [file]: "Beast Mode" — parses batch commands from a file orstdinand executes them concurrently on the shared 256-worker pool.cat <s3-url>...: Streams one or more S3 objects (including wildcards) tostdout.pipe [options] <s3-url>: Streamsstdindirectly to an S3 object via bounded-memory parallel multipart upload.head <s3-url>: PrintsHeadBucketorHeadObjectmetadata.du [options] <s3-url>: Summarizes object bytes and counts (-g/--groupby storage class,-H/--humanize).mb <s3://bucket>&rb [--force] <s3://bucket>: Create or remove buckets (--forceempties the bucket first).presign [-e <duration>] <s3-url>: Generates a SigV4 presignedGETURL.select -q <sql> <s3-url>: Executes SQL queries across JSON/NDJSON or CSV objects (supports wildcards).bucket-version [--set Enabled|Suspended] <s3://bucket>: Gets or sets bucket versioning.version: Prints version and build metadata.
curl -sSfL https://ehlerst.github.io/t3/install.sh | shOr install to a custom directory / specific version:
T3_VERSION=v0.1.2 T3_INSTALL_DIR="$HOME/.local/bin" \
sh -c "$(curl -sSfL https://raw.githubusercontent.com/ehlerst/t3/main/install.sh)"# Homebrew formula
brew install --formula https://raw.githubusercontent.com/ehlerst/t3/main/Formula/t3.rb
# Cargo (from source)
cargo install --git https://github.com/ehlerst/t3 t3-cli# Linux amd64 (or t3-linux-arm64 / t3-darwin-arm64 / t3-darwin-amd64)
curl -sSLO https://github.com/ehlerst/t3/releases/latest/download/t3-linux-amd64.tar.gz
curl -sSLO https://github.com/ehlerst/t3/releases/latest/download/t3-linux-amd64.tar.gz.sha256
sha256sum -c t3-linux-amd64.tar.gz.sha256
tar xzf t3-linux-amd64.tar.gz && sudo mv t3 /usr/local/bin/t3 implements the standard Amazon AWS credential provider chain out of the box:
- Environment Variables:
AWS_ACCESS_KEY_ID,AWS_SECRET_ACCESS_KEY,AWS_SESSION_TOKEN(andAWS_SECURITY_TOKEN). - Amazon EKS IRSA (IAM Roles for Service Accounts): Automatically exchanges Kubernetes service account JWT tokens (
AWS_ROLE_ARN+AWS_WEB_IDENTITY_TOKEN_FILE+ optionalAWS_ROLE_SESSION_NAME/AWS_ENDPOINT_URL_STS) via AWS STSAssumeRoleWithWebIdentity. - Shared Credentials & Config Files:
~/.aws/credentialsand~/.aws/config(includingAWS_SHARED_CREDENTIALS_FILE,AWS_CONFIG_FILE,AWS_PROFILE, profileendpoint_url, profilerole_arn+web_identity_token_file, and externalcredential_process). - EKS Pod Identity & ECS Container Credentials:
AWS_CONTAINER_CREDENTIALS_FULL_URI/AWS_CONTAINER_CREDENTIALS_RELATIVE_URIwithAWS_CONTAINER_AUTHORIZATION_TOKEN_FILE/AWS_CONTAINER_AUTHORIZATION_TOKEN. - Amazon EC2 Instance Metadata Service (
IMDSv2/IMDSv1): Automatically exchanges a session token viaPUT http://169.254.169.254/latest/api/token(X-aws-ec2-metadata-token-ttl-seconds: 21600) and retrieves attached EC2 IAM role credentials (respectsAWS_EC2_METADATA_DISABLEDandAWS_EC2_METADATA_SERVICE_ENDPOINT). - Anonymous Access:
--no-sign-requestfor public S3 buckets.
4. Usage with RustStack (http://localhost:4566) or AWS S3
# Create bucket
t3 --endpoint-url http://localhost:4566 mb s3://ml-datasets
# Upload a directory with execution stats
t3 --endpoint-url http://localhost:4566 --stat cp ./shards/ s3://ml-datasets/train/
# Download matching wildcard objects with exclude filter
t3 --endpoint-url http://localhost:4566 --stat cp \
--exclude "*.tmp" \
"s3://ml-datasets/train/*.parquet" ./local-shards/
# Synchronize and delete stale destination objects
t3 --endpoint-url http://localhost:4566 --stat sync --delete \
./local-shards/ s3://ml-datasets/train/crates/
├── t3-cli/ # Binary entrypoint (`t3`), s5cmd-compatible clap parser, shell completions
├── t3-core/ # Execution engine: two-tier work scheduler, DFA glob filter, speculative prefix fan-out lister, sync engine, `run` batch parser
├── t3-storage/ # Bounded slab BufferPool, pre-allocated pwrite/pread positional I/O, DashSet mkdir cache, path-traversal guard
├── t3-s3-client/ # High-throughput S3 data-plane client (cached SigV4 signing, multi-endpoint round-robin load balancing, connection pool)
├── t3-telemetry/ # Lock-free atomic stats (--stat), NDJSON structured logger (--json), indicatif progress bars (-sp)
└── t3-benchmarks/ # Automated microbenchmark & end-to-end S3 transfer performance rating suite
Licensed under either of Apache License, Version 2.0 or MIT license at your option.