ToRIS is a Rust-based LLM inference runtime that monitors token-level uncertainty in real time and adapts decoding mid-generation.
Most inference pipelines decode with fixed sampling parameters, then evaluate output quality after generation. ToRIS moves control into the generation loop itself: it tracks confidence and entropy per token, detects instability patterns across a sliding window, and applies corrective actions immediately.
- Runs GGUF models through llama.cpp from a modular Rust runtime.
- Computes token-level metrics (selected-token probability, confidence score, entropy).
- Detects unstable segments using configurable confidence/entropy thresholds.
- Adapts sampler behavior (temperature/top-p adjustments) in response to instability.
- Performs targeted regeneration by rolling back llama.cpp KV cache state.
- Captures session metrics such as latency, regeneration rate, and adaptation count.
- Prompt is tokenized and sent into the generation engine.
- Each decoded token is observed by the stream monitor.
- A sliding-window detector checks for instability.
- Control engine selects an action: continue, adapt sampling, regenerate, or stop.
- KV cache manager applies rollback when regeneration is triggered.
- Metrics collector tracks token-level and session-level outcomes.
lightweight-llm-runtime/
|- src/
| |- lib.rs
| |- model.rs
| |- tokenizer.rs
| |- generation/
| |- monitoring/
| |- control/
| |- cache/
| \- metrics/
|- tests/
|- benches/
|- python/
| |- adaptive_llm_runtime/
| |- evaluation/
| \- experiments/
|- docs/
\- Cargo.toml
- Rust 2021
- llama_cpp + llama_cpp_sys
- serde / serde_json / toml
- tracing + tracing-subscriber
- criterion (benchmarks)
- pyo3 + numpy (optional Python bindings)
- Python evaluation scripts (TruthfulQA-style benchmarking workflow)
- Core Rust runtime modules are implemented and connected.
- Adaptive control loop and KV-cache rollback path are implemented.
- Unit tests exist for monitoring and control-related behavior.
- CLI entry point exists but is still minimal.
- Python package includes runtime/evaluation scaffolding; some parts are stubbed for future binding integration.
- Rust 1.75+
- C/C++ toolchain compatible with llama.cpp builds
- Python 3.8+ (optional, for evaluation scripts)
cargo check
cargo build --release
cargo test
cargo benchuse adaptive_llm_runtime::{AdaptiveLLMRuntime, RuntimeConfig};
use std::path::Path;
fn main() {
let runtime = AdaptiveLLMRuntime::new(
Path::new("model.gguf"),
RuntimeConfig::default(),
)
.unwrap();
let result = runtime
.generate("Explain token-level uncertainty.", 128, true)
.unwrap();
println!("{}", result.text);
println!("regenerations={}", result.regeneration_count);
}python python/experiments/run_comparison.py --model path/to/model.gguf --samples 20confidence_threshold: trigger adaptation when confidence drops.entropy_threshold: trigger adaptation when uncertainty rises.window_size: number of recent tokens used for instability detection.max_regenerations: cap on rollback/retry operations per generation.
- Expand CLI commands and runtime configuration UX.
- Replace Python stubs with compiled PyO3 bindings end-to-end.
- Add stronger benchmark suites and comparative reports.
- Extend metrics export options and observability outputs.
MIT