FFFF GitHub - ItsAakif/Toris · GitHub
[go: up one dir, main page]

Skip to content
ItsAakifPublic

About

No description, website, or topics provided.

Resources

Stars

0 stars

Watchers

0 watching

Forks

Latest commit

 

History

1 Commit

Folders and files

Repository files navigation

ToRIS - Token-level Runtime Intelligence System

ToRIS is a Rust-based LLM inference runtime that monitors token-level uncertainty in real time and adapts decoding mid-generation.

Why ToRIS

Most inference pipelines decode with fixed sampling parameters, then evaluate output quality after generation. ToRIS moves control into the generation loop itself: it tracks confidence and entropy per token, detects instability patterns across a sliding window, and applies corrective actions immediately.

What It Does

  • Runs GGUF models through llama.cpp from a modular Rust runtime.
  • Computes token-level metrics (selected-token probability, confidence score, entropy).
  • Detects unstable segments using configurable confidence/entropy thresholds.
  • Adapts sampler behavior (temperature/top-p adjustments) in response to instability.
  • Performs targeted regeneration by rolling back llama.cpp KV cache state.
  • Captures session metrics such as latency, regeneration rate, and adaptation count.

Architecture

  1. Prompt is tokenized and sent into the generation engine.
  2. Each decoded token is observed by the stream monitor.
  3. A sliding-window detector checks for instability.
  4. Control engine selects an action: continue, adapt sampling, regenerate, or stop.
  5. KV cache manager applies rollback when regeneration is triggered.
  6. Metrics collector tracks token-level and session-level outcomes.

Project Structure

lightweight-llm-runtime/
|- src/
|  |- lib.rs
|  |- model.rs
|  |- tokenizer.rs
|  |- generation/
|  |- monitoring/
|  |- control/
|  |- cache/
|  \- metrics/
|- tests/
|- benches/
|- python/
|  |- adaptive_llm_runtime/
|  |- evaluation/
|  \- experiments/
|- docs/
\- Cargo.toml

Tech Stack

  • Rust 2021
  • llama_cpp + llama_cpp_sys
  • serde / serde_json / toml
  • tracing + tracing-subscriber
  • criterion (benchmarks)
  • pyo3 + numpy (optional Python bindings)
  • Python evaluation scripts (TruthfulQA-style benchmarking workflow)

Current Status

  • Core Rust runtime modules are implemented and connected.
  • Adaptive control loop and KV-cache rollback path are implemented.
  • Unit tests exist for monitoring and control-related behavior.
  • CLI entry point exists but is still minimal.
  • Python package includes runtime/evaluation scaffolding; some parts are stubbed for future binding integration.

Quick Start

Prerequisites

  • Rust 1.75+
  • C/C++ toolchain compatible with llama.cpp builds
  • Python 3.8+ (optional, for evaluation scripts)

Build and Test

cargo check
cargo build --release
cargo test
cargo bench

Rust Usage Example

use adaptive_llm_runtime::{AdaptiveLLMRuntime, RuntimeConfig};
use std::path::Path;

fn main() {
    let runtime = AdaptiveLLMRuntime::new(
        Path::new("model.gguf"),
        RuntimeConfig::default(),
    )
    .unwrap();

    let result = runtime
        .generate("Explain token-level uncertainty.", 128, true)
        .unwrap();

    println!("{}", result.text);
    println!("regenerations={}", result.regeneration_count);
}

Python Experiment Script

python python/experiments/run_comparison.py --model path/to/model.gguf --samples 20

Configuration Highlights

  • confidence_threshold: trigger adaptation when confidence drops.
  • entropy_threshold: trigger adaptation when uncertainty rises.
  • window_size: number of recent tokens used for instability detection.
  • max_regenerations: cap on rollback/retry operations per generation.

Roadmap

  • Expand CLI commands and runtime configuration UX.
  • Replace Python stubs with compiled PyO3 bindings end-to-end.
  • Add stronger benchmark suites and comparative reports.
  • Extend metrics export options and observability outputs.

License

MIT

About

No description, website, or topics provided.

Resources

Stars

0 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages

0