A live Strudel performance engine that an LLM can safely conduct in real time.
The invariant:
The LLM may propose musical and speech changes, but only the live engine validates, schedules, applies, logs, and rolls them back.
performance_state is the source of truth. Compiled Strudel and synthesized
speech audio are derived, inspectable artifacts. SQLite stores sessions, patch
log, rollback snapshots, speech manifests, and imported music plans.
LLM / agent / human operator
-> typed JSON tool calls
@mnehmos/strudel-mcp-server (Node)
-> validate / queue / apply / panic / rollback
-> compile to host commands
-> server-side OpenAI TTS (server-only .env)
WebSocket control bus (ws://127.0.0.1:8787)
-> apps/host : @strudel/web runtime, transport HUD, master/speech mix
-> apps/controller : operator panel for live overrides
packages/schemas- Zod schemas for performance state, patches, speech intents/manifests, queued patches, music plans.packages/live-engine- validation, rollback, scheduler/clock, host-command compiler.packages/speech- server-side OpenAI TTS client, manifest writer, env config.packages/storage- SQLite session store: sessions, patch log, rollback snapshots, speech manifests, music plans.packages/mcp-server- real MCP server (stdio JSON-RPC via@modelcontextprotocol/sdk) plus WebSocket control bus.packages/recorder- recording manifest + ffmpeg mux scaffolding (secondary).apps/host- browser host that evaluates compiled Strudel source through@strudel/weband plays scheduled speech with optional music-ducking.apps/controller- operator UI: transport, BPM, templates, snapshot/rollback, track ops, queue, music_plan import, speech lane.
| Tool | Actions |
|---|---|
session_manage |
create, open, save, list, close, snapshot, restore, load_template |
performance_manage |
play, stop, pause, resume, panic, set_safe_mode, set_strudel_program, clear_strudel_program |
deck_manage |
load_scene, mute, solo, set_gain |
track_manage |
add, remove, set_pattern, set_strudel, set_gain, mute, unmute, solo, unsolo |
cue_manage |
import_music_plan, fire_cue, list_cues |
patch_manage |
propose, validate, preview, apply, rollback, list_queue, flush_queue |
speech_manage |
propose, validate, synthesize, schedule, play, mute, list, rollback |
render_manage |
record_master, export_manifest, mux_with_video |
observe_live_state |
read, compiled |
set_pattern is the compatibility shorthand for role-based tracks. set_strudel
is the Strudel surface for a single track expression; the compiler adds the
reactive $: line and optionally appends the track gain stage. For top-level
Strudel code, performance_manage.set_strudel_program can install either a
track-mode prelude (register, helper functions, reusable methods) or a full
source-mode program that is evaluated as-is by @strudel/web.
Patches and speech intents are rejected for any of:
eval,fetch,import(...),require(...),Function(...), script tagsprocess,globalThis,window,document,self,Worker,XMLHttpRequest- Pattern statements (
;,{,}) or quotes in mini-notation - Strudel expression statements, assignments, function definitions, host-global access, or unsafe roots outside
s(...),note(...),n(...),stack(...), etc. - Full Strudel programs may use declarations,
register(...), arrows,$:pattern lines, and helpers, but still reject host globals, imports, network fetches, dynamic function construction, and explicit loops. - Unbalanced brackets in mini-notation or Strudel expressions
- Samples not in the allow-list for drum, sting, and duck roles
- Gain above the per-track ceiling (default 0.85)
- Master gain above ceiling (default 0.95)
- BPM jumps beyond the configured delta (default 18)
- Patches that would push active voices over the ceiling (default 12)
- Destructive ops without a prior rollback point
- Operations blocked in
safe_mode(add/remove track, BPM, master gain) - Speech missing AI disclosure, overlong duration, embedded URLs/script tags, or gain above 0.8
Panic mutes every track, hushes Strudel, and forces safe mode on.
The model proposes a SpeechIntent:
{
"id": "dubstep_tutorial_intro",
"text": "First, take a tight kick and let the bass answer it.",
"voice": "coral",
"model": "gpt-4o-mini-tts",
"instructions": "Energetic UK bass-music tutorial narrator. Short, clear, excited, not shouty.",
"apply_at": "next_bar",
"gain": 0.55,
"duck_music": true,
"duck_depth": 0.55,
"max_duration_seconds": 8,
"disclosure": "AI-generated voice"
}The server validates, calls OpenAI TTS, writes the audio to
data/recordings/speech/{session_id}/{speech_id}.wav, persists the manifest
(text hash, voice, model, gain, duck, disclosure, duration estimate), and
emits a schedule_speech host command. The host plays the artifact and ducks
music when requested. All speech actions are logged and rollback-aware.
The OpenAI API key only lives in the server-side .env. No browser bundle
imports @mnehmos/strudel-speech.
pnpm install
pnpm check
pnpm dev:mcp # MCP server + WebSocket bus on :8787
pnpm dev:host # Browser host on :5177
pnpm dev:controller # Operator UI on :5178For the MCP stdio transport:
node packages/mcp-server/dist/index.js --stdioCopy .env.example to .env and fill in OPENAI_API_KEY to enable speech
synthesis. Without a key, the rest of the system still runs; only
speech_manage.synthesize errors.
cold_briefing- low-tempo briefing pulse for video narration.war_room_escalation- denser briefing escalation with hats, bass, and alarm stings.market_panic- faster financial/news tension bed.imperial_decay- sparse drone and impact grammar.acid_glitch_livecode- kick/duck spine, acid bass, break slicing, top pressure, rare alarms.deep_acid_minimalandtrance_acid_drop- Strudel-native acid chains with filters, resonance, and distortion.
templates/style_recipes.json keeps style rules as data so the LLM proposes
lane and knob intent while the engine validates the actual patch.
examples/myanmar_junta_21/music_plan.json shows the import shape: video id,
duration, BPM, style, sections with energy curves, optional cue list. The
engine merges plan cues into the live state, persists the plan in SQLite, and
proposes section-energy/BPM patches automatically through the same validation
pipeline.
- Real MCP server (stdio JSON-RPC) with consolidated typed tools and a WebSocket bridge to host + controller.
- Performance state engine with sessions, templates, snapshots, rollback, patch queue, panic, and safe mode.
- Scheduler with musical boundaries (
immediate,next_beat,next_bar,next_4_bars,next_phrase), bar/phrase advancement, and missed-window detection. - Validator/reflex layer for patterns, Strudel expressions, gain, BPM, voices, safe-mode, and speech intents.
- Server-side OpenAI TTS lane with text-hashed manifests, music ducking, and rollback/mute.
- Strudel Web host runtime that evaluates compiled source and plays scheduled speech through the speech router.
- Operator controller UI that drives every tool over the WebSocket bus.
- Tests covering schema, scheduler, speech, runtime, MCP tool catalog, host command, and validator behavior.
Strudel itself is not vendored or mutated. Music goes through @strudel/web.
Structured MCP track control compiles into Strudel source, and source-mode
programs are passed through as complete Strudel programs after validation,
logging, and rollback capture.