Repository navigation
livekit/voice-adventure: The Drowned Bell, a LiveKit voice adventure with room webhooks through Event Gateway - #49
Merged
Merged
Conversation
LiveKit room, participant and egress webhooks through Hookdeck into a call log. Verifies with WebhookReceiver on the raw body; the log is idempotent and order-independent. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The setup script sent api_key and api_secret as top-level config keys. hookdeck-cli 3.1.0 moves a top-level api_key into config.auth as a generic API key and leaves api_secret behind, so the API would reject the LIVEKIT source for a missing config.auth.api_secret. Also correct the header comment: core verifies LIVEKIT deliveries, not the edge. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…rce auth through the API Setup now goes through _shared/hookdeck/hookdeck-cli.ts (synced), so the LIVEKIT source type's config.auth is set with a PUT to the API: the API key and secret stay off the command line. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ptures Adds the shared scene kit and topology pane (copied unchanged from the ElevenLabs branch, d771e77), a LiveKit scene (call lifecycle, duplicate dropped, bad signature, app down then retry), the page's bottom pane, failure injection, dedup and retry rules on the connection, npm run simulate, and landscape and square MP4s. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Uses LiveKit Cloud's management API with the signed-in lk user's session (or LIVEKIT_SESSION_TOKEN). Falls back to printing the dashboard steps. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…n in the pane Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The .gitignore rule for captured deliveries (captures/) also matched scenes/captures. Deliveries are written to data/, which is already ignored. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
POST /api/token mints a 10 minute room token on the server; the page joins with livekit-client 2.24.0 and the microphone. An optional test participant (the lk CLI, server-side) makes a two-person call. npm run activity stays as an optional path. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…d a session chronicle An intro agent (Marla) learns each player's name, then hands off to a game master that runs four scenes (agents-js, LiveKit Inference only). At each scene the app pulls a generated soundtrack into the room with the Ingress API (URL input). The page builds a session chronicle from the webhooks, telling players, the agent and soundtracks apart by participant kind. The live pane gets an outbound INGRESS edge. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…n the call The Join a call page published the microphone but never attached remote audio tracks, so the story agent and soundtracks were silent. Attach each subscribed audio track, start audio playback after joining, and list the participants with their role, the agent's state and who is speaking. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…nner, 3 minute soundtracks Hester Pennywhistle (innkeeper) writes each name in her ledger and fetches Old Bram Tallow (storyteller), who runs the four scenes. Different voices through LiveKit Inference. The page shows a live transcript from the lk.transcription text streams, whose turn it is, which soundtrack is playing, and when the story has ended (with a Start a new story button), from attributes the agent sets on its own participant. Bram's turn hand-overs rotate and each player gets one turn per scene. Soundtracks are regenerated at 3 minutes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…rites it in the ledger without asking A token with no name leaves the participant unnamed (it used to default to "Player"), and the innkeeper asks only those players. Named players, a second player joining later and the bot (known by its identity) are written in without a question. The choice is a pure function (agent/src/names.ts) with unit tests. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…as a mute scripted player OPENAI_API_KEY switches the language model only (agents-plugin-openai 1.10.0); speech stays on LiveKit Inference. Pixel is now a small rtc-node participant (no lk CLI, no media) that sends one scripted gesture line on lk.chat when the storyteller gives it the turn; Old Bram weaves it in on his normal turn, so it adds no model call. Hester greets Pixel at the door. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… sweep leftovers, drop the OpenAI fallback The app deletes a room's scene ingresses at room_finished and on the agent's end_story call (as it already did when the next scene starts), and sweeps leftover scene:* ingresses from gone rooms at startup. The agents use LiveKit Inference gpt-4.1-mini only. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… re-ledger known guests The earlier trigger waited for the agent to be listening, but the storyteller kept talking through Pixel's turn, so the gesture never fired. give_turn now waits for Pixel's lk.chat message and returns it as the tool result, so Bram weaves it into his reply (no extra model call). Pixel logs to data/pixel.log. Verified without a model against a fake agent participant: Pixel sends, the agent receives, the page shows the line. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…call Live, the model wrote 'functions.fetch_bram()' as speech instead of calling the tool and kept asking a named player for a name. The announcement is now spoken with session.say (no model call), the handoff is session.updateAgent after it plays, and write_in_ledger is the only tool Hester has. For a player who typed a name there is no model call at the door at all. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… with a volume slider Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…in a How it works dialog Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…lays out, real soundtracks, design-system UI Bram offers concrete choices and moves on when the players take one (2-3 exchanges per scene, nudge at 5), Pixel's gestures never decide, and he speaks plainly. end_story is called before the closing and the agent leaves only after the closing has played out plus 2 s. The soundtracks are 3 minute excerpts of public-domain and CC0 field recordings (CREDITS.md). Pixel's button disables while Pixel is in the room. The story banner, people list, transcript and volume slider use ds .message, .badge and .button. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Found in the live run: Bram offered the choice to Priya while the listener was still linked to Pixel (needed to receive its chat message), so her answer was never heard. give_turn now returns the listener to the first player who can speak once Pixel's gesture has arrived. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The model chained give_turn and next_scene with no player speaking, then wrote a tool name as speech after hitting the tool-step limit. give_turn is gone: Bram ends a reply by naming the next player and, once his speech has played out, code links that player and sets story.turn. Exchanges count only final player transcripts and Pixel's chat gesture; next_scene and end_story are refused (silently) without new play 8000 er input or before two inputs. maxToolSteps is 2, and a TTS text filter drops tool names as a backstop. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…r the player Live check of c44958f: Bram waited for the player (a user transcript before every reply) and no tool name was spoken, but a refused next_scene/end_story left the player's answer unanswered, the choice to move on was never offered (instructions were updated after the reply started), Bram opened with a question before the first scene, and he invented a Pixel who was not there. The scene note is now a system message added in onUserTurnCompleted (also for Pixel's gesture), a refusal triggers one no-tools reply, the first reply calls next_scene before speaking, and the prompt names a mute traveller only when there is one. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
In the last live run a second dispatch put a second agent in the room two minutes in, and it restarted the story with a new Hester while the first agent's LLM request was hung. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
If Bram owes a reply and nothing happens for 20 s (a hung model request), he says one short in-character line and the turn goes back to the player. The webhook record of the earlier second agent job shows it was LiveKit re-dispatching the same job (same agent identity) after the first job's process stopped responding, two minutes after Pixel joined, not Pixel's token; Pixel's token no longer asks for the agent anyway. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
_shared/theme/theme.js adds an optional demo theme to the System/Light/Dark switch (data-demo-theme on html), synced into this demo only. public/theme-story.css overrides design-system tokens on main and the dialog (parchment, ink, gilt, wax-seal crimson, candlelight; Cinzel Decorative, Cinzel, IM Fell English), with double gilt frames, a compass rose and divider, a candlelight Your turn banner, a dark The End card and a drop cap. Themed headings and microcopy sit beside the plain ones (only-plain / only-story). The Data flow pane is unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…Story type, small fixes Bram no longer pushes: no minimum exchanges, no nudge, no offer-a-choice phase; a scene changes only when a player clearly chooses to move on (still needs a player input since the last tool call). He describes what players see, hear and notice, never directs, hands over with a varied open prompt, reveals clues only on investigation and gives one gentle nudge only when asked. Story theme type is 15-20% smaller with tighter cards; swash capitals get tracking and ligatures off; Pixel's gesture lines lose their asterisks and are italic; the empty Calls card points at Join a call; README notes that a dead storyteller job restarts the story. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…aker's speech segments Speech-to-text splits an utterance at each pause, so one thing a player said showed as several entries. Consecutive segments from the same speaker now join the newest entry until someone else speaks. The demo-state Bram line describes instead of offering a choice. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ording, pane polling that logs, backs off and resumes A small separate model call decides after each player input whether the player has chosen to leave by the way out; the code then changes the scene (state, soundtrack ingress, attribute, saved metadata) and Bram describes arriving. next_scene stays as a guarded tool. The flow feed logs failing endpoints with their status, backs off, resumes on reopen and visibility change, and always ends the history phase; the pane ignores a zero-width (closed) canvas. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…iefly and shortens the longer they stay Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…), satchel on the page, you-not-names rule Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…eaker's name; test instance can run on its own agent name, port and data dir Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…aces, a per-turn action resolver, no Bram tools Places with named exits both ways (tavern, forest, hollow oak, cave, lighthouse), items that stay put, one shared satchel, and a small model call per player input that reads moves, takes, uses and gives; the code validates and applies them and ends the story when the Lantern is lit. Pixel's line is chosen by the agent from place and satchel, with a cooldown, a turn about every third round, and skips when nothing fits. State saves location, satchel and puzzle flags. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…re instead of guessing; the chest is bolted down; STORY_ITEMS for test starts Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…s and the browser not to autofill it Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…atus on its own line "Pixel travels with you" in the Story theme's spaced capitals pushed Leave onto a second row. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…generic gesture lines, answers when addressed, skip reasons logged Pixel never got a turn because players were registered once, when Hester handed over to Bram; a Pixel (or a named player) joining later was never added to the roster. A participantConnected handler now adds late joiners, updates Bram's instructions and has him acknowledge the arrival. Every place gets generic gesture lines that need no items; a player speaking to Pixel (or giving it an item) is flagged by the resolver or by name, Bram acknowledges without speaking for it, and Pixel answers on the next turn. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…he item Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…lines, destination-first resolver, blocked moves name the real exits Pixel never holds the turn and nothing waits for him: he acts only when addressed, or once on arriving somewhere, as an extra whose gesture goes into the transcript and is woven into Bram's next reply. A 5 s guard moves the listener back to a human if a bot ever holds it. Gesture lines carry puzzle-flag conditions and there are fewer of them. The resolver sees each exit with its destination and aliases and the code lets a named destination outweigh words like door; a blocked move gets a note listing where the party can go. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…g the ways out Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…te, so a changed chest, taken item or lit lamp is never described as it was Places keep only fixed ambience; item, chest and lamp phrases and clues come from the state. Already-described tracking compares against the current phrases and drops ones that are no longer true. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…r's words, the Lantern is the Mere's own flame and is kept, and Bram cannot narrate changes the code did not make Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… not obvious (name, Pixel, soundtrack volume, items, and each log section), with Story wording Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This was referenced Nov 18, 2026
…retry window: verify with a clock tolerance, pin livekit-server-sdk 2.19.3, README no longer says the LIVEKIT source path works the same way Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…l me" or "I'm", never from "It's Tuesday", "This is Pixel's idea" or "It's Bram" Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…t; an event counts as seen only once applied; store writes are atomic; deliveries.jsonl redacts the token and is capped Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…te joiners with no name are heard, a stall line keeps the turn, a stuck ending finishes with a short closing and ignores typed input; the agent signs its scene calls and restarts a finished soundtrack on arrival Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…en story rooms made by the app are joined; soundtracks restart when they end and are HEAD-checked first; local bind, caps, secret-header scene calls, guarded async handlers Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… reach the agent, safe running, redacted deliveries; HOST and caps in .env.example Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… when verified, keep the Hookdeck API key off the CLI command line and out of errors Cherry-picked from 5fe85ba (feat/vapi-agency-routing); topology-pane.js merged with this branch's own changes to it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…tional settings in .env.example, README formatting, theme test reads the demo's own theme.js Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
# Conflicts: # README.md # _shared/manifest.json
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds
livekit/voice-adventure: "The Drowned Bell", a spoken adventure run by two LiveKit agents for one or more players (with an invite link to bring friends into a running story), with every room event delivered as a LiveKit webhook through Hookdeck Event Gateway to an app that verifies it and builds a call log, a session chronicle and a live topology pane.The demo
livekit-client2.24.0). The server mints a 10-minute room token, so the API secret stays on the server. The first player's token dispatches the story agent; Pixel's doesn't. The name field locks while you're in the room; if you joined without a name, it fills with the name Hester wrote in her ledger, which also becomes your display name.?room=story-.... Opening it and selecting Join a call puts you in the same story. The app mints a token only for a room it named itself (story-and hex digits, made on the server) that is still open, so a made-up or ended link is refused with a message. Continue the story and Add Pixel accept the same rooms only, and at mostMAX_ROOMS(6) stories run at once.agent/,@livekit/agents1.10,npm run agent): Hester Pennywhistle, the innkeeper, greets each player by the name they typed and writes it in her ledger, then hands over to Old Bram Tallow, a retired lighthouse keeper who tells "The Lantern of Hollow Mere" in four scenes. Speech-to-text, the LLM (openai/gpt-4.1-mini) and text-to-speech all run on LiveKit Inference, so the demo needs only LiveKit and Hookdeck.{ move, take, use, give }, and a code resolver validates each part against the world: only real exits from the current place (a destination the player names outweighs the model's guess, and a place with no way from here is refused with one plain sentence naming the real exits), only items that lie here, and the chest only with the key. The Lantern is the Mere's own flame: it is cold when the party takes it, wakes only with the flint and wick at the lighthouse (a rule decided on the player's words too), lights the great lamp from its flame, and stays with the party. What can be seen is built from the world state, so a changed chest, a taken item or a lit lamp is never described as it was, and Bram narrates from a note the code writes (where the party is, what lies here, the satchel, the ways out, what just happened) and never describes a change the code didn't make. Code, not the model, picks the next speaker. A speech filter strips any tool name before text-to-speech. Turns end on voice activity (0.7 to 3 s), and watchdogs recover a turn that never closes or a reply that never comes. Preemptive generation and session recording are off (job memory about 250 MB). Where the party is, the satchel and the puzzle flags are saved in the room metadata./api/story/continue, an explicit agent dispatch).@livekit/rtc-nodeparticipant that never holds a turn and that nothing waits for. It acts when a player speaks to it (you can give it an item), and once when the party arrives somewhere new. The agent picks a gesture line that fits the place and the satchel, publishes it as attributes, and Bram weaves it into his next reply; Pixel sends the line as a chat message that the page shows in the transcript. Those chat messages don't reach the agent (it takes text only from the participant it is linked to), so Bram reacts through the agent's own notes. A solo tester still has a second player at no extra LLM cost.ingress_endedwebhook, or a check on arrival) the same one starts again while the story is still in that scene. It is deleted when the story or the room ends, and a late or replayedroom_finishedcan't delete a newer room's soundtrack. The excerpts are public-domain or CC0 field recordings; sources and licences are inpublic/soundtracks/CREDITS.md.npm run setup): a Hookdeck WEBHOOK source, CLI destination and connection (dedup onbody.idfor 10 minutes; exponential retry, 8 attempts from 30 s), and the LiveKit Cloud webhook through LiveKit's management API when you've signed in withlk cloud auth --user-auth, with the dashboard steps printed otherwise.WebhookReceiverfromlivekit-server-sdk(pinned to 2.19.3). LiveKit's token expires 5 minutes after it is issued and Event Gateway forwards the originalAuthorizationheader on every retry, so the app verifies with a clock tolerance that covers the connection's whole retry window (about 2 hours 7 minutes, from the retry rule) plus 5 minutes. The signature and the body hash are still checked; the trade-off is that a captured request stays acceptable for that long, and events apply idempotently. Once the LIVEKIT source type is in production, Event Gateway verifies, and the demo should rely on that instead of re-checking the forwarded token (a TODO in the README).scenes/captures/. The pane's feed logs and backs off on polling errors and catches up when reopened or when the tab is visible again.public/theme-story.cssrestyles the page as an old-world tavern ledger. The new shared_shared/themepiece adds a per-demo theme to the System / Light / Dark switch, plus per-theme wording (theme-<id>.copy.js, e.g. "Enter the inn" for "Join a call"); this demo declaresdata-demo-theme="story"and defaults to it. The theme restylesmainonly, so the Data flow pane keeps the standard Hookdeck look. Explanatory text lives in hover tips (an ⓘ next to the headings, and on the name field, Pixel and the soundtrack volume), with Story wording, so the main view stays on the use case.Live-run facts (2026-11-16)
room_started,participant_joined,track_published,track_unpublished,participant_leftandroom_finished(152 B to 1.1 KB each); every delivery passedWebhookReceiverthrough Event Gateway.kind: "AGENT", soundtrack ingresseskind: "INGRESS", and players nokind; the chronicle uses that to tell them apart. Ingress webhooks carryingressInfo, notroom.room_finishedarrives after the room's departure timeout, tens of seconds after the last leave. The webhook token expires 5 minutes after it's issued.Review fixes
A code review of the demo found these; each is fixed, with a test:
127.0.0.1(opt-inHOST);/api/sceneand/api/scene/endneed a secret header from the agent (derived from the API secret) instead of a check on the caller's address; story rooms and Pixel processes are capped; async handlers are guarded so an error is a 500, not an unhandled rejection; store writes are atomic;data/deliveries.jsonlstores theAuthorizationheader as[redacted]and keeps the last 500 deliveries.Known limits
main) doesn't resolve; setSOUNDTRACK_BASE_URLto try it from this branch.livekit/call-log; setup moves an existing LiveKit webhook to the renamed Hookdeck source, and the oldlivekit-call-logHookdeck resources are safe to delete.Shared pieces
_shared/flow-draweris byte-identical to #47. This PR changes_shared/scenes/host/flow-feed.tsand_shared/topology-pane/topology-pane.js(polling that logs, backs off and resumes); #44, #47 and #48 carry the older copies, so after merging runnode scripts/sync-shared.mjsto bring their demo copies up to date._shared/themeis new and synced into this demo only; other demos keep their owntheme.jsfor now. The WEBHOOK source reportsverified: falsefor every request, so this demo maps that in its own code rather than changing the shared flow drawer.Merging
_shared/manifest.json; keep every entry.Checks
node scripts/sync-shared.mjs --checkand both typechecks pass; 59 demo tests and 58 agent tests pass at the branch head (the scene, pane and invite-link tests run in Chrome; the invite-link test stubs the app's token endpoint andlivekit-client). The invite link and the page were also checked in a browser at 1100 and 390 px wide, and the token, scene and Pixel endpoints were exercised with curl against a test instance. The story was played end to end in Chrome by a person before the quest and review changes; those were checked by tests and short runs, and a person plays them next.Merge readiness (
5eed3f5): root README shared rows; optional settings in.env.example; README formatting; the theme test reads the demo's owntheme.jssonpm testpasses in a standalone copy.🤖 Generated with Claude Code