8000 Readiness gates do not verify JavaScript initialization on UI page leading to token burn and startup failures · Issue #1 · hengyuan-hu/via · GitHub
[go: up one dir, main page]

Skip to content

Readiness gates do not verify JavaScript initialization on UI page leading to token burn and startup failures #1

Open

Description

@hydrogenbond007

The readiness gates in run_eval.wait_for_sim_ready() and mcp_server._attach_to_record_sim_page() only verify that:

  • /env.json HTTP endpoint responds
  • Chromium CDP port is listening

They do not verify that the page's JavaScript layer initialized successfully.

If the page's inline script throws during load, both gates pass and the browser appears ready, but the UI is completely non-functional. The agent then launches and wastes a full frontier-model episode (~795k input tokens) on a dead UI before timing out, producing no useful verdict.

Evidence

run_eval.py lines 187-204:

def wait_for_sim_ready(proc: subprocess.Popen, base_port: int, ready_timeout: float, settle: float):
    env_url = f"http://localhost:{base_port + 1}/env.json"
    deadline = time.time() + ready_timeout
    while time.time() < deadline:
        if proc.poll() is not None:
            return None
        info = get_json(env_url)
        if info and info.get("demo_folder") and cdp_ready(base_port):  # ← Only checks connectivity
            time.sleep(settle)  # let the page finish its WebGL/WS handshake
            return info
        time.sleep(0.5)
    return None

mcp_server.py lines 2420-2452:

async def _attach_to_record_sim_page(pw) -> tuple[Browser, Page]:
    # Connects over CDP, finds/creates the page at UI_URL
    # ... no verification that the page's JavaScript is alive

The comment # let the page finish its WebGL/WS handshake suggests awareness of async initialization, but no actual gate checks if it succeeded.

Impact

  • Cost: ~795k input tokens burned per dead run (one full frontier-model episode)
  • Frequency: Every startup failure in the JS layer creates a dead episode
  • Root cause: Design gap, not configuration or host-specific

Solution

Add a JavaScript-layer verification gate that executes before launching the agent. Options:

  1. Preferred—lightweight CDP probe (quickest):

    • After _attach_to_record_sim_page() returns, execute a simple JS check:
      alive = await page.evaluate("() => typeof window.initializationComplete === 'boolean'")
      if not alive:
          raise RuntimeError("Page JavaScript failed to initialize")
    • Requires exposing a flag from the UI's initialization code
  2. HTTP endpoint (cleaner, decoupled from MCP):

    • Have record_sim.py expose GET /ready that returns { ready: true } only after full UI setup
    • Call it in wait_for_sim_ready() before returning
  3. Tool call probe (integration check):

    • After attach, try calling screenshot tool and verify a response
    • Catches both JS crashes and MCP attachment issues

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions

      0