You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Readiness gates do not verify JavaScript initialization on UI page leading to token burn and startup failures #1
The readiness gates in run_eval.wait_for_sim_ready() and mcp_server._attach_to_record_sim_page() only verify that:
/env.json HTTP endpoint responds
Chromium CDP port is listening
They do not verify that the page's JavaScript layer initialized successfully.
If the page's inline script throws during load, both gates pass and the browser appears ready, but the UI is completely non-functional. The agent then launches and wastes a full frontier-model episode (~795k input tokens) on a dead UI before timing out, producing no useful verdict.
Evidence
run_eval.py lines 187-204:
defwait_for_sim_ready(proc: subprocess.Popen, base_port: int, ready_timeout: float, settle: float):
env_url=f"http://localhost:{base_port+1}/env.json"deadline=time.time() +ready_timeoutwhiletime.time() <deadline:
ifproc.poll() isnotNone:
returnNoneinfo=get_json(env_url)
ifinfoandinfo.get("demo_folder") andcdp_ready(base_port): # ← Only checks connectivitytime.sleep(settle) # let the page finish its WebGL/WS handshakereturninfotime.sleep(0.5)
returnNone
mcp_server.py lines 2420-2452:
asyncdef_attach_to_record_sim_page(pw) ->tuple[Browser, Page]:
# Connects over CDP, finds/creates the page at UI_URL# ... no verification that the page's JavaScript is alive
The comment # let the page finish its WebGL/WS handshake suggests awareness of async initialization, but no actual gate checks if it succeeded.
Impact
Cost: ~795k input tokens burned per dead run (one full frontier-model episode)
Frequency: Every startup failure in the JS layer creates a dead episode
Root cause: Design gap, not configuration or host-specific
Solution
Add a JavaScript-layer verification gate that executes before launching the agent. Options:
Preferred—lightweight CDP probe (quickest):
After _attach_to_record_sim_page() returns, execute a simple JS check:
Activity