Structure-Aware Training-Free Editable Diagram Generation from Natural Language
VISTA is a training-free pipeline that turns a natural-language description into an editable diagram, released as DrawIO XML together with a rendered PNG. The output is a real vector source - every node, edge, and container remains an addressable object that can be moved or rewired - instead of a flat raster image.
VISTA assigns each subproblem to the component best suited for it:
- Language models excel at semantic abstraction but degrade when forced to produce precise coordinates.
- Classical layout algorithms are reliable on geometry but blind to semantics.
- Vision-language models can see the rendered result but should not design layouts from scratch.
The pipeline therefore lets an LLM extract semantic structure and layout intent, a deterministic algorithm compute all geometry, and a VLM inspect the rendered diagram jointly with its editable source and issue localized repairs. Because geometry is computed by an algorithm and the VLM only patches the rendered artifact, VISTA requires neither task-specific training data nor parameter updates.
prompt ──> [1. Structure extraction (LLM)] ──> structure JSON
──> [2. Layout
D733
-intent planning (LLM)] ──> grid plan
──> [3. Algorithmic layout (deterministic)] ──> DrawIO XML
──> [4. PNG rendering (DrawIO CLI)]
──> [5. Visual verification & repair (VLM)] ──> corrected XML ──> re-render
- Semantic structure extraction - an LLM converts the prompt into a typed JSON intermediate representation: nodes (with labels and optional roles), directed edges, hierarchical containment, grouping containers, and layout-related attributes. The model is never asked to commit to a position. Lightweight deterministic checks validate the JSON schema and that every edge endpoint and container child refers to a declared node; failures are reprompted with a corrective instruction within a fixed budget.
- Layout-intent planning - a second LLM call maps the structure onto a high-level spatial plan (rows/columns per container, ordering, relative spacing). Like stage 1, it contains no numeric coordinates - only structural decisions.
- Algorithmic layout - a deterministic, constraint-driven extension of Sugiyama-style hierarchical layout computes all geometry: node coordinates and sizes, container bounding boxes, and edge routing with explicit port assignment and crossing minimization (planar embedding when possible, simulated-annealing ordering otherwise). The result is serialized to DrawIO XML.
- Rendering - the DrawIO CLI rasterizes the XML to PNG.
- Multimodal verification and correction - a VLM receives the rendered PNG and
the editable source in one prompt, reports layout defects (node overlap, arrow
crossing, arrows through boxes, text overflow), and proposes localized
source-level edits, which are applied directly to the XML. The corrected source
is re-rendered and re-inspected; the loop stops as soon as the VLM reports no
issues, and runs at most
MAX_FEEDBACK_ROUNDS = 2repair rounds (vista/core/pipeline.py), matching the paper. A transient VLM failure retries the same round once before the loop gives up. If PNG rendering itself fails, a dedicated VLM call repairs the XML syntax first (up to 2 retries).
This separation is what makes VISTA training-free and bounds each component's failure modes: the LLM never sees geometry, the layout algorithm never makes semantic choices, and the VLM never designs a layout from scratch.
vista/ Python package
core/
pipeline.py End-to-end pipeline orchestration and feedback loop
structure_ir.py Intermediate representation + IR construction from LLM output
layout/
engine.py Container-aware layout engine (measure + place)
topo_layout.py Sugiyama-style hierarchical layout
planar_engine.py Planar embedding + simulated-annealing crossing minimization
grid_plan.py LLM layout-intent plan parsing/normalization
edge_router.py Edge routing and port assignment
export/
drawio.py DrawIO XML serialization
png.py PNG rendering via the DrawIO CLI
style/ Node/edge/container styling and color palettes
vlm_check/ VLM-based visual verification and repair (stage 5)
tests/ Unit tests (pytest)
llm/ LLM client (OpenAI-compatible REST / SDK, Gemini protocol)
cli/ Command-line interface
util/ Config, logging, token tracking, text measurement
prompts/ System prompts (required at runtime) and example prompts
run_vista.py CLI entry point
vista.config.example.json Configuration template (copy to vista.config.json)
- Python 3.10+
- Python packages:
pip install requests networkx - DrawIO CLI for PNG rendering. The
renderer is looked up in this order:
export.drawio_clifrom the config (path or command name), then standard install locations (C:\Program Files\draw.io\draw.io.exeon Windows, the app bundle on macOS), thendrawio/draw.ioonPATH.
git clone https://github.com/nice-grp/Vista.git
cd Vista
pip install requests networkx
cp vista.config.example.json vista.config.json # Windows: copy
export VISTA_API_KEY=sk-your-key # Windows: set VISTA_API_KEY=sk-your-key
python run_vista.py prompts/example_prompt_fsm.md outputOpen output/artifacts/<timestamp>/4.diagram.drawio in draw.io or any compatible
editor; the rendered PNGs are written next to it.
python run_vista.py <prompt_file> <output_dir> [options]<prompt_file>- a text/markdown file describing the diagram (seeprompts/for examples covering flowcharts, state machines, and swimlane architectures).<output_dir>- directory for run artifacts.
Useful options (python run_vista.py --help for the full list):
| Option | Effect |
|---|---|
--structure <file> |
Skip the structure LLM call; build the IR from an existing structure JSON |
--grid-plan <file> |
Skip the layout-intent LLM call with a fixed plan |
--force-font-size <n> |
Override the automatic font sizing |
Each run writes a timestamped directory under <output_dir>/artifacts/:
| File | Content |
|---|---|
1.structure.json |
Raw LLM response and parsed structure (stage 1) |
2.grid_plan.json, 2.ir.json, 2.id_map.json |
Layout-intent plan and intermediate representation (stage 2) |
3.layout.json |
Computed geometry for every element (stage 3) |
4.diagram.drawio |
Final editable DrawIO source |
5.diagram_initial.png, 5.diagram_round1.png, ... |
Rendered PNG, one per feedback round |
drawioCode.md |
The final XML wrapped in a markdown code block |
run.log |
Structured run log |
token_report.txt |
Aggregate LLM/VLM token usage |
With export.enable_png: false the pipeline stops after the DrawIO XML step and
skips rendering and the VLM feedback loop.
All configuration lives in vista.config.json (gitignored - never commit real
keys; use vista.config.example.json as the starting point). Keys can be provided
in two ways, and the environment variable takes precedence for any model whose
api_key is left empty:
export VISTA_API_KEY=sk-your-keyEvery entry under llm.models accepts these fields:
| Field | Meaning |
|---|---|
api_key |
Secret; leave "" and set VISTA_API_KEY instead |
model |
Model name sent to the API |
fallback_model |
Used automatically when the primary model fails |
base_url |
API root, e.g. https://api.deepseek.com (/v1/chat/completions is appended automatically; a full .../chat/completions URL also works) |
sdk |
rest (default, no extra dependency) or auto/openai (uses the openai package if installed) |
force_json |
Send response_format: {"type": "json_object"}; keep true for the structure stage |
temperature |
Sampling temperature |
timeout |
Per-request timeout in seconds |
max_retries |
Extra attempts on 429/5xx/network errors (exponential backoff) |
max_tokens / max_completion_tokens |
Output cap; 0 = provider default. Reasoning models spend output tokens on thinking - raise this or leave 0 if responses get truncated |
Protocol note: endpoints are treated as OpenAI-compatible by default. The
Google Gemini wire format (:generateContent) is used only when the URL clearly
points at a Google endpoint (googleapis.com or a /google/ path).
llm.models.defaultdrives the two text stages (structure_generation,grid_plan).llm.models.vision(optional) drives the VLM feedback loop. It must accept image input through the same OpenAI-compatible chat protocol. If omitted, the feedback loop falls back todefault- only do this if your default model is itself multimodal.- Each stage can override fields via
llm.sessions(name+model_ref, plus any model field, e.g. a differenttemperatureorforce_json).
DeepSeek (as shipped in the example config):
{
"llm": {
"models": {
"default": {
"api_key": "", "model": "deepseek-v4-flash",
"fallback_model": "deepseek-v4-flash",
"base_url": "https://api.deepseek.com",
"sdk": "rest", "force_json": true,
"temperature": 1.0, "timeout": 600, "max_retries": 2,
"max_tokens": 0, "max_completion_tokens": 0
},
"vision": {
"api_key": "", "model": "deepseek-v4-flash-vision-exp",
"fallback_model": "deepseek-v4-flash-vision-exp",
"base_url": "https://api.deepseek.com",
"sdk": "rest", "force_json": false,
"temperature": 1.0, "timeout": 600, "max_retries": 2,
"max_tokens": 0, "max_completion_tokens": 0
}
},
"sessions": [
{"name": "structure_generation", "model_ref": "default"},
{"name": "grid_plan", "model_ref": "default"}
]
}
}Any other OpenAI-compatible provider - change base_url and model, nothing
else:
- OpenAI:
https://api.openai.com+ e.g.gpt-4o - Qwen (DashScope compatible mode):
https://dashscope.aliyuncs.com/compatible-mode/v1+ e.g.qwen-plus - Moonshot Kimi:
https://api.moonshot.cn/v1+ e.g.kimi-k2.5
For the vision entry, pick a multimodal model from the same provider (e.g.
gpt-4o, qwen-vl-plus, moonshot-v1-8k-vision-preview). If you point at a
Gemini endpoint directly, the Gemini wire format is selected automatically.
A reasoning vision model produces more thorough (but slower and pricier)
analyses: its chain-of-thought is billed as completion tokens, so a single
feedback round can cost tens of thousands of output tokens. A non-reasoning VLM
is cheaper; the analysis reports let you compare quality either way.
Each round performs:
- Render the current XML to PNG (DrawIO CLI).
- Ask the VLM to analyze the PNG + XML against the original prompt and output
(a) a list of layout defects and (b) a corrected full XML behind a
---CORRECTED_CODE---separator. - If the VLM reports no issues, the diagram is accepted. Otherwise the corrected XML replaces the current source and the loop continues, for at most 2 repair rounds.
The analysis prompt prioritizes repairing layout through structure and placement changes over cosmetic tweaks, mirroring the method described in the paper.
python -m pytest vista/core/tests -qThe suite covers hierarchical layout spacing, growth directions, degenerate inputs (empty graph, single node, self-loops, parallel edges), and edge routing.
| Symptom | Fix |
|---|---|
drawio CLI not found; skip PNG export |
Install the DrawIO desktop app, or set export.drawio_cli to its full path |
VLM API not configured |
Set VISTA_API_KEY or fill api_key in vista.config.json |
| Truncated / empty LLM responses | Raise max_tokens, or increase timeout; reasoning models need headroom for thinking |
Invalid JSON in config file |
Fix the JSON syntax in vista.config.json |
| 401 / permission errors from the provider | Check VISTA_API_KEY and that the account has quota |
Every LLM/VLM call is tracked in-memory by vista/util/token_tracker.py and
summarized at the end of a run. For reasoning models the provider counts thinking
tokens as completion tokens, which dominates the total.
@misc{vista2026,
title = {VISTA: Structure-Aware Training-Free Editable Diagram Generation
from Natural Language},
author = {Han, Dongqi and Lin, Bojie and Wang, Junhan and Luo, Jiefu},
year = {2026},
note = {Preprint}
}