VLAG: Graph-Based Planning for Vision-Language-Action Models
A graph-based robot planner that connects specialized vision, language, and action models instead of relying on one large end-to-end model.
Ardalan Aryashad, Yan Jin — USC IMPACT Laboratory · ASME IDETC/CIE 2025
Abstract
VLAG is a modular framework for long-horizon robotic manipulation that combines a graph-based router with specialized vision, language, and action experts. Rather than relying on a monolithic Vision-Language-Action (VLA) backbone, VLAG performs lightweight state-transition selection from visual observations and language instructions, then activates the corresponding action expert for execution. The vision module predicts task-relevant environment states, the language module improves instruction-to-task alignment through domain-specific fine-tuning, and the action experts build on the Action Chunking with Transformers (ACT) architecture to generate control trajectories. This separation between routing and execution preserves an interpretable planning structure and supports matched baseline comparisons, diagnostics, ablations, and efficiency measurements on the CALVIN benchmark. At roughly 100M parameters, VLAG provides a compact and extensible alternative to billion-parameter end-to-end VLA models for seen-environment long-horizon manipulation.
Motivation
Monolithic VLA models—RT-1/RT-2, OpenVLA, Octo, π-0—transfer web-scale knowledge to robotic control with strong generalization, but they pay for it at deployment: billion-parameter inference at every control step introduces latency and compute requirements that are problematic when high-frequency action prediction is critical. Conventional independent action policies are efficient and precise but are trained on single, short-horizon tasks and lack the planning structure needed to chain sub-tasks together. Hierarchical methods add planning structure, yet they typically depend on generative subgoal modules (predicted images or latent waypoints) that are expensive to train and opaque at inference time.
These three paradigms share a common trade-off:
- Large VLAs — strong generalization, high computational cost.
- Hierarchical planners — added planning structure, but reliant on opaque generative subgoal modules.
- End-to-end policies — efficient and precise, but limited to single-stage tasks.
VLAG targets this trade-off directly by decomposing the monolithic VLA into distinct, efficient expert components coordinated by an explicit, graph-structured planner. The router decomposes the available dataset into a network of tasks connecting discrete environmental states, enabling long-horizon planning without billion-parameter foundation-model inference at every step. Unlike prior hierarchical methods, VLAG performs explicit state-transition routing over a pre-built graph, yielding a transparent and modifiable planning structure rather than a generated subgoal.
The primary contributions are threefold:
- Architectural — a modular Mixture-of-Experts (MoE) framework in which a graph-based router replaces the monolithic VLA backbone, enabling lightweight task selection without large foundation-model inference at every step.
- Algorithmic — a graph-construction and inference-time routing procedure that maps visual state representations and language instructions to specialized action experts, decomposing long-horizon tasks into a structured sequence of sub-task transitions.
- Empirical — a compact (~100M-parameter) system that reaches a competitive average sequence length of 2.09/5 on CALVIN D→D while remaining deployable on mid-range GPU hardware, supported by matched-baseline evaluations, diagnostics, timed efficiency measurements, multi-seed statistics, and component ablations.
System Overview
VLAG is a modular Mixture-of-Experts framework in which a graph-based router interprets visual observations and language instructions, maps the current environmental state to graph nodes, and selects the appropriate action expert for each sub-task along a long-horizon sequence. At each time step \(t\), the directed graph identifies a node—representing the environmental state—and activates the corresponding edge, which represents a specialized action expert. That expert then takes control, predicting the robotic actions required to fulfill the sub-task.
The Graph Router
Long-horizon manipulation tasks require extended action sequences and explicit planning. VLAG addresses this by mapping sub-tasks to transitions in a directed graph \(G=(V,E)\), where nodes \(V\) correspond to distinct environmental states and directed edges \(E\) represent the action experts that transition between them. Crucially, the router performs selection, not full sequential planning: given the current state and a language instruction, it identifies the single best-matching edge to activate at that moment. Long-horizon execution emerges from repeated, single-step routing—after each expert completes its sub-task, the vision module re-evaluates the updated state and the router selects the next edge, continuing until the instruction is satisfied or no matching edge is found.
The graph is constructed from a dataset \(D=\{(s_t,a_t,o_t)_{t=0}^{T},(l_i)_{i=0}^{N}\}\) of robot states, actions, observations, and language instructions. For each task trajectory, the start (\(t=0\)) and end (\(t=T\)) states are encoded by the vision module into feature vectors, and nodes are instantiated for each. A cosine-similarity check against existing nodes, with an empirically chosen merge threshold of 0.98, keeps the graph compact: states below the threshold spawn a new node, while matches append a timestamp to an existing one. Directed edges are then drawn between start and end nodes, labeled with the task’s language instruction. When multiple outgoing edges are candidate matches, the language module resolves the tie by selecting the edge whose task label has the highest cosine similarity to the user’s instruction.
The 0.98 operating point yields 18 nodes, 145 directed edges, and 20 unique task labels in the sampled graph—preserving task coverage while avoiding the excessive state merging seen at lower thresholds and the fragmentation seen at higher ones. Routing cost is modest relative to policy execution: node matching is \(O(\lvert V \rvert p)\) in feature dimension \(p\), followed by \(O(\bar{d}_{\mathrm{out}} q)\) language-based ranking over outgoing edges, where the small average out-degree makes node matching the dominant cost.
Vision, Language, and Action Modules
Vision module. A frozen CLIP encoder (ViT-B/32) processes RGB observations from the static CALVIN camera, and two MLP heads predict the states of six key elements—four continuous variables (sliding door, drawer, button, switch) and two Boolean variables (lightbulb, green light). Both heads share the same backbone architecture; the continuous head is trained with MSE loss and the Boolean head with BCE-with-logits for numerical stability. These predictions form the node feature vectors used by the router for state localization.
Language module. Sentence-BERT (SBERT) embeds both free-form instructions and canonical task names into a shared latent space, and cosine similarity selects the closest task name. Because out-of-the-box embeddings underperform on CALVIN’s domain-specific terminology, SBERT (paraphrase-MiniLM-L6-v2) is fine-tuned on instruction-task pairs with a cosine-similarity loss, sharpening routing precision.
Action module. Each action expert builds on ACT, which pairs a Conditional Variational Autoencoder (CVAE) with a DETR-based transformer to predict chunked action sequences. The CVAE encoder consumes observations, the task embedding, and ground-truth actions to infer a latent \(z\); at inference, \(z\) is set to its mean for stable execution. Each expert is trained on a task-specific subset of data grouped by language instruction, so every expert specializes in one category of manipulation. Overlapping chunks are blended with an exponential weighting scheme \(w_i = \exp(-m i)\) that prioritizes the most recent predictions.
The full pipeline is strictly modular: the language model is fine-tuned, the vision MLP heads are trained to predict key-element states, the graph is constructed from dataset trajectories, and semantically clustered task groups become the refined datasets used to train each action expert. Any component—vision, language, or an individual expert—can be independently optimized or replaced without retraining the others.
Experiments on CALVIN D→D
We evaluate on the CALVIN benchmark—34 manipulation tasks collected over 24 hours of teleoperation across four environments, with RGB-D images from a static scene camera and a gripper-mounted camera. Experiments use the D→D (“seen”) split, in which training and validation occur in identical environmental configurations, with over 512k training and 99k validation points carrying structured task annotations.
Vision module
Trained for 2k epochs over environment D, the vision module predicts all six key elements with high accuracy from visual cues alone. Within a 1% tolerance band, every key element exceeds 96% accuracy.
Language module
We built ~10k instruction-task pairs across the 34 tasks and evaluated three CLIP text encoders and eight SBERT variants, including an ablation that removes underscore (_) delimiters from task names. CLIP encoders aligned instructions to tasks poorly relative to SBERT. Fine-tuning paraphrase-MiniLM-L6-v2 with cosine-similarity loss over 4 epochs raised pairing accuracy from a 63.5% baseline to 98.4% and removed sensitivity to underscore formatting.
| Language model encoder | Accuracy with "_" | Accuracy w/o "_" |
|---|---|---|
| CLIP ViT-B/32 | 41.90% | 32.65% |
| CLIP ViT-B/16 | 47.04% | 36.82% |
| CLIP ViT-L/14 | 38.82% | 34.70% |
| all-MiniLM-L6-v2 | 58.87% | 61.70% |
| all-MiniLM-L12-v2 | 57.33% | 60.41% |
| all-mpnet-base-v2 | 57.07% | 51.16% |
| paraphrase-MiniLM-L3-v2 | 59.90% | 68.64% |
| multi-qa-MiniLM-L6-cos-v1 | 62.47% | 66.58% |
| paraphrase-albert-small-v2 | 59.50% | 61.70% |
| Fine-tuned all-MiniLM-L6-v2 | 98.46% | 98.46% |
Long-horizon benchmark
On the CALVIN D→D long-horizon benchmark, VLAG reaches an average sequence length of 2.09/5 at roughly 100M parameters. Recent billion-parameter VLA methods score higher, but at one to two orders of magnitude more parameters—the design point VLAG deliberately trades against.
| Model | Avg. sequence length | Approx. parameters |
|---|---|---|
| VLAG (ours) | 2.09 / 5 | ~100M |
| FLOWER | 4.35 / 5 | ~1B |
| RoboUniView | 3.85 / 5 | ~3–4B |
| π-0 (via StarVLA) | 2.95 / 5 | ~4B |
| π-0.5 (via StarVLA) | 3.89 / 5 | ~4B |
Matched baseline: MoE vs. single policy
To isolate the benefit of task-specialized experts, we run a controlled comparison on CALVIN D→D using 100 randomly sampled language-conditioned task chains per run. The task-specialized MoE is contrasted with a single ACT policy trained jointly on all tasks under a fixed 45,000-step budget, repeated over three seeds (11, 22, 33). The MoE configuration improves long-horizon competence dramatically—the joint single policy collapses beyond the first instruction.
| Model | Avg. seq. len | SR@1 | SR@2 | SR@3 | SR@4 | SR@5 |
|---|---|---|---|---|---|---|
| MoE (task-specialized experts) | 1.93 ± 0.08 | 0.70 ± 0.01 | 0.51 ± 0.02 | 0.35 ± 0.01 | 0.24 ± 0.04 | 0.13 ± 0.03 |
| Single (joint policy, 45k steps) | 0.16 ± 0.02 | 0.15 ± 0.01 | 0.01 ± 0.01 | 0.00 | 0.00 | 0.00 |
Mean ± population standard deviation over three seeded runs (n = 3). SR@k is the fraction of episodes completing the first k instructions. Across seeds the MoE average sequence length ranges 1.82–1.99; the single policy ranges 0.14–0.18.
Diagnostics
We instrument the evaluation to separate routing failures (no executable sub-task can be resolved) from execution failures (routing succeeds but the policy does not complete the sub-task). Routing failures are zero for both models; execution failures dominate. Performance in the seen D→D setting is therefore constrained primarily by low-level policy robustness, not by graph or language gating.
| Model | Routing failures | Execution failures | Total subtasks |
|---|---|---|---|
| MoE | 0 | 89 | 266 |
| Single | 0 | 100 | 117 |
Wall-clock runtime and GPU memory
Timing and memory were recorded on one NVIDIA A100 PCIe (40 GB) with dual AMD EPYC 7513 hosts. The MoE achieves higher subtask throughput and lower per-subtask latency than the single policy, but carries higher end-to-end runtime and peak memory. These measurements distinguish architectural compactness (~100M parameters) from runtime overhead: the modular design improves long-horizon competence but still incurs coordination and memory costs to be reduced in future work.
| Model | Runtime (s) | Subtask throughput (1/s) | Latency / subtask (ms) | Mean GPU mem. (MiB) | Peak GPU mem. (MiB) |
|---|---|---|---|---|---|
| MoE | 528.3 | 0.503 | 1986 | 9188 | 11356 |
| Single | 463.8 | 0.252 | 3964 | 7005 | 9658 |
Qualitative rollouts
Ablation Studies
We isolate each component—language retrieval, visual state estimation, graph construction, and graph-constrained routing—holding the rest of the pipeline fixed.
Language retrieval. On 34 held instruction-task pairs, fine-tuned SBERT reaches 0.9706 accuracy with or without underscore formatting, versus 0.8235 (with) and 0.7941 (without) for base SBERT. Domain-specific fine-tuning matters even for a small task vocabulary, because minor linguistic variation is enough to degrade off-the-shelf embeddings.
Vision-state prediction (oracle vs. learned). Against an oracle reference of 1.0, predicted Boolean states are reliable (lightbulb 0.9570, green light 0.9424), but continuous variables within a strict ±1% band are uneven—sliding door 0.3540, drawer 0.5189, switch 0.8512. The vision bottleneck is concentrated in fine-grained continuous-state estimation, not binary attribute recognition.
Graph construction (node-merge sensitivity). Raising the cosine merge threshold from 0.90 to 0.99 grows node count from 8 to 19 while edge count (145) and unique-task count (20) stay fixed—the threshold controls node granularity, not coverage—supporting 0.98 as the operating point.
| Threshold | Nodes | Edges | Unique tasks |
|---|---|---|---|
| 0.90 | 8 | 145 | 20 |
| 0.95 | 10 | 145 | 20 |
| 0.98 | 18 | 145 | 20 |
| 0.99 | 19 | 145 | 20 |
Graph-constrained vs. unconstrained routing. Restricting language retrieval to graph-feasible candidates raises edge-selection accuracy from 0.9379 to 0.9586 for base SBERT and from 0.9931 to 1.0000 for fine-tuned SBERT on 145 labeled transitions. The graph helps the weaker encoder most, but even the fine-tuned model benefits by eliminating residual ambiguity among semantically similar task labels.
Discussion and Limitations
The ablations and diagnostics converge on a consistent picture. Boolean state recognition and graph/language routing are robust—routing failures are zero across the matched evaluation—so gross task identification is rarely the problem. The two failure modes that remain are (i) state misclassification, when small continuous-state errors map an observation to the wrong node, which the strong vision results make comparatively rare in the seen setting; and (ii) action-execution error, the dominant mode, where a correctly routed expert fails to complete a fine-grained sub-task within its trajectory window. Because routing advances only on successful sub-task completion, a single execution failure terminates the long-horizon chain.
VLAG’s headline advantage is architectural compactness—an ~\(10^8\)-parameter system competing in a benchmark dominated by ~\(10^9\)-parameter models—and component-wise updatability, since new skills can be added without retraining a dense backbone. The honest counterpoint is that this modularity does not yet translate into uniformly lower wall-clock cost: the MoE’s coordination and memory overhead exceed a single-policy ACT baseline on the same harness, even as it achieves far higher long-horizon success.
Conclusion and Future Work
VLAG decomposes a monolithic VLA into a graph router plus specialized vision, language, and action experts, abstracting continuous environmental states into a structured transition map and routing each sub-task to a dedicated expert. The result is robust state recognition, transparent task routing, and competitive long-horizon performance at roughly 100M parameters on CALVIN D→D.
Current efforts target generalization to the unseen CALVIN long-horizon split through hyperparameter optimization and a unified expert architecture. Further directions include integrating generative foundation models as expert modules, combining reinforcement learning with graph-based routing for robustness to unseen environments and perturbations, and extending the framework to diverse sensory modalities and robotic embodiments—toward a scalable, efficient, and adaptable foundation for general-purpose robotic assistants.
Demo
VLAG executing a long-horizon manipulation sequence on CALVIN: the graph router chains specialized experts across successive sub-tasks.