A Few Words Go a Long Way:
Language Guided Robot Policy Synthesis

Anonymous Author(s)
Anonymous Institution

Paper (coming soon) arXiv (coming soon) Video Code

ARCHITECT synthesizes robot manipulation policies from natural language instructions and corrections, building a persistent skill library that transfers across tasks.

Abstract

While vision-language-action models have demonstrated impressive zero-shot manipulation capabilities, they remain fundamentally black-box policies that are difficult to interpret, adapt, or correct when they inevitably fail. In this work, we propose ARCHITECT, a framework that treats robot policy acquisition as an interactive program synthesis task.

ARCHITECT leverages the reasoning capabilities of LLM coding agents to synthesize modular robot programs that utilize a suite of perception and control tools. Unlike end-to-end models where distribution shift leads to unpredictable, cascading failures, our modular architecture allows users to isolate failures and localize feedback at the level of abstraction required.

We introduce an iterative process where a human supervisor provides natural language corrections to steer the policy. These corrections are grounded in the policy code by program execution traces and distilled into a persistent skill library, a form of long-term in-context learning which enables the agent to accumulate a repertoire of reusable, interpretable behaviors.

In a benchmark evaluation on a Franka Panda robot, ARCHITECT outperforms state-of-the-art VLA models and program synthesis baselines on complex, long-horizon tasks, including articulated object manipulation and cloth folding. Our results demonstrate that the synthesized skill library enables the system to transfer to novel tasks with decreasing human intervention, providing a steerable and data-efficient alternative to black-box robot learning.

Method

System overview of ARCHITECT.

Figure 1: System overview of ARCHITECT. The pipeline takes natural language input and synthesizes robot policies using a suite of tools including control primitives and perception modules. When execution fails, a human provides a language correction, which updates the skill library for future tasks. The agent also receives an execution trace from the robot to help localize code improvements.


Language Guided Policy Synthesis

ARCHITECT uses a pretrained LLM as an orchestrator agent capable of synthesizing executable robot programs that invoke a suite of heterogeneous tools, including pretrained foundation models, open-vocabulary detectors, grasp generators, and motion planners. The language instruction is parsed and embedded into a system prompt which calls the LLM to generate code. The policy code is grounded in the scene by using a set of control and perception primitives as well as tools such as VQA, proprioception, or CLI commands to read the available skills.

Skill Library & Memory

A challenge in human-in-the-loop robot learning is ensuring that corrections are not temporary. When a human provides a language correction during execution, ARCHITECT synthesizes it into a skill: a concise, natural language rule or code pattern that encodes the adjustment. This skill is added to a persistent collection that grows over the course of interaction. Before each subsequent code generation step, all skills in the library are loaded into the agent's context alongside the current task instruction and execution trace.

Example Synthesized Skill: Guarded Place on Surface
# Generated from correction: "keep moving down until the box is
# touching the shelf. Do not ungrasp if it would fall."

# Step 1: Move to approximate placement pose
move_ee_to_pose(place_pose)

# Step 2: Guarded descent until contact
move_ee_guarded(axis="z", distance=-0.25, force_threshold=5.0)

# Step 3: Release only after surface contact
set_gripper_width(0.085)

# Step 4: Retract upward
move_ee_to_rel_pose({"x": -0.05, "y": 0, "z": 0.1})

Tool Suite

The tools consist of control primitives (motion planning via cuRobo, gripper control, guarded moves, waypoint execution), perception modules (open-vocabulary detection via Grounded SAM2, grasp generation via AnyGrasp, placement estimation via AnyPlace, VQA, DIFT keypoints, ReKep trajectories), and proprioception queries (end-effector pose, joint state, gripper width). Functions are parameterized with natural language input to ground corrections in feasible behavior.

Real-World Benchmarking

Side-by-side comparisons of ARCHITECT against baselines across all 8 evaluation tasks. Videos play synchronized — use the controls to play/pause/restart all videos in a row simultaneously.

Long Horizon "There is a banana underneath the cloth. Pick up the cloth and drop it aside, then pick and place the banana into the dish plate."
Synchronized playback
ARCHITECT ✓
CaP ✗
π0 ✗
π0.5 ✗
Deformable "Fold the cloth into a triangle along the diagonal."
Synchronized playback
ARCHITECT ✓
CaP ✗
π0 ✗
π0.5 ✗
Articulated "Push the top drawer closed."
Synchronized playback
ARCHITECT ✓
CaP ✗
π0 ✗
π0.5 ✗
Clutter "Pick up the bread and put it to the side of the box."
Synchronized playback
ARCHITECT ✓
CaP ✗
π0 ✗
π0.5 ✗
Multi Pick-Place "Put the apple and the bread into the bowl."
Synchronized playback
ARCHITECT ✓
CaP ✗
π0 ✗
π0.5 ✗
Pick-Place "Pick up the banana and put it in the bowl."
Synchronized playback
ARCHITECT ✓
CaP ✓
π0 ✓
π0.5 ✓
Pick-Place "Pick up the block and put it in the basket."
Synchronized playback
ARCHITECT ✓
CaP ✗
π0 ✗
π0.5 ✓
Pick-Place "Pick up the baseball and put it in the bucket."
Synchronized playback
ARCHITECT ✓
CaP ✓
π0 ✗
π0.5 ✗

Results

80%
Average Success Rate
Across 8 tasks spanning pick-and-place, deformable manipulation, articulated objects, and long-horizon reasoning.
4.67 → 0.83
Fewer Corrections Needed
With a populated skill library, human queries drop 82%. 3/6 participants needed zero corrections.
0% → 67%
Zero-Shot Transfer
Skills from Task 1 transfer to Task 2, raising success rate without any additional corrections.

ARCHITECT outperforms generalist policies

We evaluate on 8 real-world manipulation tasks on a Franka Panda with the DROID setup, comparing against Code as Policies (CaP), Inner Monologue, MolmoAct2, π0, π0.5, and an ablation using VLM-generated corrections (ARCHITECT-VLM). Each policy is evaluated with N=10 episodes per task.

Task CaP Inner Monologue MolmoAct2 π0 π0.5 Ours-VLM Ours-HiTL
SRGCRNTP SRGCRNTP SRGCRNTP SRGCRNTP SRGCRNTP SRGCRNTP SRGCRNTP
Banana → plate Pick-Place .20.52.52 .50.90.90 .80.92.92 .80.90.90 .90.98.98 .80.88.88 .80.96.96
Block → basket Pick-Place .40.52.52 .50.76.76 .30.64.64 .40.44.44 1.01.01.0 .70.80.80 .70.82.82
Baseball → bucket Pick-Place .60.66.66 .50.84.84 .00.34.34 .00.16.16 .20.50.50 .40.62.62 1.01.01.0
Apple & bread → bowl Multi Pick-Place .10.59.47 .30.51.49 .10.43.43 .40.64.64 .40.65.65 .00.49.39 .80.88.88
Bread from box Clutter .10.38.38 .00.00.00 .20.32.32 .40.40.40 .40.44.44 .00.24.24 .70.78.78
Close drawer Articulated .00.00.00 .00.00.00 .00.00.00 .10.23.23 .00.17.17 .40.60.60 .90.90.90
Fold cloth Deformable .00.00.00 .00.00.00 .00.08.08 .00.40.40 .00.04.04 .70.62.62 .80.95.95
Banana under cloth Long Horizon .00.40.40 .00.30.30 .00.47.47 .10.36.36 .00.19.19 .00.60.60 .80.80.60

Table 1: Success Rate (SR), Goal Condition Recall (GCR), and Normalized Task Progression (NTP) across tasks. Bold indicates best per task. We compare ARCHITECT-HiTL against CaP, Inner Monologue, MolmoAct2, π0, π0.5, and ablation ARCHITECT-VLM.


ARCHITECT is robust to language perturbation

To isolate the effect of language on performance, we perturb the instruction on a single task (Apple → pan) while holding the object pose constant, and compare against π0.5 — the strongest VLA baseline from our main evaluation. Perturbations span paraphrasing (synonym, passive voice, reordered, ambiguous), length (short, medium, long), distractors (typos), and specification (under- and over-specified). ARCHITECT remains robust across nearly all variations, while π0.5 is sensitive to several — most notably under-specification, over-specification, and reordered or ambiguous prompts. In these failure cases, π0.5 exhibits undirected motion before eventually orienting toward the task; on "Put the apple away," the arm moves at random without ever targeting the apple.

Perturbation Instruction π0.5 ARCHITECT
SRGCRNTP SRGCRNTP
Canonical "Put the red apple in the pan." 1.01.01.0 1.01.01.0
Paraphrase (synonym) "Grab the apple and set it on the pan." 1.01.01.0 1.01.01.0
"Take the apple and place it onto the pan." .00.20.20 1.01.01.0
"Move the apple onto the pan." .00.20.00 1.01.01.0
"Lift the apple and rest it on the pan." 1.01.01.0 1.01.01.0
Paraphrase (passive) "The apple should be picked up and placed on the pan." 1.01.01.0 1.01.01.0
Paraphrase (order) "Onto the pan, place the apple." .00.20.20 1.01.01.0
Paraphrase (ambiguity) "Relocate the fruit so that it sits on the pan." .00.20.20 1.01.01.0
Length (short) "apple on pan" 1.01.01.0 1.01.01.0
Length (medium) "Pick up the apple from the table and place it onto the pan sitting nearby." .00.20.20 1.01.01.0
Length (long) "There is an apple resting on the surface in front of you. Pick that apple up, then move it over and place it down onto the pan that is also on the surface, so the apple ends up on the pan." 1.01.01.0 .00.80.80
Distractor (typos) "pickup teh appel an put it on the frying pan" .00.20.20 1.01.01.0
Underspecification "Put the apple away." .00.20.20 1.01.01.0
"Move the apple." .00.20.20 .00.60.60
Overspecification "Pick up the red apple and put it on the white pan; approach from directly overhead, keep the gripper aligned with the apple's long axis, and release only once it is centered over the pan." .00.20.20 1.01.01.0

Table 2: Language instruction perturbation ablation on the Apple → pan task. Bold indicates the better method per row. We compare ARCHITECT against π0.5, the strongest VLA baseline. Target object position and orientation are held constant across all conditions.


Human Evaluation

In an N=6 within-subjects study, we isolate three effects: the value of human language corrections, skill library transfer to new scenes, and human effort amortization through the skill library. Each participant completes two structurally similar tasks across six conditions that systematically toggle human corrections and skill library access.

Sample efficiency: mean corrections drop with skill library.
(a) Sample Efficiency
Mean corrections: 4.67 → 0.83 with skill library (p=0.036)
Transfer success rate rises with Task 1 skills loaded.
(b) Transfer: Success Rate
SR rises from 0/6 to 4/6 with Task 1 skills loaded
Transfer task completion improves with loaded skills.
(c) Transfer: Task Completion
Skills generalize to new objects and scenes

Figure 5: ARCHITECT results from human evaluation. (a) With a populated skill library, HiTL trials require fewer human queries. (b, c) Skills from Task 1 transfer to Task 2, raising both success rate and task completion.

BibTeX

@misc{anonymous2026architect,
  author = {Anonymous},
  title  = {A Few Words Go a Long Way: Language Guided Robot
            Policy Synthesis},
  note   = {Under review},
  year   = {2026},
}