Zero-real Agent Policy (ZAP)

Pretraining knows what, simulation teaches how

GPT-6 Astra [1] has changed what the robotics community expects from frontier vision-language models. Controlling a robot through a tool harness, it significantly outperforms state-of-the-art vision-language-action models (VLAs) and world-action models (WAMs) [2, 3], and many now expect that robot policies will need frontier-scale models [4]. Yet a trillion-parameter model is slow and costs dollars per task.

Could a much smaller vision-language model already have most of what is needed, given the right factorization and post-training?

We argue a robot policy composes two things:

Trained in simulation, tested on a real robot

ZAP never sees a real robot during training. It learns to move on simple simulated tasks, then runs zero-shot on a real bimanual robot, on tasks whose objects, goals and strategies appear nowhere in training.

ZAP is far cheaper and faster. Below, both models tackle the same task, placing a can into the grey bin, side by side.

ZAP and Astra solve tasks in a similar number of tool calls, but ZAP executes each tool call much faster and cheaper.

In this episode the model:

• Turns the tube so it points the right way, cap up.
• Brings the right arm, holding the tube, to the middle of the table, above the rack.
• Brings the left arm over the tube and descends from above, just far enough to grasp the cap but not so low that it hits the right arm.
• Closes the left gripper, then opens the right one.
• Moves the right arm out of the way.
• Lowers the tube into a hole in the rack.

We also ask the model to make the winning move in a chess position. This again requires precise manipulation of an object, in a scene it has never seen: chess pieces are small, light and top-heavy, and tip over easily if grasped badly. It also requires reasoning about the rules and strategy of chess itself, to find the one move that delivers checkmate.

None of these objects, scenes or goals appear in training, and neither does drawing, a mid-air handoff or board-game reasoning. Pretraining brings the understanding of the world; simulation only had to teach the model how to move the robot.

One harness, in simulation and on hardware

The model acts only through tools. We frame each task as a conversation between the model and its environment: the model first writes a plan and names the objects to locate, then repeatedly issues perception and movement tool calls. On every turn it sees fresh camera images, perception results and the robot's measured state. It does not write a program that then runs blind. Because it decides again after every action, it can notice when a grasp slips or an object moves, and adjust.

Everything robot-specific lives in a declarative description, so the same harness runs on the simulated YAM, the real YAM and a Franka the model never trained on.

The harness has five tools. There are no grasp macros or motion primitives: we found that they raise the floor but also cap the ceiling. Every approach, grasp, lift and release is a sequence of move_to and set_gripper calls the model plans itself.

Pretraining knows what, but not how

The harness works very well with frontier models like GPT-6 Astra but not with smaller VLMs. In the same harness, base Qwen 3.6 27B [11] succeeds on just 8.3% of RoboLab-120 scenes and 16.7% of real-world trials. A model can fail by missing either semantic reasoning, what the task asks for, or motion control, how to move the robot. We look at how base Qwen 3.6 27B does on both.

It understands the task

In 95% of scenes, the base model's plan correctly identifies the target objects, the destination and the object order. In 70% of scenes, its perception query returns every target and destination object. But even in episodes where both hold, it completes the task only 9% of the time.

It cannot move the robot

Only 36% of the base model's motion calls are correct. 32% are refused by the controller because the requested pose is physically impossible, 15% are misdirected (the right intent but the wrong pose, such as a grasp that stops above the object), and 16% are pointless, such as carrying an empty gripper. The failures mostly come from motion, not from understanding.

The same pattern holds on the real rig: the base model reads the scene and plans the right pick, but cannot turn that plan into the right motion.

Pretraining brings the semantic reasoning, the what of a task, but not the motion control, the how. So motion control is the only skill ZAP has to teach, and it teaches it from first principles, without distilling from a frontier model.

Simulation teaches how

We close that gap by teaching motion control in simulation, and nothing else. We build our simulated tasks in Isaac Sim [12], with simulated YAM arms whose cameras and arm dynamics are calibrated against the real ones. We design 40 tasks, none of which appear in RoboLab-120, that exercise everyday object manipulation: grasping and lifting, single- and multi-object pick and place, stacking, and reorientation.

The goals are deliberately simple, and none requires deep reasoning. That lets us test whether motion control learned on simple tasks can combine with pretrained semantic reasoning to solve tasks with new goals and strategies. We never introduce any real-world action or teleoperation data; all learning is performed in simulation.

We found it crucial that completing the task is rewarded far more than any intermediate step. Otherwise the model can settle for reliably collecting partial credit instead of finishing. The success checker is hardened against reward hacking, for example by requiring that the goal hold for at least half a second.

RoboLab-120 results

To test generalization to unseen tasks at scale, we evaluate on our bimanual YAM adaptation of RoboLab-120, none of whose 120 tasks are in training: ZAP reaches 74.0%, matching GPT-6 Astra and ahead of Claude Opus 5.5 [18].

Training changes motion, not understanding

If training had changed semantic reasoning, the base model and ZAP would disagree on goals and perception. They do not. The two models agree on the target objects, destination and object order in more than 90% of scenes, and their perception calls return the same objects in 70%. Even when we restrict the comparison to episodes with matching perception, the gap in task success stays large: 9.0% for the base model against 73.6% for ZAP.

The difference is motion control. 85% of ZAP's motion calls are correct, against 36% of the base model's (Figure 11). Below, both models identify the butter box and plan the same pick-and-place, but only one of them can grasp it.

Orientation

Each move_to gives an absolute position of the tool center point, in meters, and a full orientation, in a world frame shared by both arms. Perception reports object positions in the same frame and units, so a measured centroid can be used directly as a target. Orientation is given by three angles:

• Slot yaw ψ is the heading of the jaw slot, the axis a held object lies along between the fingers. It is set along the object to be grasped.
• Tilt τ is the angle of the tool axis from straight down: 0° is top-down and 90° is horizontal. It chooses between a top-down and a side approach.
• Tilt direction ϕ is the compass direction the fingertips tip toward. It picks the side the tool comes in from.

Where it still slips

The model is not perfect on these new tasks, and its failures are informative. Below are two examples where it gets most of a task right but fails on one part of it.

In the second, the model must sort fruit into bins by color: orange on the left, yellow in the middle, red on the right. It reasons correctly about color, location and manipulation, and moves most of the fruit to the right place. However, it misses the last remaining red fruit in the left bin, possibly because of the bin's shadow.

Several choices in this report came from trying the obvious thing first. These are the ones we would most want to know about if we were starting again.

• Grasp macros. Relative moves and approach primitives raised the floor but capped the ceiling. We removed them and let the model plan each phase.
• Euler angles and quaternions. Side grasps sit near the Euler singularity, so small errors looked large. Slot yaw, tilt and tilt direction fixed this.
• Binary reward. On tasks with no successes, pass/fail gives no signal. The dense reward ladder does.
• Overhead depth on the real rig. In the real world, depth from the height of the overhead camera is often not accurate enough, and the policy would falsely rely on these perception coordinates. We found that simply turning off overhead depth allowed the policy to focus on the wrist-camera perception measurements. Surprisingly, the policy had no issue adjusting to this, even though in simulation it had reliable depth from all three cameras.

With the right factorization and post-training, a small open model is enough. Pretraining already knows what a task asks for; simulation only has to teach how to move. Trained on 40 simple simulated tasks, from its own successes, with no frontier teacher and no real robot data, a 27B open model matches GPT-6 Astra on real hardware at 3× the speed and 1/17 the cost. Adding a skill to ZAP means writing new simulated tasks, not collecting new robot demonstrations.

@misc{pikus2026zap,
  title  = {Pretraining Knows What, Simulation Teaches How: Compositional Generalization for Agentic Robotics},
  author = {Pikus, Benjamin and Saunders, Daniel and Mironenco, Mircea and
            Mohammadi, Salman and Mathew, Mathew and Zhu, Sirou and Makkar, Divya and
            Huang, Xuhua and Li, Qinbo and Narasimhaswamy, Supreeth and
            Cen, Siyuan and Xu, Dejia and Buch, Shyamal and Neff, Thomas},
  year   = {2026},
  eprint = {2610.35817},
  archivePrefix = {arXiv},
  note   = {Luma AI}
}