Matthew Wnuk
← All work

Qualcomm · Staff Embodied AI Engineer · 2026 to present

World-Action Models for Robot Manipulation

Robot foundation models that predict future video and robot actions together, from multi-robot data to on-device control.

  • World-action models
  • Video generation
  • Imitation learning
  • On-device
“put the white mug on the left plateand put the yellow and white mugon the right plate”Language instructionRecent frames, scene cameraWrist cameraWorld-action modelVideo model:what happens nextAction model:what to dofeaturesPredicted future frames(dataset frames shown as stand-ins)Predicted action chunkarm motion and gripper over the next stepsRobotexecutes,thenre-plans

Overview

Organization
Qualcomm
Role
Staff Embodied AI Engineer
Period
April 2026 to present

At Qualcomm I work on robot foundation models for manipulation, end to end: from multi-robot training data, through training and evaluation, to running the policy on Qualcomm hardware onboard the robot.

The models are world-action models (WAMs). Given camera images and a language instruction, a WAM predicts what will happen next as well as what to do: it generates future video frames and the robot's actions together. Predicting the visual future lets the policy draw on the physical dynamics that large video models learn from data.

time
What the policy sees: one LIBERO demonstration from the scene camera (top) and the wrist camera (bottom).
observedgenerated togethernowfuturevideoactionchunkxyzgrippernoisedenoisedstep by step

01

Predicting the Future and the Action Together

A WAM pairs a video model with an action model. From the instruction and recent camera frames, it generates the next stretch of video and a chunk of robot actions, both by iterative denoising. The robot executes the actions, new frames arrive, and the model plans again.

Open X-Embodiment22 robot typesDROIDone arm, many scenesLIBEROsimulated benchmarkOne commonformat:cameras, actions,frame rateTrainWAMdifferences reconciled:camera count and placement,action space, control rate

02

Learning from Many Robots

Robot data is scarce and inconsistent. Public collections such as Open X-Embodiment, which pools more than a million trajectories from 22 robot types, and DROID differ in camera count and placement, action space, and control rate. Much of the work is mapping them into one common observation and action format that a single model can train on.

LIBERO scenes: the policy runs closed loop in simulationSpatialObjectGoalLongFour task suites. The score is the share of rolloutsin which the robot completes the instruction.

03

Evaluating in Simulation

Policies are evaluated closed loop on LIBERO, a public simulated manipulation benchmark with four task suites covering spatial relations, objects, goals, and long-horizon tasks. Each rollout either completes the instruction or does not, and the success rate is the measure that training decisions are judged against.

camerasonboard the robotQualcommprocessoractionsclosed control loopbudget: compute, memory, latency

04

Running on the Robot

The last step is running the policy on Qualcomm hardware onboard the robot, where the model has to fit the compute, memory, and latency budget of a closed control loop.

Robot frames and action curves are from the public LIBERO benchmark (Liu et al., 2023), via the lerobot/libero_10 dataset (Apache-2.0). Predicted frames are represented by dataset frames.