← All topics

Guide · Robot learning · Topic 19 of 32

Long-horizon tasks

Action chunks, sub-task decomposition, chain of thought and memory: how robots get through tasks with many steps.

8 minute readBuilds on: Robot learning basics, Robot policies today

Why long tasks are hard

Whole-task success against the number of sub-tasks that must all succeed, for four per-step success rates. At 95 percent per step, ten steps give 60 percent overall and thirty give 21 percent. At 99 percent per step, thirty steps give 74 percent.
Arithmetic, not an experiment: if every sub-task must succeed, the whole task succeeds with the product of their rates.

A policy that succeeds at each step 95% of the time sounds reliable. Chain ten such steps and the task succeeds 60% of the time. Chain thirty and it is 21%. This arithmetic is the central fact about long tasks, and every method in this note attacks it from one of three sides: make each step more reliable, make fewer steps, or recover when a step fails.

Two further problems come with length. The policy has to remember what it has already done, and it sees only the last frame or two. And the instruction no longer describes the next motion. “Clear the table” says nothing about which object to pick up first.

Levels of a task

Five rows on one time axis. A task lasting minutes. Sub-tasks of 5 to 30 seconds each. Motions of about a second each. Action chunks of half a second to two seconds. Individual actions of 20 to 100 milliseconds.
One task at five levels. Each method in this note works at one of the boundaries between them.
Hand position and gripper opening through a 21-second robot episode, divided into four shaded phases at the moments the gripper closes and opens: reach the first mug, carry it to the plate, reach the second mug, carry it to the plate.
A real episode divided into sub-tasks using nothing but the gripper command. It closes at 4.0 s and 16.1 s and opens at 9.4 s and 20.7 s.

The second figure is a real demonstration of the instruction “put the white mug on the left plate and put the yellow and white mug on the right plate”. The moments the gripper closes and opens split it cleanly into four sub-tasks. That is the simplest form of sub-task decomposition, and it is free: the boundaries are already in the data.

Sub-tasks in real data

Left: the number of gripper open or close events per episode across 379 demonstrations. Most episodes, 212, have four; 66 have two, 40 have one, 28 have six. Right: the length of each sub-task in seconds, with a median of 5.9.
The same gripper-based segmentation applied to all 379 LIBERO demonstrations.

Applied to the whole dataset, the gripper signal gives a picture of what “long horizon” means in a standard benchmark. Most episodes contain four gripper events, which is two pick-and-place sub-tasks. The median sub-task lasts 5.9 seconds and the median episode 26. At 10 actions a second that is roughly 60 actions per sub-task, or four chunks of 16.

These tasks are called long-horizon in the literature and they are two or three sub-tasks long. Household tasks that people consider simple, such as loading a dishwasher, run to dozens.

Action chunks: fewer, larger decisions

The lowest boundary is between single actions and chunks. Predicting a block of future actions in one call shortens the chain directly. The episode above is 214 steps long.

Actions per chunkDecisions in the episodeEach chunk covers
12140.1 seconds
8270.8 seconds
16141.6 seconds
5055 seconds

Fewer decisions mean fewer chances to drift, and a chunk produced in one pass is internally consistent, so motion is smooth. The price is reactivity: within a chunk the policy is not looking. Three refinements manage that trade.

  • Execute part, re-plan early. Predict 16 actions, execute 5, and predict again. The plan stays long and the reaction time stays short.
  • Blend overlapping chunks. Average the predictions that several successive chunks make for the same moment, which smooths the joins.
  • Plan around the delay. Generate the next chunk knowing which of the current actions will already have been executed by the time it arrives.

Chunking solves the problem at the scale of seconds. It does nothing for a task of minutes, which needs the levels above.

Sub-task decomposition

The next boundary up splits a task into sub-tasks, each short enough for a chunking policy to complete reliably. A high-level model names the next sub-task in words, and a low-level policy carries it out. The interface between them is usually plain language, which makes the system’s decisions readable.

Where the sub-tasks come fromHow it worksTrade-off
Written by handA fixed script or state machine lists the stepsReliable and inflexible; one script per task
A language model as plannerGiven the instruction and the scene, a language model proposes the next step, often filtered by what the robot can currently doFlexible; the planner can propose steps the robot cannot perform
A vision-language model trained for itA model fine-tuned on robot data labelled with sub-tasks predicts the next one from the camera viewGrounded in what the robot sees; needs sub-task labels
Learned from the data’s structureBoundaries are found automatically, from gripper events as above or from changes in the sceneFree labels; the segments may not match meaningful steps

Two details decide whether a hierarchy works. The high level has to know when a sub-task is finished, which needs a success check or a model that watches for completion. And the high level has to be able to re-plan when a sub-task fails, which is where the third weapon against compounding, recovery, comes in. A system that notices a dropped mug and issues “pick up the mug” again turns a failed chain into a longer successful one.

What recovery is worth

If a sub-task succeeds with probability pp per attempt and can be retried up to kk times, its effective success rate is

peff=1−(1−p)kp_{\text{eff}} = 1 - (1 - p)^{k}
Whole-task success against the number of sub-tasks, each succeeding 90 percent of the time per attempt, with one, two or three attempts allowed. With one attempt, thirty sub-tasks succeed 4 percent of the time. With two attempts, 74 percent. With three, 97 percent.
The compounding arithmetic again, with retries allowed.

A second attempt turns 90% into 99%, and a task of thirty such sub-tasks goes from succeeding 4% of the time to 74%. No plausible improvement to the low-level policy does as much. This is why failure detection and re-planning are the most valuable parts of a long-horizon system, and why the first question about any such system is whether it knows when a step has failed.

The arithmetic assumes that a failure leaves the world in a state from which a retry is possible. A dropped cup can be picked up again. A spilled one cannot. Irreversible steps deserve their own check before they are attempted.

Chain of thought: reasoning before acting

Two pipelines. Direct: image and instruction go straight to actions. With reasoning: image and instruction are followed by four generated steps, the plan, the current sub-task, the location of the object as a box, and the motion in words, before the actions.
The model writes its intermediate reasoning as tokens before the action. Each step conditions the ones after it.

In language models, chain of thought means having the model write out intermediate steps before its answer. Accuracy improves on problems that need several steps, because each step is produced with the earlier ones in view.

The embodied version asks a robot policy to do the same before each action: state the plan, name the current sub-task, locate the relevant object, describe the motion in words, and only then output the action. The published work on this reports clear gains in success on tasks that need generalisation, with a further benefit that is easy to overlook. When the robot fails, the reasoning shows where it went wrong: the wrong sub-task, the wrong object, or the right intention and a bad motion.

  • It needs labels. The reasoning steps must exist in the training data. They are usually generated automatically, with detectors and language models annotating existing demonstrations.
  • It costs time. Every reasoning token is generated before the action. Systems reduce this by reasoning only every few steps, or by reusing the high-level parts of the reasoning until the sub-task changes.
  • It can be visual. The intermediate step need not be text. Some models predict an image of the goal, or of the next few frames, and act toward that.
  • It can be wrong. Fluent reasoning followed by an unrelated action is possible. Check that changing the reasoning changes the action.

Memory

A policy that sees only the present cannot know whether it has already checked a drawer. Four kinds of memory are in use.

  • A longer context. Feed more past frames. Simple, expensive, and it invites the shortcut of copying past actions.
  • A written log. The high-level model keeps short notes on what has been done, and reads them at each decision.
  • Keyframes. Store a few informative past frames instead of all of them.
  • A recurrent state. Carry a fixed-size summary forward in time, as the linear recurrent layers in the attention note do.

Memory is the least settled part of long-horizon work. Tasks that depend on something no longer visible remain hard.

How they fit together

MethodWorks atFights compounding byCosts
Action chunksActions to motionsFewer decisionsReactivity within a chunk
Sub-task decompositionMotions to taskShorter chains per policy call; a place to re-planA second model, and a way to detect completion
Chain of thoughtInside one decisionMaking each decision more reliableTime, and labelled reasoning
Recovery and retriesAny levelTurning a failure into extra steps instead of the endA failure detector
MemoryAcross the taskAvoiding repeated and skipped stepsContext, or added machinery

Current systems use most of these at once. The note on system architectures covers how the levels are arranged into models that run at different speeds.

Common mistakes

  • Quoting per-step success. Report success on the whole task. The chain is what matters.
  • Lengthening chunks to fix a long task. Chunks help at the scale of seconds. Minutes need decomposition.
  • A hierarchy with no completion check. The high level must know when to move on.
  • Assuming the reasoning caused the action. Test it by editing the reasoning.

Terms

Horizon
How many steps or how much time a task takes.
Action chunk
A block of future actions predicted in one call.
Sub-task
One step of a longer task, short enough for a low-level policy to complete.
Chain of thought
Writing intermediate reasoning steps before an answer or action.
Re-planning
Choosing a new next step after observing the result of the last one.
Completion detection
Deciding that a sub-task has finished, successfully or not.
Effective success rate
The success rate of a step once retries are allowed.

Further reading