← All topics

Guide · Robot learning · Topic 16 of 32

Reinforcement learning

Learning from reward by trial and error: where it works, why it is expensive, and how it is used with imitation.

7 minute readBuilds on: Foundations that survived, Robot learning basics

Learning from reward

Imitation learning copies a demonstrator. Reinforcement learning, or RL, has no demonstrator. The agent tries actions, receives a number called the reward, and adjusts itself to collect more reward over time.

A loop between an agent and an environment: the agent sends an action, the environment returns an observation and a reward.
The whole setup. There are no labelled actions anywhere in this loop.

What the agent maximises is the return: the sum of rewards over an episode, with later rewards usually counted a little less. Two difficulties follow from this definition. The agent must explore, because it cannot learn from outcomes it never tries. And it must work out which of its many actions earned a reward that arrived much later, which is called credit assignment.

Value and policy

Two grids. Left: reward is zero everywhere except a plus one goal cell and a minus one hazard cell. Right: the learned value of every cell, shaded darker near the goal, with an arrow in each cell showing the best move.
Computed by value iteration on a small grid with walls, a goal and a hazard. The reward marks two cells. The value function spreads that signal over every cell, and the arrows are the policy that follows from it.

The value of a state is the return the agent can expect from it. The reward says what is good now. The value says what is good in the long run, and it is what makes a sparse reward usable: every cell on the right knows which way to go, although only two cells on the left carry any reward.

FamilyWhat it learnsHow it uses dataTypical use
Value-based (Q-learning)The value of each action in each stateReuses old experienceDiscrete choices, games
Policy gradient (PPO)The policy directly, nudged toward actions that did better than expectedNeeds fresh experience after every updateSimulated locomotion at very large scale
Actor-critic, off-policy (SAC)A policy and a value function togetherReuses old experience heavilyReal-robot learning, where every trial is costly

The mathematics

The setting is a Markov decision process: states ss, actions aa, a transition rule P(s′∣s,a)P(s' \mid s, a), a reward r(s,a)r(s, a) and a discount γ\gamma between 0 and 1. The objective is the expected discounted return:

J(π)=Eπ[∑t=0∞γt r(st,at)]J(\pi) = \mathbb{E}_{\pi} \Big[ \sum_{t=0}^{\infty} \gamma^{t}\, r(s_t, a_t) \Big]

The value of acting optimally satisfies the Bellman equation, which says that the value of an action is its immediate reward plus the discounted value of the best action afterwards:

Q∗(s,a)=r(s,a)+γ Es′[max⁡a′Q∗(s′,a′)]Q^{*}(s, a) = r(s, a) + \gamma\, \mathbb{E}_{s'} \big[ \max_{a'} Q^{*}(s', a') \big]

Q-learning turns that into an update. After each step, move the estimate toward the right-hand side, using the observed next state in place of the expectation:

Q(s,a)←Q(s,a)+α[r+γmax⁡a′Q(s′,a′)−Q(s,a)]Q(s, a) \leftarrow Q(s, a) + \alpha \big[ r + \gamma \max_{a'} Q(s', a') - Q(s, a) \big]

The bracket is the temporal-difference error. The grid experiments in this note use exactly this rule.

Policy-gradient methods differentiate the objective with respect to the policy’s parameters instead. The result has a simple form: raise the probability of each action in proportion to how much better than expected it turned out.

∇θJ=Eπ[∇θlog⁡πθ(at∣st)  A(st,at)]A(s,a)=Q(s,a)−V(s)\nabla_\theta J = \mathbb{E}_{\pi} \big[ \nabla_\theta \log \pi_\theta(a_t \mid s_t)\; A(s_t, a_t) \big] \qquad A(s, a) = Q(s, a) - V(s)

AA is the advantage. PPO, the method behind most simulated locomotion, adds a limit on how far one update may move the policy. With ρt=πθ(at∣st)/πold(at∣st)\rho_t = \pi_\theta(a_t \mid s_t) / \pi_{\text{old}}(a_t \mid s_t):

Lclip=E[min⁡(ρtAt,  clip(ρt, 1−ϵ, 1+ϵ) At)]L^{\text{clip}} = \mathbb{E} \big[ \min\big( \rho_t A_t,\; \mathrm{clip}(\rho_t,\, 1 - \epsilon,\, 1 + \epsilon)\, A_t \big) \big]

The advantage itself is estimated with a weighted sum of temporal-difference errors, A^t=∑l(γλ)lδt+l\hat A_t = \sum_{l} (\gamma \lambda)^{l} \delta_{t+l}, where λ\lambda trades bias against variance.

Why it needs so many trials

Success against the number of environment steps on a log scale. Imitation from demonstrations passes 90 percent after about 60 steps. Reinforcement learning stays near 13 percent for hundreds of steps and passes 90 percent after about 4,000.
Both methods on the grid above, averaged over 12 runs. Imitation reaches 90% success with about 60 steps of expert demonstration. Tabular Q-learning needs about 4,100 steps of its own experience, roughly 70 times more.

The gap is the price of having no teacher. Early on the agent moves at random and almost never reaches the goal, so there is nothing to learn from. This is on a 6 by 9 grid. A robot arm has a continuous state, a continuous action and a camera image, and the gap grows accordingly.

The comparison is not one-sided. Imitation needed an expert to exist, and it can only match that expert. Reinforcement learning found the behaviour by itself, and it can go on to exceed the demonstrator.

Where it works today: simulated locomotion

Thousands of simulated robots, each with different mass, friction, delay and pushes, feed experience to one policy updated by PPO. The trained weights are then copied to a real robot with no further training.
The standard recipe for walking and running robots.

Walking robots are trained almost entirely this way. Thousands of copies of the robot run in parallel on one GPU, so years of experience take hours. Each copy has slightly different physics, which is domain randomisation, and the resulting policy transfers to hardware without further training.

Three conditions make this work: the simulator is fast, contact between feet and ground is simple enough to simulate well, and the reward (move forward, stay upright, use little energy) is easy to write. Everyday manipulation fails all three. Contact with deformable and small objects is hard to simulate, and “the table is tidy” is hard to turn into a number.

Used together with imitation

For manipulation, reinforcement learning is now mostly a second stage after imitation.

  • Offline RL. Learn from a fixed dataset with no new trials. Unlike imitation, it can use failed and mediocre episodes, because the reward tells it which were which. Its main danger is overrating actions the dataset never tried, so the methods deliberately stay close to the data.
  • Fine-tuning on the robot. Start from an imitation policy, let it run, and reward success. A person steps in when it goes wrong, and those corrections become data too. Published systems report higher success and faster task completion than the imitation policy they started from.

The division of labour is sensible. Imitation gets the policy to the point where it sometimes succeeds, which removes the long flat start of the orange curve above. Reward then polishes it.

Reward shaping, measured

Success against environment steps for Q-learning on the grid with three reward designs. With the sparse reward, success stays below 20 percent for the first 700 steps and reaches 99 percent after about 4,000. With potential-based shaping it reaches 94 percent after about 1,000. With a naive bonus for moving closer it starts as fast and then lags, reaching 84 percent where the others are at 98.
The same grid and the same algorithm with three rewards. Mean of ten runs.

A sparse reward tells the agent nothing until it stumbles on the goal. Shaping adds a signal along the way, and how it is added matters. The safe form is potential-based: choose a function Φ(s)\Phi(s) that is higher in better states, here the negative distance to the goal, and add

F(s,s′)=γ Φ(s′)−Φ(s)F(s, s') = \gamma\, \Phi(s') - \Phi(s)

to every reward. Because whatever is gained by moving toward the goal is lost by moving away, no loop can accumulate reward, and it can be shown that the optimal policy is unchanged. In the figure it reaches 94% success where the sparse reward is at 35%.

The naive version pays a bonus for every step that moves closer and charges nothing for moving away. It helps at first and then holds the agent back, because stepping away and back again now pays. It still converges here, on a small grid with forced exploration. On a harder problem the same mistake produces an agent that circles for ever, collecting the bonus.

Reward is the hard part

  • Sparse or shaped. A sparse reward, one point on success, is honest and slow to learn from. A shaped reward adds hints such as distance to the object. It is faster to learn from and easy to get wrong.
  • The agent optimises what you wrote. If spinning in place earns shaped reward, the agent will spin. Watch the behaviour, not only the return.
  • Someone has to measure success. On a real robot the reward comes from a person, a sensor, or a learned success detector, and its mistakes become the policy’s goals.
  • Resets and safety. Each trial needs the scene put back, and exploration means trying things that fail. Both are cheap in simulation and expensive on hardware.

Common mistakes

  • Reaching for RL first. If demonstrations are available, start with imitation. Add reward when you need to exceed them.
  • Reading only the return curve. A rising return can come from an exploit in the reward. Watch rollouts.
  • Trusting the simulator. A policy can exploit simulator errors. Test on hardware early.

Terms

Reward
A number the environment gives the agent after each step.
Return
The sum of rewards over an episode, usually discounted.
Value
The return expected from a state, or from an action in a state.
Exploration
Trying actions whose outcome is unknown in order to learn about them.
On-policy and off-policy
Whether a method needs fresh data from the current policy or can reuse old data.
Offline RL
Reinforcement learning from a fixed dataset with no new trials.
Markov decision process
The formal setting of reinforcement learning: states, actions, transitions, rewards and a discount.
Bellman equation
The recursive relation between the value of a state and the values of the states that follow it.
Advantage
How much better an action is than the average action in that state.
Temporal-difference error
The gap between a value estimate and the reward plus the discounted estimate of the next state.

Further reading