Guide

Deep learning and robotics: a working guide

Thirty-two technical notes on the ideas behind today’s models and the robots they drive. Each starts from first principles and goes down to the equations, and wherever possible the figures are measured: on real models, on real robot data, or in small experiments run for the note. They favour ideas over model names, because the names change every few months and the ideas do not.

32 topicsabout 272 minutes in totalGlossary27 notebooksThe whole guide as slides (PDF)PowerPoint

Read in order, or jump to a topic. Each group builds on the one before it: how a network learns, which models exist, how they become robot policies, what bodies and data they need, and how the work is done. If you already know one side of the subject, these two shorter routes skip what you have.

Foundations

How any network learns, and the blocks everything is built from.

Models

The model families in use today, and how they are trained.

Robot learning

Turning those models into policies that move a robot.

Data and hardware

The bodies, the senses, and the data they produce.

Practice

How the work is trained, checked, debugged and shipped.

Robot camera frames and action curves are from the LIBERO benchmark (Liu et al., 2023) via the lerobot/libero_10 dataset (Apache-2.0). Robot photos are from Wikimedia Commons: Unitree G1 by Sayanesy (CC0); Franka by Ims, Flexiv Rizon by MichelleACipriano and MAiRA by Anna Schneider2021 (all CC BY-SA 4.0). Toy-model figures are from my own technical videos. The perception figures show the output of Grounding DINO, SAM 2 and Depth Anything V2 run on a LIBERO frame. Charts marked as computed or run for a note were produced for this guide, and most can be reproduced with the notebooks. Touch, pressure and force recordings are from the TaF tactile-force dataset (Voxel51/TaF-Tactile-Force, MIT). The two sound clips are public-domain recordings from PDSounds via Wikimedia Commons: pouring water by earthcalling and knocking by stephan. Generated images are from Stable Diffusion 1.5 (CreativeML Open RAIL-M licence), run for this guide. The handwritten digits are the scikit-learn digits set. The attention maps are real weights from the public PaliGemma model run on a LIBERO frame. Further photographs from Wikimedia Commons, all CC BY-SA 4.0: strain-wave gear by wdwd; gear set by Pieceofmetalwork; Spot by Jonte; TIAGo by PAL Robotics; depth camera by Marc Auledas. Hand photos are from Wikimedia Commons: Shadow Dexterous Hand by Richard Greenhill and Hugo Elias (CC BY-SA 3.0); Schunk SVH by NearEMPTiness and Allegro hand with uSkin by XELA Robotics (both CC BY-SA 4.0).