Guide
Deep learning and robotics: a working guide
Thirty-two technical notes on the ideas behind today’s models and the robots they drive. Each starts from first principles and goes down to the equations, and wherever possible the figures are measured: on real models, on real robot data, or in small experiments run for the note. They favour ideas over model names, because the names change every few months and the ideas do not.
Read in order, or jump to a topic. Each group builds on the one before it: how a network learns, which models exist, how they become robot policies, what bodies and data they need, and how the work is done. If you already know one side of the subject, these two shorter routes skip what you have.
- Robotics fundamentals
- Robot learning basics
- Reinforcement learning
- Robot policies today
- System 0, 1 and 2 architectures
- Long-horizon tasks
- World models
- Embodiments
- Senses beyond vision
- Dexterous hands
- Data for robots
- Data quality and curation
- Data efficiency and generalisation
- The data flywheel
- Evaluation
- Failure triage
- Deployment on the robot
- Understanding and safety
- Things I know
- Foundations that survived
- Sequence-to-sequence models
- The transformer
- Attention in depth
- Representation learning
- Contrastive and multimodal learning
- Retrieval and multimodal retrieval
- Generative modeling
- Stable Diffusion and latent diffusion
- Foundation models and adaptation
- Vision-language models
- Robot policies today
- Training at scale
Foundations
How any network learns, and the blocks everything is built from.
01Things I knowSix habits that outlast any model, and how silent failures are caught.
02Foundations that survivedThe training loop, learning rates, losses, overfitting and the stabilisers that make deep networks trainable.
03Sequence-to-sequence modelsOne sequence in, another out: the bottleneck, attention, teacher forcing, decoding, and why robot control has the same shape.
04The transformerOne block, stacked many times, that now reads text, images, video and robot actions.
05Attention in depthOne operation, many forms: self, cross and shared attention, heads and masks, what it costs, and the linear recurrent variants.Models
The model families in use today, and how they are trained.
06Representation learningHow models learn useful features from data with no labels.
07Contrastive and multimodal learningLearning by comparison: triplet and batch-wide losses, mining, and how two kinds of data are brought into one space.
08Retrieval and multimodal retrievalFinding instead of generating: embeddings, ranking metrics, searching across text and images, large collections, and uses in robotics.
09Generative modelingAutoregression, diffusion and flow matching, and the latent spaces they work in.
10Stable Diffusion and latent diffusionOne real image generator taken apart: the autoencoder, the text encoder, the denoiser, guidance and steps, and why robotics builds on it.
11Foundation models and adaptationPretrain once, adapt many times: the ladder from prompting to fine-tuning.
12Vision-language modelsHow a language model is taught to read images, what the image costs, what these models get wrong, and how one becomes a robot policy.
13Perception and 3DFrom pixels to objects, depth and maps, and when a robot still needs them.Robot learning
Turning those models into policies that move a robot.
14Robotics fundamentalsFrames, kinematics, rotations, controllers and cameras: the classical layer every learned policy sits on.
15Robot learning basicsThe control loop, action spaces, imitation learning and where reinforcement learning fits.
16Reinforcement learningLearning from reward by trial and error: where it works, why it is expensive, and how it is used with imitation.
17Robot policies todayHow a model that understands images and language is turned into one that moves a robot.
18System 0, 1 and 2 architecturesWhy robot policies are split into models that run at different speeds, what each layer does, and how they are connected.
19Long-horizon tasksAction chunks, sub-task decomposition, chain of thought and memory: how robots get through tasks with many steps.
20World modelsModels that predict what happens next, and models that predict and act together.Data and hardware
The bodies, the senses, and the data they produce.
21EmbodimentsThe bodies and sensors a policy actually controls.
22Senses beyond visionTouch, force, sound and body state: what each tells a robot, and how they are combined in one policy.
23Dexterous handsMulti-fingered hands in depth: real hardware and its specifications, how a hand connects to the action head, retargeting, and using touch.
24Data for robotsWhere robot training data comes from, and why quality matters more than volume.
25Data quality and curationWhat makes a good demonstration, how a large dataset is checked and scored, and how to measure what data is worth.
26Data efficiency and generalisationHow much data a robot needs, what makes it need less, and what the result carries over to.
27The data flywheelTurning what a deployed robot gets wrong into the data that fixes it: what to capture, how to use it, and how to keep the loop honest.Practice
How the work is trained, checked, debugged and shipped.
28Training at scaleA recipe for starting any training project, and the engineering around a large run.
29EvaluationHow to tell whether a model, or a robot, is actually better.
30Failure triageWhen the robot fails: how to tell whether the controller, the pipeline, the data or the policy is at fault.
31Deployment on the robotMaking the model small enough and fast enough, then learning from the field.
32Understanding and safetyLooking inside a model, knowing when not to trust it, and keeping people safe regardless.Robot camera frames and action curves are from the LIBERO benchmark (Liu et al., 2023) via the lerobot/libero_10 dataset (Apache-2.0). Robot photos are from Wikimedia Commons: Unitree G1 by Sayanesy (CC0); Franka by Ims, Flexiv Rizon by MichelleACipriano and MAiRA by Anna Schneider2021 (all CC BY-SA 4.0). Toy-model figures are from my own technical videos. The perception figures show the output of Grounding DINO, SAM 2 and Depth Anything V2 run on a LIBERO frame. Charts marked as computed or run for a note were produced for this guide, and most can be reproduced with the notebooks. Touch, pressure and force recordings are from the TaF tactile-force dataset (Voxel51/TaF-Tactile-Force, MIT). The two sound clips are public-domain recordings from PDSounds via Wikimedia Commons: pouring water by earthcalling and knocking by stephan. Generated images are from Stable Diffusion 1.5 (CreativeML Open RAIL-M licence), run for this guide. The handwritten digits are the scikit-learn digits set. The attention maps are real weights from the public PaliGemma model run on a LIBERO frame. Further photographs from Wikimedia Commons, all CC BY-SA 4.0: strain-wave gear by wdwd; gear set by Pieceofmetalwork; Spot by Jonte; TIAGo by PAL Robotics; depth camera by Marc Auledas. Hand photos are from Wikimedia Commons: Shadow Dexterous Hand by Richard Greenhill and Hugo Elias (CC BY-SA 3.0); Schunk SVH by NearEMPTiness and Allegro hand with uSkin by XELA Robotics (both CC BY-SA 4.0).