← All topics

Guide · Data and hardware · Topic 22 of 32

Senses beyond vision

Touch, force, sound and body state: what each tells a robot, and how they are combined in one policy.

12 minute readBuilds on: Embodiments

Vision is not enough

Robot policies are built around cameras, and for reaching toward things that is sufficient. Once the hand closes, the camera’s view is blocked and the questions change: is it in my grip, is it slipping, has it touched down? Those are answered by other senses.

A grid of six senses against five phases of a pick-and-place: approach, contact, grasp and lift, carry, place. Cameras are decisive during approach. Force, torque and touch are decisive from contact onward. Sound is decisive at placement.
A schematic of which sense carries the information in each phase of picking something up and putting it down. The ratings are a judgement, not a measurement.

What each sense tells the robot

SenseWhat it measuresGood forWatch out for
Body state (proprioception)Joint angles and speeds, gripper openingKnowing where the robot is; always available and precisePolicies lean on it and ignore the cameras
Joint torque and motor currentHow hard each joint is workingDetecting contact and load with no extra sensorNoisy; mixes contact with the arm’s own weight and friction
Wrist force and torqueThe total push and twist at the wristInsertion, wiping, pressing, knowing when to stopReports one total, not where contact is; drifts and needs zeroing
Touch, camera-basedA fine image of the fingertip’s skin deformingShape, texture and slip at the contactBulky; the skin wears; units differ from one another
Touch, pressure arraysPressure over a grid of cellsCovering fingers and palms; contact locationCoarse; says little about sideways force
SoundVibration through air, or through the gripper with a contact microphoneContact events, slip, pouring, clicking into placeMotor and room noise; differs between sites
Inertial sensorAcceleration and rate of turningBalance and base motion on legged and mobile robotsDrifts when integrated over time
Depth and lidarDistance to surfacesGeometry, obstacles, navigationShiny, dark and transparent objects

Less common inputs include thermal cameras, event cameras that report only changes in brightness, and radar. They solve specific problems and bring little shared data or tooling with them.

What the signals look like

It is easier to reason about a sense after seeing its raw data. These are real recordings.

Three moments of one press at 5, 17 and 30 newtons. Top row: images from a camera inside a gel fingertip, showing the gel surface deforming more as force rises. Middle row: a 12 by 12 pressure array lighting up more cells. Bottom: pressing force and sideways force from the wrist sensor over 46 seconds, with the three moments marked.
One recorded press seen by three sensors at once, from the public TaF tactile-force dataset. Top: the camera inside a gel fingertip. Middle: a 12 × 12 pressure array. Bottom: the wrist force sensor.

The three sensors answer different questions about the same contact. The gel camera shows the shape pressing into the finger in fine detail. The pressure array shows where the contact is, coarsely. The force sensor gives one clean number for how hard, and nothing about where.

Three plots over a 21-second episode: the hand's position in three axes in centimetres, its rotation in three axes in degrees, and the gripper opening, which closes at about 4 seconds, opens at about 10 and closes again at about 16.
Body state through one real robot episode: where the hand is, how it is turned, and how far the gripper is open. This robot reports 8 numbers per step.

Body state is the plainest signal of all: a few smooth numbers per step. That is why it is so easy for a policy to lean on.

Depth cameras

An Intel RealSense D435 depth camera on a small tripod, showing its two infrared cameras, infrared projector and colour camera in a row
A stereo depth camera of the kind found on many research robots. The two outer lenses are infrared cameras, and the projector between them adds texture to blank surfaces.

A stereo depth camera finds the same point in two images taken a known distance apart. The point’s shift between the images, the disparity dd, gives its depth:

Z=f BdδZ≈Z2f B δdZ = \frac{f\, B}{d} \qquad\qquad \delta Z \approx \frac{Z^2}{f\, B}\, \delta d

with ff the focal length in pixels and BB the baseline between the cameras. The second expression is the important one in practice. Depth error grows with the square of the distance, so a sensor that is accurate to a millimetre at 30 cm is accurate to about a centimetre at one metre. It also needs texture to match, which is why these cameras project an infrared pattern, and why they fail on shiny, transparent and very dark surfaces, where the pattern is reflected away or absorbed.

A closer look at sound

Two real sounds, each shown as a waveform above a spectrogram. Water poured into a mug is a broad wash of energy across all pitches that swells as the mug fills. Knocking on wood is three sharp bursts, each a set of horizontal bands that die away in a fraction of a second.
Two public-domain recordings. The waveform on top is what the microphone records. The spectrogram below shows the same sound as pitch against time, with brightness for loudness. A continuous flow and a sharp contact look nothing alike.

A spectrogram is the standard way to look at sound and to feed it to a model. The knock shows why. In the waveform the three knocks are spikes. In the spectrogram each is a column of bands that rings and fades, a pattern as recognisable as an object in a photo.

Sound is the newest of these in robot learning and the cheapest to add. A contact microphone on the gripper hears what the fingers do: the scrape of a spatula under food, the click of a plug seating, the change in pitch as a container fills. These are events a camera cannot see, and they arrive at the moment they happen.

The published work on this is direct about the conditions. Sound helps most where vision is ambiguous, in contact-rich tasks, and it has to be collected with the same microphone placement that the robot will use. Background noise in deployment differs from the collection site, so it is added during training as augmentation.

From a waveform to a spectrogram

The spectrograms above are computed with the short-time Fourier transform. The signal xx is cut into overlapping windows of length NN, each multiplied by a tapering function ww and transformed:

X[m,k]=∑n=0N−1x[n+mH]  w[n]  e−2πi kn/NX[m, k] = \sum_{n=0}^{N-1} x[n + m H]\; w[n]\; e^{-2\pi i\, k n / N}

mm indexes time in hops of HH samples and kk indexes frequency. The window length sets an unavoidable trade: a long window resolves pitch finely and blurs timing, and a short one does the reverse. The figures here use 64 ms windows with a 10 ms hop at 16,000 samples a second.

The power ∣X∣2|X|^2 is then pooled into bands spaced on the mel scale, which is roughly linear below 1,000 cycles a second and logarithmic above, matching how hearing resolves pitch, and the logarithm is taken:

m(f)=2595 log⁡10 ⁣(1+f700)m(f) = 2595\, \log_{10}\!\Big(1 + \frac{f}{700}\Big)

The same recipe applies to any fast signal. A force sensor or an accelerometer on a fingertip can be turned into a spectrogram the same way, and the result fed to the same kind of encoder.

Different senses, different clocks

Typical reading rates on a log scale. Cameras and camera-based touch sensors report tens of times a second. Inertial sensors, joint sensors and wrist force sensors report hundreds to thousands of times a second. Microphones report tens of thousands of times a second. A shaded band marks the 10 to 50 times a second at which a learned policy runs.
Typical ranges. A learned policy runs at the rate of the slowest senses, so the faster ones must be summarised before it sees them.

A policy that runs 10 to 50 times a second cannot take in a signal that arrives a thousand times a second sample by sample. Each fast signal is cut into a short window, the most recent tenth of a second for example, and encoded into a few tokens. Force becomes a short series. Sound becomes a spectrogram, a picture of pitch against time, and goes through an image-style encoder. Throwing the window away and keeping only the latest reading loses exactly what made the sense useful: the spike at impact, the vibration of a slip.

How a model takes each sense in

A transformer accepts only one thing: a sequence of tokens. Every sense therefore goes through the same three steps. It is cut into pieces, each piece is turned into a vector by an encoder built for that kind of signal, and the vectors join the sequence.

A table of seven inputs: camera image, touch image, pressure array, force and torque, body state, sound and instruction. For each: the form it arrives in, how it is prepared, the encoder used, and the typical number of tokens, from 256 for a camera image down to 1 for body state. All the tokens join one sequence that goes to the transformer.
How each input becomes tokens. The token counts are typical values. They matter because the cost of running the model grows with the length of the sequence.
  • Camera and touch images. Cut into square patches, and each patch becomes one token. A camera-based fingertip produces an image, so it uses the same machinery, usually a smaller encoder trained on touch images.
  • Pressure arrays. Small enough to treat as a tiny image or to flatten into one vector.
  • Force and torque. The last tenth of a second or so, as a short series of six-number readings, goes through a small network that outputs one or a few tokens. The window is what preserves spikes and vibration.
  • Body state. Normalised and passed through a single linear layer to become one token.
  • Instruction. Split into word pieces by the language model’s tokenizer, each looked up in a table of vectors. “put the white mug on the plate” becomes seven tokens: put, the, white, mug, on, the, plate.
Three steps for sound. One second of a knock as a waveform of 16,000 numbers. The same second as a spectrogram of 64 pitch bands by 96 time steps. The spectrogram cut by a grid into 24 patches, which become 24 tokens.
Sound, step by step, on a real clip. One second is 16,000 numbers. As a spectrogram it is a small image. Cut into patches it is 24 tokens, and from there it is handled like any image.

Sound takes one extra step, because 16,000 numbers a second is far too many to use directly. The waveform is converted to a spectrogram, and the spectrogram is then cut into patches exactly as a photo would be. This is why image models adapt so readily to audio.

Two design choices recur across all of these. The first is how many tokens to spend: more tokens keep more detail and make every pass through the model slower, which the deployment note quantifies. The second is where each encoder comes from. The image encoder arrives pretrained on the web. Encoders for touch, force and sound usually start from nothing and learn only from robot data.

Deadlines: how fast is fast enough

How quickly a robot must act on a sense is set by physics, not by preference. The clearest case is a slipping object, where gravity is the clock.

A line on log axes of how far a released object falls against the time before the robot responds, with four marked points: 74 milliseconds, the human grip reflex, 2.7 centimetres; 100 milliseconds, one step of a 10 hertz policy, 4.9 centimetres; 250 milliseconds, a person reacting by sight, 31 centimetres; 500 milliseconds, waiting out half a second of a chunk, 123 centimetres.
Free fall, from the formula distance = half of g times time squared. A response within 74 ms, the measured human grip reflex, lets an object drop under 3 cm. Half a second lets it drop more than a metre.

People tighten their grip about 74 milliseconds after an object starts to slip, a reflex that runs through touch and does not wait for the eyes or for a decision. Reacting to something seen takes three times as long. A robot that only notices a slip when its next action chunk begins has, by then, dropped the object.

Three groups of bars on a log time axis from 1 millisecond to 3 seconds. What happens at the fingers: first contact in 1 to 5 milliseconds, slip developing over tens of milliseconds, grip and load changing over tenths of a second. How fast people respond: grip reflex about 74 milliseconds, reacting to sight 200 to 250, choosing between options 300 to 500. How fast each robot layer responds: motor controller 1 to 2 milliseconds, compliant control loop 1 to 10, one policy step 20 to 100, waiting for the next chunk 0.5 to 2 seconds, high-level model 1 to 3 seconds.
Events, human responses and robot layers on one time axis. Each robot layer can only answer events slower than itself. The ranges are typical values.

The figure gives a rule for where each sense should be handled. Anything faster than about ten milliseconds, such as limiting force at first contact, belongs in the controller and never reaches the learned policy. Events of tens of milliseconds, such as a slip starting, need either a policy that can react between chunks or a fast reflex beside it. Slow changes, such as how the grasp is settling, fit the policy’s normal rate. Meaning, such as a spoken “stop”, can wait for the high-level model.

Touch has two bands

Three frequency bands on a log axis: steady force and shape below about 10 cycles per second, slip and texture vibration from tens to hundreds, and contact sounds in the thousands. A camera-based touch sensor covers only the first band. A wrist force-torque sensor reaches into the second. An accelerometer, piezo film or contact microphone covers the second and third.
Which sensor can carry which part of the signal. A sensor cannot report changes faster than half its reading rate, so a camera-based fingertip at 30 to 60 frames a second cannot carry vibration.

Touch carries two different kinds of information. The slow part is steady force and the shape of the contact: how hard the fingers press and where. The fast part is vibration, a buzz of tens to hundreds of cycles a second that appears when a surface starts to slide and that also encodes texture. Human skin has separate receptors for each.

This matters when choosing a sensor. Camera-based fingertips give a rich picture of the slow part and, at camera frame rates, cannot carry the fast part. Vibration needs a sensor that reads hundreds or thousands of times a second: a fast force sensor, a small accelerometer, a piezo film, or a contact microphone, which is why touch and sound overlap in practice. When recording a dataset, keep the raw fast signal and note the sensor type and its rate. A summary computed at collection time cannot be recomputed later.

Combining senses in one policy

In a transformer policy, combining senses is mechanically simple. Each one gets its own small encoder that turns it into tokens, and the tokens join the same sequence as the image and language tokens. The difficulty is in the training.

  • One sense dominates. Vision predicts the action well for most of every episode, so the policy learns to rely on it and gives the others almost no weight. The usual remedy is to randomly drop or blank a sense during training, so that the policy has to be able to use each.
  • Check that it is used. Run the policy with one sense removed or replaced by noise. If success does not change, that sense is not contributing, whatever the architecture diagram says.
  • Sensors differ. Two touch sensors of the same model give different readings, and a microphone moved by a centimetre hears something else. Calibrate, or collect across several units.
  • Little shared data. Almost no large public dataset includes touch or sound, so these encoders start from scratch while the vision encoder starts from the whole web.

The practical order is to add a sense when a specific failure calls for it: slipping calls for touch, missed insertions for force, events hidden from the camera for sound.

Common mistakes

  • Adding sensors before finding the failure. Each one adds calibration, wear and a new way for training and deployment to differ.
  • Feeding only the latest reading. Fast senses carry their information in how the signal changes over a short window.
  • Assuming a sense is used. Test with it removed.

Terms

Proprioception
A robot’s sense of its own joint positions, speeds and efforts.
Force-torque sensor
A sensor, usually at the wrist, that measures the total push and twist applied to the hand.
Tactile sensor
A sensor on the fingers or palm that measures contact over an area.
Spectrogram
A picture of a sound: time across, pitch up, loudness as brightness.
Modality dropout
Randomly removing one input type during training so the policy cannot depend on a single one.
Disparity
The shift of a point between the two images of a stereo camera; inversely proportional to its depth.
Short-time Fourier transform
The Fourier transform applied to successive short windows of a signal, giving frequency content over time.
Mel scale
A frequency scale spaced the way human hearing resolves pitch.

Further reading