← All topics

Guide · Robot learning · Topic 14 of 32

Robotics fundamentals

Frames, kinematics, rotations, controllers and cameras: the classical layer every learned policy sits on.

10 minute read

Everything is measured in a frame

A robot never knows “where the cup is”. It knows where the cup is relative to something: the camera, the robot’s base, or the gripper. Each of those is a frame, a point with a set of axes attached. A pose is the position and orientation of one frame as seen from another.

A robot arm, a camera on a stand and an object on a table, each with its own pair of axes. Dashed arrows show three transforms: base to gripper from joint angles, base to camera from calibration, and camera to object from perception.
Five frames and the three transforms a manipulation system uses all the time. Each transform comes from a different source, so each has its own kind of error.

A transform converts a pose from one frame to another, and transforms chain. To reach for the cup, the robot needs the cup in its base frame, which is the base-to-camera transform followed by the camera-to-object transform. An error in either link passes straight through: a camera calibration that is 2 cm off gives a grasp that is 2 cm off.

Three questions catch most frame bugs. Which frame is this number expressed in? What are the units, metres or millimetres, radians or degrees? In which order are the transforms applied? None of these mistakes raises an error. The robot just reaches for the wrong place.

Transforms in symbols

A pose is a rotation RR and a translation tt. Packing them into one 4 × 4 matrix turns “apply the rotation, then add the translation” into a single matrix product:

T=[Rt01][p′1]=T[p1]T−1=[R⊤−R⊤t01]T = \begin{bmatrix} R & t \\ 0 & 1 \end{bmatrix} \qquad \begin{bmatrix} p' \\ 1 \end{bmatrix} = T \begin{bmatrix} p \\ 1 \end{bmatrix} \qquad T^{-1} = \begin{bmatrix} R^{\top} & -R^{\top} t \\ 0 & 1 \end{bmatrix}

Write TABT_{AB} for the pose of frame BB expressed in frame AA. Transforms chain by multiplication, and the notation checks itself: the inner letters must match and cancel.

Tbase, object=Tbase, camera  Tcamera, objectT_{\text{base},\,\text{object}} = T_{\text{base},\,\text{camera}} \; T_{\text{camera},\,\text{object}}

This is the chain in the figure above. Hand-eye calibration is the problem of finding the fixed transform Tbase,cameraT_{\text{base}, \text{camera}}, or Tgripper,cameraT_{\text{gripper}, \text{camera}} for a wrist camera, from several views of a known target. It reduces to an equation of the form AX=XBA X = X B, where AA and BB are measured motions of the robot and of the target and XX is the unknown.

A rotation matrix has nine entries and three degrees of freedom. It must satisfy R⊤R=IR^{\top} R = I and det⁡R=1\det R = 1. That redundancy is the root of the representation problems in the next section: a network’s raw output will not satisfy those constraints unless it is projected back onto them.

Kinematics: joints and the hand

Forward kinematics turns joint angles into the pose of the gripper. It is a fixed calculation with one answer. Inverse kinematics goes the other way, from a wanted gripper pose to joint angles, and it is harder: there can be several answers, or none.

Left: a two-link arm reaching the same target with the elbow up and with the elbow down, and a point marked out of reach. Right: the hand's path from A to B is curved when the joints move evenly and straight when the hand moves evenly.
Computed for a two-link arm. Left: two joint solutions for one hand position, and a target with none. Right: the same start and end, interpolated in joint space and in hand space.

This matters as soon as a policy outputs gripper poses. A solver underneath has to pick one of the solutions, and near a singularity, where the arm is stretched out or two joints line up, a small move of the hand needs a very large move of the joints. A policy that outputs joint angles avoids the solver, at the price of being tied to one robot.

The right-hand panel shows a second consequence. Moving the joints evenly does not move the hand in a straight line. If the hand has to stay on a line, for example to slide along a table, the motion has to be planned in the hand’s space.

The Jacobian

Forward kinematics is a function x=f(q)x = f(q) from joint angles to the pose of the hand. Its derivative, the Jacobian J(q)=∂f/∂qJ(q) = \partial f / \partial q, relates small changes, and through them velocities and forces:

x˙=J(q) q˙τ=J(q)⊤F\dot x = J(q)\, \dot q \qquad\qquad \tau = J(q)^{\top} F

The first equation says how fast the hand moves for given joint speeds. The second says what joint torques produce a force FF at the hand, which is how a robot pushes with a chosen force without a force sensor. For the two-link arm with link lengths l1,l2l_1, l_2:

J=[−l1sin⁡q1−l2sin⁡(q1+q2)−l2sin⁡(q1+q2)     l1cos⁡q1+l2cos⁡(q1+q2)     l2cos⁡(q1+q2)]J = \begin{bmatrix} -l_1 \sin q_1 - l_2 \sin(q_1 + q_2) & -l_2 \sin(q_1 + q_2) \\ \;\;\, l_1 \cos q_1 + l_2 \cos(q_1 + q_2) & \;\;\, l_2 \cos(q_1 + q_2) \end{bmatrix}
A two-link arm in three configurations, each with an ellipse at the hand showing how far it can move in each direction for a unit of joint motion. Elbow bent: a rounded ellipse, manipulability 0.80. Partly extended: a narrower ellipse, 0.67. Nearly straight: the ellipse has collapsed to a line, 0.07.
Computed from the Jacobian above. The ellipse shows the hand velocities that unit joint velocities can produce. As the arm straightens it flattens: the hand can still move sideways and can barely move along the arm.

Inverse kinematics is usually solved by iterating on the Jacobian: Δq=J+Δx\Delta q = J^{+} \Delta x, with J+J^{+} the pseudo-inverse. The size of the ellipse, w=det⁡(JJ⊤)w = \sqrt{\det(J J^{\top})}, is called manipulability. Where it approaches zero the arm is at a singularity, the pseudo-inverse blows up, and a small requested motion of the hand demands enormous joint speeds. The damped form J⊤(JJ⊤+λ2I)−1J^{\top} (J J^{\top} + \lambda^2 I)^{-1} trades a little accuracy for staying finite. An arm with seven joints has one more than a pose requires, and that spare freedom, the null space of JJ, is used to keep away from singularities and joint limits.

Rotations are not ordinary numbers

Position is three numbers that behave well. Orientation is not. Every compact way to write a rotation has a place where it breaks.

Top: an angle written as one number rises steadily and then jumps from plus 180 degrees to minus 180 degrees. Bottom: the same rotation written as a cosine and a sine changes smoothly throughout.
A gripper turning steadily. Written as one angle, the value jumps where nothing physical happened. Written as two numbers, it is smooth.

A network asked to output the top curve must reproduce a jump exactly, and averaging two nearly identical poses on either side of it gives a pose that points the opposite way. That is why modern policies avoid single angles.

RepresentationNumbersIts problemWhere you will see it
Euler angles3Depends on the axis order; wraps around; loses a degree of freedom at some posesDatasets, user interfaces
Axis-angle3Wraps at half a turnMany robot action formats
Quaternion4Two quaternions describe every rotation; must be kept at unit lengthSimulators, robot drivers
Rotation matrix9Redundant; a network’s output is not a valid matrix without clean-upMaths and libraries
6D (two matrix columns)6Needs a small step to rebuild the matrixNetwork outputs

Two habits follow. Find out which representation a dataset uses before training on it. And never add rotations: a relative rotation is applied by multiplying, not by adding angles.

From an action to motor torque

A learned policy produces targets 10 to 50 times a second. A classical controller runs underneath at 500 to 1,000 times a second and turns each target into motor torques. The controller’s settings decide what an action physically does.

Two simulated runs of the same position command into a surface. The stiff controller reaches the surface quickly, hits it hard and holds 78 newtons. The compliant controller arrives later and holds 8 newtons.
A simulated one-joint model. Both controllers receive the same command to a point just past a surface. The stiff one presses with 78 N, the compliant one with 8 N.
  • Position control. Go to this position and resist anything in the way. It is precise in free space and harsh in contact.
  • Impedance control. Behave like a spring and damper around the target. The stiffness sets how much force each millimetre of error produces, so contact is gentle and position is less exact.
  • Torque control. Command the motor torques directly. It gives the most freedom and needs a fast loop and a good model of the robot.

The lesson for learning is that an action is only meaningful together with the controller that executed it. Demonstrations recorded with a compliant controller will not replay correctly through a stiff one. The commanded position and the position actually reached also differ, and a dataset may store either.

Controllers in symbols

The lowest loop turns a position error e=qd−qe = q_d - q into a motor command. The classic law has three terms:

u=Kp e+Ki∫e dt+Kd e˙u = K_p\, e + K_i \int e \,\mathrm{d}t + K_d\, \dot e
Step response of one joint carrying a constant load under three controllers. Proportional only overshoots to 1.5 and oscillates. Proportional plus derivative settles smoothly at 0.9, short of the target of 1.0. With the integral term added it settles at 1.0.
A simulated joint holding a constant load, such as its own weight. Proportional control alone oscillates. A derivative term damps it and leaves a steady error. The integral term removes that error.

A better controller does not wait for an error to appear. The arm’s dynamics are known:

M(q) q¨+C(q,q˙) q˙+g(q)=τ+J⊤FextM(q)\, \ddot q + C(q, \dot q)\, \dot q + g(q) = \tau + J^{\top} F_{\text{ext}}

with MM the inertia, CC the velocity-dependent terms and gg gravity. Computing g(q)g(q) and adding it to the command, gravity compensation, removes the load that the integral term was fighting, and it is why a torque-controlled arm can be moved by hand and stays where it is left.

Impedance control specifies the relationship between motion and force directly. The controller makes the hand behave as a spring and damper around its target:

F=K (xd−x)+D (x˙d−x˙)τ=J⊤F+g(q)F = K\, (x_d - x) + D\, (\dot x_d - \dot x) \qquad\qquad \tau = J^{\top} F + g(q)

KK is stiffness in newtons per metre. With K=400K = 400 a 2 cm error produces 8 N; with K=4,000K = 4{,}000 it produces 80 N. That single number decides whether a contact is gentle, which is the comparison in the figure above.

Cameras and calibration

A pinhole camera diagram: a ray from a point in the scene passes through the image plane to the camera centre, with the formula u equals f times X over Z plus c x.
The pinhole model. A point’s pixel position depends on its sideways offset divided by its depth, which is why one image cannot give distance on its own.

Intrinsics describe the camera itself: focal length, image centre and lens distortion. They convert a pixel into a ray. Extrinsics are the camera’s pose relative to the robot. Finding them is called hand-eye calibration. With both, and a depth value, a pixel becomes a point the robot can reach for.

End-to-end policies do not use calibration explicitly. They learn the mapping from pixels to motion from data. That hides the dependency without removing it. A policy trained with the camera in one place has absorbed that placement, and a camera bumped by a few centimetres is a quiet cause of a drop in success.

Time is part of the state

Every sensor and motor has a delay. A camera frame is tens of milliseconds old when it arrives, and a command takes time to become motion. Two rules keep this under control.

  • Timestamp everything at the source. Then align streams by time, not by the order they arrived in.
  • Keep the training and deployment timing the same. If images lagged the joint readings by 100 ms in the data and by 20 ms on the robot, the policy sees a relationship it was never trained on.

The note on deployment covers what to do about the delay that remains.

Common mistakes

  • Mixing frames or units. A pose in the camera frame fed to a function that expects the base frame gives no error and a wrong reach.
  • Adding rotations. Angles do not add. Compose rotations by multiplication.
  • Assuming the command was reached. Check whether the dataset stores commanded or measured positions.
  • Moving the camera. Treat camera placement as part of the model, and check it before blaming the policy.

Terms

Frame
A point with axes attached, in which positions and orientations are expressed.
Pose
The position and orientation of one frame relative to another.
Forward kinematics
Computing the gripper pose from the joint angles.
Inverse kinematics
Finding joint angles that give a wanted gripper pose.
Singularity
An arm configuration where some hand motions need very large joint motions.
Impedance control
A controller that makes the robot act like a spring and damper around its target.
Intrinsics and extrinsics
A camera’s internal geometry, and its pose relative to the robot.
Homogeneous transform
A 4 × 4 matrix that holds a rotation and a translation, so that poses compose by multiplication.
Jacobian
The matrix relating joint velocities to hand velocity, and hand forces to joint torques.
Manipulability
A measure of how freely the hand can move in every direction at a given configuration.
Gravity compensation
Adding the torque needed to hold the arm’s own weight to every command.

Further reading