The problem
A walking policy is normally trained for one robot. Change the robot and you start over. If one network could walk many bodies, a new robot could start from what the others already know.
The difficulty is that the bodies don't match. Cassie has 12 motors and no arms. The H2 has 31. A network has a fixed number of inputs and outputs.
What I built
One layout that fits every robot
I mapped every robot's joints onto one shared list ("left knee" means the same slot on all of them). A robot that lacks a joint leaves its slot empty. A masked network (MaskedMLP) ignores empty inputs and blocks learning signals from empty outputs, so a robot without arms doesn't teach the network anything wrong about arms.
Adding the H2
The H2 is the newest and largest robot in the set. Its first training runs failed in three ways:
- It leaned 62° forward. Leaning that far earned speed reward while it caught itself with its arms and knees. A stronger upright term removed it.
- It was punished 2.6 times harder than Cassie. Penalties were summed over joints, and the H2 has 2.6 times as many. Its penalties drowned out the reward for walking. I rescaled them per robot.
- It kept being reset for no reason. The pelvis sits low and brushed small bumps, which counted as a fall, about 45 times per training iteration. I moved the fall check to the torso.
Motor gains from physics, not from guessing
Each joint has a stiffness and damping setting. Instead of hand-tuning 31 pairs, I computed them from each motor's inertia, with a target response speed and damping, inside the limits the real motor can deliver. With that and a symmetry reward, the difference between left and right steps fell from 17.5% to 4.0%.
A crash that only showed up after hours
Training occasionally died with NaN values. The cause was an angle computed with arccos, whose slope is infinite when two orientations are identical. I replaced it with a distance that has no such point.
Results
| Measure | Before | After |
|---|---|---|
| Left/right step difference (H2) | 17.5% | 4.0% |
| Forward lean (H2) | 62° | upright |
| False "fell over" resets per training iteration | ~45 | 0 |
| Robots on one policy | 4 | 5 |
What I learned
- A policy optimizes the reward you wrote, not the one you meant. Watch the video before reading the curve.
- When robots share a reward, anything summed over joints quietly favours the small robot.
- Reset rules are part of the task. A wrong one can stop learning without any error message.
This walking work is what the pole-balancing project was built on.