The problem
Balancing a broomstick on your palm is easy to describe and hard to do. The stick is always falling, and the only fix is to keep moving your hand under it. Now do it while walking, when every step shakes the hand.
For the robot there are three more constraints:
- The pole is free. It is 1.5 m long and stands on an open palm. Nothing clamps it.
- The robot only sees it through its head camera. There is no sensor on the pole. The camera is noisy, slow, and sometimes wrong.
- One controller moves all 31 joints. The legs have to walk without shaking the arm more than the arm can correct.
It's a good research task because it can't be faked. A robot that carries a box can be stiff and slow. A pole falls in under a second if the estimate or the reaction is off.
What I built
Pure reinforcement learning didn't work
The first plan was the usual one: reward the robot for keeping the pole up and let it learn everything. It learned to stand with the pole. It never learned to walk with it: 0.2% of walking episodes succeeded.
A classical controller underneath, learning on top
Pole balancing is a textbook control problem with a known solution (LQR). So I put that solution inside the arm, with inverse kinematics to turn "move the palm here" into joint angles. The learned policy then only has to add corrections for what the simple model gets wrong, and make the legs cooperate. This is called residual reinforcement learning.
With that change, the first run went from 0.2% to 75.7% success while walking. Everything after that was about making the numbers honest and then making them high.
A learned estimator
The balance controller needs to know how fast the pole is tipping, and a camera can't measure that directly. A Kalman filter estimates it from a simple model. I trained a small network to do the same job by imitating a "teacher" that sees the true pole state in simulation. The method is DAgger: the student drives, the teacher labels what it should have said, and the student retrains on its own mistakes. The result is about 30% more accurate than the Kalman filter.
The robot's real software interface, before the real robot
Training happens in Isaac Lab. Before asking for robot time, I ran the exported policy in a different simulator (MuJoCo), talking to it through the same messaging layer the real H2 uses (DDS). A policy that only works in the simulator it was trained in has learned that simulator.
Results
All numbers are from 1,024 test episodes on a random seed the model was never trained or selected on, with the realistic camera model.
| Test | Pole still up |
|---|---|
| Standing | 98.8% |
| Walking at 0.3 m/s | 99.0% |
| Walking at 0.5 m/s | 95.1% |
| Pole handed over carelessly, then released | 86.7 – 92.0% |
| Second simulator, through the robot interface, walking | 30 / 30 |
The 30 / 30 row uses a later checkpoint, retrained with the robot-side velocity estimator after I traced early failures in that simulator to a bug in it.
Where the walking number came from, same test each time:
| Version | Walking success |
|---|---|
| Reinforcement learning alone | 0.2% |
| + LQR and inverse kinematics underneath | 75.7% |
| + honest camera, learned estimator (first version) | 75.8% |
| + error handling for bad camera frames, stepping allowed while standing | 99.0% |
What went wrong
These cost the most time and taught me the most.
The estimator that looked twice as good as it was
On recorded data, my learned estimator beat the Kalman filter by a factor of two. In the loop it was much less impressive. The cause was a leak: one input was the palm's velocity, and during data collection the arm had been driven by the true pole state. So the palm's motion gave away the answer. I re-collected the data with the arm driven by the estimate, which is what happens in real use. The honest gain is about 30%.
A better camera made the robot drop the pole
In the second simulator the policy failed within two seconds, but only when the camera was less noisy than in training. The filter adapts to how noisy the camera looks. With clean frames it trusted them fully and passed small contact vibrations straight into the "how fast is it tipping" signal. The fix was one line: never let the filter believe the camera is better than a set floor. Standing through the full camera pipeline went from 0 of 3 to 26 of 30.
A camera model that was too kind
An early version of the simulated camera reported the pole's tipping speed directly. No real camera does that. Results dropped a lot when I replaced it with an honest model, and that drop was the real starting point.
A teacher the student couldn't copy
I tried training a "teacher" policy with perfect information and then a "student" to imitate it from camera data. It failed: the teacher's actions depended on things the student's sensors can't see, so there was nothing consistent to imitate. Only teach from a teacher the student could, in principle, become.
What I'd tell someone starting this: training curves aren't results. Every claim here comes from held-out test runs, and several ideas that looked good in training lost to their own starting point when tested the same way.
What's next
- Run it on the real H2 this winter, standing first, then walking.
- Measure the real arm's friction and delay and feed them back into training.
- Turning in place still tracks only about 70% of the commanded rate.
The code is private while the work is in progress. I'm happy to walk through it.