kamila kuatova and i built within reach for the nvidia physical ai sprint. it’s an assistive object-retrieval system for people with limited upper-body mobility: someone asks for an item, and an so-101 robot arm picks it up and puts it somewhere they can actually reach. it was my first real project in physical ai, and almost everything about it worked differently from the software i was used to building.
the scene is set up like a kitchen counter. the easy case is when the requested item is sitting in the open. the case we cared about is when it isn’t, because another object is in the way and the arm has to deal with that before it can do what it was asked.
we split the system so the decision-making never knows where it’s running. a planner decides what needs to happen, a shared executor carries it out one phase at a time, and a monitor checks the result. all three talk to a backend interface instead of to the simulator directly. one backend is a digital twin in isaac sim, and the other drives a physical so-101 through lerobot. the demonstration we built and showed ran in isaac sim. the lerobot backend is the path toward real hardware, and i’m not claiming a result on the physical arm here.
the core of the planner is how it handles blockers. before a delivery, the planner checks the path between the gripper and the target for other objects. if something is in the way, it doesn’t try to squeeze around it. it pushes a new goal, relocate the blocker, on top of a goal stack, finds a clear spot to put the blocker down, and moves it there. once that goal is done it gets popped, and the original delivery is back on top, exactly where it was left.
the stack kept the planner simple. relocating a blocker is really just another pick-and-place, so it didn’t need its own code path, and the original request never got lost while the arm was busy with something else.
both kinds of goals run through the same executor: approach, grasp, lift, carry, place. sending a command isn’t the same as it happening. a grasp can close on nothing, and an object can slip while it’s being carried. so after every phase, the monitor checks the physical conditions that phase was supposed to create before the next one starts, and a failed check goes to recovery logic instead of carrying on as if nothing happened.
the bug that taught me the most had nothing to do with planning. our first inverse kinematics model passed its own test: give it a target point, get back joint angles, run those angles through the same model’s forward kinematics, and you land right on the target. it looked finished. then we calibrated it against the twin, commanding the simulated arm to those angles and measuring where the gripper actually ended up, and it was 380 mm away from where the model thought it was.
the round trip passed because it only proved the model agreed with itself. if the model’s idea of the arm is wrong, the inverse and forward directions share the same mistake, and the check can’t see it. the only thing that exposed it was measuring the arm the model was supposed to describe. after that, calibration against the twin became the test we actually trusted. the same lesson applies one level up: the twin is also a model, so moving to the physical so-101 would mean measuring again instead of assuming the simulation carries over.
building something outside traditional software made me want to keep exploring physical ai. most of what i learned came from the moments where the math said one thing and the arm did another.