last year, i trained my first full-body robot policy on a boston dynamics (bd) spot robot, focusing on the grasp phase of a drawer-opening task.

when we first started this project, i struggled to find practical resources on training/deploying learned policies on spot. this post is the guide i wish i had at the time – a walk through of how we broke down the task, collected demos, and deployed the grasp roll-out.

spot is a particularly approachable robot because it abstracts away many annoying low-level control problems including balance, feet placement, and obstacle-avoidance. it comes with a tablet you can use to walk the robot around the room and maneuver its arm. bd also exposes a handy sdk for devs/researchers to access spot’s capabilities.[A]

in this experiment, we used act (action chunking with transformers)[B] to learn the grasp portion of the drawer-opening behavior through imitation learning. at a high level, a human teleoperated spot through the drawer-opening motion several times, and we trained act on the grasp section of those demos. we did not have time to train and validate the full drawer-opening policy end to end on spot.

i. breaking down the task

we began by building an “engineered skill”: a scripted drawer-opening pipeline that combined spot’s sdk, graphnav, the hand camera, and gemini.

the steps below describe the high-level robot motion. in the engineered implementation, perception and body alignment happened before the manipulation api executed the grasp, so “pre-grasp” and “grasp pose” were conceptual phases rather than two explicit fixed-pose commands.

here are the steps:

1. move the arm toward a pre-grasp pose
2. move the arm to the grasp pose
3. close the gripper around the handle
4. walk backward while maintaining the grasp
5. open the gripper to release the handle
6. reset the arm position
7. done!

programming an engineered skill helped verify that spot was physically capable of completing the drawer-opening task. engineering checkpointed stages also let us validate perception, navigation, grasping, and release separately before collecting demonstrations.

ii. representing the task as data

to train act, we first collected demos of a human teleoperating spot through the task. each demo was recorded as a sequence of timestamps. at each timestep, we captured 2 types of data:

• robot-state data, which described how spot’s arm, gripper, and body were moving
• image data, which showed the drawer, handle, and robot from different viewpoints

together, these data points allowed the policy to connect what spot saw with how it should move. this kind of visuomotor imitation setup follows the broader line of work on learning manipulation from demonstrations.[B]

ii a. robot-state data

at each timestep, we recorded the following robot-state data:

• 6 arm joint positions
• gripper open fraction, normalized from 0 (closed) to 1 (open)
• absolute body z position in the odom frame
• body-frame x velocity
• body-frame y velocity
• body pitch in radians

together, these values formed an 11-dimensional representation of spot’s state and action space.

the arm joints and gripper fraction captured the manipulation itself, body-frame x and y velocity captured the backward walking motion, and absolute body z plus pitch captured the body configuration used to reach the drawer handle.

ii b. image data

at each timestep, we also recorded images from two external cameras that provided complementary views of the task:

• a zed camera captured a third-person view of spot and the surrounding scene, and
• an iphone running the kiwi app and mounted on spot’s arm captured an egocentric view of the interaction with the handle.

multi-camera observations are a common design choice in act-style systems, where complementary viewpoints help the policy resolve ambiguity during contact-rich manipulation.[B]

iii. data collection

during each demo, a human teleoperated spot through the drawer-opening sequence while the system recorded the robot-state and camera streams in parallel.

the collection system was split across two computers:

• a mac connected directly to spot’s wi-fi network read data from the robot.
• a gpu computer recorded the two camera streams and combined them with the robot-state data.

this gave the gpu three independent data streams:

• robot-state measurements
• zed camera frames
• iphone camera frames

because the streams were produced independently, they did not arrive at exactly the same frequencies. before the data could be used for training, the gpu aligned them onto a shared 20 hz timeline. for each target timestamp, it used nearest-neighbor sampling for image frames and linear interpolation for robot state.

iv. making act architectural changes

act is a transformer-based imitation-learning policy that predicts chunks of future actions rather than generating one action at a time.[B] it also uses a conditional variational autoencoder (cvae)[C] to model variation across demos.

the original act pipeline was designed for a different robot setup, so we adapted act++ to work with spot. related work on mobile manipulation with act-style policies similarly adapts the method beyond a fixed tabletop bimanual setup.[D]

the changes were:

• accept arm_camera and zed_camera observations
• resize images to 480 × 640 for training
• represent spot using a fixed 11-dimensional state and action space
• predict coordinated arm, gripper, and body commands
• train with chunks of 8 future actions

during training, the model receives the current robot state, the 2 camera observations, and a sequence of future actions from a demonstration. the cvae encodes the demonstrated action sequence into the default 32-dimensional act latent representation, and the policy learns to reconstruct the original actions from the observations and that latent variable.[B][C]

during deployment of the grasp policy, the model received the latest robot state and camera observations and predicted a chunk of future actions. the runtime re-queried the policy at 20 hz with fresh observations rather than executing an entire predicted chunk open-loop.

predicting actions in chunks helps the policy produce smoother, more temporally consistent behavior than predicting each command independently. this is one of the main practical advantages of act over single-step behavior cloning.[B]

v. deploying the policy

the final step was connecting the learned grasp policy to the physical spot robot.

deployment was split across two computers. the gpu ran camera streaming and policy inference. the mac, connected to spot, ran a flask server around the spot sdk and an execution client that sent predicted actions to the robot while streaming the latest 11-dimensional state back to the gpu over tcp.[A]

the video below shows the final grasp rollout. with roughly 50 demonstrations, it confirmed that our data, training, and deployment pipeline worked on the physical robot. the motion repeats because the grasp-only policy had no learned stop or post-grasp transition, so the 20 hz control loop kept producing similar action chunks.

i left boston shortly afterward and lost access to spot before i could collect more data or train the full sequence. more varied demonstrations would likely reduce repetition and improve robustness.

next steps

in a follow-up project, i will show how we successfully extended this work to demonstrate the broader behavior. more on that project soon!

acknowledgments

this project was a collaboration with Lucy Cai and Aditya Agarwal.