ULTRA Lets Humanoid Robots Perform Whole-Body Tasks From Sparse Goals
For humans, carrying a box from the floor to a target location is a routine task. For a humanoid robot, however, the same action requires tightly coordinated control of the entire body.
The robot must approach the object with an appropriate stance, squat while maintaining balance, establish stable contact with the box, stand up without losing its grip, and then walk toward the destination while coordinating its legs, torso, and arms. Small errors in any of these stages can cause the object to fall or destabilize the robot.
Traditional humanoid control pipelines often depend on pre-generated reference trajectories that specify motion frame by frame. This works when detailed motion data is available, but it leaves an important question unanswered: what should the robot do when it receives only a high-level goal?
ULTRA (Unified Multimodal Control for Autonomous Humanoid Whole-Body Loco-Manipulation) addresses this problem by combining precise reference-motion tracking with autonomous, goal-driven control in a single policy framework.
Evaluated on the Unitree G1 humanoid robot, the work has been accepted to IROS 2026 and selected as a candidate for the IROS 2026 Mobile Manipulation Paper Award.
🤖 One Controller for Dense and Sparse Commands #
ULTRA uses a shared set of policy parameters to support two different control regimes.
Motion tracking with dense references #
In dense-control mode, the system receives detailed frame-by-frame reference trajectories describing the desired motion of the robot and manipulated object.
The controller’s task is to reproduce those trajectories as accurately as possible while maintaining physically valid whole-body behavior.
This mode is useful when demonstrations or motion-capture trajectories provide an explicit description of how the task should be performed.
Goal-driven control with sparse references #
Sparse-control mode removes the requirement for an explicit intermediate trajectory.
Instead of specifying every step, the task can provide a high-level goal such as the desired position and orientation of an object. The controller must then generate the intermediate walking, bending, reaching, grasping, and whole-body coordination required to reach that state.
This distinction is important: ULTRA’s autonomous mode does not mean that the robot operates without perception or task information. Rather, it means the controller can generate the required full-body motion from task goals and available sensory inputs instead of a pre-scripted sequence of motion frames.
The same policy can also operate with substantially different sensing conditions, ranging from highly accurate external motion-capture measurements to noisy onboard egocentric depth perception.
🧠 Physics-Driven Neural Retargeting #
Human motion-capture data provides valuable examples of whole-body coordination, but human movement cannot simply be copied directly onto a humanoid robot.
Differences in body proportions, joint limits, actuator capabilities, center of mass, and contact mechanics can cause direct trajectory mapping to fail. A motion that is physically valid for a human may result in unreachable poses, foot slipping, unstable contacts, or excessive joint constraints on a robot.
ULTRA addresses this mismatch with physics-driven neural retargeting.
Reinforcement learning for embodiment-aware retargeting #
The retargeting policy is trained through reinforcement learning in simulation.
Instead of treating human motion as a trajectory that must be copied exactly, the policy learns to preserve the underlying intent of the demonstration while adapting the movement to the robot’s physical embodiment.
The resulting behavior explicitly accounts for factors such as:
- Joint and kinematic constraints.
- Balance and stability.
- Robot-specific body proportions.
- Contact dynamics.
- Reachability.
- Whole-body coordination.
Once trained, the retargeting policy can generalize to previously unseen motions and support data augmentation across different object dimensions and motion ranges without requiring a separate retraining process for every trajectory.
🎯 Two-Stage Learning From Tracking to Goal Fulfillment #
ULTRA uses a two-stage training strategy to transform reference-based whole-body skills into a deployable controller capable of operating under sparse and partially observable conditions.
Teacher-student distillation #
The first stage uses a Teacher Policy trained in simulation with access to privileged state information and dense reference trajectories.
This teacher learns the underlying whole-body coordination required for motion tracking, contact handling, and balance.
Its capabilities are then distilled into a deployable Student Policy.
A central component of this process is the Input Availability Mask. During real-world operation, some inputs available during training may be unavailable. For example, the robot may not have external motion-capture references when operating outside a laboratory.
The mask explicitly indicates which modalities are currently available to the policy, allowing the same network to adapt its behavior based on the information it can actually observe.
Reinforcement learning fine-tuning #
Imitation alone can struggle when the robot encounters states that differ from the training demonstrations.
Examples include:
- Unexpected initial configurations.
- Execution drift.
- Perturbed object locations.
- Previously unseen target positions.
- Accumulated control errors.
ULTRA therefore applies reinforcement learning fine-tuning in simulation.
Rather than optimizing exclusively for frame-by-frame imitation accuracy, the fine-tuned policy is trained to correct deviations and complete the underlying task objective.
This changes the controller from a system primarily focused on reproducing a trajectory into one capable of closed-loop task execution.
👁️ Egocentric Depth Perception for Real-World Control #
A humanoid operating outside a motion-capture laboratory cannot depend on external tracking infrastructure.
ULTRA therefore incorporates egocentric depth perception, allowing the robot to estimate relevant environmental information using onboard sensors.
Point-cloud-based object perception #
Depth-camera images are processed to extract object point clouds containing information about the object’s geometry and spatial position.
The controller combines these visual features with proprioceptive information, including:
- Joint positions.
- Robot orientation from the IMU.
- Other internal state information.
- High-level task goals.
This multimodal representation gives the controller both an understanding of the external object and the robot’s own physical configuration.
Closed-loop control from onboard perception #
The resulting system allows the Unitree G1 to generate whole-body actions using egocentric perception rather than relying exclusively on externally provided reference trajectories.
According to the reported experiments, reinforcement learning fine-tuning substantially improves task success when the controller operates using onboard point-cloud observations.
This provides an important bridge between laboratory demonstrations and autonomous robot operation in less controlled environments.
🦾 Toward Practical Humanoid Loco-Manipulation #
ULTRA combines three capabilities that are often handled separately in humanoid robotics:
- Physics-aware motion retargeting converts human demonstrations into physically feasible robot movements.
- Unified multimodal control supports both dense trajectory tracking and sparse goal-driven behavior.
- Egocentric perception allows the controller to operate using onboard sensory information in real-world environments.
Together, these components move toward a control paradigm in which a user specifies what should happen, while the robot determines how its body should move to accomplish it.
For humanoid loco-manipulation, this distinction is particularly important. A high-level command such as moving an object to a target location does not uniquely determine the robot’s intermediate poses. The controller must continuously reason through balance, contact, reachability, locomotion, and manipulation while responding to the state of the environment.
ULTRA’s approach attempts to unify those requirements within a single learned control framework rather than relying entirely on manually designed intermediate trajectories.
📄 Paper and Project Information #
Paper: ULTRA: Unified Multimodal Control for Autonomous Humanoid Whole-Body Loco-Manipulation
ArXiv: https://arxiv.org/abs/2603.03279
Project Page: https://ultra-humanoid.github.io/
Authors #
Xialin He is a PhD student in Computer Science at the University of Illinois Urbana-Champaign (UIUC) and an alumnus of Shanghai Jiao Tong University’s ACM Class. His research focuses on robot learning, embodied AI, humanoid whole-body control, autonomous loco-manipulation, and sim-to-real transfer.
Sirui Xu is a PhD student in Computer Science at UIUC and an alumnus of Peking University. His research focuses on embodied AI and physical simulation.