NVIDIA and Hugging Face Detail Agent-Driven Robot Simulations Using SimReady OpenUSD Assets

A new technical walkthrough published on Hugging Face by NVIDIA engineers Dillon Bailey, Pomi Lee, and Rishabh Chadha outlines an end-to-end pipeline for building physical robot simulations using validated OpenUSD assets and AI agents. The workflow demonstrates a beverage can pick-and-place task inside NVIDIA Isaac Sim 6.2, assembling a Universal Robots UR17G arm and a Robotiq 2F-140 gripper to manipulate an object using autonomous agent controls.

The project highlights the role of SimReady assets—standardized 3D models configured with simulation-critical parameters such as mass, collision meshes, kinematics, physical joints, and joint drives.

Standardizing Simulation Assets with SimReady Foundation

In robotics, visual 3D fidelity does not guarantee simulated real-world behavior. A model may look accurate when static but clip through geometry, lack proper mass distributions, or suffer from misconfigured joint limits once physics engines engage.

The SimReady Foundation framework resolves this by defining three structured levels for OpenUSD assets:

• Profiles: Feature groups tailored to specific use cases, such as a full robot body or end-effector.
• Features: Specific functional capabilities, including controllable drive joints.
• Requirements: Rigorous configuration checks, such as joint drive settings.

In the demonstration, assets downloaded from the open SimReady Central library on Hugging Face carry profile-validation evidence. The UR17G arm passed checks under the Robot-Body 4.0.0 profile, while the Robotiq 2F-140 gripper was validated against the Robot-Gripper 3.0.0 profile. Both validation logs confirmed passing PhysX driven-joint tests, ensuring that drive configurations meet baseline simulator requirements before scene composition begins.

AI Agents Orchestrating Isaac Sim

Rather than manually scripting every step of the robot's mechanical assembly and motion planning, the team utilized a frontier AI model—specifically GPT-6 Sol—interfacing directly with Isaac Sim.

To enable the interaction, Isaac Sim was launched with a remote Python server enabled via the command: ./isaac-sim.sh --enable isaacsim.code_editor.python_server

Equipped with Isaac Sim Agent Skills, the AI agent executed Python commands against the simulator to inspect joint limits and assemble the UR17G arm and Robotiq gripper into a single controllable articulation. For assembly, the model leveraged the isaac-sim-remote and usd-articulation skills.

Once the scene contained the assembled manipulator and the beverage can, the agent was prompted to generate pick-and-place controls using contact-based grasping. During execution, the agent called upon isaac-sim-remote, manipulaiton-ik, and physics-simulation skills to calculate inverse kinematics and command the trajectory.

Task Execution and Simulation Parameters

The pick-and-place sequence required the arm to approach the beverage can, execute a horizontal grasp, lift the object, transfer it across the workspace, and release it onto a designated target marker.

The authors noted several controlled parameters and simulation boundaries:

• Grasp Mechanics: The gripper utilized a horizontal grasp capped at a 2 N·m drive-effort limit, selected through simulation testing rather than manufacturer hardware fingertip calibration.
• Motion Planning: Contact and friction held the can as the arm tracked fixed waypoints without active replanning or vision inputs.
• Physics Assumptions: The can was modeled without liquid dynamics or structural deformation, omitting real-world fluid motion and material crushing tests.

The developers emphasized that while SimReady profile validation establishes a consistent baseline for physics assets, developers must still inspect generated code and verify contact behavior to ensure models do not drop payloads or encounter unexpected collision errors during complex workflows.