NVIDIA Developers Combine Frontier AI Agents With Omniverse to Accelerate Robotics and Physical Simulation

Turning an initial simulation concept into a fully functional application typically requires assembling 3D assets, calibrating physics, handling rendering pipelines, and validating real-world sensor behavior. NVIDIA has demonstrated how developers are now streamlining this process by pairing frontier AI models—such as GPT-6 Astra and Claude Fable 5—with NVIDIA Omniverse libraries to build, test, and refine physical simulations through natural-language prompts.

By combining AI agents with GPU-accelerated Omniverse tools, engineers guide the generation of application code, physics configurations, and scene updates while iteratively evaluating the results.

Humanoid Robotics: Warehouse Tasks and Hurdle Trials

Interactive environments are essential for evaluating robot behaviors before deploying physical hardware. Frank DeLise, an Omniverse product manager at NVIDIA, used Astra to convert a SimReady warehouse and humanoid robot into an interactive simulator featuring both first- and third-person viewpoints. Under DeLise’s direction, the agent integrated several Omniverse libraries, including ovphysx for physics, ovstage for scene updates, ovrtx for rendering, and ovui for the user interface, alongside simready-foundation to build the physical scene and generate animation code.

In another robotics project dubbed Robo Olympics, Tae Kim, head of NVIDIA Omniverse engineering and product, guided Astra using sports video footage and natural-language prompts to test simulated Unitree G1 humanoid robots. The agent configured controllers evaluated through physics trials running on the Newton Physics Engine, accelerated with the open-source NVIDIA Warp framework, and rendered via ovrtx. In one hurdle experiment, the simulated humanoid successfully cleared a single hurdle in 64 out of 100 trials, providing direct kinematic and timing data to refine robot control policies.

Autonomous Driving and Sensor-Driven Digital Twins

Frontier agents are also managing end-to-end simulation pipelines for autonomous mobility. Doyub Kim, a simulation technology manager at NVIDIA, directed Astra to construct "Zero to Alpamayo," a simulation environment modeled after Market Street in San Francisco. Astra mapped out the workflow in stages, connecting asset generation, traffic simulation, Omniverse RTX sensor outputs, and Alpamayo driving models. In a complementary experiment, Kim used Cosmos3-Nano to alter weather and lighting parameters across recorded simulations, testing how the autonomous model reacted under changing environmental conditions.

To bridge the gap between simulation and reality, Ashley Reid, an RTX sensor validation specialist at NVIDIA, deployed Astra and Claude Fable 5 agents to evaluate discrepancies between simulated sensor data and real-world inputs. Over an iterative three-day workflow, the agents compared raw LiDAR and ovrtx camera data against recorded physical metrics to build two OpenUSD digital twins from scratch and refine two existing scenes, adjusting missing geometry and material properties based on quantitative sensor thresholds.

Tool Design, Room Capture, and Orbital Telemetry

The integration between AI agents and Omniverse extends into computer-aided design (CAD) and spatial reconstruction:

• Robotic Disassembly: Jens Jebens, senior product manager for OpenUSD, instructed Astra to evaluate suspension disassembly on a vehicle model inside PTC Onshape and NVIDIA Isaac Sim. Astra analyzed spatial constraints and designed a custom wrench that enabled the simulated robot to reach and remove a suspension component bolt.
• Spatial Room Reconstruction: Chirag Majithia of the Isaac engineering applications team directed Astra to convert stereo camera footage into editable OpenUSD environments. The pipeline utilized PyCuSFM, FoundationStereo, and nvblox, using USD Content Agents and Isaac Sim to configure collision dynamics and interactive physics for doors and drawers.
• Browser-Based Space Telemetry: Nic Johns, an engineering director at NVIDIA, prompted Astra to assemble NASA assets into an interactive, browser-streamed OpenUSD model of the International Space Station linked to live telemetry. Johns created the application in a single prompt before issuing a follow-up command to rotate the scene toward Earth's daytime side, utilizing Blender for asset prep alongside Omniverse ovstage, ovrtx, and ovstream libraries.

These projects highlight an evolving paradigm in physical AI development, where natural-language orchestration by frontier agents bridges the gap between disparate 3D simulation tools, CAD systems, and physics frameworks.