Nvidia wants to create universal AI agents for all worlds with NitroGen
Nvidia introduced NitroGen, an open vision action model trained on extensive gameplay data to create universal AI agents.Summary
Nvidia has released NitroGen, a new open vision action model designed to function as universal AI agents across diverse virtual environments. This model was trained using 40,000 hours of gameplay videos sourced from YouTube and Twitch, where player inputs were extracted using template matching and a fine-tuned SegFormer model on visible controller overlays. Building upon Nvidia's GR00T N1.5 robotics model, NitroGen is the first to show that robotics foundation models can operate universally across virtual worlds with varying physics and visual styles, handling genres like action RPGs and platformers. When tested on unfamiliar games, it outperformed models trained from scratch by up to 52 percent. The research team, involving members from Nvidia, Stanford, and Caltech, has made the dataset, model weights, paper, and code publicly available.
(Source：The Decoder)