The VR training system allows human teachers to see what the robot is seeing live, in 3D, from its sensors. The teacher can instruct the robot and annotate the 3D scene, for instance adding a note on how to grasp a handle. This allows human trainers to teach robots arbitrary tasks with a variety of objects, instead of specific tasks like they would perform in a more controlled setting.

TRI’s system allows the robots to be more flexible. They don’t require a complete map of the house. Instead, they only need to understand the objects that are relevant to a behavior being performed. And thanks to fleet learning, once one robot is trained in a task, they all learn it.

The system isn’t perfect yet. In its video, TRI reminds viewers that it creates research prototypes, not product concepts. Still, the VR-based system could change the way robots learn and how we’re able to use them in different settings.