A division of Google, DeepMind releases RT2 model, a first-of-its-kind vision-language-action (VLA) model that can enable robots to do novel tasks without specific training.

RT-2 uses text and images from the web to understand distinct real-world concepts and transfer that knowledge into generalised instructions for robotic operations, similar to how language models learn general ideas and concepts from web-scale data.

When enhanced, this technology could lead to context-aware, flexible robots capable of doing multiple tasks in a variety of contexts and environments with significantly less training than is currently required.

What makes DeepMind’s RT-2 special?
DeepMind unveiled RT-1 in 2022, a multitask model trained on 130,000 examples that allowed Everyday Robots to complete 700-plus activities with a 97% success rate. The company has now trained the model’s successor, RT-2, using the robotic demonstration data from RT-1 and web datasets.

The Potential use of DeepMind’s RT2
When matured, vision-language-action models like RT-2 can lead to context-aware robots that can reason, problem-solve, and interpret information in order to perform a variety of real-world activities based on the scenario at hand.

Instead of robots performing the same repetitive operations in a warehouse, organisations may see machines that handle each thing differently, taking into account criteria such as the object’s nature, weight, fragility, and other factors.

RT-2 is not a robot in and of itself; it is a model for controlling robots more efficiently than ever before. Using visual and verbal input, an RT-2-enabled robot may do activities ranging in complexity, such as organising files alphabetically by reading the labels on the documents, sorting them, and then putting them away in the relevant places.

It could also handle more difficult tasks. For example, if you said, “I need to mail this package, but I’m out of stamps,” RT-2 may figure out what has to be done first, such as finding a Post Office or merchant nearby that sells stamps, then take the package and manage the logistics from there.

DeepMind Releases RT2 – Learn more about it

SUGGESTED
Assigning multiple characters to single character
Assigning multiple characters to single character
Importing character from mixamo to unreal
How to create 3d animations?
A Beginner’s Guide to 3D Animation