Holding gauze and grasping needles, NVIDIA cooperates with many universities to develop surgical robots

Editor | X

NVIDIA is working with academic researchers to study surgical robots.

NVIDIA teamed up with researchers from the University of Toronto, UC Berkeley, ETH Zurich, and Georgia Institute of Technology to develop ORBIT-Surgical, a simulation framework for training robots that improves the skills of technical teams while reducing surgical Cognitive load on physicians. ORBIT-Surgical is an artificial intelligence-based simulation framework that achieves highly realistic surgical simulation through a virtual surgical environment and intelligent coaching system. Doctors can interact with this system to simulate the various situations and complexities of real surgeries. This simulation technology can not only help with training

"Inspired by training courses in laparoscopic surgery (also known as minimally invasive surgery), it supports more than a dozen operations, such as grabbing small objects like needles and moving them from one The physics-based framework is built using NVIDIA Isaac Sim, a tool for designing, training and testing AI-based robots. simulation platform.

Researchers trained reinforcement learning and imitation learning algorithms on NVIDIA GPUs and used NVIDIA Omniverse, a platform for developing and deploying advanced 3D applications and pipelines based on the Universal Scene Description (OpenUSD), To achieve photorealistic rendering.

ORBIT-Surgical will be presented at IEEE International Conference on Robotics and Automation (ICRA) 2024.

GitHub open source code:

https://orbit-surgical.github.io/

Paper address:

https ://arxiv.org/abs/2404.16027

In the video below, the ORBIT-Surgical research team demonstrates how a digital twin trained in simulation is transferred to a physical robot in a laboratory environment.

ORBIT-Surgical is a modular framework for robot learning built on Isaac Orbit. Orbit supports various reinforcement learning and imitation learning libraries, where artificial intelligence agents are trained to imitate real expert demonstrations. By using Orbit, experts can design and optimize surgical procedures and translate them into executable sequences of robotic operations. The core concept of Isaac Orbit is to transform expert knowledge into robot behaviors that can be executed automatically. The system contains the following components: 1. Data collection: Use experts to perform surgeries and record their operations and decisions to build a training data set. 2. Data preprocessing: Preprocess and specialize the collected data. This surgical framework enables developers to train robots such as the da Vinci Research Kit (dVRK) and is used on NVIDIA RTX Reinforcement learning and imitation learning frameworks running on GPUs to manipulate rigid and soft objects.

ORBIT-Surgical introduces more than a dozen baseline tasks for surgical training, including single-handed tasks such as picking up a piece of gauze, inserting a shunt into a blood vessel, or raising a suture needle to a specific location. It also includes bimanual tasks such as passing a needle from one arm to the other, threading a threaded needle through a looped rod, and reaching both arms into specific locations while avoiding obstacles.

One of ORBIT-Surgical’s benchmark tests is the insertion of a shunt (real-world robot on the left, simulation on the right .)

By developing a surgical simulator that leverages GPU acceleration and parallelization, the team was able to increase the robot's learning speed by an order of magnitude over existing surgical frameworks. They found that after training, the robotic digital twin could complete tasks such as inserting a shunt and lifting a suture needle in two hours on a single NVIDIA RTX GPU.

With the visual realism enabled by Omniverse rendering, ORBIT-Surgical also allows researchers to generate high-fidelity synthetic data, which can help train AI models to perform perception tasks, such as segmenting surgeries in real videos captured in the operating room tool.

The team’s proof-of-concept shows that combining simulation and real-world data significantly improves the accuracy of artificial intelligence models in segmenting surgical needles from images, helping to reduce the need for large, expensive training of such models. requirements for realistic data sets.

Reference content: https://blogs.nvidia.com/blog/orbit-surgical-robotics-research-icra/

The above is the detailed content of Holding gauze and grasping needles, NVIDIA cooperates with many universities to develop surgical robots. For more information, please follow other related articles on the PHP Chinese website!