Surgical procedures with augmented reality data glasses – The app combines MRI images with the real surgical situation

Revolution in neurosurgery: Augmented reality data glasses as a navigation system in the operating room

Medicine is advancing relentlessly, and one particularly exciting area is the integration of augmented reality (AR) into the operating room. This is not science fiction, but a reality that is becoming increasingly tangible and has the potential to make surgical procedures safer, more precise, and less invasive for patients. Neurosurgery, in particular, a field that demands the utmost precision due to the complexity of the brain, benefits significantly from these technological advances.

A promising approach is the use of smart glasses that overlay preoperative image data, such as from magnetic resonance imaging (MRI), onto the actual surgical field in real time. This technology allows the surgeon to virtually "see through" the patient's body, thus gaining a more precise view of the region to be operated on. Unlike conventional navigation systems, which are often bulky and expensive, AR smart glasses open up a new dimension of spatial orientation in the operating room. This is particularly relevant for procedures where access to the surgical field is limited, such as brain tumor removal via the nose. The minimally invasive technique is supported by the improved visualization provided by the AR glasses, potentially leading to less tissue damage, shorter recovery times, and reduced complications.

The app as the key to precise navigation

At the heart of this innovative technology is a specially developed app that synchronizes the MRI image acquired before the operation with a real-time view of the surgical field. This development is the result of years of research and development in a collaboration between the LEGEND research group at the Department of Neurosurgery of Leipzig University Hospital (UKL) and the Fraunhofer Institute for Machine Tools and Forming Technology IWU in Zittau. The app essentially functions as a "GPS system" for the surgeon, showing not only the position of the surgical target but also the optimal, i.e., least invasive, access route. This form of navigation aid represents a significant improvement over previous methods that relied on static image data and the surgeon's spatial reasoning.

Another significant advantage of this new technology is the ability to integrate surgical instruments into the navigation system. By precisely tracking the instruments' positions in real time and displaying them in the smart glasses, the surgeon can guide them with even greater precision and safety. This real-time display, enabled by the app, minimizes the risk of errors and allows the surgeon to plan and execute movements with pinpoint accuracy. Furthermore, crucial additional information, such as the distance to the target area, is displayed directly in the surgeon's field of vision, optimizing information flow and enabling a rapid and safe response. This not only increases the safety of the procedure but also potentially reduces operating time, thus relieving the burden on both the patient and the medical staff.

Real-time precision: A quantum leap in neurosurgery

One of the research group's most remarkable achievements is the system's virtually instantaneous operational readiness. "Our team has achieved a worldwide first with fully automated registration for neurosurgical spatial computing navigation," reports PD Dr. habil. Ronny Grunert, research associate at Fraunhofer IWU and head of the "Legend" research group at UKL, enthusiastically. "Calibration and registration are completed within a second, and navigation for real-time instrument position detection is ready to go. The developed system is very intuitive to use and closely resembles GPS assistance." This speed and intuitiveness are crucial for maintaining workflow in the operating room and providing surgeons with an intuitive and reliable navigation aid.

Another important detail is the system's user-friendliness. The user interface was developed by physicians for physicians and is limited to displaying essential information, thus minimizing operating errors. An example of the clear display logic is a green crosshair that indicates the position of the instrument tip and is perfectly integrated into the MRI image displayed in the smart glasses. This clear and concise presentation is a crucial factor for the system's acceptance and success in the operating room, as it allows surgeons to focus on what matters most – the successful execution of the operation.

Efficiency and accessibility: A revolution for all

In addition to the technical innovation, another key aspect of this project is the economic viability and the resulting accessibility of the technology. While conventional navigation systems suitable for use in hospitals often cost several hundred thousand euros in neurosurgery, the development group is focusing on standard smart glasses priced at consumer levels. “These glasses cost a fraction of computer-assisted navigation systems for neurosurgery,” explains Grunert. This cost reduction is a crucial step in making the technology accessible not only to well-funded healthcare systems and institutions, but also to countries and regions with limited resources. This democratization of the technology will allow more patients to benefit from precise and minimally invasive neurosurgical treatment.

The development of the handpiece, which holds the instruments and enables their precise positioning, is also a key aspect of the project. At Fraunhofer IWU, special markers were developed whose geometries and patterns are recognized by the smart glasses. These markers can take various forms, such as spheres, cuboids, or other shapes, and serve to determine the position of the instruments in three-dimensional space. The plastic handpieces are manufactured using 3D printing in Zittau and Leipzig, allowing for a high degree of flexibility and adaptability to the needs of surgeons.

The path to clinical practice: An outlook

The first pilot course for training on the anatomical model took place at Leipzig University Hospital (UKL) in autumn 2024. This step is an important milestone on the path to clinical application of the technology. The team's next focus is on completing the prototype, which will then have to undergo the approval process according to the Medical Device Regulation (MDR) for the European market and the regulations of the American Food and Drug Administration (FDA) for the USA. The goal is to be able to use the system on patients in approximately two years.

The development of AR-based navigation systems for neurosurgery is not only a technological advancement but also a paradigm shift in medical practice. The integration of real-time image data, precise instrument guidance, and cost-effective solutions has the potential to fundamentally transform neurosurgical treatment, making it safer, more precise, and more accessible. With each step of this development, the vision of a future where innovative technologies like AR smart glasses are standard in the operating room is drawing closer. This opens up new perspectives not only for physicians and surgeons but, above all, for patients, who can benefit from improved treatment quality and less invasive procedures. With this technology, the future of neurosurgery has taken an exciting direction, where precision and innovation go hand in hand.

Suitable for:

• Germany is a leader here: 'Iteration Metaverse' solutions with XR technologies such as Extended, Augmented, Virtual and Mixed Reality
• Augmented and Virtual Reality – into a new dimension