Surgical procedures with augmented reality data glasses – The app combines MRI images with the real surgical situation

Medicine is advancing inexorably, and a particularly exciting field is the integration of augmented reality (AR) into the operating room. This is not about science fiction, but about a reality that is within reach and has the potential to make surgical procedures safer, more precise and gentler for patients. Neurosurgery in particular, a field that requires maximum precision due to the complexity of the brain, benefits significantly from these technological advances.

Medicine is advancing inexorably, and a particularly exciting field is the integration of augmented reality (AR) into the operating room. This is not about science fiction, but about a reality that is within reach and has the potential to make surgical procedures safer, more precise and gentler for patients. Neurosurgery in particular, a field that requires maximum precision due to the complexity of the brain, benefits significantly from these technological advances.

A promising approach is the use of data glasses that overlay preoperative image data, such as from magnetic resonance imaging (MRI), with the real surgical field in real time. This technology allows the surgeon to “see through the patient’s body” and thus gain a more precise insight into the region to be operated on. In contrast to conventional navigation systems, which are often bulky and expensive, AR data glasses open up a new dimension of spatial orientation in the operating room. This is particularly relevant in procedures where access to the surgical field is limited, such as brain tumor surgeries that are performed via the nose. The minimally invasive method is supported by improved visualization through AR glasses, potentially resulting in less tissue damage, shorter recovery times and reduced complications.

The app as the key to precise navigation

The heart of this innovative technology is a specially developed app that synchronizes the MRI image created before surgery with the real-time view of the surgical field. This development is the result of years of research and development work in a collaboration between the LEGEND research group at the Clinic and Polyclinic for Neurosurgery at Leipzig University Hospital (UKL) and the Fraunhofer Institute for Machine Tools and Forming Technology IWU in Zittau. The app acts as a “GPS system” for the surgeon, not only showing him the position of the surgical target, but also the optimal, i.e. most gentle, access route. This form of navigation aid is a significant improvement over previous methods that relied on static image data and the surgeon's spatial imagination.

Another significant advantage of this new technology is the ability to integrate surgical instruments into the navigation system. By precisely recording the position of the instruments in real time and displaying them in the data glasses, the surgeon can guide them even more precisely and safely. This real-time visualization, enabled by the app, minimizes the risk of errors and allows the surgeon to plan and implement his movements precisely. In addition, important additional information, such as the distance to the target area, is displayed directly in the surgeon's field of vision, optimizing the flow of information and enabling a quick and safe reaction. This not only increases the safety of the procedure, but also potentially shortens the operation time, reducing the burden on the patient and the medical staff.

Precision in Real Time: A Quantum Leap in Neurosurgery

One of the research group's most notable achievements is the almost instantaneous operational readiness of the system. “Our team has achieved a worldwide first with fully automated registration for neurosurgical spatial computing navigation,” says PD Dr. habil. Ronny Grunert, research associate at Fraunhofer IWU and head of the “Legend” research group at UKL, reports enthusiastically. “The calibration and registration is completed within a second and navigation for real-time detection of the instrument position is ready to start. The system developed is very intuitive to use and comes very close to using GPS assistance.” This speed and intuition are crucial in order not to disrupt the workflow in the operating room and to offer surgeons an intuitive and reliable navigation aid.

Another important detail is the user-friendliness of the system. The user interface was developed by medical professionals for medical professionals and is limited to displaying essential information, thereby minimizing operating errors. An example of the clear display logic is a green crosshair that indicates the position of the instrument tip and is perfectly integrated into the MRI image displayed in the smart glasses. This clear and concise presentation is a decisive factor for the acceptance and success of the system in the operating room, as the surgeons can concentrate on what is important - the successful execution of the operation.

Economics and Accessibility: A Revolution for All

In addition to technical innovation, another central aspect of this project is the economic viability and the associated accessibility of the technology. While conventional navigation systems in neurosurgery suitable for use in clinics often cost several hundred thousand euros, the development group relies on standard data glasses whose prices are in the consumer sector. “These glasses cost a fraction of the cost of computer-aided navigation systems for neurosurgery,” explains Grunert. This cost reduction is a critical step in making the technology accessible not only to well-funded health systems and institutions, but also to countries and regions with limited resources. This democratization of technology means more patients can benefit from the benefits of precise and gentle neurosurgical treatment.

The development of the handpiece that holds the instruments and enables their precise positioning is also an important aspect of the project. Special markers were developed at Fraunhofer IWU, whose geometries and patterns are recognized by the data glasses. These markers can come in different shapes such as spheres, cuboids or other bodies and are used to record the position of the instruments in three-dimensional space. The plastic handpieces are manufactured in Zittau and Leipzig using 3D printing, which enables a high level of flexibility and adaptability to the needs of surgeons.

The Path to Clinical Practice: An Outlook

The first pilot course for training on the anatomical model took place at the UKL in autumn 2024. This step is an important milestone on the way to clinical application of the technology. In the next step, the team focuses on completing the prototype, which then has to go through the approval process in accordance with the Medical Device Regulation for the European market or in accordance with the regulations of the American Food and Drug Administration (FDA) for the USA. The goal is to be able to use the system on patients in about two years.

The development of AR-supported navigation systems for neurosurgery is not only a technological advance, but also a paradigm shift in medical practice. The integration of real-time image data, precise instrument guidance and economical solutions has the potential to fundamentally transform neurosurgical treatment, making it safer, more precise and more accessible. The vision of a future in which innovative technologies such as AR data glasses are standard in the operating room gets closer with every step of this development. This not only opens up new perspectives for doctors and surgeons, but above all for patients, who can benefit from improved treatment quality and a more gentle procedure. The future of neurosurgery has taken an exciting direction with this technology, where precision and innovation go hand in hand.

Suitable for:

• Germany is the leader here: 'Iteration Metaverse' solutions with XR technologies such as extended, augmented, virtual and mixed reality
• Augmented and virtual reality – into a new dimension