Editor’s note: While Nissan Elimelech founded Augmedics and invented its foundational technology, he is no longer affiliated with or represents the company.
In 2014, Nissan Elimelech founded Augmedics, the company behind an augmented reality surgical navigation platform for spine procedures. He is credited with developing the xvision Spine System, which uses a transparent near-eye display headset to project a 3D visualization of a patient’s spine – generated from pre-operative CT scans – directly into a surgeon’s retina.
Elimelech spoke with MobiHealthNews about the development of the technology, its impact on surgeons and patients, and his latest project: medically focused augmented reality glasses that enable companies to quickly build AR applications.
MobiHealthNews: Can you tell our readers about Augmedics’ foundational technology?
Nissan Elimelech: Augmedics is a pioneer in augmented reality surgical navigation systems. We started the company in 2014, and Augmedics basically was the first company ever that received FDA clearance for an augmented reality surgical navigation system.
The company was formed to address a need for an accurate surgical navigation system for spine surgery, but instead of projecting the navigation data on a screen, on a remote screen, where surgeons will have to take their heads up and look at a screen to navigate, what we have done at Augmedics is developed a built-in surgical navigation system inside the augmented reality glasses that the surgeon can put on and see the navigation data directly projected to their eyes in 3D and in real time as if they had X-ray vision.
MHN: So, you actually take 3D images of an individual patient, and that is what the surgeons work from?
Elimelech: Well, with the xvision glasses, we have designed a system for medical procedures only, to do a real surgery with that. So, the system is for spine operations, that they can take the pre-op imaging of the patient, the CT scans or the intraoperative CT scans of a patient, and just register it to the real patient lying down on the surgical table to see the anatomy, to see the spine of the patient directly projected to their eyes while they are upright.
MHN: So if they are doing an epidural or something like that, they can use the scan to see where to put the needle and it is specific to that patient?
Elimelech: Yes, absolutely. We are using transparent lenses. So, unlike VR headsets, which actually occlude the reality and just project like virtual reality, we are using fully transparent lenses that you can still see the reality and then we are projecting the anatomy, a 3D match of the anatomy of the patient, directly onto the retina of the user, so users can see both the real patient and also what is inside that patient.
The analogy will be as if they are like Superman that can see through the patient’s body and see the anatomy of the spine and put the epidural or screws, if they’re operating, and any other instruments or implant what they want to put in.
MHN: How was adoption of the technology when released? Were surgeons open to it?
Elimelech: Yes, well, the introduction of the technology was in 2019 and, like I said, I mean, Augmedics was a pioneer in this field.
Surgeons in the U.S. accepted this technology relatively easily, and they liked to see new technologies. We marketed it very aggressively, working with PR in hospitals, and did a lot of collaborations with a lot of companies. So the adoption was very, very good – much more accepted than what I anticipated.
MHN: How has this technology changed the ability of surgeons to perform operations or for patients to not have as many complications during surgery?
Elimelech: The main purpose of introducing a system like that to a physician is to reduce the risk of having any issues with inaccuracy of placing a screw in the spine, or any other complications that you can think of during the surgery.
So Augmedics basically developed a highly accurate surgical navigation system that tracked the instrument and the implants that surgeons are putting in the spine, and that alone is a great benefit for surgeons that want to improve the surgical outcome for a patient.
The difference between what we were doing to a conventional navigation system was that, unlike certain traditional systems, we were projecting the navigation data directly to the surgeon’s retina.
So, in fact, we are not only providing the accuracy for a surgeon to place a screw in spine surgery, but we also do it in a very comfortable and intuitive way, so they do not have to look up to a remote screen to see their instruments, but rather they can just look straight at their patients, keep their hand and eye coordination and place the screws or any other implants that they want to use during the surgery accurately and conveniently.
MHN: What is next?
Elimelech: Well, after the company grew to 100 employees, 50 on the commercial team and an additional 50 in Israel for R&D, I passed the CEO position to a U.S. CEO, who took over the company and focused on growing the company and expanding it. I left Augmedics in 2022, and, since then, I’ve been searching what I want to do next.
The thing is that, like Augmedics, I realized that there are many, many companies that wanted to switch to augmented reality or to embrace this technology. And I’m talking about dozens of other companies that actually looked into augmented reality, maybe even developed products in augmented reality for their systems and mainly they used, unlike Augmedics, general-purpose augmented reality glasses like the HoloLens and Magic Leap. I’m saying unlike Augmedics because we developed proprietary glasses for spine navigation.
But many other companies have been relying on a general platform. Those consumer products were not fit for medical purposes, and so there is a lot of challenges, I would say, developing medical products on a consumer product.
But not only that, one or two years ago, somewhere between that range, the most popular augmented reality manufacturers, Microsoft HoloLens and Magic Leap, announced that they discontinued their manufacturing.
So, right now, there are dozens of medical device companies that are still using obsolete glasses or platforms for augmented reality that do not have any other solution.
What I did is, last year, I decided to address that problem, and together with a partner that I cannot disclose right now, we have formed a new company to develop an open platform, augmented reality glasses for medical application, something very similar to the Magic Leap and the Microsoft HoloLens but designed for medical applications.
So, unlike Augmedics where we developed an application for spine and tools and markers and everything, here we are only focusing on a platform that can be used by thousands, maybe hundreds of medical device companies that they can take the glasses and instantly build their own applications using a lot of different technologies that we have put in the glasses. Very easy, very convenient, and designed for medical use.
Some of the features that we are developing in the glasses, which will be very unique, include AI. We do have an AI accelerator and dedicated processor for AI processing that will include object detection and segmentation of human tissue organs, whatever you want to do with that.
We do have two RGB cameras that will be able to detect those objects and also for magnification. During a surgery, usually they put on loops, so we will have the ability to magnify the full view for the surgeon or any physician. We will have two surgical trackers built into the glasses. So, it will allow users to use that as a navigation system as well. We have slam cameras. Slam cameras are basically a special positioning system that allows the users to place virtual objects in the room, and the users who are putting the glasses on can just walk around the room and still see the object as if they are actually there. They are moving, and there is, like, no latency at all.
We also are adding a time of flight camera, a 3D reconstruction, real time of any real object, of any object. So it means that we will be able to do, for example, on-the-fly registration just by looking, for example, at the patient’s face.
So you are looking at the patient’s face, you are reconstructing the face immediately, like instantly, and then you can register the patient to the pre-op MRI scan or CT scan just by looking at a patient. So, that’s a few examples of the technologies that we are introducing in the glasses.
We already have the glasses’ first prototype ready. So, we are progressing very, very fast. The launch date of the glasses is expected in the middle of 2027. So, about two years from now we are going to be ready for the product launch.
MHN: When you say that it can put any object in a space, what type of object are you talking about?
Elimelech: The glasses are not specifically meant for surgery or for any use case that we can imagine. I mean, we do envision various use cases, but because we are not making any content on or any application, we can think that it can be used, for example, for training or education.
So, if you want to teach or if you want to show the anatomy of a patient, or you want to plan your surgery, for example, right? So you want to have the organ rotated around. We have hand tracking and hand gestures so you can move the object, or you can scroll through imaging of the patient.
Or even during surgery, you want to place a virtual monitor in the room, not a physical monitor, but just a virtual monitor so you can always look there, you know, to the side and see whatever you want to see. That is going to be one of the use cases for the slam, for the special positioning system.
We are also envisioning a use case on how to position a patient on the surgical table. So, before surgery, a very important step before surgery is how to position the patient on the table. Sometimes you want to place the patient on the side and then to take their head a little bit tilted down or up. That’s very challenging for an inexperienced surgeon, and we’re facilitating that by just visualizing how the patient is going to be laid down on the table. And we can do that by using the slam cameras.
That’s exactly the idea here. I mean, everyone who is thinking about what is going to be the next use case with augmented reality can just take our glasses and just start using it. It is as simple as just developing an app for an iPhone. I mean, you do need developers’ skills, and a lot of companies do have very good developers, but it is not going to be the barrier of the hardware, which is the glasses. I mean, the investment is huge.
Any developer who is thinking about an application that is needed for whatever, you know, PTSD or any other thing that you can imagine, you can just take our glasses and relatively easily develop whatever you want specifically for medical purposes.
