AR and VR Are Poised to Revolutionize Medtech—Are You Ready?

Virtual and augmented reality technologies are set to disrupt not just medical devices themselves but also the way they're developed.

Jamie Hartford

From Oculus Rift to Pokémon Go, once-futuristic virtual reality (VR) and augmented reality (AR) technologies are fast becoming mainstream. And like many technologies that take root in the consumer world, VR and AR are also making their way into medtech.

MD+DI caught up with VR and AR expert Brandon Bogdalek, consultant for advanced development with Minneapolis-based design firm Worrell, to get his take on why these technologies are catching on now, what they mean for medical device developers, and more. Below are his answers, which have been edited for clarity and brevity. For more, don't miss Bogdalek's talk on how AR and VR are moving the needle for medical devices at the BIOMEDevice San Jose Conference on December 6, 2017.

MD+DI: When it comes to these immersive technologies, a lot of different terms are getting thrown around--VR, AR, mixed reality. Can you explain the differences?

Bogdalek: Let's start with virtual reality. Here at Worrell, we are experimenting with 360 video, where you take a 360-degree camera that pieces together and stitches together footage from 360 degrees. You then put that footage into a virtual reality headset, and you would be able to look around as if your head is the camera. That's one way to create a virtual reality experience. Another way is a computer-generated experience, where you literally have computer scientists and developers go into computer programs and create virtual environments. You're actually creating the environment on the computer. So those are traditionally the two different ways to create a virtual reality experience.

Augmented reality is an overlay of content into the real world, but the content overlaid is not anchored to or a part of the real world. Pokemon Go, the game that was very popular on the cell phone, that's a form of augmented reality. Another form of augmented reality would be floating text on a screen. So, imagine you're watching a football game and you have statistics that pop up. Those aren't anchored to the field itself, but that's literally a content overlay or something that's real but not anchored to the real world.

Mixed reality, on the other hand, is able to track space. So, for instance, the mixed reality technology that we use here at Worrell is the Microsoft HoloLens. It actually has the capability to track the room that you're in and will place holograms relative to what's tracking. The use case for a surgical setting, for example, would be a mixed reality approach, where I'm using a head-mounted display--let's use the Microsoft HoloLens for example here. So, let's say a patient has a tumor that's located in the liver and I input that CT or MRI data into the HoloLens. It should be able to pretty accurately detect where that tumor is in approximation to the body and overlay that information.

MD+DI: Why is there so much interest in AR and VR technologies right now?

Bogdalek: The technology has really improved over the past 10 years, and industries--whether it be gaming or entertainment or healthcare--are starting to see the potential business benefits from having augmented or virtual reality as a primary platform. I think it's a combination of the technology, industries starting to recognize the benefits of AR and VR, and people becoming more fascinated with the technology and wanting to really dig deeper.

MD+DI: There has been buzz about AR and VR starting, really, back in the 1930s, with the View-Master, and continuing through the 1980s and '90s, but it always seems to be a passing fad. Will this time be different?

Bogdalek: I think AR and VR is here to stay, and I think the main reason for that is consumer adoption. It has been around for quite some time, but it was nowhere near consumer adoption like it is today. We are much closer to the mass population being able to utilize this technology than we ever have been before, and I think everybody is starting to get a taste of that. For that reason, I think it's beyond its fad stage now.

MD+DI: Why should medical device developers be paying attention to AR and VR?

Bogdalek: Here at Worrell, we look at how this technology can improve our design process. We like to make our designers feel like Tony Stark. By empowering them using a head-mounted display, they have an entirely new medium to explore the design process, so rather than spending time on 3-D CAD or computer-based design programs, you're quite literally able to take those designs in a virtual format and view them in the real world, which has been nothing that designers have really been able to do before.

From the medtech perspective, that's extremely powerful because you can you can view these designs in relation to important things like anatomy. Medical device manufacturers can design on 3-D CAD all they want, but they need to be able to see what that device looks like in the leg or bolted into the femur or the tibia. I think the keyword here is perspective. Designers and engineers a lot of the time will get what's called CAD-vision. After you're looking at a design on a computer for so long, you kind of lose all sense of scale and what is happening around the device because you're so narrowed in on one specific thing. Augmented and virtual reality can really help that by taking your design from a computer to the real world, and that gives you a totally different perspective of what you're looking at. You can design in relation to anatomy. 

MD+DI: What about end uses for AR and VR in healthcare?

Bogdalek: I think it's being explored everywhere right now. We're looking very seriously at using it for instructions for use. Pharmaceutical companies put a number of different protocols in place to ensure that the devices or the injectors their patients are using are safe. You get a bunch of paper instructions and what we commonly find is you look at the instructions, you look at the injector, you look at the instructions, you look at the injector. You go back and forth, so using augmented reality as a heads-up display for that process could be extremely beneficial and potentially even mitigate risk.

For surgical planning, it's kind of like giving your surgeons x-ray vision. If I can enter my CT or MRI data into a mixed reality headset and overlay patient specific anatomy on top of their body, I can literally see their organs and bones inside of them.

Or using something like a head-mounted display where you can monitor the patient's blood pressure while you're actually interacting with them could be really beneficial. I see it as something like tagging along with computer software like Epic, where you actually have healthcare information at an eye's reach. You don't need to access a computer or a tablet to do that; it's right there on your head and right there in your vision, so you're freeing up your hands to do more patient-focused things.

MD+DI: When it comes to medtech, will AR and VR be more useful as enabling technologies or as end-use technologies?

Bogdalek: I think it could be a very good combination of both, and I think innovative companies will utilize it as such. Using it internally in designing and using it to empower your designers and engineers to improve the overall design, I think that's a huge benefit. On the flip side, if you're going to look at this technology externally, I think there's tons of opportunity, limitless opportunity for augmented and virtual reality in a surgical situation on a healthcare practitioner's head in the operating room to have a heads-up display for patient information, or even something like instructions for use, teaching somebody how to do self-injection. These are all things that are externally facing rather than kind of helping the design process. So, to answer your question, wholeheartedly, I think it will be a very good combination of both.  

MD+DI: Who's leading the way when it comes to AR and VR in healthcare?

Bogdalek: I would say that the pharmaceutical industry, as far as my exposure, is leading the way in terms of using some of these innovative technologies. But I am also aware of a couple medical device companies that are also doing it.

Stryker comes to mind as an example. They're using augmented reality to take a look at the way an operating room is laid out. Another company that comes to mind is one out of California by the name of Osso VR. They're a virtual reality surgical simulator company, so basically they create a content suite that allows orthopedic surgeons to practice an operation before they actually go into it. So there's a variety of different medtech companies out there that are playing in this industry.  

MD+DI: Which has been quicker to embrace AR and VR, startups or more established medtech companies?

Bogdalek: I would say it's a combination of both. Obviously, a lot of these larger companies are definitely looking at this technology and trying to figure out ways it can benefit their business, but they're typically a little bit more quiet about it. A lot of large companies are kind of taking a look at the field and saying, "OK, this this may not be a fad anymore. We should really try and dig into ways that we can benefit from this technology and beat it to the punch."

I would say that startups right now are really are leading the way in terms of what they're doing. Startup companies are being very creative.