The gaming industry ceased to be something unserious or just for children long ago. It is a huge market with pharma-level budgets, top-tier development teams, advanced R&D units, and extremely fine-tuned work with human attention, motivation, and behavior. It is only logical that medicine is looking more and more in this direction – if games can keep people engaged for hours, why not use the same mechanics when a patient needs help getting through treatment, rehabilitation, or complex learning?
Just 10-15 years ago, talk of “gaming technologies in medicine” sounded like a marketing trick or a hobby for enthusiasts. Today it is a fully fledged field: Kinect in operating rooms, Joy-Con and Wii Remote in rehabilitation, Ring Fit as a training tool, VR and AR for working with pain, anxiety, and phobias, and game engines as the foundation for surgical simulators. Clinicians increasingly recognize that behind these “toys” there is serious engineering, powerful hardware, and decades of user behavior research.
We spoke with physician, researcher, and expert in the convergence of technology and medicine Alexander Morozov about why the boundaries between games and medicine have become so porous, how gaming hardware and software are turning into tools of clinical practice, and where gamification ends and medical technology begins.
2digital: In your view, why have the boundaries between the gaming industry and medicine now become so permeable? Where did this synergy come from?
Alexander: There are several fundamental reasons for this.
The first reason is that a generation of children who understood and played on game consoles have grown up and are now in the most active phase of their lives, so they can imagine what advantages and interaction mechanisms exist in this industry.
Games in one form or another have become something natural for this generation; many still play and have no intention of stopping. By and large, younger people do not see games as a pointless waste of time – they see them as just another way to spend time, like watching films or reading books. In effect, this is already part of the cultural code of today’s generations.
The second one is the availability of gaming technologies. Modern gaming systems can compete in power and capabilities with most of the “smart” tools used in literally all spheres of our lives. And it is obvious that clinicians with gaming experience quickly realized that it is easier to involve patients in medically important activities through gamification. People are more willing to take tests, spend time on rehabilitation, and track the necessary health parameters when they actually enjoy doing so. Clinicians, by the way, also learn more actively with interactive simulators.
The third one is a greater understanding and a move away from stigmatization. It is interesting to see how research at the intersection of medicine and the gaming industry has evolved. Initially it was largely aimed at casting gaming culture in a negative light: saying that games cause aggression, addiction, almost dementia … but very little of this was actually supported by data.
This opened the door for other studies, which showed that computer games can help improve reaction speed and sometimes even reduce aggressive behavior patterns and encourage helping behavior and cooperation. After that, gamification was adopted into the design of educational and rehabilitation programs, and so on.
From this follows the fourth reason – medicine has become interested in the very areas that the gaming industry is seriously exploring: mechanisms of engagement, attention retention, behavior in stressful situations, and motivation.
All of this is important in medicine, rehabilitation, and illness prevention.
2digital: We’ve been talking about the software side, but are there also points of contact when it comes to hardware?
Alexander: If you think about it, game companies now want to interact with the player through the same channels a therapist uses at an initial appointment: visual contact, auditory, and even kinetic. In essence, this is exactly what I, as a clinician, am dealing with in my research work – I look for ways to record and process a person’s medical data as effectively as possible: heart rate, blood pressure, cerebral blood flow.
We feed these data into the creation of a virtual reality environment, trying to find ways to use it in anti-anxiety and even pain-relief therapy.
In fact, for millennia medicine has been trying to “read” a person as thoroughly as possible, and that same goal is now bringing it closer to the gaming sphere. For example, Valve is already developing its own neurointerface that reads a substantial number of brain activity parameters and adjusts the individual pace of the game accordingly.
2digital: To what extent are gaming R&D units really comparable with scientific laboratories in medicine in terms of the scale of experiments, data work, and UX research?
Alexander: If we are talking about the largest companies, they are quite comparable. There are full-fledged scientific laboratories working in the gaming industry, and they achieve impressive results. They hire specialists of a very high level to work there. Many game consoles, the gadgets that go with them, and VR/AR headsets can, without exaggeration, be called marvels of scientific and engineering thought.
At the same time, the scientific labs inside gaming companies often act as a frontier – they explore areas that are unlikely to be studied extensively in medical labs.
Here is a personal example: in Poland, there is a laboratory called XR Lab that studies AR and VR and all the methods of interacting with them. Initially, they were primarily focused on the gaming industry. But gradually, their scope of interest expanded to the point that they are now working with NASA as well. And together with me, they are exploring how VR and AR technologies can be introduced into medicine.
2digital: Can we clearly define where the line is drawn between gamification and real medical technology?
Alexander: This is actually quite easy to answer because the market for medical devices, medical interventions, and medical software is tightly regulated: in the US by the FDA, in Europe by the MDR. If any intervention in any way affects diagnosis or treatment, or goes through clinical validation, then it requires certification in all cases – and that makes it a medical technology.
Some companies that are now trying to use ChatGPT or other LLMs to make diagnoses, while claiming they have nothing to do with medicine, are doing this precisely to avoid falling under fairly strict regulation. However, sooner or later, they will all be faced with a choice: either go through that process or stop their activities.
If, on the other hand, someone is simply encouraging you to drink a couple of glasses of water, walk 5,000 steps, and get some sort of reward for it – and this in no way affects diagnosis or treatment – then yes, that can reasonably be called gamification without medicine.
2digital: Let’s go back to the hardware: the Kinect camera for the Xbox game console tracks movement, has “depth vision,” and also operates in the infrared spectrum. It became one of the first gaming devices to make its way directly into the operating room. How did that happen?
Alexander: In general, sterility is a very important issue in medicine, especially in the operating room. When monitors appeared there, and it became necessary to interact with them, it created difficulties. It got to the point where a surgeon had to break sterility, go into a separate room, work with the images on a computer, then scrub in again and continue the operation.
Kinect made it possible to control the information on the screen with gestures. That turned out to be very convenient. Moreover, implementing such technologies did not require major costs. Keep in mind that, because of heavy regulation and established, let’s say, traditions, any product with a “medical” label is instantly an order of magnitude more expensive. I can hardly imagine how much Kinect would cost if it were a medical product. As it was, for ordinary gamers, a price of $100–150 was expensive, but for operating rooms, it was inspiringly cheap. At one point, there was a genuine hunt for Kinects, including among clinicians.
There is a concept of the “touchless operating room” – the surgeon does not touch anything except the patient. Within that concept, Kinect became one of the key sensors in the broader idea of a contactless OR, where the surgeon controls images, navigation, and sometimes lighting or recording — all with gestures and voice alone.
A review by a Microsoft Research group and clinicians (“Touchless interaction in surgery”) describes how Kinect lowered the barrier to entry in this area: computer vision stopped being “an exotic technology for a couple of prototypes,” and a lot of projects appeared around touchless control of medical applications.
A systematic review in J Med Internet Res analyzed 86 studies and concluded that:
– Kinect is one of the two most popular gesture sensors (along with Leap Motion);
– Most studies are prototypes and pilots rather than large-scale deployments;
– The main advantages are low cost, contactless interaction, and suitability for training;
– The drawbacks are arm fatigue, limited accuracy, and the need to design a very simple set of gestures so that everything works reliably in a real operating room.
Of course, there are now other, much more advanced technologies, and you are unlikely to see Kinect in an operating room today, but at the time it did become genuinely popular and pushed MedTech manufacturers to look for new solutions.
2digital: Joy-Con controllers for Nintendo Switch, Ring Fit, and the Wii Remote are parts of gaming systems that make it possible to track a person’s movements. How did they end up as components of serious rehabilitation programs?
Alexander: The most important point is that, by the time these devices appeared, the quality of their motion tracking had increased many times over compared with all previous analogs. In addition, the capacity for all the necessary computational processing of these movements had grown. In other words, data obtained, for example, from a Wii Remote can be accurately captured, processed, and then used to compare statistics and see what parameters have changed.
This is extremely well-suited to tracking everything in rehabilitation. On top of that, patient engagement in the process increases significantly because everything is presented in a game format. Obviously, this helps a lot when working with children, but adults also far more willingly do meaningful exercises that change something in the virtual world, rather than just monotonously waving their arms.
Look at Ring Fit – an extremely simple plastic-ring form factor that can be squeezed, stretched, lifted, and lowered. But what possibilities it offers for exercise!
2digital: Let’s talk about VR and AR – what became the main “trigger” for their medical use?
Alexander: In principle, it is a logical continuation of everything we have just discussed: these devices include motion detection, but at the same time, they provide immersion in VR, which means they influence how the patient perceives the surrounding environment.
This immediately opened up many possibilities. As I mentioned earlier, it became possible to influence how a person experiences pain: it is one thing when a patient comes in for a medical procedure and focuses entirely on their sensations, and quite another when the surrounding environment “overloads” hearing and vision, forcing them to concentrate on something else.
VR and AR also work well with various mental states. There is substitution-type therapy for phobias and anxiety, where the object of fear is shown to a person in a highly schematic form, allowing them to get used to it, and then the level of realism is gradually increased.
There are also solutions that help people with autism spectrum disorders. For example, a person may be unable to go outside because they are frightened by the cacophony of stimuli, sounds, light, and so on. Such a person can gradually adapt to the situation in the following way: first they can “go outside” in complete silence, then realistic colors are added, then people, then cars, and later realistic sound, and so on. I have seen in several Polish rehabilitation centers how successfully such programs work.
2digital: Why have game engines like Unreal Engine or Unity become the platform for surgical simulators, rather than specialized medical environments?
Alexander: Why reinvent the wheel? Building a good graphics and physics engine is extremely expensive, especially if some medical company has to do it from scratch. On the other hand, the gaming market is huge: hundreds of specialists are constantly working on creating and improving Unreal Engine and Unity, and tens of thousands more are testing them all the time and coming up with new ways to interact with them.
Let’s say we want to create a simulator for laparoscopic surgeons. What matters for this type of surgery, what issues arise beyond the obvious visual component? Realistic physics. It is important that the simulation includes not just gravity but realistic tissue physics: density, elasticity, response to different types of manipulation. All of this already exists in modern game engines.
A striking example: Dutch surgeon Henk ten Cate Hoedemaker came to the Grendel Games studio with a complaint – classic laparoscopic trainers are expensive, boring, and surgeons have no desire to sit at them unless they are forced to. That is how the game Underground for Nintendo Wii U was born. In the game, the player rescues little robots in a mine but plays using two Wii Remotes inserted into custom “tubes” designed to mimic laparoscopic instruments. All actions (grasping, moving, working with both hands in a reversed perspective, depth perception) replicate the basic mechanics of laparoscopy.
I have seen some laparoscopic simulators that today are indistinguishable from a video feed of a real operation. Moreover, they connect to a control system that looks like, for example, a patient’s abdomen, with the instruments inserted into it just as they would be during surgery.
And this kind of training produces good results. There is already a classic study, conducted 20 years ago, in which surgeons trained to perform laparoscopic cholecystectomy (gallbladder removal) on a simulator. It turned out that dissection time for the gallbladder in the VR group was about 29% shorter. Residents without VR training were nine times more likely to have “stops with no progress” and five times more likely to damage the gallbladder or burn non-target tissues. At the same time, the average number of errors was about six times lower in the VR group (1.19 versus 7.38 errors per operation).
2digital: Do you see any risks resulting from the fact that gaming devices – including gamepads, Kinect, and VR, which are essentially consumer-grade and relatively inexpensive – are gradually entering clinical workflows?
Alexander: I believe in the vigilance of regulators and the responsibility of clinicians themselves. On the other hand, I see a problem in a somewhat different, perhaps unexpected, angle.
Some companies do everything they can to win a tender to develop a training program for a given clinic and ultimately sometimes forget that promises to make things cheaper and faster help to win the bid but often reduce quality. I have repeatedly seen programs created this way that formally met all the stated requirements but were, in practice, absolutely useless.
To avoid this, there needs to be an understanding of how such programs are built – both on the developers’ side (who should be advised by clinicians) and on the clinic’s side (so it understands what level of investment is required for real results). So far, this balance is still far from being consistently achieved.
