Only 7 percent of a message is based on the words it contains. The rest, 93 percent, comes from the speaker’s tone of voice, body language and facial expressions (1).
Imagine a visit to the doctor. You describe your situation and symptoms while the doctor looks and listens carefully. Imagine that this consultation is being recorded on video.
This video will then contain your speech (text, words) that can be automatically extracted as a transcript and analyzed. Your voice and its related characteristics like tone, volume or tremor can also be extracted and analyzed. Similarly, your movements including facial expressions, fidgeting, hand gestures, posture and physical distance can be analyzed.
Let’s consider the video constituents that can be analyzed separately:
• The text can be summarized with main symptoms extraction which then can be fed to an internal medical knowledge base (ontology) module through which diagnostics insights and related recommendations could be provided.
• Your voice and its characteristics, or vocal biomarkers (a medical biomarker is a medical sign that indicates a patient’s medical state as observed from “outside” the patient) can be used to explore the likelihood of the presence a specific disease. Among various related studies that illustrate this, in a double-blind study carried out by Beyond Verbal and the Mayo Clinic which involved 120 patients and a group of controls in the context of Coronary Artery Disease (CAD) undergoing angiography, a mobile app was used to measure their voice signal prior to a coronary angiograph. One voice characteristic in particular (not audible by the human ear) indicated an almost 20-fold increase in the likelihood of CAD.
• The video in itself can be used to analyze body language and facial expressions. As an example, there is no specific test that definitively diagnoses Parkinson's disease. Neurologists clinically diagnose Parkinson's disease based on medical history, a review of signs and symptoms, and a physical examination. If we train a machine learning based AI system to learn body movement, facial expressions, tremor and so on from a number of Parkinson patient video interviews, this can be used to potentially diagnose the disease given newly recorded (or even live streamed) patient consultations. The accuracy that can be achieved given for example a training and testing data set of 10.000 interviews would be surprising.
Intelligent Digital Medical Assistants
The AI powered processing of patient interaction using text, voice and video along with a knowledge base of medical knowledge, patient history and an automated communications module are the tools that constitute an intelligent digital medical assistant. Eventually, most clinicians and private practices will use this technology to initially interact with their patients.
Importantly however, digital medical assistants will never replace experienced doctors. They will simply make some doctors better than others that are not using this technology! Doctors will have more time for meaningful consultations with patients.
Inevitably the future of healthcare will involve patients initially getting in touch with intelligent medical digital assistants through the web, mobile or IOT interfaces that will automatically analyze patient speech, voice and videos. Unless comfortably addressed, the raised issues will then be automatically transferred to a medical professional for further consultation.
In summary, Intelligent digital medical assistants will:
• Reply to questions using an internal medical knowledge ontology (for example it will store all we know about sore throats in order to respond to related questions)
• Use Machine Learning and Big Data to provide insights based on previous patient health parameters and outcomes (i.e. given patient X with Y blood tests and underlying disease Α it would be useful to run test Z, or to watch out for condition Β)
• Track patient health parameters (fluctuations, repeated symptoms)
• Manage medication
• Provide simple actionable recommendations
• Provide personalized health tips and lifestyle coaching.
• Immediately provide first line help in emergency situations
Artificial Intelligence (AI) will fully transform health care. It can improve outcomes and patient experience while democratizing access to healthcare services.
AI can help improve the experience of healthcare practitioners, enabling them to reduce burnout and spend more time in serious direct patient care. AI can help healthcare systems manage population health proactively through the allocation of resources with a view to maximum impact.
Using a mobile or web based digital medical assistant through which to provide remote video based medical consultations and almost instantaneously extracting knowledge from the video in order to support or suggest a diagnosis, while at the same time using the same system to organize and follow up on these interactions, is the way to use AI technology to improve outcomes and efficiency in primary healthcare, particularly in the midst of a global pandemic!
If you want to learn more and be at the forefront of health care technology, join
the definitive free online healthcare community.
Albert Mehrabian’s study
Image 1 & 2 Attribution